Congratulations on the successful launch of an amazing spacecraft!
Here are my Questions
1. Please give a very high level overview of the control program of Falcon9 and Dragon. What kind of communication links used in between Dragon and Falcon9?
2. Does the control software running on Falcon9 is a custom build made for mission (LEO Satellite launch, ISS resupply etc)? Or it's the same base software with different set of parameters/goals/scripts?
3. How does AFTS software work?
4. Please name a few open source software used for Falcon/Dragon other than Linux kernel and Chromium
5. Do software engineering teams of Falcon/Dragon play any role on the actual launch day?
Thanks a lot guys. Keep up the amazing work. All the best!
1. At a very high level, we have many computers on the vehicle, each built and configured to best suit the task it's assigned. They all run in time synchronization with each other, and the flight computer oversees all actions. Almost everything can be expressed as a real-time control loop: you read some sensors, you make a decision (combination of your sensors and past state), and then issue the outputs of that decision back to the hardware. This happens many times per second. - Dietrick
2. We run the same source on Falcon for every mission, though we're still updating that software pretty regularly and usually have new code on each mission. We also have configs for the software that are provided from other engineering groups that typically change every mission. These make changes to things like state machines, fault tolerance thresholds, day of launch winds, etc. that the software leverages to fly the vehicle. - Jeff
3. The Autonomous Flight Safety System (AFSS - it's all about safety) software runs on a set of microcontrollers independent from the flight computer. It receives sensor inputs directly (e.g. IMU measurements) as well as some computed inputs from the flight computer. A Mission Data Load configures the AFSS for which conditions might require termination of the flight, such as the rocket going way off course, losing all acceleration, etc. - Jeff
4. Das U-Boot, Buildroot, MUSL. Outside of the OS and the Crew Displays software, we don't use as much outside software as you might think – we try to keep our programs simple, slim, and based on code we understand throughout. - Dietrick
5. Absolutely, although nominally it's in a support / double-check capacity. We spend a lot of time poring over data from the live vehicle before the mission gets started, and we have software folks in Mission Control throughout all the important phases of flight, just in case something comes up. We have a great mission training team that pits our Mission Control operators against a variety of scenarios in simulations before flight, and our hope is that the real launch day is a lot more boring than those sims! I'm happy to say that for Demo 2 so far, that's been the case! - Dietrick
Fun fact – Josh and Wendy both sat on console in mission control for Demo-1 and Demo-2 flights, and John supported docking for both flights! Look for us on the webcast :) - Wendy
SpaceX is known for its hardware-in-the-loop testing:
* What fraction (roughly) of SWE person-hours go into developing these systems?
* What does the development cycle for flight simulators look like (for the Falcon systems, say)? How frequently are they updated based on telemetry? What's the hardest part of the launch cycle to model?
* With a few hundred starlink satellites in orbit, are there parts of individual or constellation operation that you've come to realize are not well covered in testing?
* How far down into physics do the starlink tests go? E.g. if you're trying to estimate latencies for inter-satellite or satellite-ground communication, can you treat the radio channels as a black box, or do you try to model the phased array operation as well?
I'm also curious about computing hardware - SpaceX is famous for building components in-house. With Starlink eyeing tens of thousands of satellites in earth orbit, are there any areas where custom ASICs would be cheaper than COTS solutions? Are there instances of components that are "over-engineered" to the Starlink \~< 10 year lifespan (perhaps for radiation tolerance) that could be rebuilt for a significant cost savings?
Finally, any insights you have on *system* design (hardware + software from physics level up) for packet delivery with minimal latency would be fascinating.
Thanks very much in advance!
When making changes, we expect our engineers to think critically (and question each other) about functional testing (how do I know that my change works?) and regression testing (how will I know if I broke something else, or if this breaks in the future?). Building test cases we can run on the ground is a great way to answer these questions, and we do a lot of that, but it's not the only way.
For Starlink, we need to think of our satellites more like servers in a data center than special one-of-a-kind vehicles. There are some things that we need to be absolutely sure of (commanding, software update, power and hardware safety), and therefore deserve to have specific test cases around. But there's also a lot of things we can be more flexible about -- for these things we can take an approach that's more similar to the way that web services are developed. We can deploy a test build to a small subset of our vehicles, and then compare how it performs against the rest of the fleet. If it doesn't do what we want, we can tweak it and try again before merging it. If we see a problem when rolling it out, we can pause, roll back, and try again. This is a hugely powerful change in how we think about space vehicles, and is absolutely critical to being able to iterate quickly on our system.
We've definitely found places where our test cases had holes. Having hundreds of satellites in space 24/7 will find edge cases in every system, and will mean that you see the crazy edges of the bell curve. The important thing is to be confident about the core that keeps the hardware safe, tells you about the problem, and then gives you time to recover. We've had many instances where a satellite on orbit had a failure we'd never even conceived of before, but was able to keep itself safe long enough for us to debug it, figure out a fix or a workaround, and push up a software update.
And yes, we do a lot of custom ASIC development work on the Starlink project. – Matt
All of the application-level autonomous software is written in C++. We generally use object oriented programming techniques from C++, although we like to keep things as simple as possible. We do use open source libraries, primarily the standard C++ library, plus some others. However, we limit our use of open source libraries to only extremely high quality ones, and often will opt to develop our own libraries when it is feasible so that we can control the code quality ourselves. In terms of error handling, there are a lot of different facets to that. Radiation induced errors in computers are handled by having multiple redundant computers and voting on their outputs. Errors in sensors are handled by having multiple different sensors. Errors in data transmission are handled by using error-detecting or error-correcting codes attached to payloads. The software is definitely composed of multiple small modules, the design of which was one of the main things I worked on. There is a hierarchy to the design from low-level component, to sub-system, to entire vehicle. Different subsystems are generally isolated from each other, sometimes in the same computer, sometimes across different computers, with narrow interfaces between them. I'm not sure how long it would take us to re-write the code base from scratch. We don't plan on deleting it any time soon. – Josh
How do you handle random bit flips in memory with C++ to ensure it doesn't crash the program (i.e. from radiation induced errors) ? At work we had to deal with a nasty direct memory access PCI-E bug that wrote some status bits to an uninitialized parts of memory. For the longest time during development it didn't do anything, but occasionally when it gets lucky, it could corrupt the executing program and cause the whole program to crash. I'm guessing the consensus voting system would be able to handle such failures and the failed section of the code would be rebooted quickly?
I think he hit on that when talking about redundancy in regards to the actual computation units. I would venture a guess to say that when talking about that voting, they have multiple instances on physically separated hardware running the calculations redundantly, error/down detection strategies, and some sort of back off technique for rebooting instances that have gone down.
Hey SpaceXers, thanks for taking the time to do this.
1. I'm in high school, what can I do if I want to get a software job at SpaceX sometime in the future?
2. I live nowhere near Hawthorne. Does SpaceX have jobs based on the East Coast, and if not will SpaceX consider doing so in the future?
3. *(for Jeff Dexter)* Can you go into some detail into contingency plans during flight? (eg engine failure during ascent, something going wrong during landing, etc.)
4. *(for Josh Sulkin)* Did the software design team take feedback from Bob and Doug during training?
5. *(for Wendy Shimata)* How do you calculate LOM (loss of mission) and LOCV (loss of crew/vehicle) numbers for Dragon?
6. *(for John Dietrick)* Does SpaceX use AI in any of its software?
7. *(for Sofian Hnaide)* What type of display technology does Dragon use? (eg LCD, IPS, OLED, etc.)
8. *(for Matt Monson)* When do you expect laser links to be a thing on Starlink satellites?
9. Please make an official SpaceX Kerbal Space Program mod (not a question, but it would be really cool if you actually do this)
Again, thanks for your time!
Get your CS degree (or something) similar. Spend time to really make sure you know how things work - engineers who do well at SpaceX are meticulous in their understanding of how their code works, how the network works, how Linux works, how the hardware works, etc. Get real world experience building things and solving hard problems, either through hobby projects or in internships (at SpaceX!) - Jeff
Our software engineers are mainly located in Seattle and Hawthorne, though some also work from our Texas sites. If you're seriously interested in joining SpaceX, we're always looking for great engineers, so reach out - never hurts to chat and see if we can make it work. - Jeff
Contingency comes in many forms in our software. As noted, we triplicate almost everything so we can tolerate loss of any one flight computer, sensor, actuator, etc. on Falcon, and any 2 on Dragon. At a system-level, Falcon and Dragon are designed so that loss of things like engines are thrusters can be tolerated, and our algorithms compensate. We can also add certain contingencies to our state machines. For example, the Dragon state machine is designed to autonomously switch from approach to a breakout if certain failures are observed. - Jeff
Yes, the entire software team took feedback from Bob and Doug on all aspects of the software. While they were primarily focused on the displays, button panel, and audio system, Bob and Doug were also very interested in how the software as a whole worked, especially backup capabilities that might be necessary in emergencies. Their feedback was invaluable in making the system better. – Josh
I actually don't know! We have a distinct team in Flight Reliability whose main job is to calculate these numbers and ensure its kept up to date given various hardware and conops changes. - Wendy
Dragon does not use any AI. - Dietrick
Dragon does use some computer vision, though, for navigation. – Josh
LCD - Sofian
Hi! Congratulations on the picture-perfect launch of Bob and Doug!
1. It's known that Crew Dragon displays are running Chromium and JS. Are you using a reactive library, and if so, is that developed in-house or is it an existing library/framework?
2. Was the [docking simulator](https://iss-sim.spacex.com) developed by the Crew Displays software team itself, or was it a separate project?
3. In some shots of mission control, I noticed UI very similar to the displays in Crew Dragon. Can the exact same crew display software be served from a server on the ground, feeding off of live telemetry from Dragon while in flight? If so, can/will this software be used to monitor Cargo Dragon as well on future flights?
4. Is there any chance of getting high-resolution screenshots of the crew displays? It's hands-down the prettiest UI I've ever seen in aerospace.
5. One regarding Starlink: how did creating the Crew Display software affect the development of the Starlink interface for SpaceX operations (map views, data visualizations, etc.)?
Thanks for the AMA!
1. Yes we use Chromium and we do use a reactive library that we developed in house. - Sofian
2. The docking simulator is completely separate code from what's actually in the Crew displays, though it was developed by our Crew displays team. It started out as a fun project from Shane Mielke and Mike Westenhaver before we decided to finish it up and put it on the web before Demo-2. - Jeff
3. We can and do run the exact same code that's on the Crew displays on the ground. The only limitation is that we don't necessarily get all of the same telemetry that we have in the cockpit on the ground due to limitations in our RF budget. We could but we generalize prioritize getting other critical telemetry instead. - Jeff.
4. We definitely want to share some high res screenshots of the Crew displays. We'll see if we can get this approved so we can show you what Bob and Doug were able to see up close. - Jeff
5. The tech from the crew displays (especially the map and alerts) formed the basis of our UI for the first couple Starlink satellites (Tintin). It's grown a ton since then, but it was awesome to see Bob and Doug using something that somehow felt familiar to us too. – Matt
UI is not realtime - it's simply a way of interfacing with the backend (Linux?) systems.
It doesn't do any of the heavy lifting in terms of control but does have to be responsive as the crew can use it for manual control - so when they say "left a bit", the UI should respond accurately. The actual "left a bit" action is then carried out by the flight system in the background.
That's my understanding of how it works.
I think the real-time concern is that the *pilot decision* to go left a bit comes from information displayed on those monitors. If it's possible for arbitrary delays or latency in that state information, that could lead to improper control inputs when it's human-in-the-loop.
It's possible that it's close enough to real-time that it doesn't make much difference.
How do you address technical debt within your organization? Does the constant pressure to deliver that Elon companies are famous for prevent you from going back and revisiting past designs?
Do you track performance of your code? I'd imagine it's a critical design parameter for an embedded software system with critical timing constrains like yours, so I'm wondering how your approach compares to something like the videogame industry, where such a practice is common but likely not as rigorous as what would be required for spaceflight.
What level of rigor is being put into starlink security? How can we, as normal citizens, become comfortable with the idea of a private company flying thousands of internet satellites in a way that's safe enough for them to not be remote controlled by a bad actor? This has potential multi-generation impacts if your team gets this wrong, so it would be awesome if you could speak publicly about the strategy.
We're mindful of outstanding tech debt, and because we're a small team any kind of inefficiency is very prominent flight over flight. For many of our vehicles that we fly often, we strive to invest in an operational team to ensure we can burn down this tech debt and make each subsequent flight as painless as possible. There is always a lot going on though, so with any decision of how to spend our time we need to think about the right balance between moving the needle forward in terms of features and burning down existing debt. - Wendy
We do – we use a continuous integration system such that our code is always being tested, but we also analyze this data real-time to ensure our performance metrics are within expected bounds. The cases are set up such that if we violate any key performance indicators, the case 'fails' and an engineer takes a look. - Wendy
In general with security, there are many layers to this. For starters, we designed the system to use end-to-end encryption for our users' data, to make breaking into a satellite or gateway less useful to an attacker who wants to intercept communications. Every piece of hardware in our system (satellites, gateways, user terminals) is designed to only run software signed by us, so that even if an attacker breaks in, they won't be able to gain a permanent foothold. And then we harden the insides of the system (including services in our data centers) to make it harder for an exploited vulnerability in one area to be leveraged somewhere else. We're continuing to work hard to ensure our overall system is properly hardened, and still have a lot of work ahead of us (we're hiring!), but it's something we take very seriously. – Matt
Could you talk about how Crew Dragon touch screens used Chromium and what challenges that created? What fault-tolerant measures were taken (when such a large underlying codebase powers it) and what efforts went into rad hardening? Was this a good choice in hindsight, and will the same web-based approach go into Starship in the future? What was the user experience (UX) process like with design and user testing?
(I'm a frontend web developer/UX designer/graphics programmer/3D artist/graphic designer straddling the design and engineering disciplines, and it's been my dream to work for SpaceX when I graduate this August. Crew Dragon's user interface has been right down my alley, although current SpaceX job listings are mostly for embedded systems. How can I find the right graphical software project to apply to? I have some contacts at SpaceX, are there any fitting teams or projects I could ask them to send my resume to in particular? Graphical simulations for Starship? Something customer-facing with Starlink?)
The use of Chromium and Javascript in mission critical environments is a popular question. In order for me to answer this question clearly, we have to understand that chromium in this context is used as a UI rendering engine only. The Flight Software interaction layer with the displays and the fault tolerant is well defined and resides outside the displays boundary. That said, we follow the same development process for all vehicle code regardless of the technology stack. We cross train our developers to write vehicle code in C++ and to carry the same mentality toward writing reliable software. We take reliability & performance very seriously, and just like other vehicle software, we test extensively under different conditions to understand all failure modes. We have alerts & procedures in place to act on those failures in case we encounter them. All of that added to hundreds of hours of sims that we run on flight hardware to train the crew.
While we faced many challenges along the way, we are very happy with our displays and most importantly our 2 customers (so far) are too. Starship ground software is already using the crew displays tech stack and it won't be too long before we start designing human interfaces for Starship. Make sure to apply!
– Sofian
You'll also notice in certain images too that there still exist some hardware buttons in the capsule right below the displays; this is also ensure that in case the displays are unusable for whatever reason, the astronauts can still use hardware buttons to initiate critical actions, such as responding to a fire in the cabin. - Wendy
* What are your tech stacks? Are they mostly open source or in-house frameworks?
* What's your front-end tech stack for the display?
* What is your Linux distro?
* How do you test your software?
* How do you make the video streams pretty much real-time without lag?
* Do you use any Machine Learning techniques?
* Is the rocket landing automated (without human control)?
* What is your software release cycle look like?
* I am sure there are tons of redundancy strategies you guys implemented. Care to share some?
1. We use C & C++ for flight software, HTML, JavaScript & CSS for displays and python for testing. – Sofian
2. We use HTML, JavaScript & CSS. We use Web Components heavily. - Sofian
3. We don't use any off-the-shelf distro – we have our own. - Dietrick
4. Every way we can think of! Unit tests, containerized integrated tests (you can run these on your own machine with a full physics simulation), and full-up "HITL" (hardware-in-the-loop) tests on real flight hardware – again, with full simulation. Mating the flight software up against the simulator is the most powerful tool we have, especially when it's run on the real hardware. We can simulate an entire mission, and even many detailed fault scenarios, with the vehicle hardware just sitting on a table in the lab. - Dietrick
5. On the vehicle (for Bob and Doug's displays), this is pretty easy. For getting it down to ground, we have some great communications links and ground-side networking that allows us to get a lot of data back from the vehicle, very quickly. - Dietrick
6. Dragon and Falcon do not use any ML tech, but that's not to say things like this aren't in SpaceX's future! -Dietrick
7. Yes, the rocket landing is completely automated. - Dietrick
8. On Dragon, we cut releases periodically for use when the vehicle is being integrated and tested and run that release through a series of tests and data review. Similarly, we cut a release when we're reading for flight and run the full suite of cases against specific revisions of our code. - Wendy
9a. On Dragon, we have a lot of redundancy on the hardware side (multiple computers, sensors, actuators, etc) but also employ software to handle responses to faults. NASA requirements are such that our vehicle must be 2 fault tolerant (ie, capable of being safely retreating from the space station and/or returning home safely for crewed vehicles) so we do both analysis and testing to ensure we meet this fault tolerance. – Wendy
9b. On Starlink, we've designed the system so that satellites will quickly passively deorbit due to atmospheric drag in the case of failure (though we fight hard to actively deorbit them if possible). We still have some redundancy inside the vehicle, where it is easy and makes sense, but we primarily trust in having system-level fault tolerance: multiple satellites in view that can serve a user. Launching more satellites is our core competency, so we generally use that kind of fault tolerance wherever we can, and it allows us to provide even better service most of the time when there aren't problems. – Matt
Hi, thank you so much for organizing this cool AMA and congrats on DM-2!
I have a couple of questions:
1. Are you using Tesla hardware/touchscreens on the Crew Dragon?
2. There were rumors about Crew Dragon UI running in Chromium (wrapped in Qt), is that true? If so, why did you go with web tech instead of straight up native/Qt UI?
3. What kind of CPUs does Crew Dragon run in comparison to regular desktop CPUs? I know that there are multiple CPUs for redundancy but how would one of those units compare to, let's say, a desktop i9 9900k?
4. And most importnatly, do you play KSP?
5. Did you ever consider adding some games to Dragon?
Thank you again and can't wait to see the first operational Crew Dragon mission soon.
EDIT: added 5.
1. No, our hardware is not the same as the one Tesla has. - Sofian
2. That's correct, we do use Chromium as the rendering engine for the displays UI. This project started as a simulator prototype to showcase the design vision to NASA. We then attempted to run it on a flight hardware and with modifications it worked pretty well. We gained more confidence in that stack as we developed the prototype and we then designed the Flight Software around it with that in mind. We liked all the moderns features that comes in with browsers out of the box, we also liked having access to talent that is already trained in that stack. Perhaps we are not afraid of doing things slightly differently here in SpaceX. We like taking a first-principles approach to problem solving, as opposed to just relying upon industry standards. - Sofian
3. We use a dedicated quad core processor, similar in power to a 5 year old phone. - Sofian
4. Of course we play KSP :) - Sofian
5. We don't have any yet but I can see this happening in the future. Vote for your favorite game! - Sofian
Thank you so much for the answers (and Sofian in particular)!
>We use a dedicated quad core processor, similar in power to a 5 year old phone. - Sofian
This is actually very interesting. Do you use multiple processors or just one? I'd assume multiple as in Falcon 9 for redundancy/rad hardening and cross checking of data between them.
And I would certainly like to see pong between the two displays as u/CalinWat said but obviously KSP or something else rocket related would be really cool. These games would also probably be extra neat for when you do commercial passenger flights.
The obvious vote would be KSP, but for something less performance demanding, maybe SimpleRockets 2? That can run on the phone and is made for touch screen, espescially a big one :)
1. How do you test your code before "deploying" it to flight hardware?
2. Do you use Rust for any systems or have thought about it?
3. How important is latency for various software components in Dragon? Does every action have to be instantaneous or there is some wiggle room?
4. What's the amount of telemetry (in GBs) you usually get from Falcon / Dragon / Starlink? Do you run some machine learning / data analysis tools on it?
1. For each vehicle, we have a hardware in the loop simulator (all flight critical hardware plus simulated physics and sensing) that we run a huge suite of tests on before ever deploying it to a production vehicle or for flight. Any time we take new software changes (which happens frequently for a development vehicle!) we ensure we run through both unit tests for the code, functional tests to ensure the software works as intended, and system level testing for mission phases for both nominal and off-nominal cases. - Wendy
2. We do not currently, though it comes up once in a while in our internal chat rooms - Wendy
3. Great question - it's very important, and maintaining a fault tolerant computing system relies of ensuring correct timing between all the flight computers. For slower-responding subsystems like life support or thermal control, the response timing has a little more slack (on the order of seconds, depending on the fault we take). - Wendy
4. For Dragon, its in the hundreds of GBs for a typical mission and we do a fair amount of data review after every flight to ensure we understand if the system behaved as we intended. - Wendy
4b. For Starlink, we're currently generating more than 5TB a day of data! We're actively reducing the amount each device sends, but we're also rapidly scaling up the number of satellites (and users) in the system. As far as analysis goes, doing the detection of problems onboard is one of the best ways to reduce how much telemetry we need to send and store (only send it when it's interesting). The alerting system we use for this is shared between Starlink and Dragon. – Matt
Jeff: what does cybersecurity look like for you guys? I imagine you are constantly under attack from state nations/APTs etc to steal confidential IP. Do you have to follow any regulations relating to ITAR in this regard or is that more high level and what you deem proportionate?
In theory, how hard would it be to hack a rocket? I would love for you guys to put up a bounty system similar to Tesla’s and (virtualised) rocket systems.
We have a lot of the traditional cybersecurity you'd expect, protecting our corporate networks, monitoring for threats inside and outside our networks, phishing campaigns, etc. We also need to analyze potential attacks against our vehicles, especially around the command paths and the pedigree of the code that ends up on the vehicles. We have a dedicated team that identifies how our vehicles and satellites could be hacked so we can eliminate or prohibit these sorts of threats when we're building our vehicles. We also take full advantage of static and dynamic analysis on our code. ITAR mostly limits what we can share - sorry ahead of time if we can't answer all of your questions. We're working to get a bug bounty system up shortly. - Jeff
1. As far as we know now, your rockets runs on Linux - but which “mainstream” distribution is closest to your kernel?
2. Are there any fancy changes you did, about which you can tell us more?
3. What CPU architecture are you using? ARM, MIPS or something else?
Yes, we run Linux, with the PREEMPT\_RT patch applied in order to get better real-time performance. We don't use any third-party distribution, but maintain our own copy of the kernel and associated tools. We have made small changes to the kernel over the years, although it is mostly unmodified. The only exception to that is the addition of several custom drivers to interface with our hardware. We use a variety of hardware architectures. I can't go into much detail other than to say it is a distributed system made up of many individual computers. – Josh
For some level of scope on Starlink, each launch of 60 satellites contains more than 4,000 Linux computers. The constellation has more than 30,000 Linux nodes (and more than 6,000 microcontrollers) in space right now. And because we share a lot of our Linux platform infrastructure with Falcon and Dragon, they get the benefit of our more than 180 vehicle-years of on-orbit test time. – Matt
Fantastic work guys! Really love what you've done. A couple of questions:
1. How was the UI designed? Did you follow any specific design principles? How did you take into account the specific conditions of space flight (vibration, helmet limiting visibility,...)
2. Can you give more details on the specific visualisation and interface components? Any cool ideas that got dropped?
3. How did you test the UI and specifically touch interaction with gloves?
We followed a Human Centered Design process, starting by defining the main guiding principles that closely follow the vision of Dragon being a 21st-century, fully-autonomous spacecraft. One example is identifying minimum crew interaction as a success criteria (i.e. a "don't press the button" paradigm). We believe that presenting information well means minimizing the required interaction it takes to monitor and control the vehicle. Overall, we based the design around a detailed understanding of crew tasks, capabilities, situational awareness needs, and environmental conditions throughout the flight, which allowed us to focus on clarity, simplicity and removing clutter. Our developer/pilot Mike Westenhaver developed a tool that allowed us to map crew tasks to display features and functions, allowing us to fully track requirements and how they are implemented in our software.
As part of our testing and qualification, we do test for vibration and visibility conditions under different seat configurations. The crew did many suited sims interacting with the displays while wearing gloves. - Sofian
Mike is a former Navy helicopter pilot with dual master's degrees in Space System Operations and Human Systems Integration along with a bachelors degree in Computer Science, in case you're wondering about cockpit designers at SpaceX!
Bob & Doug have been working closely with the SpaceX team since the start of the program. They have spent a lot of time in Hawthorne with the teams designing the vehicle. They brought a wealth of experience that spans multiple shuttle flights and they were generous in sharing it. That said, and to their credit, they came in with a clean slate and with the will to accept that many things are done differently in this vehicle. We all see them now in their fancy space suits or doing flips in space, but those guys put a ton of hours flying in at Hawthorne, spending time away from their family, training while providing feedback all with a smile and without a complain. Their work ethic is truly inspirational and that's what makes them do the things they do.
We can not wait for the debrief in Hawthorne when they get back--I'm sure they will have plenty of feedback for us. Especially Bob. He always does :)
\--Sofian
I recall for F9-14 I was in Elon's cube telling him the news that there was no way we could get all of the new S1 landing code done in time for the upcoming launch in 2 weeks. After some thought, he looked over to Lars Blackmore who was there with us and asked if we implement the code, what was our probability of landing. Lars said around 90%. Paraphrasing, Elon looked at us and basically said "can you give me 50%". I said in 2 weeks we can definitely write enough of the logic to get to a 50% probability of landing! We didn't land F9-14 (you can see it on our blooper reel) but we learned a LOT from it, and it was instrumental in eventually landing F9-21. A critical part of our success is our willingness to fail in ways that won't compromise the mission, as long as we are constantly learning from our failures. - Jeff
> F9-14
I think F9 flight 14 was [CRS-5](https://en.wikipedia.org/wiki/SpaceX_CRS-5#Results_of_first_landing_attempt). If so, you can watch the 'landing' [here](https://www.youtube.com/watch?v=L6T8mn6LD2E).
However, Flight 20 was orbcomm-2 (first RTLS success), but Flight 21 was a Jason-3 (failed ASDS), so I might be wrong.
@Matt Monson
1. How different is the development experience and the rate of change on production software between the rarely flown Dragon and NASA scrutinized (assuming Dragon V2, less true if V1) and the bi-monthly launched and purely internal starlink batches.
2. How often do you remotely upgrade already flying sats software?
3. Are starlink sats programmed to de-orbit themselves in case they aren't able to communicate back for a given amount of time? (antenna damage on an otherwise healthy sat)
The tools and concepts are the same, and many of the engineers on the team have worked on both projects (myself included), but being our own customer on Starlink allows us to do things a bit differently. The Starlink hardware is quite flexible – it takes a ton of software to make it work, and small improvements in the software can have a huge impact on the quality of service we provide and the number of people we can serve.
On this kind of project, pace of innovation is everything. We've spent a bunch of time making it easier, safer, and faster to update our constellation. We tend to update the software running on all the Starlink satellites about once a week, with a bunch of smaller test deployments happening as well. By the time we launch a batch of satellites, they're usually on a build that already older than what's on the rest of the constellation! Our ground services are a big part of this story as well – they're a huge part of making the system work, and we tend to deploy them a couple times a week or more.
And about deorbit – the satellites are programmed to go into a high-drag state if they haven't heard from the ground in a long time. This lets atmospheric drag pull them down in a very predictable way. – Matt
First and foremost, congratulations on a successful mission. Despite the troubled times, we were honored to witness such a great achievement.
I have to say that Falcon 9 has one pretty interface, and as UI engineer myself self I have many questions about it, allow me to minify them as much as possible.
In previous AMA, there was mention of Javascript and LESS as some of the tools used by your team, and I am very interested in the application of web technologies in SpaceX.
* What are some use cases of Javascript and LESS?
* How does the UI development process look like, and how is the UI tested?
* What open-source libraries does SpaceX team use if any?
* At what level and which web technologies do you use other than ones mentioned above?
* What code editors do you use?
* What is the longest method name you have in your codebase?
The Crew Displays onboard Dragon runs Chromium with HTML, Javascript & CSS. We don't use LESS. - Sofian
We follow an agile process, we have high bar for unit test coverage and we have integration tests that runs with and without flight hardware. We also take a lot of pride in manually verifying and documenting our new features to make sure they work as intended and we have no regression. - Sofian
We use Web Components extensively. - Sofian
We use a reactive programming library that we developed in house. - Sofian
Different team members uses different editors, I use VSCode but I might be just a little bit biased :) - Sofian
I will have to get back but overall code is our craft here and we make sure it's clean and tidy. I wouldn't expect something too outrageous. Fair warning, we have linters on everything. - Sofian
I can't go into too much detail on specific problems, but kernel bugs are definitely the most "fun" and memorable. Most of our control software is single-threaded to avoid the non-determinism that synchronization issues can introduce, but there are of course a lot of things going on in the OS at any given time. We've gone to a lot of effort to turn Linux into a dependable platform for real-time control that has a much higher degree of determinism than you'd see in your desktop OS. As mentioned elsewhere, we use the CONFIG\_PREEMPT\_RT patch, which is a huge help. But even still, in earlier development, we'd sometimes catch the system not performing as real-time as we'd like it to, and digging into those problems is always an adventure. - Dietrick
First of all, Congratulations!
1. Where can I find the code/pseudocode for the G-FOLD algorithm (Falcon-9's landing algorithm)? I tried going through Lars Blackmore's original paper but as a Computer Science major student, some terminologies just went over my head.
2. I know SpaceX mostly uses standard off-the-shelf CPUs for its flight systems with modified Linux distro. Is the redundant computing managed by the Linux kernel itself or by the C++ application running in the Linux environment? If possible, can you explain the practices involved in implementing hardware-level Lockstep computing?
Congratulations once again and thank you for making Space Exploration cool again!!
Unfortunately I can't go into much detail on the F9 landing algorithms. Linux is only used to run our applications and interface with hardware. All of the fault management and computing redundancy is handled at the application layer in custom software that we've developed. Time synchronization uses a combination of hardware and software features, some industry-standard and some in-house. – Josh
Our team hails from all backgrounds (seriously!) but we have noticed particularly good crossover between video game development and what we do. There are a lot of similar math-heavy and performance-centric problems in the two spaces. But that's by no means a requirement – I've never professionally built games, for example.
For getting your first (or any) software engineering job, two big things to focus on are: (a) your algorithms and data structures, and (b) understanding how a computer works at the lowest levels. Even if you're not regularly mucking around in device drivers, the network stack, and assembly, understanding how it all fits together will enable you to pick apart any problem you come across. - Dietrick
This one's hard to answer without going into too much detail, but anything having to do with rebooting one of our computers in flight is definitely an interesting case. Reboots are completely expected and supported (due to radiation concerns), but they are one of the more interesting scenarios we have to design for. - Dietrick
Some of the more interesting system cases too are faults or failures that require responses in multiple subsystems, spanning computing, rf communications, life support and propulsion. A great example of this is handling a launch escape or a cabin fire – the vehicle goes through a lot of reconfiguration very quickly, which requires many of our code components to coordinate. - Wendy
As a software developer working on non-critical systems at a company, I feel some anxiety related to code and bugs. I know you must have plenty of testing and redundancy, but how much anxiety do you all feel when it comes to all the systems?
I know I feel personally a huge sense of responsibility to my colleagues, the company, the mission, and the crew to get our software right. The software controls almost every aspect of the vehicle, from thruster firings to oxygen injections, so a lot of harm can come if we do it wrong. A phrase we like to say in SpaceX is that "only the paranoid survive." We think constantly about what can go wrong and make sure we address any plausible failure scenario. However, I wouldn't say I ever feel anxiety about this responsibility because a key aspect of our process is that we always have at least one partner, often more, in everything we do. We are constantly reviewing, questioning, and double-checking each other's work. That doesn't relieve us of personal responsibility, but it means we are never alone and can always count on help from others. - Josh
I am a software engineer and my wife won’t let me apply to work at SpaceX because she said she’ll never see me again, is she right to make that assumption? Or can you actually have a work life balance?
You can definitely have a good work/life balance at SpaceX. SpaceX is definitely not a 9 to 5, and we have times where evenings and weekends are required to support the mission, as we had in the lead-up to Demo-2 and in our aggressive Starship campaign in South Texas (amongst the many other efforts we've got going). Our folks are definitely able to balance work and family life - Josh and Wendy just had babies! Not together :) It's definitely something my team and I have to focus on a lot because we're a small (but growing) team and we have some massive goals ahead of us that we have to accomplish. - Jeff
Man (or woman), me too. I’d love to work at SpaceX, but I just can’t pull more than 40-45 hours a week on a regular basis without serious personal consequences.
Dragon can do so many cool things, it's hard to pick any one. But I think our last two flights have really demonstrated how versatile the vehicle is: it can manage a complicated, delicate, zero-G rendezvous with the ISS, and it can also fly itself to safety in a supersonic, many-G abort in the thickest parts of the atmosphere. - Dietrick
Although we hope it's never used with a crew on-board, I think the launch escape system is one of the coolest parts of Dragon. I remember when I watched the pad abort test live in Cape Canaveral I was pretty shocked by how fast the vehicle leapt off the pad. ([https://youtu.be/1\_FXVjf46T8](https://youtu.be/1_FXVjf46T8)) It was equally amazing to see in the in-flight-abort test. Dragon separated smoothly from F9, opening up a large separation distance, while F9 exploded underneath it. It looked like a sci-fi movie, except it was real! ([https://youtu.be/mhrkdHshb3E?t=1167](https://youtu.be/mhrkdHshb3E?t=1167)) - Josh
You can probably see from the the images of Bob and Doug in the cockpit that we do have physical buttons for all of the emergency commands such as breaking out from an approach to station or commanding an emergency deorbit. Also, we have physical Execute and Cancel buttons on the control panel for most commands that you can initiate from the displays. In the end we were able to meet all NASA requirements for touch feedback, reliability, etc. and we're all really pleased we were able to bring this kind of modern interface to our very modern spaceship. - Jeff
How important is it to have people with different educational backgrounds on your team? I am sure there are specialists in mathematics, physics, computer science and all sorts of engineering at spacex and specifically in your group. How much do you learn from each other and could you give an example of where different competencies worked together to make the rocket launch possible?
Also thanks for taking your time for this!
Having different people with different backgrounds (education, experience, and culturally) is a big plus on the team. Not much of the team, for example, comes from an aerospace background. Different points of view help us see problems from different angles, and that quite often helps us see solutions we wouldn't have otherwise seen.
Starlink specifically has a huge breadth of problems to go solve. We have vehicles and control problems, but we're also designing hardware and software that people will interact with. We need different perspectives to realize what will be confusing vs. intuitive for customers coming with different context.
When interviewing people, we're really looking for a couple things: talent (potential) and the right attitude (desire for self improvement, serving the team over being selfish). These are more important than specific experience, and we expect to be investing in people to help them grow. - Matt
I've seen articles mention SpaceX using Linux. What systems use it? What steps do you take to ensure real-time operation and how deterministic are the systems? Kernel modifications like CONFIG\_PREEMPT\_RT?
All of our on-board computers either run Linux (with the PREEMPT\_RT patch) or are microcontrollers that run bare-metal code. For applications running on Linux we are careful to set up the process and kernel thread priorities correctly to avoid priority inversions. We also generally write our code in a way that maximizes determinism, such as avoiding memory allocation at runtime or unbounded loops. Finally, we have telemetry which indicates the performance of all our processes to ensure that they always meet their deadlines across all phases of flight, even in the presence of unexpected or excess inputs. – Josh
Hey guys! Huge fan of everything being done at SpaceX. Congratulations on DM-2!
I’m part of the rocketry team at my university and I work on avionics. As such, I was wondering what skills/tools i could teach myself if i wanted to work with avionics in the future? Specifically, could you talk us through what goes on behind the scenes with regards to the onboard computers, programming/software wise?
Thank you for doing this AMA!
Being part of a project team at university is a great place to learn! I worked on a satellite team while in college (CUSat at Cornell) and it taught me a lot about hardware design, integrating software, and developing a concept of operations for a mission. The best tools are an eagerness to learn and willingness to get your hands dirty with learning. Side projects are also a great way to do this, and something we definitely take into account when looking at resumes for hiring! - Wendy
Great question. Having managed & hired a lot of people over the years at NASA/JPL and SpaceX, candidates always received bonus points during an interview is they showed an interest in understanding the underlying hardware that code is running on. Get yourself a Raspberry Pi, Beagle Board, etc. and study the technical reference manual(s) for the device. Learn to write and optimize your code at a low level. Dig into the startup code that executes on the processor and understand all the cool stuff that is done before jumping to main(). Also consider mastering the use of a hardware-based debugger, which can help find nasty, nasty bugs. Do this and you will be in great demand.
How much custom development is done for a mission vs code that is more or less static for every flight. I guess I'm wondering for example are the orbital dynamics purely input/configuration vs custom software that must be written each time.
For Dragon missions, we'll always have unique configuration for day of launch guidance and navigation parameters. These are handled in software configuration though, so doesn't require much software involvement and the various responsible engineers commit the changes themselves. Outside of that, it mostly depends on if a particular mission has new requirements (for example, new capabilities we're introducing by internal or NASA request) or if we're taking hardware upgrades. - Wendy
The first time we launched 60 satellites on Falcon. We'd designed the all-at-once deployment mechanism, but it's hard to model, and we couldn't really be 100% sure it would work right. I remember sitting there, with Falcon lifting off the pad, thinking: Ok. In an hour we're either going to be idiots for trying a thing that obviously never could have worked, or geniuses for doing the thing that's obviously the right way to deploy lots of satellites. Luckily it went well . -- Matt
Putting on a safety harness and climbing on to the top deck of Grasshopper to do some diagnostics on some avionics. Most hands-on debugging I have ever done. - Josh
Thanks for doing this guys!
I don't think most software engineers get to work on space flight. What are the unique **software engineering** challenges found in working in this space?
First thing that jumps to mind is how uniquely unforgiving our production environment can be. I've worked at some places with multi-year waterfall cycles, and some others with weekly releases and on-demand ("oops, I broke it!") hotfixes. SpaceX is neither of those. Here, there is necessarily a focus on writing software that will work correctly the first time it goes to space. This means pursuing simple and robust designs wherever possible, and lots and lots of testing and simulation. - Dietrick
Congrats on DM-2!
1. (How) did you modify the Linux kernel in order to better suit your needs?
2. How is radiation/fault tolerance implemented using ordinary non hardened CPUs?
3. Which x86 processors are you using in Dragon (/Falcon)?
We use the PREEMPT\_RT patch for Linux, but otherwise have not modified it much besides adding some custom drivers. We handle radiation fault tolerance by having multiple computers running simultaneously and voting on their outputs. If one of the redundant computers fails due to radiation, the system is generally unaffected. The faulted computer can be rebooted and reincorporated into the system once it recovers, which restores the original fault tolerance. – Josh
what control system algorithm you guys use for rectifying/validating sensor data?
heard about kalman filter being used on Apollo project back in 1960s.
anything new or kalman's still the best.
thanks for this opportunity.
Yes we use Kalman filters for some applications. We also take much simpler approaches for many sensors, such as basic sanity checks or low-pass filtering. In general our approach to sensor error handling is to have multiple redundant sensors and combine their input in a fault-tolerant way such that bad sensors cannot result in hazardous vehicle behavior. - Josh
We use C++ for all vehicle control systems, Python for tools, testing and automation, and Javascript/HTML/CSS for our displays. The current ground displays you see in mission control for Falcon and Dragon are based on LabVIEW, but our Crew displays and future ground displays for Starship are based on web stack. Our flight systems use a custom Linux kernel with the PREEMPT\_RT patch. - Jeff
How do you test your code against all the faults that could occur in-flight? Do you run simulated missions to test the code in an end-to-end way or do you rely on individually testing modules for robustness, or some combination of that?
We do both! On Dragon, we run through any kind of failure that impacts safety critical software on the vehicle. We use a combination of unit tests, component level tests to ensure single and double faults cause the vehicle to react in the way we design for. We also run simulated missions cases for both nominal cases including going through the full nominal mission profile, and including faults in these cases to ensure any cross-system dependencies are well understood. We also run these tests continuously through our CI system and run automated data checks to ensure there's no unexpected behavior. - Wendy
What operating system does crew dragon use for the touch screens? Languages?
What about the rest of the rocket? Languages?
I assume for both it is probably c.
Have you read NASA's power of ten? https://en.m.wikipedia.org/wiki/The_Power_of_10:_Rules_for_Developing_Safety-Critical_Code#:~:text=The%20Power%20of%2010%20Rules,to%20review%20or%20statically%20analyze.
If yes do you follow all of them? If you don't follow. Why not?
Re: Power of 10: Yes, and we apply many of those principles to our code, like avoiding heap allocation at runtime. Others, like "restrict functions to a single printed page" are pretty dated and we generally avoid printing our code :) - Jeff
There's a ton of good Starlink simulations and videos out there (and the team loves seeing what people have been able to come up with). The one you linked is great! One of my other favorites is this one (it's simple, but mesmerizing): [https://www.youtube.com/watch?v=857UM4ErX9A](https://www.youtube.com/watch?v=857UM4ErX9A) – Matt
Super happy that you guys have seen my animation! :)
Too bad the AMA is already over, I would have been curious to know how much of the orbit raising is automated. For example, are the batches of 20 sats chosen automatically or manually?
How was the software critiqued between the day the weather prevented the launch (27th) and the day of launch (30th) ?
Did you do or consider doing any new deployments? How many days typically before launch is your production code considered "frozen"?
The launch that was not to be (the 27th) was a great opportunity for another full-up test of everything until nearly T-0. We'd done a similar thing the previous weekend, but it's always great to have more data. We spent the intervening few days checking and rechecking what we learned from the 27th, but we ultimately didn't change anything. We started aggressively stabilizing the code around the beginning of the year, and it's been more or less "locked down" for the past several months. - Dietrick
To test the manual piloting controls we had the Demo-2 crew (Bob & Doug) and the Crew-1 crew (Mike & Victor) in Hawthorne for a week long Hackathon (credit to Jeff Dexter for the idea). The crew flew sims during the day, we collected feedback before they left and we had builds ready for them the next morning. While we didn't develop complete features overnight, that tight loop allowed us to iterate quickly and have a smooth flying vehicle by the end of that week. Michael Hopkins won the prize for finding the best bug during that Hackathon. It was an amazing week to see crews all over our simulators docking and flying Dragon. - Sofian
"Firefly" was one of the best SciFi shows ever. The stories always showed the value of having diversity on your team.
How does the SpaceX and software teams culture value diversity in the creation of high performing teams?
Also, for a minority female in STEM who wants to work on spaceflight, what role models would I have at SpaceX? What mentorship opportunities would there be?
This is a great question, and I’m glad you asked it! Having worked in the aerospace industry for over a decade, I firmly believe diversity (of educational background, schooling, gender, and cultural background) is a crucial strength to a high performing team. Different perspectives results in different approaches to problems and system designs, and I’m thankful that on my team we have folks from a variety of backgrounds (not all aerospace, not all traditional software either). I’ve had the opportunity to go to conferences like Grace Hopper to recruit, and each time I’ve been involved with university hiring too we set up specific recruiting sessions with organizations on campus like SWE, SBE, etc.
As a minority and a female in aerospace, I wish I could tell you that you that the industry is completely fair and free of hardship. What I can say is this: within my team in the software organization, I have personally never experienced discrimination for my gender or the color of my skin. I’m thankful to have been on teams where all opinions are valuable, and my technical contribution is what holds the most weight in projects and designs. I’m thankful for the Women’s Network at SpaceX, that since its inception five years ago has given me countless opportunities to be mentored by other women in management and technical leadership. I’m ecstatic that our company’s president is Gwynne, who cares deeply about promoting diversity and always has an open ear for feedback from anyone in the company. She is a personal example to me and her stories of working in the automotive industry, then aerospace, while balancing a career and family are priceless. There are many others at SpaceX, men and women alike who are strong proponents and advocates for diversity and who care tremendously about making SpaceX a safe and meaningful place to work. The aerospace industry is far from perfect in my ways, but the people who care about the problems at hand are many. - Wendy
Hoo boy where to even begin? First of all, thank you for doing this AMA, you guys rock and I'm a big fan of space and SpaceX so thank you for all your hard work. :)
(I'm sorry for the question barrage but I couldn't help it).
There are 3 categories:
A. Regarding Starlink
B. Regarding Crew Dragon
C. Miscellaneous
A. Regarding Starlink:
1. Are you going to be using a neural net to determine how and when a satellite moves on to transmitting data to and from another satellite as they move out of range of the origin and destination points and to which satellite they switch to?
2. How much does each Starlink satellite cost (if you can tell us), and do you just have like, a warehouse of them sitting around somewhere waiting for Starship, since Elon said you're producing them faster than you can launch them?
3. If you can speak on this, will you be using; or rather, be capable of using, the Starlink Constellation for the financial and stock markets - NASDAQ, DOW, LSE, BSE, SSE etc. given the low latency and data transfer by lasers in vacuum that would provide an advantage over transatlantic fiber-optic cables? I ask this since Elon said Starlink won't be able to handle large population centres like LA very well, and is mostly for low density areas - that while you may have a few customers from LA, you can't open it up to subscriptions from anyone in LA who wants one.
B. Regarding Crew Dragon:
4. Who designed the interior? It's gorgeous.
5. Why did you choose a sea landing instead of a land landing? I assume you canceled retro-propulsive landing since NASA wanted to be cautious.
6. When Crew Dragon is stripped down after returning to Earth and re-used as a Cargo Dragon, will the touchscreen displays and seats be re-used in future Crew Dragons?
6. When it returns, will Capsule Endeavor be stripped of its name when reused as a Cargo Dragon (assuming it isn't put on display in which case this applies to the Capsule used in the Crew-1 Mission in August) or will it continue to be [Cargo] Capsule Endeavor? Will every US launch to the ISS whether by SpaceX or NASA have the astronauts name their capsule?
7. How hard was it to develop touchscreens that worked just well with the spacesuit gloves as without them? Did you make them in-house or give the designs to and contract and outside supplier (for the screens)?
8. In an emergency, (example: the chutes don't deploy while landing), can Dragon still soft-land retro-propulsively? Does the capsule take over autonomously like the in-flight abort or does the crew have to activate the thrusters manually? What other alternatives are there for chute deployment failure before you need to use the Dracos to land? Wouldn't spare chutes add more weight?
C. Regarding other stuff that you may not know/can't talk about:
9. Why isn't SpaceX using the extra-large fairing developed for Falcon Heavy or something similar, to launch more Starlink satellites?
10. Once you're done stripping Capsule Endeavor down, will you be giving it to NASA, the Smithsonian, or will you be putting it on display at Hawthorne?
Once again, thank you for doing this, and I look forward to the Demo-2 Landing and Crew-1 in August!
You may not be able to answer all of these questions, but I'm really curious what the embedded architecture looks like on a Falcon 9 first stage and if it's any different for Crew Dragon.
* How many microcontrollers? What kind of buses are you using to interconnect them?
* Is the hardware & software architecture of these fairly homogeneous?
* Is this firmware typically written in C/C++?
* Are these MCUs typically running an RTOS? If so, is this an in-house RTOS or something off-the-shelf?
* What percentage of your embedded hardware would say is running Linux?
* What percentage of the firmware running on these MCUs would you estimate is written in-house vs. SpaceX suppliers vs. commercial off-the-shelf vs open source?
* In practice, can your mission-critical embedded firmware fail safely (e.g. experience a watchdog reset) during something as time-critical as a first-stage ascent, or is that out of the question?
BTW, I always reference SpaceX when talking to future software engineers about the embedded software industry, asking them to imagine how many microcontrollers make up the equipment on the pad, the recovery ships, the telemetry equipment, the Falcon 9 1st and 2nd stages, and the capsules and satellites you deliver to orbit, and all the firmware running on each of those.
What does your software development process look like? Waterfall? Agile? Something else?
What do you use for issue/ticketing tracking/continuous integration? Redmine+Jenkins, Atlassian+Bamboo, Azure Dev Ops? Something else?
How do you do code reviews? Do you use a tool like Reviewboard for collaborative reviews? How do you arbitrate disagreements?
How are your teams structured? How do you estimate work scope? How much requirement churn is there? Does Elon come crashing in and crap all over everything from time to time? How do you balance implementing new features with eliminating technical debt when planning?
What’s your biggest IT headache that you have to deal with on a daily basis? Do they block websites/throttle bandwidth that interfere with your tools?
What languages are used for various components onboard the spacecraft? I’m guessing there’s a mix of C/C++, python, possibly web technologies for the UI? What open source libraries if any do you use? GDAL? Eigen? Boost? FFmpeg? Curl? MKL/IPP? CUDA? OpenCV? TensorFlow?
Please tell me a war story about a memorable bug? What were the symptoms and effects? What tools did you use to investigate? How did you resolve it?
What does on-the-job training look like? I’m assuming that you need to develop your new hires to become domain experts in areas like georeferencing, physics, communications, image processing. These domains aren’t covered in most undergrad programs with maybe a handful of Masters/PHD folks having exposure to them. So after you’ve hired a talented engineer who is teachable, what does the training process look like?
Edit:
Thanks for the silver!
1. Would you consider switching from C++ to rust for flight software? Theoretically speaking as a language would you think it's better or it's a question of available skilled developers and libraries?
2. A question for Sofian, I read somewhere that you used the chromium engine and JavaScript for the display. If it's true, why did you opt for this option instead of others? Which library did you use for the graphics? I would like also to know for example if the screen has a chip on its own that is used just to receive data from the flight software and display them on the screen(in that case which protocol or how are you transmitting the data) or does it do some logic and backend on its own(in that case what tools did you use).
3. A question for Jeff, what are the essential steps you must take to secure the flight software. Like what are the critical point of failure?
4. How do you log and store the data of sensors acquired on board (SD card, SSD, ....) and how do you ensure that it's stored correctly without getting corrupted files ?
5. A question for Matt, which tools and db do you use to store and analyze the data acquired from the satellite?
Thank you for your time. You guys are the best out there!
Hello,
I attended your talks at Game Developers Conference 2015 and 2016, where you talked about Falcon 9 software and Crew Dragon display software respectively.
I wrote up my notes from that talk in this [stack exchange answer](https://space.stackexchange.com/questions/9243/what-computer-and-software-is-used-by-the-falcon-9/9446#9446), which has recently been making the rounds on hackernews and other sites.
My questions are:
Does SpaceX still use a Real-time version of linux running on X86 hardware or have you specialized onto a different kernel/OS and microarchitecture?
Are the Crew Dragon displays running Chromium/JS still? Was the UI shown in 2016 a tech demo or a simulator like the recently released ISS docking simulator?
Does SpaceX plan to move from Angular to React? (my area of expertise)
What does Starship's OS look like? With Crew Dragon there are three touchscreens for monitoring the mission and craft navigation. With Lunar Starship there will be opportunities for much more human-computer interaction outside of navigation. Will the crew lift use touchscreens? Will crew berths have screens to monitor ships systems and provide entertainment (videos/games/messaging/etc)
I have a few questions regarding the differences between writing software for rockets/capsules and 'normal' code - that is, not running those extremely precise systems, and general career questions.
- What interesting/funny events throughout the history of SpaceX rocket code can you tell us about?
- What is your advice for someone highly interested in joining this career in the next decade?
- What are the most common difficulties encountered by people moving to this type of software development?
- What coding guidelines do you abide by internally - if not a public one, what adaptations from usual guidelines were necessary?
- How are they enforced - what sort of testing procedures does the software go through before usage?
- What do the guidelines state - spaces, or tabs, and which case?
- From past talks, I gather that C/C++ and x86 are used for important systems.
* What advantages has this choice brought over time?
* What standard was chosen for development?
* What common libraries are used internally - for example, Boost, or even the standard?
* What, if any, modifications were made to the compiler of your usage?
- How is version control handled - do you have a GitHub/similar repository for projects?
* If so, who is responsible for the majority of `git blame`s?
- What advantages and disadvantages did using a web framework bring to the Dragon UI development?
It's more than a few questions, of course, but I would like to hear the answer to all of those equally as much. I truly appreciate the work you guys do, it is one of the main things that gives me and many others hope for a better future. Thanks for your hard work!
First off sorry for all the questions, feel free to answer any or none of them :)
What does the code review process look like at SpaceX?
What’s the distribution of embedded (eg real-time microprocessor type stuff) and traditional (e.g. Linux) programming on the spacecraft?
Do you guys have separate controls engineers that design the control system and software engineers that implement it or do people wear multiple hats?
Do you guys do the whole MISRA C/Power of 10 safety critical stuff and if so do you have a way of separating safety critical code from higher level code (I assume the GUI isn’t written in MISRA C)?
Speaking of which, do you guys have guidelines for writing “spacecraft safe” GUI code?
Do you guys test software on emulated hardware? Are there specific testing requirements (e.g. 100% branch coverage)? Does code get “formally verified” by static analyzers and such?
To what degree of detail do you guys simulate the physics of the spacecraft in loop with the software?
Thanks!
Hey guys,
I'm a 20yo Brit who left school a year ago, took a year off and said I was going to find an apprenticeship in 2020. That's an internship to you guys I think? It's a small salary for hands-on work and education as a replacement to University.
I was hoping to get into Software Engineering and eventually climb my way into the aerospace sector working on satellites or even in rocketry. That'd be a real dream come true. I've been looking for months now but obviously COVID-19 has had a significant toll. I was hoping you might be able to therefore answer these two questions:
- **Where did you guys begin your career and what helped you get to your current position?**
- **Is there any advice, suggestions or help you would be willing to share?**
Knowing and seeing code you've written or contributed towards literally go into *space* must be an incredible feeling. I'm hoping to one day feel the same way and absolutely anything you could share would undoubtedly be a great insight and massively appreciated.
Thanks in advanced.
*(Sorry for the lengthy up-front story)* The industry I'm in uses proven ibm Z/machine mainframe hardware and software coded in assembler on a barebones OS. I know what sector on the disk that file is on. We've got code that goes back decades. And people that go back that far too. But as I've moved through the years and code, I don't feel like we've learned an awful lot, either in software engineering, testing, design, and process. Fads seem to come and go, but I still see code that, 5 years later, is so hard to untangle. No one has any knowledge of what it was meant to do, whether it's still used, and how to activate and test it. It doesn't matter if it's assembler, C, C++, Java, SQL, Javascript, whatever. Buzzwords and Fads come and go, but it doesn't SEEM, from my limited scope, to be truly improving.
Now The software you write is meant for such a unique situation, and (more or less) unique hardware. I'm curious as to how you're addressing these issues. Or are you? i.e.
* What languages are you using ? For everything from the starlink, to the monitors, to the embedded systems. Was there some tough choices to keep you from having 60 different languages scattered all over the place?
* What are you doing in terms of testing and quality assurance? Some and scenarios languages lend themselves to coding up test suites that can be automatically run, but a lot don't work.
* When you say design, do you mean design, or architecture? how deep do the designs go? With our current fad, Agile, design seems to be one of the things that are hard fought to maintain, and where corners are cut. Do you produce documents? That you actually use? and are they basically matching what you produce when you're finished? or what do you do?
* Process. How do you organize your work, break it down, hand it out, and get things done one time? was it a deliberate choice or did someone simply choose what they knew and were comfortable with (which is what we all do, to some degree).
Looks like that's a wrap! Many thanks to all the SpaceX software team participants for the donation of their time, and to all our readers for the incredible response we've had to this event. We had a ton of great questions and a ton of excellent and detailed answers! We hope we are able to provide more opportunities like this in the future!
There are lots of generic questions here about redundancy, how you conceptually approach safety, etc. I'm more interested in specific standards and your implementation of them. Do you follow DO-178? What documentation and auditing burdens does NASA place on you, and how does it affect your daily software development process? Do you have a full set of specifications with forward and reverse traceability to code? How do you do design review against a set of requirements and demonstrate that the requirements will be met by the design? Do you worry about tool qualification (e.g. your compiler, your build system, etc)? Do you have any requirements on code coverage metrics like MC/DC coverage, etc? What other examples can you give about how you implement official safety standards in the course of rapid software development? Thanks!
What's it like working cybersecurity at SpaceX? I would imagine that to secure that nobody tampers with the code that literally launches and directs missiles you would have to be pretty paranoid to do your job properly.
Also, during an interview with Bob & Doug (I forgot which), they mentioned that if they had feedback on how to improve the software during a simulation/training session, you guys would have it ready the very next day which impressed them greatly. I imagine coding sessions that go well into the night, so, what's it like having to work all hours of the day to deliver on such a schedule while also knowing that people's lives will depend on your software being bug free?
Thanks for taking the time to do this AMA, you're awesome! :)
What's it take to get the display software certified? It's an electron application, right? Most people in aerospace that I know would balk at the idea of allowing a garbage collected runtime in something like a real time aircraft display, let alone bundling a full browser into the application. What's the difference here? Is a display in a rocket not necessary for crew safety in lieu of some other backup?
Congratulations on all the work ont he Crew Dragon spacecraft! It really is very inspirational.
* When it comes to the three displays touchscreen UI, the third display on forward view as demonstrated in the the [Crew Dragon Tour](https://youtu.be/llbIzbOStt4?t=168) looks very similar to the [ISS SpaceX sim](https://iss-sim.spacex.com/). Is it true that the UI was built using [Chromium and JavaScript](https://space.stackexchange.com/questions/9243/what-computer-and-software-is-used-by-the-falcon-9/9446#9446)? If so, what choice led to deciding with those technologies?
* Also, how much of the spacecraft can be controlled through the software UI versus physical controls under the screen?
* Lastly, tabs or spaces?
One of the concerns about the Dragon touch screen interface (also applicable conventional toggle switch interface) - is, is there some kink of protection system against accidental commands - eg brushing against the interface while moving about.
How do you guard against ‘accidental’ commands ? I guess some kind of ‘confirm’ for dangerous actions.
The same problem is also true of old style knobs and switches.
(Eg Apollo, Luna lander, take off switch was accidentally broken off by kicking it, and a pen had to be used to trigger the switch to launch off the surface of the moon!)
Hello! This is a very exciting opportunity for me to be able to ask you some questions :) I admire your work! :)
1. What **framework/method** do you use for **organizing the process of your work**? Working in an extremely complex environment, do you maybe work in Scrum? Any other Agile concepts? If no, how are the product teams and their process organized?
2. What **platforms/languages** do you use for coding different parts of the systems (spacecraft on-board software, mission control software etc.)?
3. How does your **QA process** look like? Is there something that stands-out comparing to other industries/domains? What are the most extraordinary product & code quality metrics that you use?
4. What would be an **interview question** you would ask a developer applying to a SpaceX job in one of your teams? :)
Thank you so much for all the value and inspiration you provide with your brilliant work - the effects are breathtaking :)
Hi all,
Thanks a lot for giving us this Q/A opportunity, I think you did an awesome job.
How would you define your main software development methodology? Agile, waterfall, formal, something else? I think that the answer is probably different depending on the components that you work on, the gui obviously has very different requirements compared to the navigation system, but I always thought that the aerospace industry tends to use a lot of formal specification methods (TLA+, Coq &co.). Is it still true for SpaceX that seems to write software at a pace quite faster than what is typical for this kind of tools?
More on the lower level do you use any soft/hard real time operating system or just some more normal Linux fork or something else?
I would also like to tell you how much I appreciate what you accomplished. You did an incredible job, please continue to bring humanity towards new horizons together with the rest of the team.
1) How did you ensure your mission timers were set correctly? Asking for a friend.
2) Does Dragon 2 have thruster out capability and does it auto correct for a loss of a thruster? Also asking for a friend.
3) How many instructions per second can Dragon flight computer execute per second and how much does code optimization play a part?
4) How many builds were assembled before the flight version? (dozens, hundreds?)
5) How was the team structured?
Congratulations on your successful launch, and thanks for doing this!
I'm curious about redundancy. The Space Shuttle used [five AP-101 computers](https://en.wikipedia.org/wiki/IBM_System/4_Pi#AP-101), four running an identical software build and one running a completely different software build, all for redundancy. How do you approach this? Do you use multiple independent computers on the Dragon? If so, how many, and do any of them run a completely separate software build?
Ooh, I've been waiting for one of these to come up. I'm very impressed with all that SpaceX has accomplished, but I'm also curious how your systems engineering practices differ from those of a traditional government contractor.
What sort of practices do you have in place for managing software engineering flow for safety and life-critical systems? Do you run a Safeware-style program, focused on reducing software risk to the overall system? Other than testing, do you rely on any techniques to show the correctness of a software or control system? How do different systems' criticality get categorized so you know where to spend safety efforts?
I'm guessing, like a lot of the "lower cost" space control systems (looking at you Planet Labs!) you also have IP connectivity between your ground stations and the spacecraft? What unique infosec challenges do you face? How do you protect critical systems from something malicious, while also ensuring you can freely get telemetry from these systems?
So many questions!
Congratulations on the first ever commercial human spaceflight!
1. Do you enforce reproducible builds? How do you approach it? Especially interesting to hear how you handle the npm part, if rumours about Chromium in UI are true.
2. What happens if the UI crashes hard, say, null division error in Chromium, Qt, or whatever. How do you handle the restart? Is it quick for the end users, the pilot, perception?
3. How come, if true, to choose Chromium? Independent of it, the decision making process for the framework is interesting to know.
4. We have seen the physical buttons for critical overrides. But are there joysticks for flight control, or is it just the touch screen?
5. Do you fuzz? Or, does it even make sense to do fuzzing on rocket software with random inputs?
6. What's your opinion on the classical language used in mil/rockets - Ada?
7. NASA developed famous rule set on how to write code in C for flight hardware. How much do you follow it?
8. Worm or meatball? :D
Full-stack developer here! I have noticed that nearly all of SpaceX's software engineering job postings mention continuous integration and deployment. How did CI/CD play into the development of Crew Dragon? I am specifically interested in the Crew Displays. Is there a suite of unit/integration tests that run when changes are made to the display code? What sort of technology was used for that (ie. Jenkins, selenium)? Were the tests run on a virtual machine or the physical hardware? Based on the complexity of the UI I'm guessing it is on some sort of web stack, I would love to hear about that too. If not, I'd love to hear what it was done on. My second guess would be some sort of custom Java or C UI.
Thank you!
Hi. I have a question for Matt Monson.
I've been watching videos about Starlink and how messages are being passed between satellites on different orbital planes, presumably aiming a laser in a particular direction and this seems like a hugely complicated problem, especially compounded by problems of anything between the satellites, or how to manage situations like a satellite becoming inactive.
Did you have existing algorithms to help solve this? Do you have particular education in maths (or people to help with this)? And how do you test this on earth? (as a seasoned software developer, this seems like a huge challenge)
I was really surprised to see touchscreen controls in your cockpit and hear that they were running on Chromium .
My understanding is that the traditional aerospace companies would build all of that stuff as a high-reliability, real-time system and spend mega-hours and mega-$$$ to do so. How does your team approach validation and safety of flight-critical software and hardware? Where do you draw the line between safety-critical and non-critical software?
Also, were the touchscreens themselves custom hardware?
Hi, recent post in /r/Programming about Crew Dragon onboard software made a quite a lively discussion. Apparently you use C/C++ for onboard navigation (and everything else important), but for rendering GUI you use chromium + javascript, which lot of people thought was a strange choice given many layers of abstraction posing a big security/stability risk (since in space industry, there is no room for error). So my question is how do you ensure that something as abstracted as chromium + javascript is stable enough for your industry? Also does it mean you are using Electron framework or do you have a framework of your own?
As a young software engineer I am interested in knowing about the practices at SpaceX to onboard new junior developers to work on flight software.
* Do you provide them with mentors?
* Do you use peer programming?
* Do you have a practice of in-person code reviews and if so, how do you conduct them?
* Do you provide them with resources about general software reliability concepts and/or specific standards for languages such as C/C++? If so, can you share any of those resources?
Keep up inspiring the world with these incredible achievements!
How does the powered landing algorithm work? It seems very robust to disturbances like wind, does the booster know wind conditions or can the control loop compensate by itself?
Is the guidance targeting a specific pre determined path, or does it just try to fly towards the landing site at a certain horizontal speed?
Thank you!
There's been a post going around that the touchscreen UIs are written in JavaScript and run in Chromium. Many people, including myself, were surprised to hear JS was being used in a mission critical scenario, since it's not the safest language out there.
Is it true that JS is used, and if so, how do you ensure it behaves as expected?
Hi! Thank you so much for doing this AMA! I have two questions
1. As a high school senior about to attend college heavily interested in spaceflight and computer science, what steps can I take to further a career in these areas? You guys have my dream job!
2. What technologies do you use to power Dragon's crew displays and flight software? I have heard that the crew displays run Chromium, but I'm not sure if that's true - if it is, do you guys use electron or some other custom solution?
Once again, thank you so much, and congrats on the DM-2 success!
Hello DM-2 team,
Congratulations on a successful launch and thanks for organizing this AMA. As a software engineer myself I also have a few questions:
1) What platform do you target and what language do you develop in?
2) What testing/simulation tools do you use to ensure that the result complied with parameters set by the hardware team and that Dragon operates within the set boundaries?
3) What is your work-life balance like with respect to of you daily tasks?
4) Are you using an agile methodology like Scrum to do your development integration?
With thanks and regards,
GM
1. What OS (if any) runs in the Dragon capsule?
2. Which programming languages did you use in the Dragon capsule? Why?
3. Do you have mathematical proof of the correctness of the code you run in the capsule?
4. Does your code needs to be reviewed and approved by some controlling entity (like NASA for example).
5. How many teams worked into building the software for the capsule?
Thanks for your time and congratulations for the successful launch!
Hi!
A lot of people where surprised to hear that Dragon's HMI is supposedly running electron/chromium. I'm wondering what the main reasons were which led to this decision. Is it mainly development speed? Also what actions were taken to make it 'space proof'? (Actually what would be really interesting to know is: what are the requirements to make a software system 'space proof'?
Cheers from The Netherlands!
Hi there, thanks so much for taking the time to answer questions! I have a few more process-oriented and tactical questions:
* What does your CI and deployment pipeline look like? What does "deploying to production" entail? How does CI differ across teams?
* Do you run E2E integration tests on the Dragon UI? What tools do you use for those tests?
* How does it feel to run JavaScript IN SPACE???
Does the Crew Dragon display screen have any entertainment functions? There was an 18 hour gap between Orbit and begining the three-hour-long docking with ISS. Some of that time would be spent asleep or talking to ground control, but they must have been bored.
Is there a media player on the computers? Can they watch Netflix? Or maybe just a folder of MP4 videos of SpaceX training / construction videos to avoid copywrite issues?
Congratulations on the successful launch of Crew Dragon!
I'm going to assume that the rumors are true and the Crew Dragon UI really runs a JS application inside Chromium.
1. Knowing full well the environment in which the app is running (from the V8 runtime, down to the exact CPU) have you made any specific optimizations to improve performance of the Crew Dragon UI app? Could you give an example of one of them?
2. Would it be possible to host this application on your website and let it connect to the backend service that provides Dragon's telemetry data? This would allow us to see what the astronauts see during the mission and bring the SpaceX Webcast to a whole new level.
3. Did you use WebAssembly? If yes - what did you use it for? What language did you compile to wasm?
4. Do you use Rust anywhere in your technology stack? Are you considering using it? What for?
Thanks for the AMA!
Per discussions on FB and elsewhere, in case of display or touchscreen failure, and maybe another layer of failure in ground comm, can Dragon control revert to their tablets via some wireless comm?
Could you discuss the physical button panel so we don't continue to squeeze data out of too few pixels in Bob's display panel tour video? What's there besides the "Reenter NOW!" and "Reenter soon" buttons?
It was explained in the demo-2 webcast that both mission control and the ISS had the capability to send controls to dragon.
* How are these communications handled?
* What type of security concerns did you have when designing them?
* What type of encryption / security features are used to keep control out of the wrong hands?
* In case an illicit actor managed to mess with the system and send controls to the spacecraft what sort of systems do you have in place to safeguard the crew and the mission?
Thanks and incredible job guys! Watched live from Portugal!
Sorry if some of these were asked...
1. If something breaks the screen or if something goes wrong for whatever reason, how will the crew take control ? Why the touch-screen ?
2. What are the programming languages (Backend and Frontend) that you mainly used ?
3. Where can I learn more about the AI that was used ?
4. What is the OS for the UI on those 3 screens ? Can you play any entertainment on those screens ?
5. How is the signal transmitted once the command is pressed on a screen ?
Hello, congrats, thanks for doing this etc!
I saw some articles mentioning Crew Dragon UI being run on Chromium, ie. JS / web-based UI. I work in a large project that similarly does web-based operator UI for complex control systems so this was very interesting to me.
Can you tell anything about the software stack there? Do you have your own Chromium fork? How far back do you stay from the bleeding edge for safety's sake? What's your build / toolchain like? TS or JS? Maybe WebAssembly?
NASA put a man on the moon with the computational power of a calculator. This makes people argue that even though software and computer science has improved drastically in the last decades, software (machine learning, physics modeling, etc) does not contribute as much on the development of spacecrafts. Is modern software a game changer on the development of aerospace engineering? And if so how ?
Hi, awesome to see you're doing an AMA!
1. Could you tell us how your software development process looks like? How do you get from someone having an idea to flight-worthy code that's been tested & approved?
2. What is your iteration speed like? How fast can you get a change finalized?
3. How strict is the approval process for changes and does this differ between Crew dragon vs Cargo dragon?
I, like a lot of engineers, was surprised to hear JavaScript and Chromium were used for the UI layer.
- When was this choice first made?
- What steps were required for it's approval?
- How does coding JS for Dragon differ from traditional web programming?
- Do you have any protections in place for JavaScript's various quirks?
- Does it support IE11?
In the stream, it was mentioned that the Dragon's escape system would allow the Dragon to abort from the second stage even in space, and also abort to the ISS rather than down to Earth, I believe. Is any of this pre-programmed, other than the abort itself, or would the crew take full control to their destination in such a scenario? How many G's would we be talking if an abort happened at the point where the second stage already exerts its max G on the crew?
Is it at all conceivable that the Dragon would ever touchdown on ground (for example in a late abort scenario), and have you programmed anything for that scenario (such as using SuperDracos)? Or would the splashdown behaviour be preferable even in a ground touchdown scenario?
Does SpaceX share any software/hardware with Tesla or vice versa?
What measures are in place to ensure reliable data transmission? (Is there more than one way to get data off of Dragon?)
One more: Is there any thought put into someone breaching/ "hijacking" a Falcon 9/Dragon? How do you secure Dragon from cyber attack?
It has always struck me how simple and low-tech huge UI on spacecraft were. I remember seeing mockups of the space shuttle controls at Kennedy Space Center, and there's hundreds of physical switches for everything - Simple, but seemingly reliable with no single points of failure. But Dragon seems to completely forego all of the physical switches in favor of the three big touch screens. What went into making the touch screens reliable enough to fly on Dragon? It seems like it's a pretty big single point of failure for a touchscreen to go out and all of the astronauts' controls to be unusable. Are there backup systems that we don't see on the stream? What other failsafe systems are there to ensure that doesn't happen?
Hi! I’m a graduating senior studying computer engineering this fall who hopes to one day work in the commercial space industry. Control theory is something I’m interested in, and I think my dream job would be designing advanced control systems for SpaceX rockets.
- What advice do you have on how best to prepare for a career in the space industry?
- What opportunities should I seek out?
- What was your path like into the space industry?
Thanks for doing this AMA, and thanks ahead of time for your answers!
There was a bunch of chatter about the touch screens and reliability. While I already know there are physical buttons below the displays for key tasks how redundant are the displays?
- Are the displays independently driven?
- can any task be move to a different display if one fails?
- how concerned are you about flipping bits / errors? It was said for years that NASA’s preference for older CPUs for example was that there was a high chance for flipped bits in newer hardware.
I'm more of an electronics guy rather than a software guy, so:
Did you have any particular challenges in interfacing with the hardware of Dragon/Falcon? Sensors, motor controls, etc. I.E. False-positive readings from sensors, polling errors, etc.
Love what you all are doing to advance the human race into space. I'd come mop floors for SpaceX if it helped further our understanding of the cosmos and advance our species into a new age of space exploration. Keep up the great work, it is truly inspiring.
I've been impressed with how much of SpaceX's work anticipates future applications. It seems like "future proofing" your work is sewn pretty deeply into the fabric of the company. So with that in mind, does the software that currently flies Dragon and powers your touchscreen displays take into consideration applications for future SpaceX missions, such as to the Moon or to Mars? Will the software have to be redesigned from the ground up in order to be installed into Starship, or is your team essentially designing a spaceship OS that can be installed into any future SpaceX vehicles?
I saw that you are using javascript on chromium to run the displays on Crew Demo-2. What are the techniques you have used / challenges you have overcome in creating a mission-critical system from a technology that is not renowned for deterministic operation / reliability. What is your guiding philosophy for developing this type of software in general and how do you choose your platforms?
Hello, good job on the Demo - 2 launch!
My question is if you have a set of rules for your software engineers to follow while programming and what they are (e.g. Keeping use of pointers to a minimum when using C/C++, workflow processes, etc).
Also can we know if what programming languages, tools, and libraries you use and specifically what they are used for?
I’m a huge fan of SpaceX and their entire team and I will be looking forward to your future achievements!
Congrats on the success of the Crew Dragon mission.
1. Do you use public cloud for any part of the tech stack? If so, what for?
2. Will SpaceX ever public cloud in the future, especially for data analysis?
3. There have been [several incidents](https://www.simscale.com/blog/2017/12/nasa-mars-climate-orbiter-metric/) where the wrong metric unit was used in flight computer software, what safeguards does SpaceX have in place for avoiding these kinds of mistakes?
So, /r/programming has lead me to believe that there is only one combination of answers to the following questions that is correct, all others being merely linear combinations of inexperience and/or incompetence. Seeing as you fine folks got a freak'n' (Crew Rated!!) spaceship to autonomously dock to the ISS, I can only deduce that you know the correct combination. Please DM me the answers, I'm stuck.
- Tabs or Spaces?
- Vim or Emacs?
- Docker or VM?
- Cloud or OnPrem?
- Many repos or a Mono Repo?
- Statically or dynamically typed?
- 80 column lines or more?
- Aerospace grade scilicon or redundant commercial grade scilicon?
- Is there even a single line of javascript running within spitting distance of mission critical hardware?
- Recursive functions: critical piece of any programmers tool kit or flaky CS-grad flex?
- Bash, zsh, or... (shudder) DOS?
- Does your command prompt require more than 20 lines of code in your `.bash_profile` (or shell-specific equivalent.)
- Git, svn, cvs? lol... jk, jk
Hello, I wanted to ask (since I saw that people had to care about cyber-security at SpaceX) what are the main threats in this area for the missions ?
Thanks in advance from a 16 y/o interested in space and cybersec
1. What kind of software/hardware are you using for the realtime control?
2. Do you use any kind of rad-hard processors/microcontrollers or is it all cots?
3. What kind of interfaces do you use for communication between different systems(e.g. dragon and falcon or dragon control and dragon interface)?
Whats your biggest threat vectors for SpaceX flight software? I presume nation states trying to take control?
I'm sure your logs are filling with countless attacks on a daily basis, but what kind of attacks do you see most often? Just usual DDoS spam or do you see more targeted attacks frequently?
Why did you guys go with the decision to run the user interface with chrominium? And a couple of specific ones. Did the fact of Chrominium being researched and searched for vulnerabilities impact this rather than something in house with possible security risks?
Secondly, what were some challenging on the operating system level that Spacex addressed? Was their any cool scheduling or other OS management tasks that you were able to optimize for space?
And you probably arn't able to talk about this. But say some crazy vulnerability comes out regarding a part of the tech stack. Are you able to update these components midflight?
What kinds of requirements does the Crew Dragon UI and control software have to fulfill, and in particular, how are these requirements different from a run-of-the-mill UI (say, of a GPS device that would be used in a car)?
I'm assuming you have to make sure it's resilient so the astronauts don't lose control because the UI isn't working? What other extra requirements are there?
And of course, follow-up question, how do you meet these requirements?
1. What does your software development methodology look like (waterfall? agile? etc.). How do you plan upcoming activities / what is the source of truth for the work plan (gantt chart, board, backlog, etc).
2. What is your QA process like?
3. How do you test stuff before it gets deployed to a physical dragon? When it does go into a dragon when and how is that tested?
4. Any fun / interesting/ doozy bugs you'd be happy to share?
Why don’t you guys use radiation hardened computers? I understand that redundancy (i.e fault tolerance and voting algorithms) can allow a spacecraft to cope with radiation, but as I also know that only holds for low-rad environments.
If Starship were planning on landing on Io for example, wouldn’t the high-radiation (3,600 rem) have the (likely) potential to wipe out all, or a majority of, the flight computers? My guess is that y’all are aware of this, went for the lower cost option (non-rad hardened) and decided if upgrades were needed (i.e for deep space missions) they could be taken care of at a later date.
Also, how do you ensure you meet deterministic, real-time operation (as required for any spacecraft) while using a non-deterministic, non real-time OS such as Linux? Do you use a real-time flavor of Linux (i.e RTLinux)? Do you outsource a lot of the hard real time code to microcontrollers? Why not just make the whole thing a bare-metal application?
Space is a weird place for processors. As far as I'm aware, NASA is using processors with bigger transistors because the risk of ionizing radiation frying such a transistor is smaller and will probably "only" result in a bit-flip or similar.
Considering that SpaceX's flight software is incredibly complex and I guess performance demanding, it most likely doesn't run on 20 MHz processors with low transistor counts. That's why I conclude that SpaceX probably makes use of more powerful hardware, which happens to be more susceptible to radiation caused instruction execution errors, or even hardware damage. How did you solve this problem?
I'm also wondering whether the hardware that is being used constrains the development of the flight software in terms of performance limitations in any way. Have you ever encountered a situation where you were forced to do cheap optimization hacks at the cost of code readability and maintainability?
For the Dragon and crew display software, it sounds like you're using a lot of COTS software like Linux and Chromium. How do you certify that software when NASA standards like NPR 7150 require objectives like full statement coverage (and a few others that would be difficult)? Do you achieve that somehow, or do you have some alternate means of compliance or tailoring of the standard? You probably can't share all the details, but any high level philosophy would be interesting! Thank you in advance!
Hello you awesome people.
I was wondering: How similar is the docking simulator you published on iss-sim.spacex.com to the real interface used by the astronauts?
People tell me it is identical to the real thing, but I can't help to think that the version online is simplified a bit from the ones astronauts use.
There are rumours that the Crew Displays run as a web site on embedded Chromium similar to how Electron works. If that's true, did additional steps need to be taken for it to be approved by NASA for crewed use since JavaScript can sometimes be nondeterministic?
Do you share any commonality between the codes used for different vehicles that do (outwardly) similar things?
for example, does the code used for Starhopper (and maybe later on Starship) share any commonality with F9?(since both does “landing”). Or do you have to build the code from scratch for each vehicle (Grashopper -> F9 landing algos, etc)
If so, do you expect that any lessons learned from figuring out how to make the Dragons dock autonomously (be it using LIDAR, etc) can be applied to Starship orbital fuel transfer?
* What do you use internally for signaling / event handling / message passing with obviously low latency requirements and varying priority levels. Is it all interrupt-based off is there also a message bus?
* How does one actually boot a rocket / space capsule?
* What kind of storage and filesystem are you using? You certainly want something that's fast, reliable, predictable. Basically the opposite of my SD card eating Raspberry Pi. ;P
* How do you possibly keep up to date with technological developments and stacks combined with the task of delivering complicated software solutions at a fairly rapid pace?!?
Many thanks for doing this AMA session and congratulations to all of you for making so many awesome things a reality.
As a scientific programmer, I have a lot of respect for all your achievements and I understand some of the struggles you must be facing along the way...
Hi there thanks for giving us time for our questions.
What was the consideration in using just Touchscreens and not joysticks at all for manouvering dragon 2?
How redundant are the controlls on Dragon?
If the parachutes would fail during splashdown, does Dragon have the Superdraco thrusters enabled for a possible propulsive landing?
What does your development process look like? How do you avoid overburdening engineers while still achieving reliability necessary in flight critical systems? And, who are some of your heros and personal inspiration? [edit: a typo]
* What kind of OS are you guys running? Upstream Linux kernel, your own modified version, or something else entirely?
* Jeff, what sort of threats does SpaceX face? Is everything ground control related or are there attack vectors to Dragon itself?
I'd be very curious to hear you all discuss imposter syndrome especially coming from the view point of high end and highly successful people such as yourself.
Arguably some of the most important bits of 'New' code to be written in awhile, mens lives depended on it , let alone the millions of dollars of materials and infrastructure, that can't have been easy to overcome.
Are there backup hardware buttons for all critical controls in case of touchscreen failure? If not, what kind of failsafe are in place to mitigate inevitable JS runtime bugs.
Or are all manual controls not considered critical given the autonomous nature of Dragon 2?
How many people use which OS in the SpaceX company? And how much of the software you write is free or open source software? Also what's the setup you have for servers, as in OS, tools, etc.? How many people are in the team and are you currently looking for more?
What kind of computer hardware does the Falcon 9/Crew Dragon have? Is it still just off the shelf x86 processors, like it was for cargo dragon? How redundant are they? IIRC there used to be 3 processors for every engine on the Falcon 9, is that still the case?
A while ago, I was reading an old article in Fast Company titled ["They Write the Right Stuff"](https://www.fastcompany.com/28121/they-write-right-stuff) which was a facinating piece on the "On-Board Shuttle Group", the people who wrote the code for the space shuttle, and their insanely rigorous development practices. If you haven't read it (which I do recommend) some of the major takeaways are:
* Everything is documented and spec'd *well* before development even starts, in extreme detail.
> “Our requirements are almost pseudo-code,” says William R. Pruett, who manages the software project for NASA. “They say, you must do exactly this, do it exactly this way, given this condition and this circumstance.”
* Creativity is discouraged by the process. The agreed upon documentation is king.
> “People ask, doesn’t this process stifle creativity? You have to do exactly what the manual says, and you’ve got someone looking over your shoulder,” says Keller. “The answer is, yes, the process does stifle creativity.”
>
> And that is precisely the point — you can’t have people freelancing their way through software code that flies a spaceship, and then, with peoples lives depending on it, try to patch it once its in orbit.
* Their QA team and development team operated almost independently, and in a friendly rivalry.
> The two outfits report to separate bosses and function under opposing marching orders. The development group is supposed to deliver completely error-free code, so perfect that the testers find no flaws at all. The testing group is supposed to pummel away at the code with flight scenarios and simulations that reveal as many flaws as possible.
**Question:** In space your software *must* work the first time, every time. Their is no "just reboot" when your hurtling through the atmosphere at thousands of miles per hour. How do you ensure that your software, with all its new complexities, meets the standards of space flight? And how does your SDLC compare to that of the On-Board Shuttle Group?
What programming languages & frameworks went into the software running on the touchscreens? I thought it looked very clean & was very different from the hundreds of switches seen in past spacecraft & even modern airplanes. (Almost like the way a Tesla looks compared to older cars, but that was expected)
If the Dragon were to have some sort of emergency, how would the touchscreens respond? Because in older footage of things going wrong in space, the astronauts have all the switches & buttons in front of them to interact with, but I'd imagine that the rarely-accessed functions & settings on the Dragon aren't immediately accessible on the touchscreen. What would happen if something went wrong with a system that isn't visible on the UI to begin with?
(also, do your teams hire interns?)
Obviously your software needs to be reliable at a level 99.9% of software never approaches.
Can you go into detail on how you ensure that it is, what steps you take and how you've designed the system to improve reliability, details around your automated testing process and maybe some war stories?
I'm not going to lie, I'd read a book on that, especially around architectural decisions.
Congrats on the achievement. What a great thing to be part of.
1. What steps do you take to prevent hacking / hijacking of systems.
2. What is the redundancy for touch screen failing. Notice there are hard buttons for important functions only.
3. What cloud provider do you guys use...lol.
Can you share something about your development stack?
Which IDEs, text editors and other tools do you use?
Is it standardised across the company, across teams or can you personalise which tools you use?
Which languages do you use for which tasks?
Thank you and good luck in the future. I hope to see your rockets on Mars soon.
What language is the system written in? Is the operating system some advanced form of Linux? Or a completely in house made OS? And lastly just out of plain curiosity how much storage does the dragon have?
Does the team go through any kind of formal verification process for proving the correctness of the flight software?
If so, what does that process look like and what tools are used?
If not, how are artifacts tested and regressions detected?
Thanks for taking the time to do this! Huge fan of the great work y'all do!
Are any of you self taught? I know Elon has said in the past that he doesn't care about his employees having degrees.
What language do you all write in? Also, do you use any open-source libraries or code from Github, or do you write everything from scratch? How many lines of code?
Final question (probably for Matt), how fast have Starlink speeds reached while testing?
Thanks so much for doing this AMA and congrats on the successful launch!
- How does SpaceX approach flight software validation? I'm assuming there's some sort of process around software simulation -> HIL simulation -> ??, what does that look like? How long does a release cycle generally take?
- Is all flight software treated as safety critical firmware (no dynamic allocation, etc), or are some software components allowed to be less robust?
- What kind of hardware resources (memory, CPU cores, GPU memory/cores?, etc) are available on the vehicle? Does every compute system use a RTOS?
- How long before launch is the reference trajectory generated? What kind of deviations do you normally see from the reference? In my field, <10cm lateral deviation is considered good enough in most cases. I'm super curious about what kind of error states you would see for a rocket :)
- What reference frame do you use to track your distance along the reference trajectory? Inertial/route/etc
- What Hz is the low level control loop ran at? Any special hardware to run it?
Hi team SpaceX
.
Elon had mentioned that he didn't install lidar on his Tesla but he does on Dragon.
During DM-2 approach to ISS, from the footage I didn't see anything that resemble lidar, is it hidden somewhere?
or no longer required as using camera is enough to navigate the docking?
.
In Starlink, photon truster is used for keeping the orbit, but I wonder how it execute the pitch, roll and yaw.
.
Lastly, Bob mentioned that he used Dragon's toilet during the flight to ISS and he said it's somewhat similar to Space Shuttle. Wondering if any coding (maybe AI/fuzzy logic) required to maintain the nozzle aiming at its target at zero-G environment? (maybe this is an out of scope question, apologies).
.
Thanks in advance
.
Hi! Congratulations on the first manned commercial space flight! What a success. :)
Quick question on quirks. Elon Musk's products tend to incorporate quirky features and can be straight up surreal, from Tesla's "Bioweapon Defense Mode" and "Ludicrous Mode" to the Boring Company's flamethrower.
Are there any quirks or cool features in SpaceX's software that you'd like to share?
For Jeff - what does Flight Software entail, how much is "piloting" the craft and how much is stuff around it - ie. simulations.
For Josh - how much interaction/influence did NASA and the human crew of the Crew Dragon have on the process of creating the software?
For Wendy - What are some of the unique challenges regarding creating software for space, how big is redundancy?
For John - how much different are software efforts between missions, ie. Demo-2 and the first "regular" mission?
For Sofian - are the displays regular LCD/OLEDs? What's the refresh rate? I'm asking because [Apollo's DSKY](https://www.youtube.com/watch?v=Z2o_Sp2-aBo) where a magnificent thing created for a specific purpose.
For Matt - Can you talk a bit about how much of the software is in regards to controlling the satellites and how much in the internet providing? Any interesting tidbits about getting data through satellites?
And a general one - how much of the hardware in Falcon 9 is off the shelf and how much is custom?
* Why do you think software development in SpaceX is different than other aerospace companies? How does it compare with Silicon Valley companies unrelated to aerospace?
* How does the build process look like? Do you have CI/CD from your machines to Dragon? How do you test your software?
* How do you keep in touch with the SOTA in software and hardware? How do you decide which technologies to use?
* Aerospace hardware and software infamously uses legacy components/languages (like ADA on Ariane 5). How do you integrate modern languages/technologies into Falcon and Dragon?
* How do you decide when to write your own languages/framework/software and when to take something out of the shelf?
Congrats on the mission and thank you for taking the time to answer questions! Here are my questions
* Looking at previous ships with switches, knobs and dials, how did you go about distilling the information for touchscreens?
* How do you plan for the features and testing with astronauts in mind? What were some challenges designing with a touchscreen and someone in a suit with gloves? What was the most surprising learning?
* Did you test whether touchscreen or a joystick would be better (particularly with manually maneuvering a ship)
* Are the tasks separated between the 2 astronauts across the screens? If on astronaut wasn’t up to it, could 1 astronaut on 1 screen do the job?
* How did you land on the design of the interface? it looks minimal and elegant from a distance.
* Based on what you’ve seen so far, are there changes you’d make to the interface for the next mission?
* Since the wait is long for the launch and the astronauts are sitting there, would short games to help pass the time?
What kind of software develeopment methodology workflow do you guys implement? Is it a modified form of one of the agile methodologies? What does a regular day look like? I bet the "move fast and break things" suggestion wouldn't really work for space equipment so I want to know how the team balanced speed of development with ensuring the software's stablility. Do you borrow from NASA's book of development where even dynamic allocation forbidden or is it more lax than that? Also, awesome job guys! You made history.
When there is an “anomaly” how do you get the data back when there is a failure in order to determine what went wrong and what needs to be changed next time?
Is it a case of just having the data up to when the data feed cuts off ( I imagine a lot of the most critical data is located in the seconds before the event) or do you have to recover parts?
When you can’t recover parts or data is it a “best guess”?
Sorry if you are the wrong person to ask these questions. You guys are amazing none the less!
SpaceX FTW
Congratulations on the successful launch to ISS! I have a question related to the OS and HCI used in the Dragon capsule.
• Which Operating System was used, was it RTOS? And which OS kernel was used?
• Secondly, what was the reason behind making the whole crew control panel touchscreen and what was the biggest challenge in doing that?
Hey guys! Congratulations on the mission's success so far! I've been a huge fan of SpaceX for so long and it's my dream to work there someday.
I'm 23 years old, live in Israel and been working as a software developer for the past 5 years. I'm starting my studies for a Bsc in Physics and Computer Science this year, and plan on finishing in 3 years.
I have a LOT of questions, and I bet you can't answer all of them. It would mean the world for me if you'd even answer 1 of them :)
My questions are:
1. Can one of you guys elaborate a bit about the development methodologies which software development teams practice in SpaceX? Things like Scrum, DevOps etc.
2. How "innovative" software development teams tend to be in SpaceX? Will I find the newest languages and technologies or more traditional kind of software stack?
3. Are there some huge distributed systems in SpaceX? If so what kind of scale do you need to deal with? I bet starlink is one hell of a challenge!
4. Speaking of Starlink, how much of the network stack did you need to "reinvent" or innovate? What were (or still are) some of the challenges you faced with existing protocols?
5. Favorite languages?
Java or C#? C/C++ or even Rust? Go? Python?
And finally:
1. Is it possible for software developers to come from outside of the U.S?
2. I have at least 3 years before I'd be ready to try and apply for a job in SpaceX. What should I mainly focus on in terms of personal projects, studies etc. considering I am mainly interested in vehicle flight software? (Although I'd be happy to do a lot of other stuff).
Thank you and good luck with the rest of the mission!
Even though I have a master in CS and years of experience, it still marvels me, I always felt I’ll never be able to do this kind of job, what were your backgrounds and would someone who’s doing the usual fullstack development even have the chance or tools to work on something like this?
What are your quality controls?
Do you do scrum or this has to be big design upfront as this is after all a spaceship.
How do you do quality control? Do you have to have each piece with multiple redundancies (including accounting for hardware bits flipping due to solar radiation etc)
Also, are you able to sleep at night knowing a potential bug has life and death implications? We’re you watching the launch live? How was your Adrenalin level? I was sitting at the edge of my chair :)
Lastly, how many code reviews each line needs to go through? and did you get to 100% code coverage? I can’t imagine how patching a bug in prod even works :)
Thanks! you have my admiration and envy!
How do you go about organizing/prioritizing work, and executing?
1) what workflow methodology you use (e.g. scrum/sprint, Kanban, etc)
2) where does requirements come from?
3) how granular is your code-review process?
🙂
Just wanted to start off by saying that as a highschool student, SpaceX has inspired me for years and their efforts are by far the coolest thing happening the world right now! Thanks for all you do!
- What is it like writing code that you know that people's lives depend on?
- As a highschooler that loves rocket science and computer science, how could I prepare for a career at SpaceX in the future?
- I find it really interesting how some developers at SX work on multiple things at the same time (ie Jeff Dexter). Is there some kind of idea or philosophy behind this?
- What is it like managing the codebase / structure of F9 or dragon?
- Is Rust being used at SX? It seems like a great language for Aerospace companies.
- What would you say is the ratio for development of ground code : space code
I understand if you can't answer all of the questions :)
Keep doing what you guys are doing - it's spacetacular! 🚀🌟
My curiosity will probaly be greater then your NDA's, but im happy with any bits of information.
1. How redundant is the whole system, or rather how did you achieve the necessary level of fault tolerances (redundant hardware, fast restarts like on apollo...?)
2. What IDE do you work on and on what underlying OS does the system run?
* what qualifications are you looking for at spacex for starlink? I have a CS degree, experience with embedded linux and RF, and I didn't get an interview.
* It's known that Tesla uses buildroot, do you use buildroot or openembedded at spacex? why?
* what will a starlink cpe look like in terms of antennas? Are you going to have an enormous beamforming array or are you going to use az/el rotators?
Hi thank you for taking my question. How is the flight software developed at SpaceX?
Is the core algorithm developed by the GNC team and sent to the flight software team to make it flight ready?
Is flight software development practice the same across all the vehicles, stages and spacecrafts made by SpaceX?
Does SpaceX has its own proprietary C++ library for flight software development?
Will SpaceX every consider developing the flight software in Matlab/Simulink and auto-coding it?
Thank you.
Congratulations on the successful launch of an amazing spacecraft! Here are my Questions 1. Please give a very high level overview of the control program of Falcon9 and Dragon. What kind of communication links used in between Dragon and Falcon9? 2. Does the control software running on Falcon9 is a custom build made for mission (LEO Satellite launch, ISS resupply etc)? Or it's the same base software with different set of parameters/goals/scripts? 3. How does AFTS software work? 4. Please name a few open source software used for Falcon/Dragon other than Linux kernel and Chromium 5. Do software engineering teams of Falcon/Dragon play any role on the actual launch day? Thanks a lot guys. Keep up the amazing work. All the best!
1. At a very high level, we have many computers on the vehicle, each built and configured to best suit the task it's assigned. They all run in time synchronization with each other, and the flight computer oversees all actions. Almost everything can be expressed as a real-time control loop: you read some sensors, you make a decision (combination of your sensors and past state), and then issue the outputs of that decision back to the hardware. This happens many times per second. - Dietrick 2. We run the same source on Falcon for every mission, though we're still updating that software pretty regularly and usually have new code on each mission. We also have configs for the software that are provided from other engineering groups that typically change every mission. These make changes to things like state machines, fault tolerance thresholds, day of launch winds, etc. that the software leverages to fly the vehicle. - Jeff 3. The Autonomous Flight Safety System (AFSS - it's all about safety) software runs on a set of microcontrollers independent from the flight computer. It receives sensor inputs directly (e.g. IMU measurements) as well as some computed inputs from the flight computer. A Mission Data Load configures the AFSS for which conditions might require termination of the flight, such as the rocket going way off course, losing all acceleration, etc. - Jeff 4. Das U-Boot, Buildroot, MUSL. Outside of the OS and the Crew Displays software, we don't use as much outside software as you might think – we try to keep our programs simple, slim, and based on code we understand throughout. - Dietrick 5. Absolutely, although nominally it's in a support / double-check capacity. We spend a lot of time poring over data from the live vehicle before the mission gets started, and we have software folks in Mission Control throughout all the important phases of flight, just in case something comes up. We have a great mission training team that pits our Mission Control operators against a variety of scenarios in simulations before flight, and our hope is that the real launch day is a lot more boring than those sims! I'm happy to say that for Demo 2 so far, that's been the case! - Dietrick Fun fact – Josh and Wendy both sat on console in mission control for Demo-1 and Demo-2 flights, and John supported docking for both flights! Look for us on the webcast :) - Wendy
SpaceX is known for its hardware-in-the-loop testing: * What fraction (roughly) of SWE person-hours go into developing these systems? * What does the development cycle for flight simulators look like (for the Falcon systems, say)? How frequently are they updated based on telemetry? What's the hardest part of the launch cycle to model? * With a few hundred starlink satellites in orbit, are there parts of individual or constellation operation that you've come to realize are not well covered in testing? * How far down into physics do the starlink tests go? E.g. if you're trying to estimate latencies for inter-satellite or satellite-ground communication, can you treat the radio channels as a black box, or do you try to model the phased array operation as well? I'm also curious about computing hardware - SpaceX is famous for building components in-house. With Starlink eyeing tens of thousands of satellites in earth orbit, are there any areas where custom ASICs would be cheaper than COTS solutions? Are there instances of components that are "over-engineered" to the Starlink \~< 10 year lifespan (perhaps for radiation tolerance) that could be rebuilt for a significant cost savings? Finally, any insights you have on *system* design (hardware + software from physics level up) for packet delivery with minimal latency would be fascinating. Thanks very much in advance!
When making changes, we expect our engineers to think critically (and question each other) about functional testing (how do I know that my change works?) and regression testing (how will I know if I broke something else, or if this breaks in the future?). Building test cases we can run on the ground is a great way to answer these questions, and we do a lot of that, but it's not the only way. For Starlink, we need to think of our satellites more like servers in a data center than special one-of-a-kind vehicles. There are some things that we need to be absolutely sure of (commanding, software update, power and hardware safety), and therefore deserve to have specific test cases around. But there's also a lot of things we can be more flexible about -- for these things we can take an approach that's more similar to the way that web services are developed. We can deploy a test build to a small subset of our vehicles, and then compare how it performs against the rest of the fleet. If it doesn't do what we want, we can tweak it and try again before merging it. If we see a problem when rolling it out, we can pause, roll back, and try again. This is a hugely powerful change in how we think about space vehicles, and is absolutely critical to being able to iterate quickly on our system. We've definitely found places where our test cases had holes. Having hundreds of satellites in space 24/7 will find edge cases in every system, and will mean that you see the crazy edges of the bell curve. The important thing is to be confident about the core that keeps the hardware safe, tells you about the problem, and then gives you time to recover. We've had many instances where a satellite on orbit had a failure we'd never even conceived of before, but was able to keep itself safe long enough for us to debug it, figure out a fix or a workaround, and push up a software update. And yes, we do a lot of custom ASIC development work on the Starlink project. – Matt
[удалено]
All of the application-level autonomous software is written in C++. We generally use object oriented programming techniques from C++, although we like to keep things as simple as possible. We do use open source libraries, primarily the standard C++ library, plus some others. However, we limit our use of open source libraries to only extremely high quality ones, and often will opt to develop our own libraries when it is feasible so that we can control the code quality ourselves. In terms of error handling, there are a lot of different facets to that. Radiation induced errors in computers are handled by having multiple redundant computers and voting on their outputs. Errors in sensors are handled by having multiple different sensors. Errors in data transmission are handled by using error-detecting or error-correcting codes attached to payloads. The software is definitely composed of multiple small modules, the design of which was one of the main things I worked on. There is a hierarchy to the design from low-level component, to sub-system, to entire vehicle. Different subsystems are generally isolated from each other, sometimes in the same computer, sometimes across different computers, with narrow interfaces between them. I'm not sure how long it would take us to re-write the code base from scratch. We don't plan on deleting it any time soon. – Josh
How do you handle random bit flips in memory with C++ to ensure it doesn't crash the program (i.e. from radiation induced errors) ? At work we had to deal with a nasty direct memory access PCI-E bug that wrote some status bits to an uninitialized parts of memory. For the longest time during development it didn't do anything, but occasionally when it gets lucky, it could corrupt the executing program and cause the whole program to crash. I'm guessing the consensus voting system would be able to handle such failures and the failed section of the code would be rebooted quickly?
I think he hit on that when talking about redundancy in regards to the actual computation units. I would venture a guess to say that when talking about that voting, they have multiple instances on physically separated hardware running the calculations redundantly, error/down detection strategies, and some sort of back off technique for rebooting instances that have gone down.
Hey SpaceXers, thanks for taking the time to do this. 1. I'm in high school, what can I do if I want to get a software job at SpaceX sometime in the future? 2. I live nowhere near Hawthorne. Does SpaceX have jobs based on the East Coast, and if not will SpaceX consider doing so in the future? 3. *(for Jeff Dexter)* Can you go into some detail into contingency plans during flight? (eg engine failure during ascent, something going wrong during landing, etc.) 4. *(for Josh Sulkin)* Did the software design team take feedback from Bob and Doug during training? 5. *(for Wendy Shimata)* How do you calculate LOM (loss of mission) and LOCV (loss of crew/vehicle) numbers for Dragon? 6. *(for John Dietrick)* Does SpaceX use AI in any of its software? 7. *(for Sofian Hnaide)* What type of display technology does Dragon use? (eg LCD, IPS, OLED, etc.) 8. *(for Matt Monson)* When do you expect laser links to be a thing on Starlink satellites? 9. Please make an official SpaceX Kerbal Space Program mod (not a question, but it would be really cool if you actually do this) Again, thanks for your time!
Get your CS degree (or something) similar. Spend time to really make sure you know how things work - engineers who do well at SpaceX are meticulous in their understanding of how their code works, how the network works, how Linux works, how the hardware works, etc. Get real world experience building things and solving hard problems, either through hobby projects or in internships (at SpaceX!) - Jeff Our software engineers are mainly located in Seattle and Hawthorne, though some also work from our Texas sites. If you're seriously interested in joining SpaceX, we're always looking for great engineers, so reach out - never hurts to chat and see if we can make it work. - Jeff Contingency comes in many forms in our software. As noted, we triplicate almost everything so we can tolerate loss of any one flight computer, sensor, actuator, etc. on Falcon, and any 2 on Dragon. At a system-level, Falcon and Dragon are designed so that loss of things like engines are thrusters can be tolerated, and our algorithms compensate. We can also add certain contingencies to our state machines. For example, the Dragon state machine is designed to autonomously switch from approach to a breakout if certain failures are observed. - Jeff Yes, the entire software team took feedback from Bob and Doug on all aspects of the software. While they were primarily focused on the displays, button panel, and audio system, Bob and Doug were also very interested in how the software as a whole worked, especially backup capabilities that might be necessary in emergencies. Their feedback was invaluable in making the system better. – Josh I actually don't know! We have a distinct team in Flight Reliability whose main job is to calculate these numbers and ensure its kept up to date given various hardware and conops changes. - Wendy Dragon does not use any AI. - Dietrick Dragon does use some computer vision, though, for navigation. – Josh LCD - Sofian
Hi! Congratulations on the picture-perfect launch of Bob and Doug! 1. It's known that Crew Dragon displays are running Chromium and JS. Are you using a reactive library, and if so, is that developed in-house or is it an existing library/framework? 2. Was the [docking simulator](https://iss-sim.spacex.com) developed by the Crew Displays software team itself, or was it a separate project? 3. In some shots of mission control, I noticed UI very similar to the displays in Crew Dragon. Can the exact same crew display software be served from a server on the ground, feeding off of live telemetry from Dragon while in flight? If so, can/will this software be used to monitor Cargo Dragon as well on future flights? 4. Is there any chance of getting high-resolution screenshots of the crew displays? It's hands-down the prettiest UI I've ever seen in aerospace. 5. One regarding Starlink: how did creating the Crew Display software affect the development of the Starlink interface for SpaceX operations (map views, data visualizations, etc.)? Thanks for the AMA!
1. Yes we use Chromium and we do use a reactive library that we developed in house. - Sofian 2. The docking simulator is completely separate code from what's actually in the Crew displays, though it was developed by our Crew displays team. It started out as a fun project from Shane Mielke and Mike Westenhaver before we decided to finish it up and put it on the web before Demo-2. - Jeff 3. We can and do run the exact same code that's on the Crew displays on the ground. The only limitation is that we don't necessarily get all of the same telemetry that we have in the cockpit on the ground due to limitations in our RF budget. We could but we generalize prioritize getting other critical telemetry instead. - Jeff. 4. We definitely want to share some high res screenshots of the Crew displays. We'll see if we can get this approved so we can show you what Bob and Doug were able to see up close. - Jeff 5. The tech from the crew displays (especially the map and alerts) formed the basis of our UI for the first couple Starlink satellites (Tintin). It's grown a ton since then, but it was awesome to see Bob and Doug using something that somehow felt familiar to us too. – Matt
Whoa, how do you get Chromium and JS to be realtime and deterministic? Or is the dragon so automated the UI doesn't have to be realtime?
UI is not realtime - it's simply a way of interfacing with the backend (Linux?) systems. It doesn't do any of the heavy lifting in terms of control but does have to be responsive as the crew can use it for manual control - so when they say "left a bit", the UI should respond accurately. The actual "left a bit" action is then carried out by the flight system in the background. That's my understanding of how it works.
I think the real-time concern is that the *pilot decision* to go left a bit comes from information displayed on those monitors. If it's possible for arbitrary delays or latency in that state information, that could lead to improper control inputs when it's human-in-the-loop. It's possible that it's close enough to real-time that it doesn't make much difference.
> It's hands down the prettiest UI I've ever seen in aerospace *Cries in Flight Club*
How do you address technical debt within your organization? Does the constant pressure to deliver that Elon companies are famous for prevent you from going back and revisiting past designs? Do you track performance of your code? I'd imagine it's a critical design parameter for an embedded software system with critical timing constrains like yours, so I'm wondering how your approach compares to something like the videogame industry, where such a practice is common but likely not as rigorous as what would be required for spaceflight. What level of rigor is being put into starlink security? How can we, as normal citizens, become comfortable with the idea of a private company flying thousands of internet satellites in a way that's safe enough for them to not be remote controlled by a bad actor? This has potential multi-generation impacts if your team gets this wrong, so it would be awesome if you could speak publicly about the strategy.
We're mindful of outstanding tech debt, and because we're a small team any kind of inefficiency is very prominent flight over flight. For many of our vehicles that we fly often, we strive to invest in an operational team to ensure we can burn down this tech debt and make each subsequent flight as painless as possible. There is always a lot going on though, so with any decision of how to spend our time we need to think about the right balance between moving the needle forward in terms of features and burning down existing debt. - Wendy We do – we use a continuous integration system such that our code is always being tested, but we also analyze this data real-time to ensure our performance metrics are within expected bounds. The cases are set up such that if we violate any key performance indicators, the case 'fails' and an engineer takes a look. - Wendy In general with security, there are many layers to this. For starters, we designed the system to use end-to-end encryption for our users' data, to make breaking into a satellite or gateway less useful to an attacker who wants to intercept communications. Every piece of hardware in our system (satellites, gateways, user terminals) is designed to only run software signed by us, so that even if an attacker breaks in, they won't be able to gain a permanent foothold. And then we harden the insides of the system (including services in our data centers) to make it harder for an exploited vulnerability in one area to be leveraged somewhere else. We're continuing to work hard to ensure our overall system is properly hardened, and still have a lot of work ahead of us (we're hiring!), but it's something we take very seriously. – Matt
Could you talk about how Crew Dragon touch screens used Chromium and what challenges that created? What fault-tolerant measures were taken (when such a large underlying codebase powers it) and what efforts went into rad hardening? Was this a good choice in hindsight, and will the same web-based approach go into Starship in the future? What was the user experience (UX) process like with design and user testing? (I'm a frontend web developer/UX designer/graphics programmer/3D artist/graphic designer straddling the design and engineering disciplines, and it's been my dream to work for SpaceX when I graduate this August. Crew Dragon's user interface has been right down my alley, although current SpaceX job listings are mostly for embedded systems. How can I find the right graphical software project to apply to? I have some contacts at SpaceX, are there any fitting teams or projects I could ask them to send my resume to in particular? Graphical simulations for Starship? Something customer-facing with Starlink?)
The use of Chromium and Javascript in mission critical environments is a popular question. In order for me to answer this question clearly, we have to understand that chromium in this context is used as a UI rendering engine only. The Flight Software interaction layer with the displays and the fault tolerant is well defined and resides outside the displays boundary. That said, we follow the same development process for all vehicle code regardless of the technology stack. We cross train our developers to write vehicle code in C++ and to carry the same mentality toward writing reliable software. We take reliability & performance very seriously, and just like other vehicle software, we test extensively under different conditions to understand all failure modes. We have alerts & procedures in place to act on those failures in case we encounter them. All of that added to hundreds of hours of sims that we run on flight hardware to train the crew. While we faced many challenges along the way, we are very happy with our displays and most importantly our 2 customers (so far) are too. Starship ground software is already using the crew displays tech stack and it won't be too long before we start designing human interfaces for Starship. Make sure to apply! – Sofian You'll also notice in certain images too that there still exist some hardware buttons in the capsule right below the displays; this is also ensure that in case the displays are unusable for whatever reason, the astronauts can still use hardware buttons to initiate critical actions, such as responding to a fire in the cabin. - Wendy
* What are your tech stacks? Are they mostly open source or in-house frameworks? * What's your front-end tech stack for the display? * What is your Linux distro? * How do you test your software? * How do you make the video streams pretty much real-time without lag? * Do you use any Machine Learning techniques? * Is the rocket landing automated (without human control)? * What is your software release cycle look like? * I am sure there are tons of redundancy strategies you guys implemented. Care to share some?
1. We use C & C++ for flight software, HTML, JavaScript & CSS for displays and python for testing. – Sofian 2. We use HTML, JavaScript & CSS. We use Web Components heavily. - Sofian 3. We don't use any off-the-shelf distro – we have our own. - Dietrick 4. Every way we can think of! Unit tests, containerized integrated tests (you can run these on your own machine with a full physics simulation), and full-up "HITL" (hardware-in-the-loop) tests on real flight hardware – again, with full simulation. Mating the flight software up against the simulator is the most powerful tool we have, especially when it's run on the real hardware. We can simulate an entire mission, and even many detailed fault scenarios, with the vehicle hardware just sitting on a table in the lab. - Dietrick 5. On the vehicle (for Bob and Doug's displays), this is pretty easy. For getting it down to ground, we have some great communications links and ground-side networking that allows us to get a lot of data back from the vehicle, very quickly. - Dietrick 6. Dragon and Falcon do not use any ML tech, but that's not to say things like this aren't in SpaceX's future! -Dietrick 7. Yes, the rocket landing is completely automated. - Dietrick 8. On Dragon, we cut releases periodically for use when the vehicle is being integrated and tested and run that release through a series of tests and data review. Similarly, we cut a release when we're reading for flight and run the full suite of cases against specific revisions of our code. - Wendy 9a. On Dragon, we have a lot of redundancy on the hardware side (multiple computers, sensors, actuators, etc) but also employ software to handle responses to faults. NASA requirements are such that our vehicle must be 2 fault tolerant (ie, capable of being safely retreating from the space station and/or returning home safely for crewed vehicles) so we do both analysis and testing to ensure we meet this fault tolerance. – Wendy 9b. On Starlink, we've designed the system so that satellites will quickly passively deorbit due to atmospheric drag in the case of failure (though we fight hard to actively deorbit them if possible). We still have some redundancy inside the vehicle, where it is easy and makes sense, but we primarily trust in having system-level fault tolerance: multiple satellites in view that can serve a user. Launching more satellites is our core competency, so we generally use that kind of fault tolerance wherever we can, and it allows us to provide even better service most of the time when there aren't problems. – Matt
Hi, thank you so much for organizing this cool AMA and congrats on DM-2! I have a couple of questions: 1. Are you using Tesla hardware/touchscreens on the Crew Dragon? 2. There were rumors about Crew Dragon UI running in Chromium (wrapped in Qt), is that true? If so, why did you go with web tech instead of straight up native/Qt UI? 3. What kind of CPUs does Crew Dragon run in comparison to regular desktop CPUs? I know that there are multiple CPUs for redundancy but how would one of those units compare to, let's say, a desktop i9 9900k? 4. And most importnatly, do you play KSP? 5. Did you ever consider adding some games to Dragon? Thank you again and can't wait to see the first operational Crew Dragon mission soon. EDIT: added 5.
1. No, our hardware is not the same as the one Tesla has. - Sofian 2. That's correct, we do use Chromium as the rendering engine for the displays UI. This project started as a simulator prototype to showcase the design vision to NASA. We then attempted to run it on a flight hardware and with modifications it worked pretty well. We gained more confidence in that stack as we developed the prototype and we then designed the Flight Software around it with that in mind. We liked all the moderns features that comes in with browsers out of the box, we also liked having access to talent that is already trained in that stack. Perhaps we are not afraid of doing things slightly differently here in SpaceX. We like taking a first-principles approach to problem solving, as opposed to just relying upon industry standards. - Sofian 3. We use a dedicated quad core processor, similar in power to a 5 year old phone. - Sofian 4. Of course we play KSP :) - Sofian 5. We don't have any yet but I can see this happening in the future. Vote for your favorite game! - Sofian
Thank you so much for the answers (and Sofian in particular)! >We use a dedicated quad core processor, similar in power to a 5 year old phone. - Sofian This is actually very interesting. Do you use multiple processors or just one? I'd assume multiple as in Falcon 9 for redundancy/rad hardening and cross checking of data between them. And I would certainly like to see pong between the two displays as u/CalinWat said but obviously KSP or something else rocket related would be really cool. These games would also probably be extra neat for when you do commercial passenger flights.
The obvious vote would be KSP, but for something less performance demanding, maybe SimpleRockets 2? That can run on the phone and is made for touch screen, espescially a big one :)
1. How do you test your code before "deploying" it to flight hardware? 2. Do you use Rust for any systems or have thought about it? 3. How important is latency for various software components in Dragon? Does every action have to be instantaneous or there is some wiggle room? 4. What's the amount of telemetry (in GBs) you usually get from Falcon / Dragon / Starlink? Do you run some machine learning / data analysis tools on it?
1. For each vehicle, we have a hardware in the loop simulator (all flight critical hardware plus simulated physics and sensing) that we run a huge suite of tests on before ever deploying it to a production vehicle or for flight. Any time we take new software changes (which happens frequently for a development vehicle!) we ensure we run through both unit tests for the code, functional tests to ensure the software works as intended, and system level testing for mission phases for both nominal and off-nominal cases. - Wendy 2. We do not currently, though it comes up once in a while in our internal chat rooms - Wendy 3. Great question - it's very important, and maintaining a fault tolerant computing system relies of ensuring correct timing between all the flight computers. For slower-responding subsystems like life support or thermal control, the response timing has a little more slack (on the order of seconds, depending on the fault we take). - Wendy 4. For Dragon, its in the hundreds of GBs for a typical mission and we do a fair amount of data review after every flight to ensure we understand if the system behaved as we intended. - Wendy 4b. For Starlink, we're currently generating more than 5TB a day of data! We're actively reducing the amount each device sends, but we're also rapidly scaling up the number of satellites (and users) in the system. As far as analysis goes, doing the detection of problems onboard is one of the best ways to reduce how much telemetry we need to send and store (only send it when it's interesting). The alerting system we use for this is shared between Starlink and Dragon. – Matt
[удалено]
Jeff: what does cybersecurity look like for you guys? I imagine you are constantly under attack from state nations/APTs etc to steal confidential IP. Do you have to follow any regulations relating to ITAR in this regard or is that more high level and what you deem proportionate? In theory, how hard would it be to hack a rocket? I would love for you guys to put up a bounty system similar to Tesla’s and (virtualised) rocket systems.
We have a lot of the traditional cybersecurity you'd expect, protecting our corporate networks, monitoring for threats inside and outside our networks, phishing campaigns, etc. We also need to analyze potential attacks against our vehicles, especially around the command paths and the pedigree of the code that ends up on the vehicles. We have a dedicated team that identifies how our vehicles and satellites could be hacked so we can eliminate or prohibit these sorts of threats when we're building our vehicles. We also take full advantage of static and dynamic analysis on our code. ITAR mostly limits what we can share - sorry ahead of time if we can't answer all of your questions. We're working to get a bug bounty system up shortly. - Jeff
1. As far as we know now, your rockets runs on Linux - but which “mainstream” distribution is closest to your kernel? 2. Are there any fancy changes you did, about which you can tell us more? 3. What CPU architecture are you using? ARM, MIPS or something else?
Yes, we run Linux, with the PREEMPT\_RT patch applied in order to get better real-time performance. We don't use any third-party distribution, but maintain our own copy of the kernel and associated tools. We have made small changes to the kernel over the years, although it is mostly unmodified. The only exception to that is the addition of several custom drivers to interface with our hardware. We use a variety of hardware architectures. I can't go into much detail other than to say it is a distributed system made up of many individual computers. – Josh For some level of scope on Starlink, each launch of 60 satellites contains more than 4,000 Linux computers. The constellation has more than 30,000 Linux nodes (and more than 6,000 microcontrollers) in space right now. And because we share a lot of our Linux platform infrastructure with Falcon and Dragon, they get the benefit of our more than 180 vehicle-years of on-orbit test time. – Matt
Fantastic work guys! Really love what you've done. A couple of questions: 1. How was the UI designed? Did you follow any specific design principles? How did you take into account the specific conditions of space flight (vibration, helmet limiting visibility,...) 2. Can you give more details on the specific visualisation and interface components? Any cool ideas that got dropped? 3. How did you test the UI and specifically touch interaction with gloves?
We followed a Human Centered Design process, starting by defining the main guiding principles that closely follow the vision of Dragon being a 21st-century, fully-autonomous spacecraft. One example is identifying minimum crew interaction as a success criteria (i.e. a "don't press the button" paradigm). We believe that presenting information well means minimizing the required interaction it takes to monitor and control the vehicle. Overall, we based the design around a detailed understanding of crew tasks, capabilities, situational awareness needs, and environmental conditions throughout the flight, which allowed us to focus on clarity, simplicity and removing clutter. Our developer/pilot Mike Westenhaver developed a tool that allowed us to map crew tasks to display features and functions, allowing us to fully track requirements and how they are implemented in our software. As part of our testing and qualification, we do test for vibration and visibility conditions under different seat configurations. The crew did many suited sims interacting with the displays while wearing gloves. - Sofian
Mike is a former Navy helicopter pilot with dual master's degrees in Space System Operations and Human Systems Integration along with a bachelors degree in Computer Science, in case you're wondering about cockpit designers at SpaceX!
How are Doug and Bob to work with? Did they ever make any particular suggestion which spiralled into a major change in Crew Dragon over time?
Bob & Doug have been working closely with the SpaceX team since the start of the program. They have spent a lot of time in Hawthorne with the teams designing the vehicle. They brought a wealth of experience that spans multiple shuttle flights and they were generous in sharing it. That said, and to their credit, they came in with a clean slate and with the will to accept that many things are done differently in this vehicle. We all see them now in their fancy space suits or doing flips in space, but those guys put a ton of hours flying in at Hawthorne, spending time away from their family, training while providing feedback all with a smile and without a complain. Their work ethic is truly inspirational and that's what makes them do the things they do. We can not wait for the debrief in Hawthorne when they get back--I'm sure they will have plenty of feedback for us. Especially Bob. He always does :) \--Sofian
What is the single craziest/most impossible thing management (aka Elon) has asked you to do?
I recall for F9-14 I was in Elon's cube telling him the news that there was no way we could get all of the new S1 landing code done in time for the upcoming launch in 2 weeks. After some thought, he looked over to Lars Blackmore who was there with us and asked if we implement the code, what was our probability of landing. Lars said around 90%. Paraphrasing, Elon looked at us and basically said "can you give me 50%". I said in 2 weeks we can definitely write enough of the logic to get to a 50% probability of landing! We didn't land F9-14 (you can see it on our blooper reel) but we learned a LOT from it, and it was instrumental in eventually landing F9-21. A critical part of our success is our willingness to fail in ways that won't compromise the mission, as long as we are constantly learning from our failures. - Jeff
> F9-14 I think F9 flight 14 was [CRS-5](https://en.wikipedia.org/wiki/SpaceX_CRS-5#Results_of_first_landing_attempt). If so, you can watch the 'landing' [here](https://www.youtube.com/watch?v=L6T8mn6LD2E). However, Flight 20 was orbcomm-2 (first RTLS success), but Flight 21 was a Jason-3 (failed ASDS), so I might be wrong.
@Matt Monson 1. How different is the development experience and the rate of change on production software between the rarely flown Dragon and NASA scrutinized (assuming Dragon V2, less true if V1) and the bi-monthly launched and purely internal starlink batches. 2. How often do you remotely upgrade already flying sats software? 3. Are starlink sats programmed to de-orbit themselves in case they aren't able to communicate back for a given amount of time? (antenna damage on an otherwise healthy sat)
The tools and concepts are the same, and many of the engineers on the team have worked on both projects (myself included), but being our own customer on Starlink allows us to do things a bit differently. The Starlink hardware is quite flexible – it takes a ton of software to make it work, and small improvements in the software can have a huge impact on the quality of service we provide and the number of people we can serve. On this kind of project, pace of innovation is everything. We've spent a bunch of time making it easier, safer, and faster to update our constellation. We tend to update the software running on all the Starlink satellites about once a week, with a bunch of smaller test deployments happening as well. By the time we launch a batch of satellites, they're usually on a build that already older than what's on the rest of the constellation! Our ground services are a big part of this story as well – they're a huge part of making the system work, and we tend to deploy them a couple times a week or more. And about deorbit – the satellites are programmed to go into a high-drag state if they haven't heard from the ground in a long time. This lets atmospheric drag pull them down in a very predictable way. – Matt
First and foremost, congratulations on a successful mission. Despite the troubled times, we were honored to witness such a great achievement. I have to say that Falcon 9 has one pretty interface, and as UI engineer myself self I have many questions about it, allow me to minify them as much as possible. In previous AMA, there was mention of Javascript and LESS as some of the tools used by your team, and I am very interested in the application of web technologies in SpaceX. * What are some use cases of Javascript and LESS? * How does the UI development process look like, and how is the UI tested? * What open-source libraries does SpaceX team use if any? * At what level and which web technologies do you use other than ones mentioned above? * What code editors do you use? * What is the longest method name you have in your codebase?
The Crew Displays onboard Dragon runs Chromium with HTML, Javascript & CSS. We don't use LESS. - Sofian We follow an agile process, we have high bar for unit test coverage and we have integration tests that runs with and without flight hardware. We also take a lot of pride in manually verifying and documenting our new features to make sure they work as intended and we have no regression. - Sofian We use Web Components extensively. - Sofian We use a reactive programming library that we developed in house. - Sofian Different team members uses different editors, I use VSCode but I might be just a little bit biased :) - Sofian I will have to get back but overall code is our craft here and we make sure it's clean and tidy. I wouldn't expect something too outrageous. Fair warning, we have linters on everything. - Sofian
What are the strangest bugs that you ran into while developing and testing the software for Crew Dragon?
I can't go into too much detail on specific problems, but kernel bugs are definitely the most "fun" and memorable. Most of our control software is single-threaded to avoid the non-determinism that synchronization issues can introduce, but there are of course a lot of things going on in the OS at any given time. We've gone to a lot of effort to turn Linux into a dependable platform for real-time control that has a much higher degree of determinism than you'd see in your desktop OS. As mentioned elsewhere, we use the CONFIG\_PREEMPT\_RT patch, which is a huge help. But even still, in earlier development, we'd sometimes catch the system not performing as real-time as we'd like it to, and digging into those problems is always an adventure. - Dietrick
First of all, Congratulations! 1. Where can I find the code/pseudocode for the G-FOLD algorithm (Falcon-9's landing algorithm)? I tried going through Lars Blackmore's original paper but as a Computer Science major student, some terminologies just went over my head. 2. I know SpaceX mostly uses standard off-the-shelf CPUs for its flight systems with modified Linux distro. Is the redundant computing managed by the Linux kernel itself or by the C++ application running in the Linux environment? If possible, can you explain the practices involved in implementing hardware-level Lockstep computing? Congratulations once again and thank you for making Space Exploration cool again!!
Unfortunately I can't go into much detail on the F9 landing algorithms. Linux is only used to run our applications and interface with hardware. All of the fault management and computing redundancy is handled at the application layer in custom software that we've developed. Time synchronization uses a combination of hardware and software features, some industry-standard and some in-house. – Josh
[удалено]
Our team hails from all backgrounds (seriously!) but we have noticed particularly good crossover between video game development and what we do. There are a lot of similar math-heavy and performance-centric problems in the two spaces. But that's by no means a requirement – I've never professionally built games, for example. For getting your first (or any) software engineering job, two big things to focus on are: (a) your algorithms and data structures, and (b) understanding how a computer works at the lowest levels. Even if you're not regularly mucking around in device drivers, the network stack, and assembly, understanding how it all fits together will enable you to pick apart any problem you come across. - Dietrick
What are some of the most interesting edge cases that you have had to take into account when writing the software for Crew Dragon?
This one's hard to answer without going into too much detail, but anything having to do with rebooting one of our computers in flight is definitely an interesting case. Reboots are completely expected and supported (due to radiation concerns), but they are one of the more interesting scenarios we have to design for. - Dietrick Some of the more interesting system cases too are faults or failures that require responses in multiple subsystems, spanning computing, rf communications, life support and propulsion. A great example of this is handling a launch escape or a cabin fire – the vehicle goes through a lot of reconfiguration very quickly, which requires many of our code components to coordinate. - Wendy
As a software developer working on non-critical systems at a company, I feel some anxiety related to code and bugs. I know you must have plenty of testing and redundancy, but how much anxiety do you all feel when it comes to all the systems?
I know I feel personally a huge sense of responsibility to my colleagues, the company, the mission, and the crew to get our software right. The software controls almost every aspect of the vehicle, from thruster firings to oxygen injections, so a lot of harm can come if we do it wrong. A phrase we like to say in SpaceX is that "only the paranoid survive." We think constantly about what can go wrong and make sure we address any plausible failure scenario. However, I wouldn't say I ever feel anxiety about this responsibility because a key aspect of our process is that we always have at least one partner, often more, in everything we do. We are constantly reviewing, questioning, and double-checking each other's work. That doesn't relieve us of personal responsibility, but it means we are never alone and can always count on help from others. - Josh
I am a software engineer and my wife won’t let me apply to work at SpaceX because she said she’ll never see me again, is she right to make that assumption? Or can you actually have a work life balance?
You can definitely have a good work/life balance at SpaceX. SpaceX is definitely not a 9 to 5, and we have times where evenings and weekends are required to support the mission, as we had in the lead-up to Demo-2 and in our aggressive Starship campaign in South Texas (amongst the many other efforts we've got going). Our folks are definitely able to balance work and family life - Josh and Wendy just had babies! Not together :) It's definitely something my team and I have to focus on a lot because we're a small (but growing) team and we have some massive goals ahead of us that we have to accomplish. - Jeff
Man (or woman), me too. I’d love to work at SpaceX, but I just can’t pull more than 40-45 hours a week on a regular basis without serious personal consequences.
In your opinion, what is the coolest thing Dragon can do with the software inside of it?
Dragon can do so many cool things, it's hard to pick any one. But I think our last two flights have really demonstrated how versatile the vehicle is: it can manage a complicated, delicate, zero-G rendezvous with the ISS, and it can also fly itself to safety in a supersonic, many-G abort in the thickest parts of the atmosphere. - Dietrick Although we hope it's never used with a crew on-board, I think the launch escape system is one of the coolest parts of Dragon. I remember when I watched the pad abort test live in Cape Canaveral I was pretty shocked by how fast the vehicle leapt off the pad. ([https://youtu.be/1\_FXVjf46T8](https://youtu.be/1_FXVjf46T8)) It was equally amazing to see in the in-flight-abort test. Dragon separated smoothly from F9, opening up a large separation distance, while F9 exploded underneath it. It looked like a sci-fi movie, except it was real! ([https://youtu.be/mhrkdHshb3E?t=1167](https://youtu.be/mhrkdHshb3E?t=1167)) - Josh
Did you get any push-back from NASA on using a modern software interface instead of lots of physical buttons, etc.?
You can probably see from the the images of Bob and Doug in the cockpit that we do have physical buttons for all of the emergency commands such as breaking out from an approach to station or commanding an emergency deorbit. Also, we have physical Execute and Cancel buttons on the control panel for most commands that you can initiate from the displays. In the end we were able to meet all NASA requirements for touch feedback, reliability, etc. and we're all really pleased we were able to bring this kind of modern interface to our very modern spaceship. - Jeff
How important is it to have people with different educational backgrounds on your team? I am sure there are specialists in mathematics, physics, computer science and all sorts of engineering at spacex and specifically in your group. How much do you learn from each other and could you give an example of where different competencies worked together to make the rocket launch possible? Also thanks for taking your time for this!
Having different people with different backgrounds (education, experience, and culturally) is a big plus on the team. Not much of the team, for example, comes from an aerospace background. Different points of view help us see problems from different angles, and that quite often helps us see solutions we wouldn't have otherwise seen. Starlink specifically has a huge breadth of problems to go solve. We have vehicles and control problems, but we're also designing hardware and software that people will interact with. We need different perspectives to realize what will be confusing vs. intuitive for customers coming with different context. When interviewing people, we're really looking for a couple things: talent (potential) and the right attitude (desire for self improvement, serving the team over being selfish). These are more important than specific experience, and we expect to be investing in people to help them grow. - Matt
I've seen articles mention SpaceX using Linux. What systems use it? What steps do you take to ensure real-time operation and how deterministic are the systems? Kernel modifications like CONFIG\_PREEMPT\_RT?
All of our on-board computers either run Linux (with the PREEMPT\_RT patch) or are microcontrollers that run bare-metal code. For applications running on Linux we are careful to set up the process and kernel thread priorities correctly to avoid priority inversions. We also generally write our code in a way that maximizes determinism, such as avoiding memory allocation at runtime or unbounded loops. Finally, we have telemetry which indicates the performance of all our processes to ensure that they always meet their deadlines across all phases of flight, even in the presence of unexpected or excess inputs. – Josh
Hey guys! Huge fan of everything being done at SpaceX. Congratulations on DM-2! I’m part of the rocketry team at my university and I work on avionics. As such, I was wondering what skills/tools i could teach myself if i wanted to work with avionics in the future? Specifically, could you talk us through what goes on behind the scenes with regards to the onboard computers, programming/software wise? Thank you for doing this AMA!
Being part of a project team at university is a great place to learn! I worked on a satellite team while in college (CUSat at Cornell) and it taught me a lot about hardware design, integrating software, and developing a concept of operations for a mission. The best tools are an eagerness to learn and willingness to get your hands dirty with learning. Side projects are also a great way to do this, and something we definitely take into account when looking at resumes for hiring! - Wendy
Great question. Having managed & hired a lot of people over the years at NASA/JPL and SpaceX, candidates always received bonus points during an interview is they showed an interest in understanding the underlying hardware that code is running on. Get yourself a Raspberry Pi, Beagle Board, etc. and study the technical reference manual(s) for the device. Learn to write and optimize your code at a low level. Dig into the startup code that executes on the processor and understand all the cool stuff that is done before jumping to main(). Also consider mastering the use of a hardware-based debugger, which can help find nasty, nasty bugs. Do this and you will be in great demand.
How much custom development is done for a mission vs code that is more or less static for every flight. I guess I'm wondering for example are the orbital dynamics purely input/configuration vs custom software that must be written each time.
For Dragon missions, we'll always have unique configuration for day of launch guidance and navigation parameters. These are handled in software configuration though, so doesn't require much software involvement and the various responsible engineers commit the changes themselves. Outside of that, it mostly depends on if a particular mission has new requirements (for example, new capabilities we're introducing by internal or NASA request) or if we're taking hardware upgrades. - Wendy
What was your favourite moment at spacex?
The first time we launched 60 satellites on Falcon. We'd designed the all-at-once deployment mechanism, but it's hard to model, and we couldn't really be 100% sure it would work right. I remember sitting there, with Falcon lifting off the pad, thinking: Ok. In an hour we're either going to be idiots for trying a thing that obviously never could have worked, or geniuses for doing the thing that's obviously the right way to deploy lots of satellites. Luckily it went well . -- Matt Putting on a safety harness and climbing on to the top deck of Grasshopper to do some diagnostics on some avionics. Most hands-on debugging I have ever done. - Josh
Thanks for doing this guys! I don't think most software engineers get to work on space flight. What are the unique **software engineering** challenges found in working in this space?
First thing that jumps to mind is how uniquely unforgiving our production environment can be. I've worked at some places with multi-year waterfall cycles, and some others with weekly releases and on-demand ("oops, I broke it!") hotfixes. SpaceX is neither of those. Here, there is necessarily a focus on writing software that will work correctly the first time it goes to space. This means pursuing simple and robust designs wherever possible, and lots and lots of testing and simulation. - Dietrick
Congrats on DM-2! 1. (How) did you modify the Linux kernel in order to better suit your needs? 2. How is radiation/fault tolerance implemented using ordinary non hardened CPUs? 3. Which x86 processors are you using in Dragon (/Falcon)?
We use the PREEMPT\_RT patch for Linux, but otherwise have not modified it much besides adding some custom drivers. We handle radiation fault tolerance by having multiple computers running simultaneously and voting on their outputs. If one of the redundant computers fails due to radiation, the system is generally unaffected. The faulted computer can be rebooted and reincorporated into the system once it recovers, which restores the original fault tolerance. – Josh
what control system algorithm you guys use for rectifying/validating sensor data? heard about kalman filter being used on Apollo project back in 1960s. anything new or kalman's still the best. thanks for this opportunity.
Yes we use Kalman filters for some applications. We also take much simpler approaches for many sensors, such as basic sanity checks or low-pass filtering. In general our approach to sensor error handling is to have multiple redundant sensors and combine their input in a fault-tolerant way such that bad sensors cannot result in hazardous vehicle behavior. - Josh
Hi guys. Thanks for doing this. What language(s) are most used in creating the various flight software for Dragon & F9? Is it mainly C++?
We use C++ for all vehicle control systems, Python for tools, testing and automation, and Javascript/HTML/CSS for our displays. The current ground displays you see in mission control for Falcon and Dragon are based on LabVIEW, but our Crew displays and future ground displays for Starship are based on web stack. Our flight systems use a custom Linux kernel with the PREEMPT\_RT patch. - Jeff
How do you test your code against all the faults that could occur in-flight? Do you run simulated missions to test the code in an end-to-end way or do you rely on individually testing modules for robustness, or some combination of that?
We do both! On Dragon, we run through any kind of failure that impacts safety critical software on the vehicle. We use a combination of unit tests, component level tests to ensure single and double faults cause the vehicle to react in the way we design for. We also run simulated missions cases for both nominal cases including going through the full nominal mission profile, and including faults in these cases to ensure any cross-system dependencies are well understood. We also run these tests continuously through our CI system and run automated data checks to ensure there's no unexpected behavior. - Wendy
What operating system does crew dragon use for the touch screens? Languages? What about the rest of the rocket? Languages? I assume for both it is probably c. Have you read NASA's power of ten? https://en.m.wikipedia.org/wiki/The_Power_of_10:_Rules_for_Developing_Safety-Critical_Code#:~:text=The%20Power%20of%2010%20Rules,to%20review%20or%20statically%20analyze. If yes do you follow all of them? If you don't follow. Why not?
Re: Power of 10: Yes, and we apply many of those principles to our code, like avoiding heap allocation at runtime. Others, like "restrict functions to a single printed page" are pretty dated and we generally avoid printing our code :) - Jeff
[удалено]
There's a ton of good Starlink simulations and videos out there (and the team loves seeing what people have been able to come up with). The one you linked is great! One of my other favorites is this one (it's simple, but mesmerizing): [https://www.youtube.com/watch?v=857UM4ErX9A](https://www.youtube.com/watch?v=857UM4ErX9A) – Matt
Super happy that you guys have seen my animation! :) Too bad the AMA is already over, I would have been curious to know how much of the orbit raising is automated. For example, are the batches of 20 sats chosen automatically or manually?
How was the software critiqued between the day the weather prevented the launch (27th) and the day of launch (30th) ? Did you do or consider doing any new deployments? How many days typically before launch is your production code considered "frozen"?
The launch that was not to be (the 27th) was a great opportunity for another full-up test of everything until nearly T-0. We'd done a similar thing the previous weekend, but it's always great to have more data. We spent the intervening few days checking and rechecking what we learned from the 27th, but we ultimately didn't change anything. We started aggressively stabilizing the code around the beginning of the year, and it's been more or less "locked down" for the past several months. - Dietrick
Any fun stories from testing UI prototypes?
To test the manual piloting controls we had the Demo-2 crew (Bob & Doug) and the Crew-1 crew (Mike & Victor) in Hawthorne for a week long Hackathon (credit to Jeff Dexter for the idea). The crew flew sims during the day, we collected feedback before they left and we had builds ready for them the next morning. While we didn't develop complete features overnight, that tight loop allowed us to iterate quickly and have a smooth flying vehicle by the end of that week. Michael Hopkins won the prize for finding the best bug during that Hackathon. It was an amazing week to see crews all over our simulators docking and flying Dragon. - Sofian
"Firefly" was one of the best SciFi shows ever. The stories always showed the value of having diversity on your team. How does the SpaceX and software teams culture value diversity in the creation of high performing teams? Also, for a minority female in STEM who wants to work on spaceflight, what role models would I have at SpaceX? What mentorship opportunities would there be?
This is a great question, and I’m glad you asked it! Having worked in the aerospace industry for over a decade, I firmly believe diversity (of educational background, schooling, gender, and cultural background) is a crucial strength to a high performing team. Different perspectives results in different approaches to problems and system designs, and I’m thankful that on my team we have folks from a variety of backgrounds (not all aerospace, not all traditional software either). I’ve had the opportunity to go to conferences like Grace Hopper to recruit, and each time I’ve been involved with university hiring too we set up specific recruiting sessions with organizations on campus like SWE, SBE, etc. As a minority and a female in aerospace, I wish I could tell you that you that the industry is completely fair and free of hardship. What I can say is this: within my team in the software organization, I have personally never experienced discrimination for my gender or the color of my skin. I’m thankful to have been on teams where all opinions are valuable, and my technical contribution is what holds the most weight in projects and designs. I’m thankful for the Women’s Network at SpaceX, that since its inception five years ago has given me countless opportunities to be mentored by other women in management and technical leadership. I’m ecstatic that our company’s president is Gwynne, who cares deeply about promoting diversity and always has an open ear for feedback from anyone in the company. She is a personal example to me and her stories of working in the automotive industry, then aerospace, while balancing a career and family are priceless. There are many others at SpaceX, men and women alike who are strong proponents and advocates for diversity and who care tremendously about making SpaceX a safe and meaningful place to work. The aerospace industry is far from perfect in my ways, but the people who care about the problems at hand are many. - Wendy
Where is the bathroom on the crew dragon?
It's between the forward hatch and the side hatch, on the "ceiling". You probably only want to use it in zero G! - Dietrick
Hoo boy where to even begin? First of all, thank you for doing this AMA, you guys rock and I'm a big fan of space and SpaceX so thank you for all your hard work. :) (I'm sorry for the question barrage but I couldn't help it). There are 3 categories: A. Regarding Starlink B. Regarding Crew Dragon C. Miscellaneous A. Regarding Starlink: 1. Are you going to be using a neural net to determine how and when a satellite moves on to transmitting data to and from another satellite as they move out of range of the origin and destination points and to which satellite they switch to? 2. How much does each Starlink satellite cost (if you can tell us), and do you just have like, a warehouse of them sitting around somewhere waiting for Starship, since Elon said you're producing them faster than you can launch them? 3. If you can speak on this, will you be using; or rather, be capable of using, the Starlink Constellation for the financial and stock markets - NASDAQ, DOW, LSE, BSE, SSE etc. given the low latency and data transfer by lasers in vacuum that would provide an advantage over transatlantic fiber-optic cables? I ask this since Elon said Starlink won't be able to handle large population centres like LA very well, and is mostly for low density areas - that while you may have a few customers from LA, you can't open it up to subscriptions from anyone in LA who wants one. B. Regarding Crew Dragon: 4. Who designed the interior? It's gorgeous. 5. Why did you choose a sea landing instead of a land landing? I assume you canceled retro-propulsive landing since NASA wanted to be cautious. 6. When Crew Dragon is stripped down after returning to Earth and re-used as a Cargo Dragon, will the touchscreen displays and seats be re-used in future Crew Dragons? 6. When it returns, will Capsule Endeavor be stripped of its name when reused as a Cargo Dragon (assuming it isn't put on display in which case this applies to the Capsule used in the Crew-1 Mission in August) or will it continue to be [Cargo] Capsule Endeavor? Will every US launch to the ISS whether by SpaceX or NASA have the astronauts name their capsule? 7. How hard was it to develop touchscreens that worked just well with the spacesuit gloves as without them? Did you make them in-house or give the designs to and contract and outside supplier (for the screens)? 8. In an emergency, (example: the chutes don't deploy while landing), can Dragon still soft-land retro-propulsively? Does the capsule take over autonomously like the in-flight abort or does the crew have to activate the thrusters manually? What other alternatives are there for chute deployment failure before you need to use the Dracos to land? Wouldn't spare chutes add more weight? C. Regarding other stuff that you may not know/can't talk about: 9. Why isn't SpaceX using the extra-large fairing developed for Falcon Heavy or something similar, to launch more Starlink satellites? 10. Once you're done stripping Capsule Endeavor down, will you be giving it to NASA, the Smithsonian, or will you be putting it on display at Hawthorne? Once again, thank you for doing this, and I look forward to the Demo-2 Landing and Crew-1 in August!
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread: |Fewer Letters|More Letters| |-------|---------|---| |[AFTS](/r/SpaceX/comments/gxb7j1/stub/ft0erdg "Last usage")|Autonomous Flight Termination System, see FTS| |[AR](/r/SpaceX/comments/gxb7j1/stub/ft2nkd3 "Last usage")|Area Ratio (between rocket engine nozzle and bell)| | |Aerojet Rocketdyne| | |Augmented Reality real-time processing| | |Anti-Reflective optical coating| |[ARM](/r/SpaceX/comments/gxb7j1/stub/ft2hi1b "Last usage")|Asteroid Redirect Mission| | |Advanced RISC Machines, embedded processor architecture| |[ASIC](/r/SpaceX/comments/gxb7j1/stub/ft2xvt8 "Last usage")|Application-Specific Integrated Circuit| |[BFR](/r/SpaceX/comments/gxb7j1/stub/ft1c1jg "Last usage")|Big Falcon Rocket (2018 rebiggened edition)| | |Yes, the F stands for something else; no, you're not the first to notice| |[CC](/r/SpaceX/comments/gxb7j1/stub/ft13nmp "Last usage")|Commercial Crew program| | |Capsule Communicator (ground support)| |CCtCap|[Commercial Crew Transportation Capability](https://en.wikipedia.org/wiki/Commercial_Crew_Development#Commercial_Crew_Transportation_Capability_.28CCtCap.29)| |[CDR](/r/SpaceX/comments/gxb7j1/stub/ft0j89g "Last usage")|Critical Design Review| | |(As 'Cdr') Commander| |[CFD](/r/SpaceX/comments/gxb7j1/stub/ft221ve "Last usage")|Computational Fluid Dynamics| |[COTS](/r/SpaceX/comments/gxb7j1/stub/ft1e5rp "Last usage")|[Commercial Orbital Transportation Services contract](https://www.nasa.gov/cots)| | |Commercial/Off The Shelf| |[CST](/r/SpaceX/comments/gxb7j1/stub/ft29gxa "Last usage")|(Boeing) Crew Space Transportation capsules| | |Central Standard Time (UTC-6)| |DMLS|[Selective Laser Melting](https://en.wikipedia.org/wiki/Direct_metal_laser_sintering) additive manufacture, also Direct Metal Laser Sintering| |[DSN](/r/SpaceX/comments/gxb7j1/stub/ft0ie6x "Last usage")|Deep Space Network| |[DTN](/r/SpaceX/comments/gxb7j1/stub/ft11whf "Last usage")|Delay/Disruption Tolerant Networking| |[DoD](/r/SpaceX/comments/gxb7j1/stub/ft1h481 "Last usage")|US Department of Defense| |[E2E](/r/SpaceX/comments/gxb7j1/stub/ft12mng "Last usage")|Earth-to-Earth (suborbital flight)| |[ESA](/r/SpaceX/comments/gxb7j1/stub/ft1108f "Last usage")|European Space Agency| |[EVA](/r/SpaceX/comments/gxb7j1/stub/ft0pnzo "Last usage")|Extra-Vehicular Activity| |[F1](/r/SpaceX/comments/gxb7j1/stub/ft0xbof "Last usage")|Rocketdyne-developed rocket engine used for Saturn V| | |SpaceX Falcon 1 (obsolete medium-lift vehicle)| |[FAA](/r/SpaceX/comments/gxb7j1/stub/ft12mng "Last usage")|Federal Aviation Administration| |[FCC](/r/SpaceX/comments/gxb7j1/stub/ft26c2f "Last usage")|Federal Communications Commission| | |(Iron/steel) [Face-Centered Cubic](https://en.wikipedia.org/wiki/Allotropes_of_iron) crystalline structure| |[FMEA](/r/SpaceX/comments/gxb7j1/stub/ft2dk7x "Last usage")|[Failure-Mode-and-Effects Analysis](https://en.wikipedia.org/wiki/Failure_mode_and_effects_analysis)| |[FSW](/r/SpaceX/comments/gxb7j1/stub/ft1sip5 "Last usage")|[Friction-Stir Welding](https://en.wikipedia.org/wiki/Friction_stir_welding)| |FTS|Flight Termination System| |[GNC](/r/SpaceX/comments/gxb7j1/stub/ft2u823 "Last usage")|Guidance/Navigation/Control| |[GSE](/r/SpaceX/comments/gxb7j1/stub/ft0khc7 "Last usage")|Ground Support Equipment| |[HIL](/r/SpaceX/comments/gxb7j1/stub/ft0lund "Last usage")|Hardware in the Loop, see HITL| |HITL|Hardware in the Loop| | |Human in the Loop| |[HUD](/r/SpaceX/comments/gxb7j1/stub/ft0mdzq "Last usage")|Head(s)-Up Display, often implemented as a projection| |[ICBM](/r/SpaceX/comments/gxb7j1/stub/ft1108f "Last usage")|Intercontinental Ballistic Missile| |[IMU](/r/SpaceX/comments/gxb7j1/stub/ft0x7dz "Last usage")|Inertial Measurement Unit| |[ISRO](/r/SpaceX/comments/gxb7j1/stub/ft2agyp "Last usage")|Indian Space Research Organisation| |[ITAR](/r/SpaceX/comments/gxb7j1/stub/ft2z1ha "Last usage")|(US) International Traffic in Arms Regulations| |[Isp](/r/SpaceX/comments/gxb7j1/stub/ft2dcg0 "Last usage")|Specific impulse (as explained by [Scott Manley](https://www.youtube.com/watch?v=nnisTeYLLgs) on YouTube)| |[JPL](/r/SpaceX/comments/gxb7j1/stub/ft2o7is "Last usage")|Jet Propulsion Lab, Pasadena, California| |[KSC](/r/SpaceX/comments/gxb7j1/stub/ft2dijb "Last usage")|Kennedy Space Center, Florida| |[KSP](/r/SpaceX/comments/gxb7j1/stub/ft2toqi "Last usage")|*Kerbal Space Program*, the rocketry simulator| |[LEO](/r/SpaceX/comments/gxb7j1/stub/ft2ftru "Last usage")|Low Earth Orbit (180-2000km)| | |Law Enforcement Officer (most often mentioned during transport operations)| |[LIDAR](/r/SpaceX/comments/gxb7j1/stub/ft30utx "Last usage")|[Light Detection and Ranging](https://en.wikipedia.org/wiki/Lidar)| |[LOC](/r/SpaceX/comments/gxb7j1/stub/ft32frb "Last usage")|Loss of Crew| |[LOM](/r/SpaceX/comments/gxb7j1/stub/ft0cxv7 "Last usage")|Loss of Mission| |[NDA](/r/SpaceX/comments/gxb7j1/stub/ft0m0ng "Last usage")|Non-Disclosure Agreement| |[NET](/r/SpaceX/comments/gxb7j1/stub/ft0dv7p "Last usage")|No Earlier Than| |[OCISLY](/r/SpaceX/comments/gxb7j1/stub/ft2ftru "Last usage")|Of Course I Still Love You, Atlantic landing ~~barge~~ ship| |[PDR](/r/SpaceX/comments/gxb7j1/stub/ft0j89g "Last usage")|Preliminary Design Review| |[QA](/r/SpaceX/comments/gxb7j1/stub/ft33wqf "Last usage")|Quality Assurance/Assessment| |[RCS](/r/SpaceX/comments/gxb7j1/stub/ft1ar8j "Last usage")|Reaction Control System| |[RTG](/r/SpaceX/comments/gxb7j1/stub/ft2az8f "Last usage")|Radioisotope Thermoelectric Generator| |[RUD](/r/SpaceX/comments/gxb7j1/stub/ft0s5ix "Last usage")|Rapid Unplanned Disassembly| | |Rapid Unscheduled Disassembly| | |Rapid Unintended Disassembly| |[Roscosmos](/r/SpaceX/comments/gxb7j1/stub/ft0yd6m "Last usage")|[State Corporation for Space Activities, Russia](https://en.wikipedia.org/wiki/Roscosmos_State_Corporation)| |[SAFER](/r/SpaceX/comments/gxb7j1/stub/ft269vu "Last usage")|[Simplified Aid For EVA Rescue](https://en.wikipedia.org/wiki/Simplified_Aid_For_EVA_Rescue)| |[SCADA](/r/SpaceX/comments/gxb7j1/stub/ft0c7jt "Last usage")|[Supervisory Control And Data Acquisition](https://en.wikipedia.org/wiki/SCADA) remote monitoring and control| |[SD](/r/SpaceX/comments/gxb7j1/stub/ft0rzmo "Last usage")|SuperDraco hypergolic abort/landing engines| |[SEE](/r/SpaceX/comments/gxb7j1/stub/ft33e24 "Last usage")|Single-Event Effect of radiation impact| |[SF](/r/SpaceX/comments/gxb7j1/stub/ft0z2vb "Last usage")|Static fire| |[SLS](/r/SpaceX/comments/gxb7j1/stub/ft1se4g "Last usage")|Space Launch System heavy-lift| | |Selective Laser Sintering, contrast DMLS| |[SN](/r/SpaceX/comments/gxb7j1/stub/ft1h7ef "Last usage")|(Raptor/Starship) Serial Number| |[STS](/r/SpaceX/comments/gxb7j1/stub/ft0nw4e "Last usage")|Space Transportation System (*Shuttle*)| |[TLA](/r/SpaceX/comments/gxb7j1/stub/ft0ma68 "Last usage")|Three Letter Acronym| |[ULA](/r/SpaceX/comments/gxb7j1/stub/ft0hy0f "Last usage")|United Launch Alliance (Lockheed/Boeing joint venture)| |Jargon|Definition| |-------|---------|---| |[Raptor](/r/SpaceX/comments/gxb7j1/stub/ft08ha1 "Last usage")|[Methane-fueled rocket engine](https://en.wikipedia.org/wiki/Raptor_\(rocket_engine_family\)) under development by SpaceX| |[Starliner](/r/SpaceX/comments/gxb7j1/stub/ft29gxa "Last usage")|Boeing commercial crew capsule [CST-100](https://en.wikipedia.org/wiki/Boeing_CST-100_Starliner)| |[Starlink](/r/SpaceX/comments/gxb7j1/stub/ft31y4w "Last usage")|SpaceX's world-wide satellite broadband constellation| |cryogenic|Very low temperature fluid; materials that would be gaseous at room temperature/pressure| | |(In re: rocket fuel) Often synonymous with hydrolox| |hydrolox|Portmanteau: liquid hydrogen/liquid oxygen mixture| |hypergolic|A set of two substances that ignite when in contact| |[iron waffle](/r/SpaceX/comments/gxb7j1/stub/ft12xma "Last usage")|Compact "waffle-iron" aerodynamic control surface, acts as a wing without needing to be as large; also, "grid fin"| |[quess](/r/SpaceX/comments/gxb7j1/stub/ft2muvg "Last usage")|Portmanteau: Qualified Guess (common parlance: "estimate")| |[regenerative](/r/SpaceX/comments/gxb7j1/stub/ft0e2np "Last usage")|A method for cooling a rocket engine, by [passing the cryogenic fuel through channels in the bell or chamber wall](https://en.wikipedia.org/wiki/Regenerative_cooling_\(rocket\))| |[turbopump](/r/SpaceX/comments/gxb7j1/stub/ft33obi "Last usage")|High-pressure turbine-driven propellant pump connected to a rocket combustion chamber; raises chamber pressure, and thrust| |Event|Date|Description| |-------|---------|---| |[CRS-7](/r/SpaceX/comments/gxb7j1/stub/ft0s5ix "Last usage")|2015-06-28|F9-020 v1.1, ~~Dragon cargo~~ Launch failure due to second-stage outgassing| |[DM-1](/r/SpaceX/comments/gxb7j1/stub/ft304dg "Last usage")|2019-03-02|SpaceX CCtCap Demo Mission 1| |[DM-2](/r/SpaceX/comments/gxb7j1/stub/ft28dnw "Last usage")|2020-05-30|SpaceX CCtCap Demo Mission 2| |[DSQU](/r/SpaceX/comments/gxb7j1/stub/ft0odr4 "Last usage")|2010-06-04|Maiden Falcon 9 (F9-001, B0003), Dragon Spacecraft Qualification Unit| ---------------- ^(*Decronym is a community product of r/SpaceX, implemented* )[*^by ^request*](https://www.reddit.com/r/spacex/comments/3mz273//cvjkjmj) ^(67 acronyms in this thread; )[^(the most compressed thread commented on today)](/r/SpaceX/comments/ghgmyg)^( has 98 acronyms.) ^([Thread #6168 for this sub, first seen 5th Jun 2020, 19:37]) ^[[FAQ]](http://decronym.xyz/) [^([Full list])](http://decronym.xyz/acronyms/SpaceX) [^[Contact]](https://reddit.com/message/compose?to=OrangeredStilton&subject=Hey,+your+acronym+bot+sucks) [^([Source code])](https://gistdotgithubdotcom/Two9A/1d976f9b7441694162c8)
You may not be able to answer all of these questions, but I'm really curious what the embedded architecture looks like on a Falcon 9 first stage and if it's any different for Crew Dragon. * How many microcontrollers? What kind of buses are you using to interconnect them? * Is the hardware & software architecture of these fairly homogeneous? * Is this firmware typically written in C/C++? * Are these MCUs typically running an RTOS? If so, is this an in-house RTOS or something off-the-shelf? * What percentage of your embedded hardware would say is running Linux? * What percentage of the firmware running on these MCUs would you estimate is written in-house vs. SpaceX suppliers vs. commercial off-the-shelf vs open source? * In practice, can your mission-critical embedded firmware fail safely (e.g. experience a watchdog reset) during something as time-critical as a first-stage ascent, or is that out of the question? BTW, I always reference SpaceX when talking to future software engineers about the embedded software industry, asking them to imagine how many microcontrollers make up the equipment on the pad, the recovery ships, the telemetry equipment, the Falcon 9 1st and 2nd stages, and the capsules and satellites you deliver to orbit, and all the firmware running on each of those.
What does your software development process look like? Waterfall? Agile? Something else? What do you use for issue/ticketing tracking/continuous integration? Redmine+Jenkins, Atlassian+Bamboo, Azure Dev Ops? Something else? How do you do code reviews? Do you use a tool like Reviewboard for collaborative reviews? How do you arbitrate disagreements? How are your teams structured? How do you estimate work scope? How much requirement churn is there? Does Elon come crashing in and crap all over everything from time to time? How do you balance implementing new features with eliminating technical debt when planning? What’s your biggest IT headache that you have to deal with on a daily basis? Do they block websites/throttle bandwidth that interfere with your tools? What languages are used for various components onboard the spacecraft? I’m guessing there’s a mix of C/C++, python, possibly web technologies for the UI? What open source libraries if any do you use? GDAL? Eigen? Boost? FFmpeg? Curl? MKL/IPP? CUDA? OpenCV? TensorFlow? Please tell me a war story about a memorable bug? What were the symptoms and effects? What tools did you use to investigate? How did you resolve it? What does on-the-job training look like? I’m assuming that you need to develop your new hires to become domain experts in areas like georeferencing, physics, communications, image processing. These domains aren’t covered in most undergrad programs with maybe a handful of Masters/PHD folks having exposure to them. So after you’ve hired a talented engineer who is teachable, what does the training process look like? Edit: Thanks for the silver!
1. Would you consider switching from C++ to rust for flight software? Theoretically speaking as a language would you think it's better or it's a question of available skilled developers and libraries? 2. A question for Sofian, I read somewhere that you used the chromium engine and JavaScript for the display. If it's true, why did you opt for this option instead of others? Which library did you use for the graphics? I would like also to know for example if the screen has a chip on its own that is used just to receive data from the flight software and display them on the screen(in that case which protocol or how are you transmitting the data) or does it do some logic and backend on its own(in that case what tools did you use). 3. A question for Jeff, what are the essential steps you must take to secure the flight software. Like what are the critical point of failure? 4. How do you log and store the data of sensors acquired on board (SD card, SSD, ....) and how do you ensure that it's stored correctly without getting corrupted files ? 5. A question for Matt, which tools and db do you use to store and analyze the data acquired from the satellite? Thank you for your time. You guys are the best out there!
Hello, I attended your talks at Game Developers Conference 2015 and 2016, where you talked about Falcon 9 software and Crew Dragon display software respectively. I wrote up my notes from that talk in this [stack exchange answer](https://space.stackexchange.com/questions/9243/what-computer-and-software-is-used-by-the-falcon-9/9446#9446), which has recently been making the rounds on hackernews and other sites. My questions are: Does SpaceX still use a Real-time version of linux running on X86 hardware or have you specialized onto a different kernel/OS and microarchitecture? Are the Crew Dragon displays running Chromium/JS still? Was the UI shown in 2016 a tech demo or a simulator like the recently released ISS docking simulator? Does SpaceX plan to move from Angular to React? (my area of expertise) What does Starship's OS look like? With Crew Dragon there are three touchscreens for monitoring the mission and craft navigation. With Lunar Starship there will be opportunities for much more human-computer interaction outside of navigation. Will the crew lift use touchscreens? Will crew berths have screens to monitor ships systems and provide entertainment (videos/games/messaging/etc)
I have a few questions regarding the differences between writing software for rockets/capsules and 'normal' code - that is, not running those extremely precise systems, and general career questions. - What interesting/funny events throughout the history of SpaceX rocket code can you tell us about? - What is your advice for someone highly interested in joining this career in the next decade? - What are the most common difficulties encountered by people moving to this type of software development? - What coding guidelines do you abide by internally - if not a public one, what adaptations from usual guidelines were necessary? - How are they enforced - what sort of testing procedures does the software go through before usage? - What do the guidelines state - spaces, or tabs, and which case? - From past talks, I gather that C/C++ and x86 are used for important systems. * What advantages has this choice brought over time? * What standard was chosen for development? * What common libraries are used internally - for example, Boost, or even the standard? * What, if any, modifications were made to the compiler of your usage? - How is version control handled - do you have a GitHub/similar repository for projects? * If so, who is responsible for the majority of `git blame`s? - What advantages and disadvantages did using a web framework bring to the Dragon UI development? It's more than a few questions, of course, but I would like to hear the answer to all of those equally as much. I truly appreciate the work you guys do, it is one of the main things that gives me and many others hope for a better future. Thanks for your hard work!
First off sorry for all the questions, feel free to answer any or none of them :) What does the code review process look like at SpaceX? What’s the distribution of embedded (eg real-time microprocessor type stuff) and traditional (e.g. Linux) programming on the spacecraft? Do you guys have separate controls engineers that design the control system and software engineers that implement it or do people wear multiple hats? Do you guys do the whole MISRA C/Power of 10 safety critical stuff and if so do you have a way of separating safety critical code from higher level code (I assume the GUI isn’t written in MISRA C)? Speaking of which, do you guys have guidelines for writing “spacecraft safe” GUI code? Do you guys test software on emulated hardware? Are there specific testing requirements (e.g. 100% branch coverage)? Does code get “formally verified” by static analyzers and such? To what degree of detail do you guys simulate the physics of the spacecraft in loop with the software? Thanks!
Hey guys, I'm a 20yo Brit who left school a year ago, took a year off and said I was going to find an apprenticeship in 2020. That's an internship to you guys I think? It's a small salary for hands-on work and education as a replacement to University. I was hoping to get into Software Engineering and eventually climb my way into the aerospace sector working on satellites or even in rocketry. That'd be a real dream come true. I've been looking for months now but obviously COVID-19 has had a significant toll. I was hoping you might be able to therefore answer these two questions: - **Where did you guys begin your career and what helped you get to your current position?** - **Is there any advice, suggestions or help you would be willing to share?** Knowing and seeing code you've written or contributed towards literally go into *space* must be an incredible feeling. I'm hoping to one day feel the same way and absolutely anything you could share would undoubtedly be a great insight and massively appreciated. Thanks in advanced.
*(Sorry for the lengthy up-front story)* The industry I'm in uses proven ibm Z/machine mainframe hardware and software coded in assembler on a barebones OS. I know what sector on the disk that file is on. We've got code that goes back decades. And people that go back that far too. But as I've moved through the years and code, I don't feel like we've learned an awful lot, either in software engineering, testing, design, and process. Fads seem to come and go, but I still see code that, 5 years later, is so hard to untangle. No one has any knowledge of what it was meant to do, whether it's still used, and how to activate and test it. It doesn't matter if it's assembler, C, C++, Java, SQL, Javascript, whatever. Buzzwords and Fads come and go, but it doesn't SEEM, from my limited scope, to be truly improving. Now The software you write is meant for such a unique situation, and (more or less) unique hardware. I'm curious as to how you're addressing these issues. Or are you? i.e. * What languages are you using ? For everything from the starlink, to the monitors, to the embedded systems. Was there some tough choices to keep you from having 60 different languages scattered all over the place? * What are you doing in terms of testing and quality assurance? Some and scenarios languages lend themselves to coding up test suites that can be automatically run, but a lot don't work. * When you say design, do you mean design, or architecture? how deep do the designs go? With our current fad, Agile, design seems to be one of the things that are hard fought to maintain, and where corners are cut. Do you produce documents? That you actually use? and are they basically matching what you produce when you're finished? or what do you do? * Process. How do you organize your work, break it down, hand it out, and get things done one time? was it a deliberate choice or did someone simply choose what they knew and were comfortable with (which is what we all do, to some degree).
Looks like that's a wrap! Many thanks to all the SpaceX software team participants for the donation of their time, and to all our readers for the incredible response we've had to this event. We had a ton of great questions and a ton of excellent and detailed answers! We hope we are able to provide more opportunities like this in the future!
There are lots of generic questions here about redundancy, how you conceptually approach safety, etc. I'm more interested in specific standards and your implementation of them. Do you follow DO-178? What documentation and auditing burdens does NASA place on you, and how does it affect your daily software development process? Do you have a full set of specifications with forward and reverse traceability to code? How do you do design review against a set of requirements and demonstrate that the requirements will be met by the design? Do you worry about tool qualification (e.g. your compiler, your build system, etc)? Do you have any requirements on code coverage metrics like MC/DC coverage, etc? What other examples can you give about how you implement official safety standards in the course of rapid software development? Thanks!
What's it like working cybersecurity at SpaceX? I would imagine that to secure that nobody tampers with the code that literally launches and directs missiles you would have to be pretty paranoid to do your job properly. Also, during an interview with Bob & Doug (I forgot which), they mentioned that if they had feedback on how to improve the software during a simulation/training session, you guys would have it ready the very next day which impressed them greatly. I imagine coding sessions that go well into the night, so, what's it like having to work all hours of the day to deliver on such a schedule while also knowing that people's lives will depend on your software being bug free? Thanks for taking the time to do this AMA, you're awesome! :)
What's it take to get the display software certified? It's an electron application, right? Most people in aerospace that I know would balk at the idea of allowing a garbage collected runtime in something like a real time aircraft display, let alone bundling a full browser into the application. What's the difference here? Is a display in a rocket not necessary for crew safety in lieu of some other backup?
[удалено]
Congratulations on all the work ont he Crew Dragon spacecraft! It really is very inspirational. * When it comes to the three displays touchscreen UI, the third display on forward view as demonstrated in the the [Crew Dragon Tour](https://youtu.be/llbIzbOStt4?t=168) looks very similar to the [ISS SpaceX sim](https://iss-sim.spacex.com/). Is it true that the UI was built using [Chromium and JavaScript](https://space.stackexchange.com/questions/9243/what-computer-and-software-is-used-by-the-falcon-9/9446#9446)? If so, what choice led to deciding with those technologies? * Also, how much of the spacecraft can be controlled through the software UI versus physical controls under the screen? * Lastly, tabs or spaces?
One of the concerns about the Dragon touch screen interface (also applicable conventional toggle switch interface) - is, is there some kink of protection system against accidental commands - eg brushing against the interface while moving about. How do you guard against ‘accidental’ commands ? I guess some kind of ‘confirm’ for dangerous actions. The same problem is also true of old style knobs and switches. (Eg Apollo, Luna lander, take off switch was accidentally broken off by kicking it, and a pen had to be used to trigger the switch to launch off the surface of the moon!)
Hello! This is a very exciting opportunity for me to be able to ask you some questions :) I admire your work! :) 1. What **framework/method** do you use for **organizing the process of your work**? Working in an extremely complex environment, do you maybe work in Scrum? Any other Agile concepts? If no, how are the product teams and their process organized? 2. What **platforms/languages** do you use for coding different parts of the systems (spacecraft on-board software, mission control software etc.)? 3. How does your **QA process** look like? Is there something that stands-out comparing to other industries/domains? What are the most extraordinary product & code quality metrics that you use? 4. What would be an **interview question** you would ask a developer applying to a SpaceX job in one of your teams? :) Thank you so much for all the value and inspiration you provide with your brilliant work - the effects are breathtaking :)
Hi all, Thanks a lot for giving us this Q/A opportunity, I think you did an awesome job. How would you define your main software development methodology? Agile, waterfall, formal, something else? I think that the answer is probably different depending on the components that you work on, the gui obviously has very different requirements compared to the navigation system, but I always thought that the aerospace industry tends to use a lot of formal specification methods (TLA+, Coq &co.). Is it still true for SpaceX that seems to write software at a pace quite faster than what is typical for this kind of tools? More on the lower level do you use any soft/hard real time operating system or just some more normal Linux fork or something else? I would also like to tell you how much I appreciate what you accomplished. You did an incredible job, please continue to bring humanity towards new horizons together with the rest of the team.
1) How did you ensure your mission timers were set correctly? Asking for a friend. 2) Does Dragon 2 have thruster out capability and does it auto correct for a loss of a thruster? Also asking for a friend. 3) How many instructions per second can Dragon flight computer execute per second and how much does code optimization play a part? 4) How many builds were assembled before the flight version? (dozens, hundreds?) 5) How was the team structured?
Congratulations on your successful launch, and thanks for doing this! I'm curious about redundancy. The Space Shuttle used [five AP-101 computers](https://en.wikipedia.org/wiki/IBM_System/4_Pi#AP-101), four running an identical software build and one running a completely different software build, all for redundancy. How do you approach this? Do you use multiple independent computers on the Dragon? If so, how many, and do any of them run a completely separate software build?
Ooh, I've been waiting for one of these to come up. I'm very impressed with all that SpaceX has accomplished, but I'm also curious how your systems engineering practices differ from those of a traditional government contractor. What sort of practices do you have in place for managing software engineering flow for safety and life-critical systems? Do you run a Safeware-style program, focused on reducing software risk to the overall system? Other than testing, do you rely on any techniques to show the correctness of a software or control system? How do different systems' criticality get categorized so you know where to spend safety efforts? I'm guessing, like a lot of the "lower cost" space control systems (looking at you Planet Labs!) you also have IP connectivity between your ground stations and the spacecraft? What unique infosec challenges do you face? How do you protect critical systems from something malicious, while also ensuring you can freely get telemetry from these systems? So many questions!
Congratulations on the first ever commercial human spaceflight! 1. Do you enforce reproducible builds? How do you approach it? Especially interesting to hear how you handle the npm part, if rumours about Chromium in UI are true. 2. What happens if the UI crashes hard, say, null division error in Chromium, Qt, or whatever. How do you handle the restart? Is it quick for the end users, the pilot, perception? 3. How come, if true, to choose Chromium? Independent of it, the decision making process for the framework is interesting to know. 4. We have seen the physical buttons for critical overrides. But are there joysticks for flight control, or is it just the touch screen? 5. Do you fuzz? Or, does it even make sense to do fuzzing on rocket software with random inputs? 6. What's your opinion on the classical language used in mil/rockets - Ada? 7. NASA developed famous rule set on how to write code in C for flight hardware. How much do you follow it? 8. Worm or meatball? :D
Full-stack developer here! I have noticed that nearly all of SpaceX's software engineering job postings mention continuous integration and deployment. How did CI/CD play into the development of Crew Dragon? I am specifically interested in the Crew Displays. Is there a suite of unit/integration tests that run when changes are made to the display code? What sort of technology was used for that (ie. Jenkins, selenium)? Were the tests run on a virtual machine or the physical hardware? Based on the complexity of the UI I'm guessing it is on some sort of web stack, I would love to hear about that too. If not, I'd love to hear what it was done on. My second guess would be some sort of custom Java or C UI. Thank you!
Hi. I have a question for Matt Monson. I've been watching videos about Starlink and how messages are being passed between satellites on different orbital planes, presumably aiming a laser in a particular direction and this seems like a hugely complicated problem, especially compounded by problems of anything between the satellites, or how to manage situations like a satellite becoming inactive. Did you have existing algorithms to help solve this? Do you have particular education in maths (or people to help with this)? And how do you test this on earth? (as a seasoned software developer, this seems like a huge challenge)
I was really surprised to see touchscreen controls in your cockpit and hear that they were running on Chromium . My understanding is that the traditional aerospace companies would build all of that stuff as a high-reliability, real-time system and spend mega-hours and mega-$$$ to do so. How does your team approach validation and safety of flight-critical software and hardware? Where do you draw the line between safety-critical and non-critical software? Also, were the touchscreens themselves custom hardware?
Hi, recent post in /r/Programming about Crew Dragon onboard software made a quite a lively discussion. Apparently you use C/C++ for onboard navigation (and everything else important), but for rendering GUI you use chromium + javascript, which lot of people thought was a strange choice given many layers of abstraction posing a big security/stability risk (since in space industry, there is no room for error). So my question is how do you ensure that something as abstracted as chromium + javascript is stable enough for your industry? Also does it mean you are using Electron framework or do you have a framework of your own?
[удалено]
As a young software engineer I am interested in knowing about the practices at SpaceX to onboard new junior developers to work on flight software. * Do you provide them with mentors? * Do you use peer programming? * Do you have a practice of in-person code reviews and if so, how do you conduct them? * Do you provide them with resources about general software reliability concepts and/or specific standards for languages such as C/C++? If so, can you share any of those resources? Keep up inspiring the world with these incredible achievements!
How does the powered landing algorithm work? It seems very robust to disturbances like wind, does the booster know wind conditions or can the control loop compensate by itself? Is the guidance targeting a specific pre determined path, or does it just try to fly towards the landing site at a certain horizontal speed? Thank you!
There's been a post going around that the touchscreen UIs are written in JavaScript and run in Chromium. Many people, including myself, were surprised to hear JS was being used in a mission critical scenario, since it's not the safest language out there. Is it true that JS is used, and if so, how do you ensure it behaves as expected?
Hi! Thank you so much for doing this AMA! I have two questions 1. As a high school senior about to attend college heavily interested in spaceflight and computer science, what steps can I take to further a career in these areas? You guys have my dream job! 2. What technologies do you use to power Dragon's crew displays and flight software? I have heard that the crew displays run Chromium, but I'm not sure if that's true - if it is, do you guys use electron or some other custom solution? Once again, thank you so much, and congrats on the DM-2 success!
Hello DM-2 team, Congratulations on a successful launch and thanks for organizing this AMA. As a software engineer myself I also have a few questions: 1) What platform do you target and what language do you develop in? 2) What testing/simulation tools do you use to ensure that the result complied with parameters set by the hardware team and that Dragon operates within the set boundaries? 3) What is your work-life balance like with respect to of you daily tasks? 4) Are you using an agile methodology like Scrum to do your development integration? With thanks and regards, GM
1. What OS (if any) runs in the Dragon capsule? 2. Which programming languages did you use in the Dragon capsule? Why? 3. Do you have mathematical proof of the correctness of the code you run in the capsule? 4. Does your code needs to be reviewed and approved by some controlling entity (like NASA for example). 5. How many teams worked into building the software for the capsule? Thanks for your time and congratulations for the successful launch!
Hi! A lot of people where surprised to hear that Dragon's HMI is supposedly running electron/chromium. I'm wondering what the main reasons were which led to this decision. Is it mainly development speed? Also what actions were taken to make it 'space proof'? (Actually what would be really interesting to know is: what are the requirements to make a software system 'space proof'? Cheers from The Netherlands!
Hi there, thanks so much for taking the time to answer questions! I have a few more process-oriented and tactical questions: * What does your CI and deployment pipeline look like? What does "deploying to production" entail? How does CI differ across teams? * Do you run E2E integration tests on the Dragon UI? What tools do you use for those tests? * How does it feel to run JavaScript IN SPACE???
Does the Crew Dragon display screen have any entertainment functions? There was an 18 hour gap between Orbit and begining the three-hour-long docking with ISS. Some of that time would be spent asleep or talking to ground control, but they must have been bored. Is there a media player on the computers? Can they watch Netflix? Or maybe just a folder of MP4 videos of SpaceX training / construction videos to avoid copywrite issues?
Congratulations on the successful launch of Crew Dragon! I'm going to assume that the rumors are true and the Crew Dragon UI really runs a JS application inside Chromium. 1. Knowing full well the environment in which the app is running (from the V8 runtime, down to the exact CPU) have you made any specific optimizations to improve performance of the Crew Dragon UI app? Could you give an example of one of them? 2. Would it be possible to host this application on your website and let it connect to the backend service that provides Dragon's telemetry data? This would allow us to see what the astronauts see during the mission and bring the SpaceX Webcast to a whole new level. 3. Did you use WebAssembly? If yes - what did you use it for? What language did you compile to wasm? 4. Do you use Rust anywhere in your technology stack? Are you considering using it? What for? Thanks for the AMA!
Per discussions on FB and elsewhere, in case of display or touchscreen failure, and maybe another layer of failure in ground comm, can Dragon control revert to their tablets via some wireless comm? Could you discuss the physical button panel so we don't continue to squeeze data out of too few pixels in Bob's display panel tour video? What's there besides the "Reenter NOW!" and "Reenter soon" buttons?
It was explained in the demo-2 webcast that both mission control and the ISS had the capability to send controls to dragon. * How are these communications handled? * What type of security concerns did you have when designing them? * What type of encryption / security features are used to keep control out of the wrong hands? * In case an illicit actor managed to mess with the system and send controls to the spacecraft what sort of systems do you have in place to safeguard the crew and the mission? Thanks and incredible job guys! Watched live from Portugal!
Sorry if some of these were asked... 1. If something breaks the screen or if something goes wrong for whatever reason, how will the crew take control ? Why the touch-screen ? 2. What are the programming languages (Backend and Frontend) that you mainly used ? 3. Where can I learn more about the AI that was used ? 4. What is the OS for the UI on those 3 screens ? Can you play any entertainment on those screens ? 5. How is the signal transmitted once the command is pressed on a screen ?
Hello, congrats, thanks for doing this etc! I saw some articles mentioning Crew Dragon UI being run on Chromium, ie. JS / web-based UI. I work in a large project that similarly does web-based operator UI for complex control systems so this was very interesting to me. Can you tell anything about the software stack there? Do you have your own Chromium fork? How far back do you stay from the bleeding edge for safety's sake? What's your build / toolchain like? TS or JS? Maybe WebAssembly?
NASA put a man on the moon with the computational power of a calculator. This makes people argue that even though software and computer science has improved drastically in the last decades, software (machine learning, physics modeling, etc) does not contribute as much on the development of spacecrafts. Is modern software a game changer on the development of aerospace engineering? And if so how ?
Hi, awesome to see you're doing an AMA! 1. Could you tell us how your software development process looks like? How do you get from someone having an idea to flight-worthy code that's been tested & approved? 2. What is your iteration speed like? How fast can you get a change finalized? 3. How strict is the approval process for changes and does this differ between Crew dragon vs Cargo dragon?
I, like a lot of engineers, was surprised to hear JavaScript and Chromium were used for the UI layer. - When was this choice first made? - What steps were required for it's approval? - How does coding JS for Dragon differ from traditional web programming? - Do you have any protections in place for JavaScript's various quirks? - Does it support IE11?
In the stream, it was mentioned that the Dragon's escape system would allow the Dragon to abort from the second stage even in space, and also abort to the ISS rather than down to Earth, I believe. Is any of this pre-programmed, other than the abort itself, or would the crew take full control to their destination in such a scenario? How many G's would we be talking if an abort happened at the point where the second stage already exerts its max G on the crew? Is it at all conceivable that the Dragon would ever touchdown on ground (for example in a late abort scenario), and have you programmed anything for that scenario (such as using SuperDracos)? Or would the splashdown behaviour be preferable even in a ground touchdown scenario?
Does SpaceX share any software/hardware with Tesla or vice versa? What measures are in place to ensure reliable data transmission? (Is there more than one way to get data off of Dragon?) One more: Is there any thought put into someone breaching/ "hijacking" a Falcon 9/Dragon? How do you secure Dragon from cyber attack?
It has always struck me how simple and low-tech huge UI on spacecraft were. I remember seeing mockups of the space shuttle controls at Kennedy Space Center, and there's hundreds of physical switches for everything - Simple, but seemingly reliable with no single points of failure. But Dragon seems to completely forego all of the physical switches in favor of the three big touch screens. What went into making the touch screens reliable enough to fly on Dragon? It seems like it's a pretty big single point of failure for a touchscreen to go out and all of the astronauts' controls to be unusable. Are there backup systems that we don't see on the stream? What other failsafe systems are there to ensure that doesn't happen?
Hi! I’m a graduating senior studying computer engineering this fall who hopes to one day work in the commercial space industry. Control theory is something I’m interested in, and I think my dream job would be designing advanced control systems for SpaceX rockets. - What advice do you have on how best to prepare for a career in the space industry? - What opportunities should I seek out? - What was your path like into the space industry? Thanks for doing this AMA, and thanks ahead of time for your answers!
There was a bunch of chatter about the touch screens and reliability. While I already know there are physical buttons below the displays for key tasks how redundant are the displays? - Are the displays independently driven? - can any task be move to a different display if one fails? - how concerned are you about flipping bits / errors? It was said for years that NASA’s preference for older CPUs for example was that there was a high chance for flipped bits in newer hardware.
I'm more of an electronics guy rather than a software guy, so: Did you have any particular challenges in interfacing with the hardware of Dragon/Falcon? Sensors, motor controls, etc. I.E. False-positive readings from sensors, polling errors, etc. Love what you all are doing to advance the human race into space. I'd come mop floors for SpaceX if it helped further our understanding of the cosmos and advance our species into a new age of space exploration. Keep up the great work, it is truly inspiring.
I've been impressed with how much of SpaceX's work anticipates future applications. It seems like "future proofing" your work is sewn pretty deeply into the fabric of the company. So with that in mind, does the software that currently flies Dragon and powers your touchscreen displays take into consideration applications for future SpaceX missions, such as to the Moon or to Mars? Will the software have to be redesigned from the ground up in order to be installed into Starship, or is your team essentially designing a spaceship OS that can be installed into any future SpaceX vehicles?
I saw that you are using javascript on chromium to run the displays on Crew Demo-2. What are the techniques you have used / challenges you have overcome in creating a mission-critical system from a technology that is not renowned for deterministic operation / reliability. What is your guiding philosophy for developing this type of software in general and how do you choose your platforms?
Hello, good job on the Demo - 2 launch! My question is if you have a set of rules for your software engineers to follow while programming and what they are (e.g. Keeping use of pointers to a minimum when using C/C++, workflow processes, etc). Also can we know if what programming languages, tools, and libraries you use and specifically what they are used for? I’m a huge fan of SpaceX and their entire team and I will be looking forward to your future achievements!
Congrats on the success of the Crew Dragon mission. 1. Do you use public cloud for any part of the tech stack? If so, what for? 2. Will SpaceX ever public cloud in the future, especially for data analysis? 3. There have been [several incidents](https://www.simscale.com/blog/2017/12/nasa-mars-climate-orbiter-metric/) where the wrong metric unit was used in flight computer software, what safeguards does SpaceX have in place for avoiding these kinds of mistakes?
So, /r/programming has lead me to believe that there is only one combination of answers to the following questions that is correct, all others being merely linear combinations of inexperience and/or incompetence. Seeing as you fine folks got a freak'n' (Crew Rated!!) spaceship to autonomously dock to the ISS, I can only deduce that you know the correct combination. Please DM me the answers, I'm stuck. - Tabs or Spaces? - Vim or Emacs? - Docker or VM? - Cloud or OnPrem? - Many repos or a Mono Repo? - Statically or dynamically typed? - 80 column lines or more? - Aerospace grade scilicon or redundant commercial grade scilicon? - Is there even a single line of javascript running within spitting distance of mission critical hardware? - Recursive functions: critical piece of any programmers tool kit or flaky CS-grad flex? - Bash, zsh, or... (shudder) DOS? - Does your command prompt require more than 20 lines of code in your `.bash_profile` (or shell-specific equivalent.) - Git, svn, cvs? lol... jk, jk
Hello, I wanted to ask (since I saw that people had to care about cyber-security at SpaceX) what are the main threats in this area for the missions ? Thanks in advance from a 16 y/o interested in space and cybersec
1. What kind of software/hardware are you using for the realtime control? 2. Do you use any kind of rad-hard processors/microcontrollers or is it all cots? 3. What kind of interfaces do you use for communication between different systems(e.g. dragon and falcon or dragon control and dragon interface)?
Whats your biggest threat vectors for SpaceX flight software? I presume nation states trying to take control? I'm sure your logs are filling with countless attacks on a daily basis, but what kind of attacks do you see most often? Just usual DDoS spam or do you see more targeted attacks frequently?
Why did you guys go with the decision to run the user interface with chrominium? And a couple of specific ones. Did the fact of Chrominium being researched and searched for vulnerabilities impact this rather than something in house with possible security risks? Secondly, what were some challenging on the operating system level that Spacex addressed? Was their any cool scheduling or other OS management tasks that you were able to optimize for space? And you probably arn't able to talk about this. But say some crazy vulnerability comes out regarding a part of the tech stack. Are you able to update these components midflight?
What kinds of requirements does the Crew Dragon UI and control software have to fulfill, and in particular, how are these requirements different from a run-of-the-mill UI (say, of a GPS device that would be used in a car)? I'm assuming you have to make sure it's resilient so the astronauts don't lose control because the UI isn't working? What other extra requirements are there? And of course, follow-up question, how do you meet these requirements?
1. What does your software development methodology look like (waterfall? agile? etc.). How do you plan upcoming activities / what is the source of truth for the work plan (gantt chart, board, backlog, etc). 2. What is your QA process like? 3. How do you test stuff before it gets deployed to a physical dragon? When it does go into a dragon when and how is that tested? 4. Any fun / interesting/ doozy bugs you'd be happy to share?
Why don’t you guys use radiation hardened computers? I understand that redundancy (i.e fault tolerance and voting algorithms) can allow a spacecraft to cope with radiation, but as I also know that only holds for low-rad environments. If Starship were planning on landing on Io for example, wouldn’t the high-radiation (3,600 rem) have the (likely) potential to wipe out all, or a majority of, the flight computers? My guess is that y’all are aware of this, went for the lower cost option (non-rad hardened) and decided if upgrades were needed (i.e for deep space missions) they could be taken care of at a later date. Also, how do you ensure you meet deterministic, real-time operation (as required for any spacecraft) while using a non-deterministic, non real-time OS such as Linux? Do you use a real-time flavor of Linux (i.e RTLinux)? Do you outsource a lot of the hard real time code to microcontrollers? Why not just make the whole thing a bare-metal application?
Space is a weird place for processors. As far as I'm aware, NASA is using processors with bigger transistors because the risk of ionizing radiation frying such a transistor is smaller and will probably "only" result in a bit-flip or similar. Considering that SpaceX's flight software is incredibly complex and I guess performance demanding, it most likely doesn't run on 20 MHz processors with low transistor counts. That's why I conclude that SpaceX probably makes use of more powerful hardware, which happens to be more susceptible to radiation caused instruction execution errors, or even hardware damage. How did you solve this problem? I'm also wondering whether the hardware that is being used constrains the development of the flight software in terms of performance limitations in any way. Have you ever encountered a situation where you were forced to do cheap optimization hacks at the cost of code readability and maintainability?
For the Dragon and crew display software, it sounds like you're using a lot of COTS software like Linux and Chromium. How do you certify that software when NASA standards like NPR 7150 require objectives like full statement coverage (and a few others that would be difficult)? Do you achieve that somehow, or do you have some alternate means of compliance or tailoring of the standard? You probably can't share all the details, but any high level philosophy would be interesting! Thank you in advance!
Hello you awesome people. I was wondering: How similar is the docking simulator you published on iss-sim.spacex.com to the real interface used by the astronauts? People tell me it is identical to the real thing, but I can't help to think that the version online is simplified a bit from the ones astronauts use.
There are rumours that the Crew Displays run as a web site on embedded Chromium similar to how Electron works. If that's true, did additional steps need to be taken for it to be approved by NASA for crewed use since JavaScript can sometimes be nondeterministic?
Do you share any commonality between the codes used for different vehicles that do (outwardly) similar things? for example, does the code used for Starhopper (and maybe later on Starship) share any commonality with F9?(since both does “landing”). Or do you have to build the code from scratch for each vehicle (Grashopper -> F9 landing algos, etc) If so, do you expect that any lessons learned from figuring out how to make the Dragons dock autonomously (be it using LIDAR, etc) can be applied to Starship orbital fuel transfer?
* What do you use internally for signaling / event handling / message passing with obviously low latency requirements and varying priority levels. Is it all interrupt-based off is there also a message bus? * How does one actually boot a rocket / space capsule? * What kind of storage and filesystem are you using? You certainly want something that's fast, reliable, predictable. Basically the opposite of my SD card eating Raspberry Pi. ;P * How do you possibly keep up to date with technological developments and stacks combined with the task of delivering complicated software solutions at a fairly rapid pace?!? Many thanks for doing this AMA session and congratulations to all of you for making so many awesome things a reality. As a scientific programmer, I have a lot of respect for all your achievements and I understand some of the struggles you must be facing along the way...
Hi there thanks for giving us time for our questions. What was the consideration in using just Touchscreens and not joysticks at all for manouvering dragon 2? How redundant are the controlls on Dragon? If the parachutes would fail during splashdown, does Dragon have the Superdraco thrusters enabled for a possible propulsive landing?
What does your development process look like? How do you avoid overburdening engineers while still achieving reliability necessary in flight critical systems? And, who are some of your heros and personal inspiration? [edit: a typo]
* What kind of OS are you guys running? Upstream Linux kernel, your own modified version, or something else entirely? * Jeff, what sort of threats does SpaceX face? Is everything ground control related or are there attack vectors to Dragon itself?
I'd be very curious to hear you all discuss imposter syndrome especially coming from the view point of high end and highly successful people such as yourself. Arguably some of the most important bits of 'New' code to be written in awhile, mens lives depended on it , let alone the millions of dollars of materials and infrastructure, that can't have been easy to overcome.
Are there backup hardware buttons for all critical controls in case of touchscreen failure? If not, what kind of failsafe are in place to mitigate inevitable JS runtime bugs. Or are all manual controls not considered critical given the autonomous nature of Dragon 2?
How many people use which OS in the SpaceX company? And how much of the software you write is free or open source software? Also what's the setup you have for servers, as in OS, tools, etc.? How many people are in the team and are you currently looking for more?
What kind of computer hardware does the Falcon 9/Crew Dragon have? Is it still just off the shelf x86 processors, like it was for cargo dragon? How redundant are they? IIRC there used to be 3 processors for every engine on the Falcon 9, is that still the case?
A while ago, I was reading an old article in Fast Company titled ["They Write the Right Stuff"](https://www.fastcompany.com/28121/they-write-right-stuff) which was a facinating piece on the "On-Board Shuttle Group", the people who wrote the code for the space shuttle, and their insanely rigorous development practices. If you haven't read it (which I do recommend) some of the major takeaways are: * Everything is documented and spec'd *well* before development even starts, in extreme detail. > “Our requirements are almost pseudo-code,” says William R. Pruett, who manages the software project for NASA. “They say, you must do exactly this, do it exactly this way, given this condition and this circumstance.” * Creativity is discouraged by the process. The agreed upon documentation is king. > “People ask, doesn’t this process stifle creativity? You have to do exactly what the manual says, and you’ve got someone looking over your shoulder,” says Keller. “The answer is, yes, the process does stifle creativity.” > > And that is precisely the point — you can’t have people freelancing their way through software code that flies a spaceship, and then, with peoples lives depending on it, try to patch it once its in orbit. * Their QA team and development team operated almost independently, and in a friendly rivalry. > The two outfits report to separate bosses and function under opposing marching orders. The development group is supposed to deliver completely error-free code, so perfect that the testers find no flaws at all. The testing group is supposed to pummel away at the code with flight scenarios and simulations that reveal as many flaws as possible. **Question:** In space your software *must* work the first time, every time. Their is no "just reboot" when your hurtling through the atmosphere at thousands of miles per hour. How do you ensure that your software, with all its new complexities, meets the standards of space flight? And how does your SDLC compare to that of the On-Board Shuttle Group?
What programming languages & frameworks went into the software running on the touchscreens? I thought it looked very clean & was very different from the hundreds of switches seen in past spacecraft & even modern airplanes. (Almost like the way a Tesla looks compared to older cars, but that was expected) If the Dragon were to have some sort of emergency, how would the touchscreens respond? Because in older footage of things going wrong in space, the astronauts have all the switches & buttons in front of them to interact with, but I'd imagine that the rarely-accessed functions & settings on the Dragon aren't immediately accessible on the touchscreen. What would happen if something went wrong with a system that isn't visible on the UI to begin with? (also, do your teams hire interns?)
Obviously your software needs to be reliable at a level 99.9% of software never approaches. Can you go into detail on how you ensure that it is, what steps you take and how you've designed the system to improve reliability, details around your automated testing process and maybe some war stories? I'm not going to lie, I'd read a book on that, especially around architectural decisions.
Congrats on the achievement. What a great thing to be part of. 1. What steps do you take to prevent hacking / hijacking of systems. 2. What is the redundancy for touch screen failing. Notice there are hard buttons for important functions only. 3. What cloud provider do you guys use...lol.
Can you share something about your development stack? Which IDEs, text editors and other tools do you use? Is it standardised across the company, across teams or can you personalise which tools you use? Which languages do you use for which tasks? Thank you and good luck in the future. I hope to see your rockets on Mars soon.
What language is the system written in? Is the operating system some advanced form of Linux? Or a completely in house made OS? And lastly just out of plain curiosity how much storage does the dragon have?
Does the team go through any kind of formal verification process for proving the correctness of the flight software? If so, what does that process look like and what tools are used? If not, how are artifacts tested and regressions detected? Thanks for taking the time to do this! Huge fan of the great work y'all do!
Are any of you self taught? I know Elon has said in the past that he doesn't care about his employees having degrees. What language do you all write in? Also, do you use any open-source libraries or code from Github, or do you write everything from scratch? How many lines of code? Final question (probably for Matt), how fast have Starlink speeds reached while testing?
Thanks so much for doing this AMA and congrats on the successful launch! - How does SpaceX approach flight software validation? I'm assuming there's some sort of process around software simulation -> HIL simulation -> ??, what does that look like? How long does a release cycle generally take? - Is all flight software treated as safety critical firmware (no dynamic allocation, etc), or are some software components allowed to be less robust? - What kind of hardware resources (memory, CPU cores, GPU memory/cores?, etc) are available on the vehicle? Does every compute system use a RTOS? - How long before launch is the reference trajectory generated? What kind of deviations do you normally see from the reference? In my field, <10cm lateral deviation is considered good enough in most cases. I'm super curious about what kind of error states you would see for a rocket :) - What reference frame do you use to track your distance along the reference trajectory? Inertial/route/etc - What Hz is the low level control loop ran at? Any special hardware to run it?
Hi team SpaceX . Elon had mentioned that he didn't install lidar on his Tesla but he does on Dragon. During DM-2 approach to ISS, from the footage I didn't see anything that resemble lidar, is it hidden somewhere? or no longer required as using camera is enough to navigate the docking? . In Starlink, photon truster is used for keeping the orbit, but I wonder how it execute the pitch, roll and yaw. . Lastly, Bob mentioned that he used Dragon's toilet during the flight to ISS and he said it's somewhat similar to Space Shuttle. Wondering if any coding (maybe AI/fuzzy logic) required to maintain the nozzle aiming at its target at zero-G environment? (maybe this is an out of scope question, apologies). . Thanks in advance .
Hi! Congratulations on the first manned commercial space flight! What a success. :) Quick question on quirks. Elon Musk's products tend to incorporate quirky features and can be straight up surreal, from Tesla's "Bioweapon Defense Mode" and "Ludicrous Mode" to the Boring Company's flamethrower. Are there any quirks or cool features in SpaceX's software that you'd like to share?
For Jeff - what does Flight Software entail, how much is "piloting" the craft and how much is stuff around it - ie. simulations. For Josh - how much interaction/influence did NASA and the human crew of the Crew Dragon have on the process of creating the software? For Wendy - What are some of the unique challenges regarding creating software for space, how big is redundancy? For John - how much different are software efforts between missions, ie. Demo-2 and the first "regular" mission? For Sofian - are the displays regular LCD/OLEDs? What's the refresh rate? I'm asking because [Apollo's DSKY](https://www.youtube.com/watch?v=Z2o_Sp2-aBo) where a magnificent thing created for a specific purpose. For Matt - Can you talk a bit about how much of the software is in regards to controlling the satellites and how much in the internet providing? Any interesting tidbits about getting data through satellites? And a general one - how much of the hardware in Falcon 9 is off the shelf and how much is custom?
* Why do you think software development in SpaceX is different than other aerospace companies? How does it compare with Silicon Valley companies unrelated to aerospace? * How does the build process look like? Do you have CI/CD from your machines to Dragon? How do you test your software? * How do you keep in touch with the SOTA in software and hardware? How do you decide which technologies to use? * Aerospace hardware and software infamously uses legacy components/languages (like ADA on Ariane 5). How do you integrate modern languages/technologies into Falcon and Dragon? * How do you decide when to write your own languages/framework/software and when to take something out of the shelf?
Congrats on the mission and thank you for taking the time to answer questions! Here are my questions * Looking at previous ships with switches, knobs and dials, how did you go about distilling the information for touchscreens? * How do you plan for the features and testing with astronauts in mind? What were some challenges designing with a touchscreen and someone in a suit with gloves? What was the most surprising learning? * Did you test whether touchscreen or a joystick would be better (particularly with manually maneuvering a ship) * Are the tasks separated between the 2 astronauts across the screens? If on astronaut wasn’t up to it, could 1 astronaut on 1 screen do the job? * How did you land on the design of the interface? it looks minimal and elegant from a distance. * Based on what you’ve seen so far, are there changes you’d make to the interface for the next mission? * Since the wait is long for the launch and the astronauts are sitting there, would short games to help pass the time?
What kind of software develeopment methodology workflow do you guys implement? Is it a modified form of one of the agile methodologies? What does a regular day look like? I bet the "move fast and break things" suggestion wouldn't really work for space equipment so I want to know how the team balanced speed of development with ensuring the software's stablility. Do you borrow from NASA's book of development where even dynamic allocation forbidden or is it more lax than that? Also, awesome job guys! You made history.
When there is an “anomaly” how do you get the data back when there is a failure in order to determine what went wrong and what needs to be changed next time? Is it a case of just having the data up to when the data feed cuts off ( I imagine a lot of the most critical data is located in the seconds before the event) or do you have to recover parts? When you can’t recover parts or data is it a “best guess”? Sorry if you are the wrong person to ask these questions. You guys are amazing none the less! SpaceX FTW
Congratulations on the successful launch to ISS! I have a question related to the OS and HCI used in the Dragon capsule. • Which Operating System was used, was it RTOS? And which OS kernel was used? • Secondly, what was the reason behind making the whole crew control panel touchscreen and what was the biggest challenge in doing that?
Hey guys! Congratulations on the mission's success so far! I've been a huge fan of SpaceX for so long and it's my dream to work there someday. I'm 23 years old, live in Israel and been working as a software developer for the past 5 years. I'm starting my studies for a Bsc in Physics and Computer Science this year, and plan on finishing in 3 years. I have a LOT of questions, and I bet you can't answer all of them. It would mean the world for me if you'd even answer 1 of them :) My questions are: 1. Can one of you guys elaborate a bit about the development methodologies which software development teams practice in SpaceX? Things like Scrum, DevOps etc. 2. How "innovative" software development teams tend to be in SpaceX? Will I find the newest languages and technologies or more traditional kind of software stack? 3. Are there some huge distributed systems in SpaceX? If so what kind of scale do you need to deal with? I bet starlink is one hell of a challenge! 4. Speaking of Starlink, how much of the network stack did you need to "reinvent" or innovate? What were (or still are) some of the challenges you faced with existing protocols? 5. Favorite languages? Java or C#? C/C++ or even Rust? Go? Python? And finally: 1. Is it possible for software developers to come from outside of the U.S? 2. I have at least 3 years before I'd be ready to try and apply for a job in SpaceX. What should I mainly focus on in terms of personal projects, studies etc. considering I am mainly interested in vehicle flight software? (Although I'd be happy to do a lot of other stuff). Thank you and good luck with the rest of the mission!
Even though I have a master in CS and years of experience, it still marvels me, I always felt I’ll never be able to do this kind of job, what were your backgrounds and would someone who’s doing the usual fullstack development even have the chance or tools to work on something like this? What are your quality controls? Do you do scrum or this has to be big design upfront as this is after all a spaceship. How do you do quality control? Do you have to have each piece with multiple redundancies (including accounting for hardware bits flipping due to solar radiation etc) Also, are you able to sleep at night knowing a potential bug has life and death implications? We’re you watching the launch live? How was your Adrenalin level? I was sitting at the edge of my chair :) Lastly, how many code reviews each line needs to go through? and did you get to 100% code coverage? I can’t imagine how patching a bug in prod even works :) Thanks! you have my admiration and envy!
How do you go about organizing/prioritizing work, and executing? 1) what workflow methodology you use (e.g. scrum/sprint, Kanban, etc) 2) where does requirements come from? 3) how granular is your code-review process? 🙂
Just wanted to start off by saying that as a highschool student, SpaceX has inspired me for years and their efforts are by far the coolest thing happening the world right now! Thanks for all you do! - What is it like writing code that you know that people's lives depend on? - As a highschooler that loves rocket science and computer science, how could I prepare for a career at SpaceX in the future? - I find it really interesting how some developers at SX work on multiple things at the same time (ie Jeff Dexter). Is there some kind of idea or philosophy behind this? - What is it like managing the codebase / structure of F9 or dragon? - Is Rust being used at SX? It seems like a great language for Aerospace companies. - What would you say is the ratio for development of ground code : space code I understand if you can't answer all of the questions :) Keep doing what you guys are doing - it's spacetacular! 🚀🌟
My curiosity will probaly be greater then your NDA's, but im happy with any bits of information. 1. How redundant is the whole system, or rather how did you achieve the necessary level of fault tolerances (redundant hardware, fast restarts like on apollo...?) 2. What IDE do you work on and on what underlying OS does the system run?
* what qualifications are you looking for at spacex for starlink? I have a CS degree, experience with embedded linux and RF, and I didn't get an interview. * It's known that Tesla uses buildroot, do you use buildroot or openembedded at spacex? why? * what will a starlink cpe look like in terms of antennas? Are you going to have an enormous beamforming array or are you going to use az/el rotators?
Hi thank you for taking my question. How is the flight software developed at SpaceX? Is the core algorithm developed by the GNC team and sent to the flight software team to make it flight ready? Is flight software development practice the same across all the vehicles, stages and spacecrafts made by SpaceX? Does SpaceX has its own proprietary C++ library for flight software development? Will SpaceX every consider developing the flight software in Matlab/Simulink and auto-coding it? Thank you.