Boeing Starliner launches to space (but not ISS) (reached ISS in 2022)

[anorlunda]

The Boeing Starliner just launched into a pre-sunrise cloudless sky.

It was fun to see the black smoke trail change to white as it lifted into the sunlight (pic 1).

It was not as spectacular as the post-sunset launch from California a few months ago. The trajectory became more horizontal after liftoff (pic 3) and the exhaust plume began rapidly expanding in diameter just 1-2 seconds before first stage cutoff (pic 2).

I'm not a photographer, but here is a couple of cell phone shots.

 

[Jonathan Scott]

Starliner has apparently experienced "an off-nominal Orbital Insertion" (apparently because at the planned time of the orbital insertion burn it was not pointing in the right direction, so the burn was either delayed or stopped) but is in a stable orbit anyway (unfortunately presumably not the right one).

 

[Jonathan Scott]

Starliner has apparently experienced "an off-nominal Orbital Insertion" (apparently because at the planned time of the orbital insertion burn it was not pointing in the right direction, so the burn was either delayed or stopped) but is in a stable orbit anyway (unfortunately presumably not the right one).

The problem is now being attributed to a mission elapsed timer software problem causing Starliner to enter a precise pointing mode too early for the insertion burn, using up a lot of thruster fuel maintaining exactly the right alignment before it was needed.

Starliner is therefore not going to ISS after all. They are planning to land at White Sands on Sunday. In the mean time they are trying to do whatever testing they can in the wrong orbit. See NASA news updates for more information.

 

[mfb]

The parachute failure in the pad abort test and now that. While both individual issues are easy to fix (and NASA highlights that astronauts on board would have performed the burn manually) it leads to the question how many of these issues are still to be discovered. In general Boeing preferred to do certification via more reviews and fewer tests than SpaceX (e.g. no in-flight abort test) but this strategy seems to have problems.
The next flight is supposed to deliver crew to the ISS for a month. I hope NASA and Boeing review all this very thoroughly.

 

[berkeman]

https://www.pbs.org/newshour/economy/boeing-ceo-to-step-down-immediately

 

[mfb]

Starliner landed: Boeing’s Starliner capsule lands after missing rendezvous with space station. This time all three parachutes worked. The capsule looks healthy and should be reusable after refurbishment*.

*in an unusual role reversal, Boeing plans to reuse their capsules for NASA flights while SpaceX will not - their Crew Dragon capsules will still fly to space again, but only to deliver cargo to the ISS or for commercial flights.

 

[nsaspook]

The parachute failure in the pad abort test and now that. While both individual issues are easy to fix (and NASA highlights that astronauts on board would have performed the burn manually) it leads to the question how many of these issues are still to be discovered. In general Boeing preferred to do certification via more reviews and fewer tests than SpaceX (e.g. no in-flight abort test) but this strategy seems to have problems.
The next flight is supposed to deliver crew to the ISS for a month. I hope NASA and Boeing review all this very thoroughly.

I wonder if NASA would really let the astronauts on board fly the craft during this phase in anything other than a life and death emergency.

 

[mfb]

They routinely did for all their history. Assisted by the capsule computers, obviously, but "stop doing what you do now, and make the orbital insertion burn" is something an astronaut can tell the capsule.

 

[mfb]

This is getting more interesting. Apparently Boeing found a bug and sent a software update to the capsule in flight that could otherwise have lead to a crash for the landing attempt - an issue different from the failed maneuvers after launch.
ars technica: Starliner faced “catastrophic” failure before software bug found
CNN: Boeing's Starliner spacecraft , built to carry astronauts, faces new safety concerns
NASA press conference (in 5 hours)

NASA will have the last word but it looks increasingly likely that Boeing will have to do another uncrewed test flight. But even if they don't have to do that the upcoming safety reviews will delay the schedule.
As SpaceX passed all major tests already it is very likely to fly NASA astronauts first.

 

[Dullard]

I think that the situation may be considerably worse than having to do another test. Design testing is (ultimately) based on the oft-incorrect assumption: 'If it works, it must be right.' It's 'oft-incorrect' because it's impossible to test every possible scenario, even for fairly simple systems. Systems like these work because:

Smart people, using a disciplined, organized approach design them carefully. Then, other smart people carefully test them. That includes incremental component, system, and vehicle-level testing.

Note that if the 1st group does their job perfectly, the 2nd group doesn't actually add anything (if confidence isn't a thing). Also note that the 2nd group can't completely back-stop the 1st group. It is simple malpractice to 'discover' stuff like this after a booster is lit. There are a lot of dead Rockwell engineers (Boeing absorbed the Rockwell Space Division) spinning in their graves right now. Were I dead, I'd be at about 3500 RPM.

 

[anorlunda]

It is simple malpractice to 'discover' stuff like this after a booster is lit.

Very good point. Is it known what they did to discover the flaw while in-flight that they did not do before liftoff?

 

[Dr Transport]

Smart people, using a disciplined, organized approach design them carefully. Then, other smart people carefully test them. That includes incremental component, system, and vehicle-level testing.

Given their track record lately, they have neither of the above groups. I agree, there are a lot of former Rockwell engineers spinning and for the past 5 years, boeing has been shedding any and all quality engineering staff because they are too expensive, replacing them with untested new hires.

 

[Dullard]

I didn't see any detail on that. I'm guessing that the honest answer is something like: We decided to take a closer look at our software right after that first (on-semi-orbit) failure. That's actually the best case. It is also possible that it was just obvious once the circumstances were right.

I just can't get past the fact that while this flight was (sloppily, I think) called a 'test,' a proper development program would regard it as a 'demonstration' (given the cost of using an Atlas booster as a test bench). For a manned spacecraft design, nothing that can be simulated on the ground should happen for the first time on-orbit - the pieces are too expensive. This (to me) is Spacex' huge advantage - they got NASA to fund (and empirically tested) most of their vehicle when it was a lowly freighter - They started with lots fewer untested systems when they created the manned variant.

 

[mfb]

ars technica: Boeing’s Starliner problems may be worse than we thought

Specifically, after the service module separated from the capsule, it would not have performed a burn to put the vehicle into a disposal burn. Instead, Starliner's thrusters would have fired such that the service module and crew capsule could have collided.
[...]
At this point, it seems that NASA and Boeing do not yet know what they don't know about the problems, and it will take some time to sort all of this out.

NASA Shares Initial Findings from Boeing Starliner Orbital Flight Test Investigation - they expect a more detailed report end of February.

If there are two bugs that showed up during the first flight, how many more bugs are in that code?

A lot of interesting quotes from the press conference, among them:

Jim (Boeing): Admits they WOULD NOT have found the second software issue that would have destroyed Starliner in reentry if the first Mission Elapsed Timer issue didn't occur".
[...]
Kathy confirms that had the second software issue not been found, the Starliner and Service Module would have recontacted. NASA statement from today says this would have caused a Loss Of Vehicle of Starliner.

 

[Dr Transport]

If there are two bugs that showed up during the first flight, how many more bugs are in that code?

Given their recent history of coding issues, i.e. the 737MAX... your guess is as good as anyone's.

 

[cosmik debris]

Given their recent history of coding issues, i.e. the 737MAX... your guess is as good as anyone's.

Yes, I don't think I would want to fly in anything Boeing has a hand in.

Cheers

 

[hutchphd]

From the NASA release:

There was no simple cause of the two software defects making it into flight. Software defects, particularly in complex spacecraft code, are not unexpected. However, there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects. Due to these breakdowns found in design, code and test of the software, they will require systemic corrective actions. The team has already identified a robust set of 11 top-priority corrective actions. More will be identified after the team completes its additional work.

I believe this constitutes a stinging rebuke in NASAspeak. Very sad.

 

[Dullard]

How (short of a re-write) do they get out of this situation? No amount of duct tape is going to restore confidence at this point and a re-write seems unfeasible in terms of time and money. Program termination (after a polite, face-saving delay) seems very possible. A 'new' program (using the same hardware ) is also a standard NASA move in a situation like this.

 

[mfb]

Bugs can be fixed. It will take months to study all the software in detail, but it's better than relying on a single spacecraft . We should know more in a few weeks. I could imagine another uncrewed flight towards the end of the year, followed by a crewed flight early 2021.

 

[Dullard]

I hope you're right.

 

[hutchphd]

NASA is clearly (from the tone of their press release) looking at reverification of the system. If they have good sense they not involve any Boeing middle management in any substantive supervisory capacity for this undertaking.

 

[Dullard]

That's the reason for my concern about the viability of the program. I've been the '3rd party' programmer trying to sort out an OP code mess (in a much lower stakes/scale situation). The trick will be finding enough programmers smart enough to do the job and dumb enough to confidently declare themselves 'done' with the job any time soon. This isn't a web page - this code interacts with hardware - there are a lot of errors that aren't easily discovered in a cursory 'review.' I don't claim that it's impossible - only that it may take longer than a re-write (done thoroughly). I guess it comes down to what they think they don't need to analyze in detail. Bigger picture - It's also hard to imagine that an engineering management program which delivered this software mess (to orbit) did much better on their hardware.

 

[mfb]

Boeing didn’t perform full end-to-end test of its astronaut capsule before troubled mission, ‘surprising’ NASA safety panel

“Since the two noted problems [during the flight] occurred at system interfaces, one would have to speculate that there was some weakness in the integrated testing,”

Boeing and NASA officials are expected to release the results of an independent investigation into the set of issues that occurred during Boeing’s late December test of Starliner, its astronaut crew capsule, within the next week.

Meanwhile the two astronauts who will fly on Crew Dragon are getting additional ISS and space walk training. NASA considers extending the mission (which was originally expected to be just one week).

 

[hutchphd]

That's the reason for my concern about the viability of the program. I've been the '3rd party' programmer trying to sort out an OP code mess (in a much lower stakes/scale situation)...

What one needs to hope is that the fundamental software is well engineered (not unlikely I think) but the test program was minimized by idiots with good management haircuts. The errors seem to me to be the foolish "surface" mistakes that always occur upon software integration and are quickly discovered by an adequate test program. i am a firm believer in "black box" testing performed by disinterested parties. If so tested I'll bet there will only be a few more changes required (but of course they are vital changes!) to produce good product.

This is in contradistinction to the 737 fiasco which was a bad idea implemented stupidly.

 

[Dullard]

I hope that you're right, but don't share your optimism.

 

[mfb]

Boeing provides update, path forward for Starliner
- Boeing can prepare another capsule for a flight this year and ULA can provide an Atlas rocket to launch it (not that surprising, they wanted to fly crew this year anyway)
- Boeing highlights things that went well. Electricity consumption/demand had more contingency than expected, the heat shield was stressed less than planned, thrusters were able to operate beyond their planned performance (which turned out to be necessary to put the capsule into a good orbit...), ...
- Boeing will perform a launch-to-docking and an undocking-to-landing software test
- The deorbit software bug was not caught because the simulation assumed a wrong thruster configuration (and presumably the code used by the capsule did so as well). Legacy software also made the Ariane 5 maiden flight fail.
- NASA will give an update on March 6. However:

Boeing officials cautioned that NASA might not be ready to make an announcement at that briefing as to whether a reflight of the Orbital Flight Test will be required.

 

[Dr Transport]

If NASA is smart about it, they'd take into account Boeing's entire history (recent) and make them do a hardware test to prove that the system actually works as opposed to taking their word for it. the culture in the space division is no different than in the commercial airliner division.

 

[mfb]

Update from Boeing.

The Boeing Company is honored to be a provider for the Commercial Crew mission. We are committed to the safety of the men and women who design, build and ultimately will fly on the Starliner just as we have on every crewed mission to space. We have chosen to refly our Orbital Flight Test to demonstrate the quality of the Starliner system. Flying another uncrewed flight will allow us to complete all flight test objectives and evaluate the performance of the second Starliner vehicle at no cost to the taxpayer. We will then proceed to the tremendous responsibility and privilege of flying astronauts to the International Space Station.

Quite sure Boeing "chose" to do that because NASA insisted on it.[/size]

 

[Dullard]

An Atlas is (tentatively) reserved to launch a Starliner at the end of July ('21).

 

[Dullard]

Trying again tomorrow evening.

 

[Dullard]

Just checked in with my guy in the firing room. All is Go (so far). Headed to the porch to watch the festivities.

 

[Dullard]

Booster worked. Now for the tricky part...

 

[mfb]

Its first burn was a success - that's where the previous mission had a problem. The following phasing burn was a success, too. It's now in an orbit similar to the ISS, approaching it slowly while Boeing and NASA run more tests.

 

[mfb]

Boeing's capsule encounters propulsion issues on way to International Space Station

Two of its 12 maneuvering thrusters failed. There is enough redundancy, the capsule is still on the way to the ISS using the other 10, but it's certainly not ideal. One thruster failure is bad luck, two is another systematic problem that was missed.

 

[Dullard]

It's going to take some guts to go ahead with docking with 2 thrusters out, the rest suspect, and an untested system. I'm guessing that there is some serious indigestion at NASA.

 

[berkeman]

It's going to take some guts to go ahead with docking with 2 thrusters out, the rest suspect, and an untested system. I'm guessing that there is some serious indigestion at NASA.

Can they use one of the ISS robotic arm systems to aid in the docking if needed? I don't know if they can reach near the docking hatch at all...

https://en.wikipedia.org/wiki/Mobile_Servicing_System

 

[Dullard]

No idea, however: To dock with the station, they'll have to move toward the station. They can be cute with their vector until they get close, but eventually they will be in a situation where the thrusters absolutely need to work properly (that's ON and OFF). One major problem with not knowing exactly what's wrong is that you also don't know what's right.

 

[Dullard]

Hmmm.

https://www.space.com/nasa-boeing-hail-starliner-launch-success-despite-glitch

"The first failed after only one second. Its backup immediately kicked on and was able to fire for another 25 seconds before it also failed."

"Nappi emphasized that the issue was not one that needed to be resolved prior to the completion of the OFT-2 mission."

 

[anorlunda]

The proof of the pudding will come when astronauts are asked to risk their lives in that vehicle.

 

[mfb]

Successful docking.

Here it is at the ISS

The approach path is designed to always miss the station if the capsule stops firing at any point in time, excluding the last meters when the approach is so slow that a collision should do larger damage (~5 cm/s for the last 7 meters, they dock with the ISS at that speed). Leaving the ISS is easier.

They'll have to find out what happened with the thrusters but that's likely something that can be done on the ground before the next crewed flight. The primary goal - reaching the ISS - has been achieved. After the flight Boeing will have to convince NASA that the risk of a loss of mission and the risk of a loss of crew are below their thresholds (1 in 270 for a loss of crew. Spaceflight is still dangerous).

 

[hutchphd]

They'll have to find out what happened with the thrusters but that's likely something that can be done on the ground before the next crewed flight.

Are the bad thrusters on the capsule or the service module? There are some on each, right?

 

[mfb]

Are the bad thrusters on the capsule or the service module? There are some on each, right?

The OMAC thrusters, responsible for orbit raising (where we saw two failures), are on the service module.
The RCS thrusters, responsible for docking (where we also saw two failures), seem to be partially on the capsule (12) and partially on the service module (28). Source for the numbers. I don't know where the two failed RCS thrusters are. That system has a massive redundancy so losing two isn't a big deal here.

A sublimator in the cooling system started later than planned
An issue with the NASA Docking System delayed docking by an hour.

It's still on track to have a successful mission, but it's more things Boeing and NASA will have to work on.

 

[hutchphd]

The service module does not survive re-entry. How will they troubleshoot this I wonder? You don't want to be doing functional tests while attached to ISS (the Crew Dragon Super Draco firing event comes to mind). Seems to me this analysis could be a nontrivial problem.

 

[anorlunda]

A sublimator in the cooling system started later than planned
An issue with the NASA Docking System delayed docking by an hour.

I had to search for the word sublimator. I was surprised to find it in a description of the Apollo Lunar Module.
https://history.nasa.gov/SP-368/s6ch4.htm

 

[Nik_2213]

Upside, much improved.
Downside, no Boeing executives seem willing to hitch a ride...

 

[hutchphd]

I had to search for the word sublimator. I was surprised to find it in a description of the Apollo Lunar Module.

Also used to cool the space EVA suits They are really clever devices. As I understand them they are essentially self regulating because ice accumulation slows the cooling (subliming) rate as the feedwater gets colder and vice-versa.

 

[mfb]

The service module does not survive re-entry. How will they troubleshoot this I wonder? You don't want to be doing functional tests while attached to ISS (the Crew Dragon Super Draco firing event comes to mind). Seems to me this analysis could be a nontrivial problem.

Maybe some tests after undocking. Now they had a few days to study it.

Undocking is planned 25 May 2022, 18:36 UTC, reentry is planned 25 May 2022, 22:49 UTC.
It is currently 24 May, 06:20 UTC, so undocking is in 1 day 12 hours.

 

[Dullard]

Speculation:
They may already know what's wrong. There is a dispute between Boeing and Aerojet-Rocketdyne (the RCS system supplier) that is on the verge of becoming a lawsuit. I found NASA's willingness to continue with the docking (After several apparently unexplained thruster failures) to be extremely un-NASA-like. If they semi-expected these problems (and understood the cause), that decision would make a lot more sense.

 

[hutchphd]

If they semi-expected these problems (and understood the cause), that decision would make a lot more sense.

Did they know this before launch? Then it makes no technical sense at all to me that the flight proceeded. These are the re-entry thrusters, right? Fiasco.

 

[Dullard]

As I said, I'm just speculating. It is a fact that Boeing and AJRD are about to go to the mattresses. Boeing has stated that the RCS system has design flaws (and blames AJRD). AJRD has a different story.

It seems possible that they took whatever steps that they could to 'mitigate' the known issues and decided to fly it - There almost certainly wasn't time to fix it (they just 'discovered' it during the last launch attempt). It's important to remember that egg on Boeing's face is also egg on NASA's face - Everyone needed a win.