The Story behind the Apollo 11 and the 1202 Computer Error Code

[jedishrfu]

TL;DR Summary

Perhaps the most dramatic moment of Apollo 11's mission to the moon was when the Eagle began its final descent to the lunar surface and the Apollo Guidance Computer became overloaded. Few were more nervous than the young computer programmer who had written the code for the landing. On the Apollo 11's 50th anniversary, WSJ sat down with programmer Don Eyles.

 

[hutchphd]

Summary: Perhaps the most dramatic moment of Apollo 11's mission to the moon was when the Eagle began its final descent to the lunar surface and the Apollo Guidance Computer became overloaded. Few were more nervous than the young computer programmer who had written the code for the landing. On the Apollo 11's 50th anniversary, WSJ sat down with programmer Don Eyles.

Interesting film clip... the history of that computer is rich with the best of human endeavor.
I take exception with your use of the term "failure" in the title. The error handling of the operating system was a resounding success!

 

[jedishrfu]

Thanks for the correction, the title is now fixed.

 

[hutchphd]

I think this is a fabulously interesting. We have been feedingf off it for half a century:

 

[anorlunda]

The story I heard (sorry, no link) was that the most interesting part was not the 1202 error itself, but the architecture of the on board systems.

Because of limited storage, the code for the onboard computer was uploaded by ground link for each execution. Therefore, even if a program crashed, it would be immediately replaced by the next program anyhow. The same program may be uploaded and executed again later. Remember that they said the errors were momentary.

By analogy, that was similar to a CTRL-ALT-DEL reboot of the machine after each program execution. Several different programs would run interleaved. That kind of robustness may have been an unintended consequence of the architecture.

Of course if the algorithms could not use the state or calculated results from prior executions, that would be limiting on the scope of the functionality. But those restrictions may have contributed to reliability, even if unintentionally. Call it KISS by necessity.

 

[hutchphd]

Truth be told I love to look at the hardware in the MIT clip I posted (particularly at the end of it) and consider the rate at which this seed grew over my lifetime. Almost makes you believe in the power of science...
I thought the programs were what was on the "rope" memory...fast but hell to change.

 

[jedishrfu]

I remember learning about these machines and thought it would be cool to have one but the price was prohibitive and a few years later the 8080 appeared on the market and the PC revolution took off. Personally I got a MITS 6800 (still works at least it lights up) that I programmed from the switch panel but after a bit of a struggle decided it wasn't worth the effort (televideo tubes were more expensive that the micro too) and got a TRS-80 from Radio Shack which was absolutely awesome.

 

[Dr. Courtney]

Great stuff. Thanks for sharing.

 

[Greg Bernhardt]

Unbelievable courage, continuing the mission after getting those errors and not really knowing what they meant! I'd need a new space suit!

 

[anorlunda]

Unbelievable courage, continuing the mission after getting those errors and not really knowing what they meant! I'd need a new space suit!

That might not be the whole story. Their ability to abort at that point may have seemed more risky than continuing. Remember that they admitted later that they did not have much faith in the ascent engine. Aborting would have meant maintaining control of the descent plus starting the ascent engine at the same time.

 

[neilparker62]

- thanks for the memories. 7 years old living in Nairobi at the time and very proud also to be "Neil" !
- a bit worrying to be the programmer sitting in Houston control centre with Neil Armstrong on moon descent asking for clarity on error code "twelve-two". No time to de-bug!
- to any Apollo crew and/or support staff out there congratulations even now. You guys are legends!

 

[jedishrfu]

Imagine also the lack of search capability in these programs:
- no visual editor
- no internet to look up the answer

just the symbol index in the program listing if you chose to generate one and hopefully the programmer used a symbol like ERR1202.

 

[anorlunda]

just the symbol index in the program listing if you chose to generate one and hopefully the programmer used a symbol like ERR1202.

It is hard for us today to imagine just how constraining the memory size was in those days. When I started in 66, the cost was about $1 per bit. (I worked at GE's Apollo Support Department in Daytona 1966-68) 1kB was an engineer's annual salary and more than one year's salary for a programmer. In a space module, cost was unimportant, but size and mass were still severe constraints.

When code is not modular, when there were no libraries or OS or compilers or assemblers available, just your program and a naked machine, situations like the one below were very common.
if X=A then CASE=1 elseif X=B then CASE=2 else ERR=1202 'because there should only be 2 cases

How the ERR=1202 got communicated to the pilot and to the ground is a very complex subject. The simplest way for a single error would be to light a lamp. But if there were thousands of possible errors, some kind of numeric indicator would be better than thousands of lamps, but the indicator could only indicate one error at a time. Lots of trade offs. An easy out was simply to not check for abnormal conditions, but that extracts a robustness penalty.

In more modern languages we might say
assert X=A or X=B
and we might even be able to specify compile-time assertions and/or run-time assertions. There would be a library to handle assertion errors. But this takes more code more memory, and more time. So we didn't do stuff like that back then.

 

[jedishrfu]

Kind of reminds you of car technology today an indicator light or that mystery error code and the need for a device to plug into a data port to figure out what is really going on.

 

[neilparker62]

This is ground control to Neil Armstrong
Error 12 02 and there's nothing I can do
This is ground control to Neil Armstrong



With apologies to Dave Bowie

 

[mfb]

That might not be the whole story. Their ability to abort at that point may have seemed more risky than continuing. Remember that they admitted later that they did not have much faith in the ascent engine. Aborting would have meant maintaining control of the descent plus starting the ascent engine at the same time.

Aborting the descent is what Apollo 10 did on purpose - as dress rehearsal for the landing, because that was considered to be safer than actually landing.

 

[hutchphd]

Aborting the descent is what Apollo 10 did on purpose - as dress rehearsal for the landing, because that was considered to be safer than actually landing.

That is not really accurate. Apollo 10 LM was happily in lunar orbit (albeit a low one)) the whole time. At no time did the perform a true abort procedure although they did separate the lower stage and briefly fired the ascent engine to test it. I assume they did not perform an "abort" procedure because it was, in fact, too dangerous!

 

[mfb]

They didn't get that close to the Moon but they separated the descent stage in flight and fired the ascent engine - just as Apollo 11 would have done for an abort. With the difference that Apollo 11 would have needed the ascent engine to work quickly and fire for a while, it would have crashed into the Moon otherwise.

 

[hutchphd]

Yes. There was also an issue if they had much residual downward/backward velocity when they fired the ascent stage... not much margin of extra fuel to make that up. Superbly competent humans

 

[jedishrfu]

That or they had a lot of free time to play the lunar lander game beforehand. :-)

https://en.m.wikipedia.org/wiki/Lunar_Lander_(1979_video_game)
Of course, they’ed have to have traveled into the future a few years.

 

[anorlunda]

They didn't get that close to the Moon but they separated the descent stage in flight and fired the ascent engine - just as Apollo 11 would have done for an abort. With the difference that Apollo 11 would have needed the ascent engine to work quickly and fire for a while, it would have crashed into the Moon otherwise.

Not just as. Apollo 11 would either had to fire both descent and ascent engines simultaneously, or shut down the descent engine then start the ascent engine. If they lost attitude control during that maneuver, the ascent engine could have propelled them downward. With the pilot requiring intense focus to control just the descent, the abort maneuver may have been seen by the pilot as more risky than continuing.

There was a special system Abort Guidance System dedicated to the abort maneuver.

https://en.wikipedia.org/wiki/Apollo_Abort_Guidance_System#Use_of_AGS

Use of AGS[edit]
There are few actual descriptions of the use of the AGS, as a landing abort was never needed during the Apollo missions. There were, however, four cases in which the AGS was used.

Its first use was for testing of the Lunar Module descent stage in Earth orbital flight during the Apollo 9 mission[6]. It was used again in the Apollo 10 mission, following separation of the Lunar Module descent stage prior to the APS burn. An incorrect switch setting[7] leaving AGS in Auto rather than Attitude Hold mode led to a prompt and pronounced deviation in attitude moments before staging[8]. The situation was quickly brought under control.

The next use of the AGS was during the lunar ascent phase of the Apollo 11 mission, when the LM crew performed a sequence of rendezvous maneuvers that resulted in gimbal lock; the AGS was subsequently used to acquire attitude control.[2]

The AGS played an important role in the safe return of Apollo 13 after an oxygen tank explosion left the Service Module crippled and forced the astronauts to use the Lunar Module as a "lifeboat." Supplies of electrical power and water on the LM were limited and the Primary Guidance and Navigation System used too much water for cooling. As a result, after a major LM descent engine burn 2 hours past its closest approach to the Moon to shorten the trip home, the AGS was used for most of the return, including two mid-course corrections.

Edit: By the way, that AGS Wikipedia article list the complete instruction set (27 instructions) of the AGS computer. Never saw that before in a wiki.

 

[Aufbauwerk 2045]

1202 alarm shortly after 10:00.

There is one for Apollo 12 also. It's a very enjoyable video.

 

[Aufbauwerk 2045]

Unbelievable courage, continuing the mission after getting those errors and not really knowing what they meant! I'd need a new space suit!

They asked about the error codes but accepted the answer that came back from their colleagues. In those days the astronauts were working with people they knew and trusted. Before that they had been in many dangerous situations in the military as fighter pilots or test pilots. It was an environment in which men relied on each other and trusted each other with their lives. They certainly had great courage, but it was based on knowledge of the men they were working with. They also knew the system at NASA and had confidence in it, even after the Apollo 1 disaster. I can't help wondering if people today can still have the same level of confidence in any system.

 

[Aufbauwerk 2045]

Summary: Perhaps the most dramatic moment of Apollo 11's mission to the moon was when the Eagle began its final descent to the lunar surface and the Apollo Guidance Computer became overloaded. Few were more nervous than the young computer programmer who had written the code for the landing. On the Apollo 11's 50th anniversary, WSJ sat down with programmer Don Eyles.

Fascinating to actually hear from the programmer involved. Good for him that it was not his fault. But also good that others made the go/no go decisions, and not one of us nervous programming types.

I think he is right regarding the skill level needed to get anything done with such limited hardware. Also their development environment was rather primitive by today's standards. But they also had intelligent and hard working people like Mr. Eyles. Also they had a great organization.

I'm intrigued by a couple of facts. One is that they hired someone who had never programmed, and he ended up programming the flight computer for the first lunar landing mission. The other is that he was not thrown under the bus so people higher up would not look bad. Kudos to NASA.

There have been many stories of software malfunctions causing major problems and even loss of life. But sometimes the problem is not the programmers, but the system, including communication between various departments or even within the same department. For example, in 1962 the Mariner I Venus probe was lost when the Atlas rocket veered off course and had to be destroyed. The reason was that the original specification used an average velocity, which was indicated by R with a bar over it. But the programmer was given handwritten equations to follow, and in that handwritten document that bar was left out. The programmer coded according to the handwritten equations, using instantaneous velocity instead of average velocity. The computer tried to correct from velocity fluctuations but this resulted instead in loss of control. (There are alternative explanations. This one is from Expert C Programming by Van Der Linden, p. 61. According to Wikipedia, the missing bar over R is "the most detailed and consistent account". Not that I rely on Wikipedia as the ultimate authority!)

Of course even today with our much more advanced technology we have occasional problems which I'm sure many people here will remember from recent news stories. We have the problem of managing ever more complex human and technological systems. Clearly we have not solved that problem.