When the Error Message Comes From 50 Million Miles Away

…spacecraft engineers have to get creative.

The Galileo spacecraft was already far from Earth when problems cropped up with its antenna. (NASA)



Sending a machine out into space requires taking many, often mysterious, risks, so preventable human error makes a failure that much more painful. In June 1998, Bernhard Fleck and his wife had just moved near NASA’s Goddard Space Flight Center in Maryland to work on the Solar and Heliospheric Observatory, or SOHO, which had already completed its primary two-year mission: studying the structure and changes of the sun. Fleck, a project scientist with the European Space Agency, which had joined NASA on the solar probe, was unpacking boxes when he got a call from his team. During a routine maneuver conducted every three months, SOHO lost its attitude lock on the sun. The loss triggered a safe mode called Emergency Sun Reacquisition, or ESR, which fires the spacecraft’s thrusters in an attempt to get the craft back facing the sun.

“At first I wasn’t worried,” Fleck says. “I knew it would be a long night, but SOHO had gone into ESR before. We were bringing it back, and had a second ESR. Then a third. After five minutes someone in the control room said, ‘Oh shit! We made a mistake!’ It was our fault.”

Recovering SOHO from safe mode would have gone smoothly the first time if it weren’t for two crucial errors. A pre-programmed command code made it so one of the gyroscopes, which was supposed to activate during ESR to get the spacecraft realigned with the sun, failed to turn on. A second pre-programmed code was sending the team incorrect readings from another gyroscope. To make matters worse, the team had rushed to get SOHO out of safe mode instead of analyzing the problem, during which time they might have found the command errors. With the thrusters firing but the gyroscopes uncontrollable, SOHO was left spinning wildly, unable to collect solar energy. Eventually, the spacecraft switched off, terminating the communication link. One can imagine a shrug and maybe a robotic wish that the humans would get it together.

“The feeling was very grim,” Fleck says, “But the team did not give up.” The first “long night” turned into long months searching the skies to find the spacecraft, and Fleck updated a website every day on the recovery process. Then came the update on August 4: “The search was finally successful!” Using a combination of the Arecibo Radio Telescope in Puerto Rico and NASA’s Deep Space Network, the team found SOHO floating out in space and reestablished communications.

It was a few more months before the team could return SOHO to full operations, slowly charging its solar batteries and defrosting its hydrazine fuel. At any time, the tank or pipes could have ruptured. As the spacecraft emerged from its icy state, the team was thrilled to find that all 12 science instruments were still operational. The gyroscopes, however, could not be salvaged, so Fleck’s team improvised a unique stabilizing system using the spacecraft’s reaction wheels, a type of flywheel.

Seventeen years after launch, SOHO continues to make significant contributions to our understanding of the sun. “It was a blunder with a very lucky outcome,” Fleck says. “Though it was a nightmare, I learned you do everything you can, and you can do what you think is impossible.”

Comment on this Story

comments powered by Disqus