The Ground
Astronauts get the glory, but flight directors run the show.
/https://tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/mission_control_388-nov06.jpg)
Michael Moses Remembers feeling giddy that day in February 2005 as he walked into chief flight director Milt Heflin’s office at NASA’s Johnson Space Flight Center in Houston to accept his new job. Among space engineers, becoming a flight director is a crowning career achievement, and Moses half-expected Heflin, known as Uncle Milty, to give a round of high-fives to the nine newly selected directors gathered in the room. But Heflin’s words were sober. “We got an hour-long lecture that this is dangerous business, that we are on the pointy end of the sword, and that if we screw up, somebody dies,” Moses recalls.
Not exactly welcoming, the lecture at least had an impact. “That night I hardly slept,” says Richard Jones, who like Moses had worked for years in mission control before being promoted to flight director. Another new flight director, Holly Ridings, whose previous job in mission control had been monitoring the attitude of the International Space Station (ISS) in orbit, says that now, “every time I sit down in the flight director chair, there is a little piece of my mind that thinks, ‘If things go really wrong today, the U.S. space program could be over—or at least grounded for a very long time.’ ”
Months later, when I asked Heflin why the shock treatment was necessary when all the new recruits were already battle-hardened veterans, he answered, “Humility was something they needed to pay attention to right away. They could not leave pounding their chests.”
For one thing, he says, “there were a number of people who weren’t in the room [who] were just as qualified to be sitting there with them.” And the new directors will still rely heavily on those people’s judgment. The flight director sees the big picture during a spaceflight, but it’s the individual flight controllers sitting at 19 consoles—from the propulsion engineer to the booster systems engineer to the flight surgeon—who supply the critical details. Moses, Jones, Ridings, and all but one of the other new flight directors had been reared as NASA flight controllers, in fact. Last year, a spate of retirements and promotions in mission control freed up nine coveted slots for flight director, bringing the total number to 30. It was the second largest collective hire since the job was created in the early days of the space program.
Last December I spent a day at JSC with the new recruits as they went through their training exercises. By 7:30 a.m., Dempsey is already “on console” in the Flight Control Room (FCR, or “ficker”) for the space station. He hands me a set of headphones so I can listen to the comm loop, an open channel between mission control and the orbiting astronauts.
The station ficker is called the Blue Room. There’s also the Red Room, used for ISS training, and the White Room, the shuttle ficker, the room Hollywood usually depicts in films. Today’s task in the White Room is a simulated docking between the shuttle and the station, with new flight director Michael Sarafin in charge. Before his promotion Sarafin worked in mission control for 10 years as a guidance, navigation, and control officer. He was on duty during the 2003 Columbia accident, but doesn’t seem eager to discuss it, at least not with a journalist. “It was a hard day to lose our friends” is all he says.
In today’s scenario, the shuttle (a simulator in another room) will perform a two-minute burn of its twin Orbital Maneuvering System engines. The aim is to boost the vehicle into a higher orbit to rendezvous with the station. Two floors above, the simulation supervisor, or “simsup,” and his cohorts will try to confound the flight controllers with any number of curve balls. “They might cause a fire, cabin leak, loss of communications, loss of critical flight control systems, or loss of jet thrusters,” says Sarafin. “They are pulling the puppet strings behind the scenes. Our job is to react to what they do.”
The engines fire and mission control falls silent. The exact positions of the shuttle and ISS are projected as a high-resolution graphic on a 10- by 12-foot overhead screen. Another large display plots the orbit of the shuttle and station onto a world map. A third screen relays telemetry data, caution messages, and emergency warnings from the shuttle and ISS to mission control. It’s only a simulation, but I’m transfixed by the realism, knowing that all hell is about to break loose. Suddenly, a yellow warning code indicates three separate temperature spikes in one of the avionics bays that house the shuttle’s flight control and computer systems. The EECOM (emergency, environmental, and consumable operations manager), pronounced "ee-com," calmly flips through her mission rulebook (“Our bible,” says Sarafin) to find the right protocol and determines—with the help of other flight controllers—that the spikes are caused by a small blockage in the water-cooling loop. Switching to a redundant cooling loop returns the temperatures to normal.
It’s this particular EECOM’s first simulation in the ficker front room, a significant milestone for a flight controller. She’s already endured hundreds of similar exercises in the “back room,” where junior flight controllers work in specialist teams, training on each system in year-long rotations like med school students alternating between the psych ward, gastroenterology, and pediatrics. So goes the arc of a career in mission control—back room to front room to flight director.
Shuttle simulations might last three or four days, and to get certified to work an actual launch and landing, a flight director may have completed hundreds of simulations before doing it for real. “They give you all these failures—to the point where it is almost unrealistic—to see how we’ll react,” says Ridings.
Still, even after hundreds of hours of practicing far-fetched disaster scenarios, real missions occasionally serve up surprises. One frequently cited by flight directors is the March 1992 STS-49 mission to rescue an Intelsat communications satellite stranded in an unusable orbit. The shuttle crew’s task was to attach a new rocket motor to the satellite that would boost it to the proper altitude. The original plan had been for a spacewalking astronaut at the end of the shuttle’s robot arm to snare the satellite with a special capture bar and bring it into the shuttle cargo bay. When that failed several times, the astronauts themselves proposed a workaround: Send out three spacewalkers, which had never been tried before, to grab the two-ton beast by hand and coax it into the bay.
It worked. But after the astronauts attached a booster rocket to the satellite and reentered the shuttle, another glitch occurred. When the crew flipped a pair of switches to activate a spring that would eject the satellite from the shuttle, nothing happened. “Now we’re sitting there with a satellite in our payload bay, with a rocket motor that might be getting ready to fire off,” recalls Phil Engelauf, who was a flight director for the mission.
Stumped by a potentially dangerous situation they’d never trained for, the ground controllers set to work. Engelauf remembers Jeff Hanley, then a mission control payload officer responsible for the vehicle’s cargo, poring over the shuttle’s wiring diagrams. “He was sitting in front of me with these long, fold-out drawings tracing through the entire system,” says Engelauf. Hanley had a hunch: The arming and firing circuits could have been wired backward. “Instead of arming the A circuit and firing the B circuit, Hanley wanted to arm the B circuit and fire the A circuit. So we read the instructions up to the crew and I remember you could cut the tension and suspense with a knife as we counted down…three…two…one…fire…and sure enough, the satellite left the payload bay.”
Intuition like Hanley’s amounts to a sixth sense, says Gene Kranz, the legendary NASA flight director who handled mission control’s most famous “save” after an oxygen tank on the Apollo 13 spacecraft exploded on the way to the moon in 1970. “There’s a gut feeling, an almost intuitive response to things that are happening around you,” he says. “It’s like chess,” says former astronaut and NASA head of spaceflight Bill Readdy. “You have to always think several moves ahead.”
So it was with an emergency on the space station on Super Bowl Sunday, February 3, 2002. When a software glitch took down the station’s Russian-run computers shortly before midnight, there wasn’t much Bryan Lunney, the flight director on duty, could do but watch and wait. The Russian computers fed data to the station’s gyroscopes, which were essential for holding the station’s position stable. Two hours after the first computer failed, the backup system also crashed. “We’d seen similar things in the past,” says Lunney, whose father was the Apollo-era flight director Glynn Lunney. “It’s normally no big deal. But when the second computer failed, we started thinking about what to do if the third one failed too.”
Sure enough, the third computer also shut down. With data no longer being supplied by its gyroscopes, the ISS began a very slow tumble. Consequently, the huge solar arrays were no longer pointing at the sun. And without a steady supply of solar-derived electricity, the controllers knew things would get dark and cold pretty fast. “If the station runs out of power, that’s bad,” says Lunney. “There is no good way to jumpstart it.”
Not sure how long the computers would be down, and never having trained for a triple-computer failure, Lunney turned for help to his PHALCON (power generation, storage, and power distribution) flight controller, who offered a simple, almost primitive solution: Have the crew look out the window, find the sun, then manually rotate the arrays toward the light. “He came up with this procedure on the fly,” says Lunney. “It had never been done before.”
The PHALCON controller devised a crude table that divided the station into four viewing areas—deck, forward, overhead, and aft—with corresponding angles for each section. For example, if the sun was visible through the aft portal, the crew should tilt one solar array to 90 degrees and the other to 270. Because radio communications on the drifting station were also intermittent, mission control would have less than five minutes to read the instructions up to the astronauts. “We practiced reading it aloud a couple of times to see how long it took,” remembers Lunney. “I was nervous and the adrenalin was certainly flowing.” But the solar arrays eked out enough juice to keep the station powered up until the Russians finally got their computers back online. Visible relief swept through mission control. Says Lunney, “I felt like a fireman who’d walked out of a burning house having just rescued the kids from the bedroom.”
Although the improvised procedure worked, space engineers hate to wing it when astronauts’ lives and billions of dollars of hardware are at stake. That’s why flight directors are among the most systematically and thoroughly trained professionals in any field. “They are Spartans, tough and competent, like the Navy SEALs,” says Ron Dittemore, the now-retired flight director who was in the public spotlight as NASA’s shuttle program manager at the time of the Columbia accident.
It was Gene Kranz who first scribbled “tough and competent” on a mission control chalkboard after another disaster—the fire that killed Apollo 1 astronauts Gus Grissom, Ed White, and Roger Chaffee during a ground test in 1967. Kranz demanded that flight directors write the words on the blackboards in their offices, never to be erased. That combat mentality—Kranz was a fighter pilot before joining NASA—engenders camaraderie and mutual loyalty that extend well beyond working hours. “We knew each other very personally as a family,” Kranz says of his time at the space agency. “We partied together. We had the largest party fund in all of the federal government. We would take over the Astrodome. It basically helped us maintain our bond.” Not much has changed. “Being a flight director is not a job but a lifestyle,” says Ridings. “You know everybody’s kids, wives, and husbands. People call you at two in the morning, or on vacation, it doesn’t matter. When you’re working a big mission, you get home after a shift and immediately turn on NASA TV. It’s addictive.”
It’s also demanding. The job is grueling mentally, and the nine-hour shifts go round the clock. Flight directors regularly have to adjust their sleep schedules. “Everyone has a really good set of blinds at home,” quips Dana Weigel, who was a flight controller for extravehicular activities (spacewalks) before she was promoted to flight director.
The key to success, Sarafin explains, is learning how to handle potentially debilitating stress. He describes a condition called “scope lock” that sets in when a frazzled flight director or controller gets too focused on the minutiae of a particular problem “and forgets that there is a spacecraft flying up there.” They lose sight of the big picture, hypnotized by the dizzying stream of data spewing from their consoles. “Others start to ramble when the pressure is on,” adds Sarafin. “Or they might just stop talking entirely.”
Burnout is a danger in most high-stress professions. For flight directors, it’s endemic. The attrition rate for all flight controllers is in excess of 20 percent, says Sarafin. That’s why, according to Heflin, the flight director culture has had to become more “touchy feely” than it was in the Apollo days—being driven is okay; being obsessed is not. “When I was chief of the flight director office [he was recently promoted to deputy director of NASA’s mission operations directorate], I made a point of keeping track of how many hours they were working,” says Heflin. “If I needed to, I’d tell them to slow down, to go home, and find a way when they get in their cars at the end of the day to leave the job behind.” Not everyone makes it through unscathed. “The divorce rate within the flight director community is fairly high,” Sarafin says.
For the Class of 2005, the job of flight director is about to change radically. Fewer than 20 shuttle missions remain before NASA’s 25-year-old workhorse is retired. The agency has already begun to shift its focus from flying in Earth orbit to returning astronauts to the moon by 2020 and, someday, heading on to Mars. Not since Kranz ran the Apollo missions have flight directors faced such daunting missions.
This time, when it comes to directing the astronauts, they aim to be less…well, controlling. The purpose of NASA’s new moon program is to stay for the long haul—to learn how to live on the lunar surface. That will mean rethinking the crew’s relationship with mission control. “Right now we schedule the crew’s time in five-minute chunks,” says flight director Ginger Kerrick. “We know that’s going to have to change.”
Dempsey has been involved in planning an exercise for an upcoming space station mission “where we say to the crew, ‘Okay, here are the key things to get done, here are your time constraints, now you plan it yourselves.’ ” The idea is to give the astronauts the freedom they’ll need for lunar missions lasting months or even years.
An important part of this approach will be onboard autonomy, software systems that enable the crew to resolve a crisis when lengthy communications delays—in the case of Mars, up to 22 minutes, depending on the positions of the planets—make it impossible to talk to the ground in real time. “Instant problem resolution will have to fall on the astronauts’ shoulders, not ours,” says Dempsey. “When there’s an onboard fault, it can’t just be a light that turns on. Today, when that light comes on, the guy sitting next to me tells the crew what page of what book to turn to. In the future, the software systems will have to do that for them.”
Both Dempsey and Dittemore, who retired from NASA shortly after the Columbia accident, make the inevitable comparison to HAL 9000, the omnipotent, omnipresent computer in 2001: A Space Odyssey. “Of course the system wouldn’t be psychotic,” assures Dempsey, “or necessarily super-intelligent, or even completely autonomous. But smarter than the software we have now.” Dittemore says that some small steps have been taken in this direction, such as the Integrated Vehicle Health Monitoring system, flown on shuttle mission
STS-95 in October 1998. IVHM uses a network of pressure, temperature, strain, and other sensors to monitor the vehicle’s condition without aid from mission control, and even to do some limited troubleshooting.
The nine new flight directors are helping to plan the next generation of NASA space vehicles, even while their main responsibilities still lie with the shuttle and space station. I accompany Weigel and another new director, Brian Smith, to a meeting with the deputy project manager for the Crew Exploration Vehicle, or CEV, the upgraded Apollo-style capsule that will ferry astronauts to the moon and back beginning in the next decade.
We gather around a circular table inside JSC’s Building 9, a massive hangar that houses, among other things, a life-size training mockup of various space station modules. When we arrive, instructors are conducting a simulation of a cabin fire, complete with billowing white smoke and blaring alarms. Directly behind us is a metal skeleton of the CEV, the framework for a full-scale model. For now, it’s empty except for a pair of fuzzy dice dangling from its ceiling.
The purpose of the meeting is to solicit from the flight directors specific suggestions for the spacecraft’s interior layout, software systems, and other design features. Being in on these early decisions should give flight directors a tremendous advantage downstream, says Kranz, and will make them “far better at coping with the complexities of the new exploration program.” He points to the colorful console displays that allow flight directors and flight controllers to keep tabs on every system in the spacecraft, such as navigation, guidance, communications, payload, life support, software, and extravehicular activity. Instead of strings of numbers, the modern CEV displays have easy-to-interpret graphics. “It gives flight directors a three-dimensional visualization of what’s going on,” says Kranz. “With Apollo 13, it took us almost 25 minutes to figure out we had an explosion. Whereas in today’s world, they would have known it instantly.”
But improved or not, these are still just tools. For the foreseeable future, flight directors will continue to rely on their own intuitions and instincts more than on technological wizardry. “It’s the human element that allows them to be successful,” says Readdy. The danger is in losing sight of that, of relying too much on technology. During the 1990s, when the shuttle was flying several times a year, he says, “there was an attempt to depersonalize the flight director’s job. It wasn’t about the people; it was about the science and the hardware.” It all got “antiseptic and routine,” he observes.
Even as they drill endlessly, routine is something the flight directors try to avoid. When it comes to spaceflight, a sense of routine can lead to trouble. “I’m not going to sugarcoat it,” says Jones. “Ronald Reagan deemed the shuttle operational after the third or fourth flight. ‘Operational’ means we have nothing else to learn about the vehicle. That set a tone. Apathy set in. And NASA got in that mode that leads to accidents.”
Being a flight director, says Kranz, is “about staying on the edge,” always mindful that the U.S. space program could come to a halt because of something that happens on your next shift. No wonder, then, that Milt Heflin’s “Welcome to your new job” speech was so somber.