ERAM continues to undergo critical failures

The $2.4 billion En Route Automation Modernization air traffic control modernization software project has continued to experience critical system failures--so much so that the Federal Aviation Administration has made moves to extend the life of a backup system originally meant for only temporary deployment.

In a Transportation Department office of inspector general report (.pdf) dated Sept. 13, auditors say the FAA has made a "recent discovery of a previously unidentified serious hidden software defect" in ERAM that results in brief loss of air control.

ERAM replaces a four-decade-old long-range radar tracking system known as HOST; it is slated for deployment to all 20 U.S. air route traffic control centers in 2014--a schedule delay of 4 years. The report says the FAA has "paused" with further deployment of the system. "A prudent decision," auditors say.

The defect stems from the fact that ERAM has two identical and interlinked data paths for processing flight and radar data and displaying it on air traffic control displays. Each path provides full functionality, auditors say, but one is meant as a primary and the other as a backup. But, one channel has sometimes interfered with the other, or both have simultaneously failed, the audit says.

As part of the transition from HOST to ERAM, the FAA has used another system it developed as part of the ERAM contract, Enhanced Backup Surveillance that itself replaces the old HOST backup system, Direct Access Radar Channel. The FAA has planned to remove EBUS and leave ERAM without an external backup (i.e., with backup consisting only of the "backup channel" within ERAM).

Now, the ERAM program office has made modifications to EBUS necessary to retain it after HOST is decommissioned, auditors say. It's a decision auditors again approve of, "given ERAM's history of repeated system failures."

Currently, the Salt Lake City, Seattle, Denver, Minneapolis and Albuquerque centers use ERAM full time to control traffic, while the Chicago, Oakland, Los Angeles, and Houston centers use it under limited conditions.

The system's persistent problem raises questions whether the design of functions related to tracking aircraft and displaying information are flawed, auditors say. A study commissioned by the FAA identified potential tracker issues, including ones that could impact integration of signals from the Automatic Dependent Surveillance-Broadcast system. When fully implemented, ADS-B will transmit the Global Positioning System-derived real-time position of most national airspace aircraft.

Auditors also cite other ERAM problems, including incorrect aircraft identification tags that cause air traffic controllers to manually re-key the data block on their display. Doing so increases risk of data entry errors--and "most importantly, takes the controller's focus away from the primary task of managing and separating aircraft," auditors note.

They also criticize FAA program management, stating that the agency and prime contractor Lockheed Martin have continued to add new capabilities while simultaneously fixing core functionality issues. That overlap of software development and deployment increases the likelihood of introducing problems, or of re-introducing previous problems, they say.

A key milestone the FAA has used to measure ERAM progress is misleading, auditors add. The milestone dubbed Initial Operating Capability only means that the system is ready for "very limited control of live air traffic," and must still overcome very site-specific problems before it can actually continuously handle traffic. Use of the IOC metric gave FAA officials "a false sense of confidence in the maturity" of ERAM.

In fact, FAA and Lockheed Martin underestimated from the start the difficulty of fielding ERAM in the field and assumed they would make a one-to-one replacement of center HOST systems with ERAM, ignoring the fact that centers have accumulated changes over HOST's 40-year operational lifetime, auditors say.

The agency dismissed or ignored early warning signs of trouble, and bad news wasn't transmitted up the reporting chain to senior FAA management. Site personnel, in turn, told auditors they felt pressure from the program office to stay on schedule, and "were uncertain of the program office's commitments to fix discovered problems."

For more:
- download the report, AV-2012-179 (.pdf)

Related Articles:
Porcari: FAA could have better managed ERAM
FAA DataComm delayed by 2 years
Incomplete FAA consolidation plan could have NextGen budget impact, say auditors