NOTAM system down

[PeterNSteinmetz]

Looks like GA aircraft are still flying around and flights resuming. Anyone have an idea what went wrong?

Air travel across US thrown into chaos after computer outage

NEW YORK (AP) — The world’s largest aircraft fleet was grounded for hours by a cascading outage in a government system that delayed or cancelled thousands of flights across the U.S. on Wednesday. The White House initially said that there was no evidence of a cyberattack behind the outage that...

apnews.com

 

[PeterNSteinmetz]

Lack of an administrator a contributing factor?

FAA has operated without a Senate-confirmed director for nearly a year

The Federal Aviation Administration, the agency to blame for thousands of flight delays Wednesday, has operated without a Senate-confirmed director for nearly a year because Biden's nominee was never confirmed.

www.foxnews.com

 

[Bob Noel]

Lack of an administrator a contributing factor?

no.

Who would have been able to do anything about the NOTAM infrastructure mess?

Of course, the clown's administration hasn't shown any ability to nominate capable people... so that didn't help.

 

[Jim Logajan]

Possibly unrelated story: A few years ago when I was with https://www.iwl.com/ I did a demo of our KMAX network emulator to a group of FAA guys in their communications group. They had just started working on switching some of their internal data communications to an IP network. To test how their systems handled various network problems they needed to be able emulate national network conditions in a lab. (They did buy one of our products.)

 

[PeterNSteinmetz]

Jim would likely know more, but I assume that with mission critical databases there are techniques such as fault checking and fall backs which could be used to prevent this sort of thing?

Damaged Database Likely Cause for NOTAM Outage, FAA Says - FLYING Magazine

The FAA said a damaged database file—and not a cyberattack— was the root cause of the NOTAM system crash that ground air traffic to a halt.

www.flyingmag.com

 

[PeterNSteinmetz]

File replacement flub ?

Report: Human Error and Software Meltdown Caused Bumbling FAA's Shutdown Fiasco

The FAA ordered a nationwide halt to domestic flights in the United States on Wednesday morning due to a major software meltdown. The error that led to this failure reportedly was caused by an engineer replacing one file with another while working on the system. One FAA official stated, "It was...

www.breitbart.com

 

[Jim Logajan]

Jim would likely know more, but I assume that with mission critical databases there are techniques such as fault checking and fall backs which could be used to prevent this sort of thing?

Damaged Database Likely Cause for NOTAM Outage, FAA Says - FLYING Magazine

The FAA said a damaged database file—and not a cyberattack— was the root cause of the NOTAM system crash that ground air traffic to a halt.

www.flyingmag.com

I'm not privy to FAA systems but they likely do create periodic backups. But when something fails it takes some time to detect and diagnose the problem before deciding to restore the system from backups. And the restore process takes some time. Sounds like they restored things in about the amount of time I would expect.

Apparently the FAA is in the process of updating their NOTAM systems and currently has an API and plans to distribute services:
https://www.faa.gov/about/initiatives/notam

Lastly many mission critical systems in industry developed in the last twenty years or so are designed with mirroring or similar distributed services so a failure in the active system can be switched to the backup system in seconds or even seamlessly.

 

[RandomDroneGuy]

no.

Who would have been able to do anything about the NOTAM infrastructure mess?

Of course, the clown's administration hasn't shown any ability to nominate capable people... so that didn't help.

Please don't insult clowns that way. Thanks!