Scrambling to avoid a repeat of a systems meltdown last month that snarled travel for tens of thousands of international passengers at LAX, U.S. customs officials have fast-tracked an overhaul of their operations here and around the nation.
Los Angeles International Airport officials say they are encouraged by the response of customs officials, who were put in a hot seat after their widely publicized system failure Aug. 11.
About $15.3 million has been allocated to refurbish technical systems at the country's 104 major airports and international border crossings within the next six to nine months, bringing the upgrade project to a close about nine months ahead of schedule, said Ken Ritchhart, assistant commissioner in the Office of Information and Technology with U.S. Customs and Border Protection.
"Everything has been moved up," Ritchhart said. "We recognize the fact that having folks sit on airplanes for hours is not acceptable, so we have to look at new procedures . . . to make sure it doesn't happen again."
The outage at LAX that led to the speedup of the nationwide overhaul was the result of a single malfunctioning computer that prevented customs from screening international passengers for security risks.
Officials had no suitable backup system, and more than 17,000 arriving passengers were stranded in planes for hours while about 16,000 departing passengers waited at their gates for updates from carriers with little information.
By Thanksgiving, or Christmas at the latest, the entire customs system at LAX will be redone, with not only new workstations, network switches, routers and cables, but also a snazzy new satellite backup system that will allow screeners to access network databases should local routers fail, Ritchhart said.
Airport officials said they have seen signs that the outage -- a huge embarrassment for customs officials and a source of outrage from passengers -- has touched a nerve at the agency's highest levels.
"The tangible steps that have been taken, they're encouraging, and, frankly, they're impressive," said Paul Haney, deputy executive director for airports and security for Los Angeles World Airports, the agency that operates LAX. "We actually have tangible evidence that this is being taken very seriously and there's a commitment to ensure that there is never a repeat of this nature."
Customs officials traced the source of the LAX system outage to a malfunctioning network interface card on a desktop computer in the Tom Bradley International Terminal.
The card, which allows computers to connect to a local area network, experienced a partial failure that started about 12:50 p.m., slowing down the system, said Jennifer Connors, an official in the office of field operations for the customs agency.
As data overloaded the system, a domino effect occurred with other computer network cards, eventually causing a total system failure a little after 2 p.m., Connors said. The system was restored about nine hours later, only to give out again late the next day, a Sunday, for about 80 minutes, until about 1:15 a.m. that Monday. The second outage was caused by a power supply failure, Connors said.
During the Saturday outage, customs officials implemented their slower and more limited backup system, which included setting up laptops at workstations and using the laptops to search through copies of files from the downed network on a CD-ROM and scan lists of names.
Last week, airport officials received 100 laptops, weeks earlier than promised. In addition, they are being allocated 30 more customs officials to help process passengers more quickly.
Customs officials said that, instead of just one information technology supervisor and one technician on duty as was the case Aug. 11, the agency now has seven such personnel on staff at LAX. They also plan to hire at least three more systems technicians and will explore other ideas, such as having another airport handle some of LAX's screening load.
"Maybe Atlanta's not too busy and can handle these five flights, and then phone it in, to redistribute the workload," Ritchhart said.
In addition to the new satellite site, Ritchhart said customs is also looking at a more long-term idea of hiring additional Internet service providers to ensure more redundancy in the backup system.
Ritchhart said the LAX workstations were about 4 years old, with networking switches and routers 6 years old and cables about 20 years old. Nearly all had reached the end of their life expectancy, he said.
Local airport agency officials also have been dealing with the results of the outage that left them fielding dozens of angry phone calls and e-mails from passengers seeking reimbursement for missed cruises or hotel rooms.
Officials recently held a three-hour meeting to brush up on emergency procedures, including notifying people via the airport agency's website (which was not done during the Aug. 11 incident).