Two-hour 9-1-1 Disruption in Massachusetts’s City
Thomas Caywood; Shaun Sutner, Worcester Telegram & Gazette (Mass.)
Imagine you’re having chest pains. Or think you hear an intruder downstairs late at night. Maybe you see smoke wafting from a neighbor’s house.
What if you called 9-1-1 but couldn’t get through?
It’s not supposed to be possible.
But that’s what happened to between three and eight emergency calls earlier this month during a locally unprecedented two-hour disruption of the city’s 9-1-1 system caused by a Verizon equipment failure downtown, a Telegram & Gazette investigation found.
“Was someone being abused or assaulted in a domestic incident, or was someone having a heart attack?” wondered David W. Clemons, Worcester’s emergency communications director. “To this day, we don’t know what they were about, and we may never know.”
The Verizon malfunction also knocked out dedicated telephone links between emergency medical service radio towers around Central Massachusetts and area hospitals. EMTs throughout the region could not reach hospital emergency rooms for about 15 minutes at the beginning of the disruption, according to Central Massachusetts Emergency Medical Systems Corp., the nonprofit agency that manages the system.
The state’s 9-1-1 network, where every call could be a matter of life and death, was supposed to be engineered with enough redundant systems, alternate answering points and backup fiber optic connections to ensure that, even in the midst of a major equipment breakdown, every call for help gets through.
But they didn’t.
The State 9-1-1 Department in Taunton is awaiting a final report from Verizon detailing what happened on Thursday, Jan. 17, at 3:45 p.m., when a piece of networking gear in the Verizon building on Chestnut Street malfunctioned.
For some period during the two-hour disruption, the malfunction cut off access to all eight dedicated connections, called “trunk lines,” that carry emergency calls from landline phones in Worcester to the emergency dispatch center in the police station, according to state regulators. Some of the trunk lines that carry emergency calls made from mobile phones apparently went down as well, according to the state.
The state pays Verizon more than $20 million a year – raised through surcharges tacked on to telephone bills – to provide the backbone of a bulletproof emergency telecommunications network.
“We do have failures in the network from time to time, but normally you don’t hear about them because the calls are still getting answered. Worcester not getting their calls, that’s very troubling,” said Tom Ashe, deputy director of the State 9-1-1 Department. “I can’t think of another time this happened, and I’ve been here 10 years.”
Here’s how the system is supposed to work:
The everyday public telephone network that we use from landlines at home and mobile phones on the go was designed to recognize an outgoing 9-1-1 call and funnel it into a separate, ultra-reliable emergency network. That network determines what city or town the call originated from and sends it, along with the location information, down a dedicated line to the appropriate dispatch center.
If the dedicated line is overloaded or down for some reason, the call shifts automatically through a series of secondary lines until it connects. If none of those lines work, the call then bounces through yet other lines to an alternate dispatch center, often in an adjacent town.
Verizon spokesman Philip G. Santoro said the company’s engineers are still trying to determine precisely why some Worcester emergency calls went awry despite all the network redundancy. Some emergency calls from the city were successfully re-routed as the system was designed to do, he said, but an unknown number of callers got a busy signal instead of a dispatcher’s voice asking, “What’s your emergency?”
Verizon engineers haven’t been able to determine the precise number of blocked emergency calls, but they can tell it was between three and eight, Mr. Santoro said.
“That’s very rare. It should never happen, and we take great pains to make sure it doesn’t,” said Mr. Santoro, who credited Verizon technicians with working as fast as possible to resolve the problem.
As the 9-1-1 system crisis played out in the city that afternoon, managers at the Central Massachusetts Emergency Medical Systems office in Holden were scrambling to re-route calls from ambulances. In the case of serious injuries or heart attacks, EMTs may call ahead over the radio to the emergency room to request a trauma team or cardiac specialists stand by, a step that can shave crucial minutes off the time it takes to treat the patient.
The ambulance-to-hospital calls are carried by dedicated telephone lines from the receiving radio tower to a facility in Holden that then patches the ambulance crew in to the hospital over dedicated telephone lines. All but one of the special lines leading to and from the Holden office went down, cutting off communications for about 15 minutes, said Ed McNamara, CMEMS executive director.
“We’ve never had a situation like this happen before. Nothing this bad has ever happened to us in our 30 years,” he said.
The timeline provided by Verizon indicates the company got an internal alarm warning of an “abnormality” in the company’s Chestnut Street network switching center at 3:45 p.m. The company notified state regulators of the failure 10 minutes later, but didn’t bring the matter to the city’s attention until 4:30 p.m. Network service was restored at 5:55 p.m., according to Verizon.
The city’s Mr. Clemons expressed frustration with Verizon for the delay in notifying him about the breakdown and for what he considers a lack of straight answers from the company.
“We’re held to a standard of seconds. If I don’t answer your call in seconds, then I’m not doing my job,” Mr. Clemons said. “How is our vendor not held to the same standard?”
Mr. Clemons was the dispatch supervisor in Hopkinton in 2005 when a 49-year-old woman died, apparently from an asthma attack, after a different kind of malfunction in Verizon’s 9-1-1 system.
The Federal Communications Commission, which regulates emergency communications at the national level, released a critical report this month on a failure that knocked out 9-1-1 service to a swath of northern Virginia last summer.
“This inquiry has raised significant concerns about the extent to which Verizon has followed its stated maintenance procedures in the past,” noted the agency’s report.
After a major 9-1-1 network failure in Maryland in January 2011, FCC Public Safety and Homeland Security Bureau Chief Jamie Barnett issued a statement saying, “We are particularly concerned that this problem may be widespread across Verizon’s footprint.”
In the recent Worcester case, Mr. Clemons said all 9-1-1 calls that didn’t get through in the city should have been routed to the state police dispatch center in Framingham. But the state police call center happened to be closed that day for routine maintenance.
Given that, the calls should have been re-directed again to the state police’s alternate center, a State 9-1-1 Department facility in Maynard, Mr. Clemons said.
City Manager Michael V. O’Brien said any failure of Verizon’s system is unacceptable.
“I know that appropriate agencies at the state and federal governments are reviewing what happened and why it happened,” Mr. O’Brien said. “I intend to work with them closely so I understand the cause and what they will require Verizon to do to upgrade systems to correct this so it never happens again.”
The Verizon malfunction here also affected parts of the 9-1-1 network serving about 40 other cities and towns, according to the state. In those cases, however, emergency calls connected over alternate trunk lines or ricocheted to dispatch centers in other locations.
Mr. Ashe said he and other members of his department have been talking to Verizon officials daily since the network failure. He expects a final report from the company within a few weeks.
“They are fairly guarded and don’t want to send out any guesses,” he said. “They stated to me that it’s a very complicated failure, and they have their engineering folks looking at it.
Copyright © 2013 LexisNexis, a division of Reed Elsevier Inc. All Rights Reserved. ”