Attention: This is Not a Drill. The Logicalis
Cincinnati-based EBOC Stood Eye-to-Eye with Hurricane Ike...and
Never Blinked.
Logicalis EBOC operations manager, Tom Sweeney, knew he had a
good disaster recovery plan in place. But he didn’t really know how
good it was until Hurricane Ike roared through Ohio on Sunday,
September 14, 2008, ripping gaping holes through the Cincinnati
metropolitan area power grid. By Monday, Ohio Governor Ted
Strickland had declared a state of emergency and reported that
nearly two million customers were without power in Ohio. There were
power lines down, transformers bursting into flames, traffic
signals knocked out, and roads closed around the state, while
inside the Logicalis EBOC, it was network monitoring and management
as usual.
Sweeney summarized the situation, saying, “We had a plan and it
worked; and that’s exactly what I expected.” According to plan, a
back-up natural gas generator promptly took over from the downed
power grid, and kept all of Logicalis’ customer network monitoring
and management equipment fully powered. During the 30 hours it took
for commercial power to be restored, the generator never went
beyond 13% of its capacity. As a matter of course, all customers
were immediately notified of the situation outside the EBOC, but
there were no interruptions of service.
More than Technology
Three huge monitors dominate one wall in the EBOC, giving the
facility the feel of a command center. There is more than
technology involved in managed services, however. There are as many
as 22 service delivery engineers available 365 days a year.
Logicalis service delivery engineers hold among them the highest
level of certifications in Cisco, Microsoft, Unix, IP telephony,
and other technologies. Some have as many as 20 years of experience
in IT operations.
As soon as the storm hit, senior management contacted everyone on
the Managed Services team to maintain constant status updates. Not
everyone was able to navigate through the devastation to get to
work, but according to plan, those who had power at home were able
to log in remotely and carry out their responsibilities. The
Managed Services team was also available via cell phone or
Blackberry, so communication was maintained at all times.
Those who could make it to work safely, swapped shifts with those
who could not, and senior management worked 12-hour shifts at the
EBOC so that they would be available to make critical decisions.
Senior level engineers were also enlisted from Logicalis’ Columbus,
Chicago, Phoenix, Seattle, and De Pere offices.
One of the additional responsibilities to be carried out, was the
notification of clients with operations in the region where the
monitoring equipment showed that they had lost power. As a matter
of policy, any time a connection goes dark, Logicalis engineers
make calls to the cell phones and private lines of designated
individuals, persisting until news of a power outage is
confirmed.
Uninterrupted
Meanwhile, across town where Logicalis maintains a co-location site
for its clients, a similar level of foresight kept all systems
running throughout the storm and its aftermath. A diesel generator
was employed here, and according to plan, a full 13 days worth of
fuel was on hand to ensure uninterrupted service. Each day that the
generator ran, an additional day’s fuel was booked to maintain a
13- day buffer. Logicalis clients in the Cincinnati area who were
co-located at this facility, might have lost power at their
headquarters, but their IT infrastructure hummed on
uninterrupted.
“We don’t keep all our eggs in one basket,” Sweeney notes. Had it
been necessary, Logicalis could have switched over to a duplicate
site at its facility in Indianapolis that provides full replication
of the equipment used to monitor and manage our customers. No plan
can cover all contingencies in a catastrophic event like a
hurricane – that’s when training and experience count the most.
“One thing we all learned from this is to be flexible, and respond
to the situation at hand, rather than concentrating on preconceived
notions of what could happen,” Sweeney says.
A Post-Mortem Review of the Day
Hurricane Ike descended on Cincinnati, and did reveal merely one
shortcoming in the overall disaster plan; while flashlights were
available when the lights went out in the EBOC restrooms, no one
had thought to hook up the electrical outlet for the coffee pot to
the back-up generator. It would have been nice to have been able to
make some fresh coffee during those late-night shifts. That
oversight, Sweeney says, has since been corrected.