Disaster Recovery Planning: Sadder But Wiser
Reducing Exposure
MetLife already has a geographically spread-out workforce that works in its favor during times of crisis, says Sheinheit. ""We have many offices and our data centers are no longer in Manhattan,"" he says. ""Having a data center in a tall office building with many windows is not the ideal spot. We have our data centers in more rural areas,"" and in buildings more suited to housing a data center—high security and fewer windows. ""Cost was another reason why we moved the data centers out of New York long ago. Why pay top dollar for real estate if you don't have to?""
Empire relied on other offices during the disaster recovery effort. ""We had more than one facility to fall back on,"" including the company's main data center in Staten Island, says Snow. ""Now, if all 6,000 employees had been in the World Trade Center and we didn't have a remote data center, it would have been a different story.""
Jacksonville, FL-based Life of the South's (more than $300 million in premiums) data center is on the third floor of the company's headquarters. ""It is not the ideal location because there are windows,"" says Robert Fullington, CIO. ""Behind the windows is high-impact Plexiglas. We can't have the data center on the first floor because we are right next to a river.""
Fullington says that his company's hot site could be in a number of places. ""Our hotsite vendor is IBM and they have sites all over the country,"" he says. ""We are an AS/400 shop, so we can go to any one of IBM's facilities. Two years ago our disaster recovery test was in Atlanta. The last test was in Dallas.""
Even with hotsite contracts and redundant technology in place, companies are still having to strengthen DR plans based on the 9/11 events and its aftershocks, says Jim Grogan, vice president of alliances at SunGard Availability Services (Wayne, PA), a technology hotsite and back-up services provider. ""There was a huge interruption caused by the grounding of air travel,"" Grogan says. ""Most plans assumed that hardware and even people could be moved overnight.""
Furthermore, some firms were relying on employees from another location who could fly to New York to enact the DR plan, Grogan adds. ""Companies are now looking at everything more closely because they can no longer make certain assumptions about what is normal.""
Empire's disaster recovery plan also called for replacement equipment to be flown to New York, says Snow. ""Basically, everything ended up being trucked to us,"" he says. ""IBM was tremendous. They got us PCs, laptops and servers very quickly—all on truck. Cisco San Jose, CA sent us routers and Aspect Communications San Jose replaced our lost phone switches. We had the technology staff available, we just needed the equipment.""
OppenheimerFunds had 600 employees, primarily traders, in the WTC, says Modie. ""We had to get data tapes from New York to our Denver data center,"" she recalls. ""Luckily we were able to get a plane MassMutual's corporate jet in the air on September 12, one of the few flights that day.""
However, while OppenheimerFunds was able to get the data where it needed to go, it still needed office space for 600 employees. ""We spent a lot of energy relocating people,"" Modie says. ""Some went to OppenheimerFunds' hotsite facility at SunGard in New Jersey, and about 150 came to Hartford,"" where a LAN was set up and PCs—procured from MassMutual's inventory for desktop upgrades—were configured for OppenheimerFunds' use.
Office space, more than telecom or hardware, was the biggest problem for many of the companies displaced from the WTC. ""We lost 480,000 square feet of space in the World Trade Center,"" says Empire's Snow. ""The real bottleneck was not telecommunications or IT, it was finding space for everyone. It wasn't until the end of November that we got everyone into an office.
""All of a sudden, we needed 480,000 square feet of space,"" Snow adds. ""But there is no way you can carry 480,000 square feet of office space in reserve. Other than the office space, the disaster recovery plan is receiving a lot of accolades. We passed the supreme test."" Empire is currently building an office in Brooklyn, where the employees will move when it is completed. For now, the ""biggest hangover from 9/11 is what we are paying for real estate,"" Snow says. Empire had a long-term lease at the World Trade Center.
Location, Location, Location
Immediately following 9/11, Empire had employees located in five offices in Manhattan. Currently, Empire has moved all employees to a single location on 42 Street.
MetLife employees currently have the capability to work at home, or telecommute, and that functionality could come in handy if the insurer was to lose one of its offices, says CTO Sheinheit. ""From a systems perspective, we can offer the ability to work at home,"" he says. ""That is a flexible option that we could use if there was a disruption with transportation.""
Another option is to incorporate non-traditional office facilities into a DR plan, says Life of the South's Fullington. He is quick to point out that his DR plan accounts for his company's location. ""We're in hurricane country,"" he says. ""I doubt we'll be a target of a terrorist attack, but a hurricane is much more likely.""
Life of the South has contracts with a number of hotels where the company could move its operations at a moment's notice. ""With a big hurricane, you have advance warning,"" Fullington says. ""We have a number of locations under contract that are farther inland,"" as far north as Atlanta and as far south as Orlando. ""Based on the track of the hurricane, we contact the places we think the hurricane will bypass and we put them on alert. Hotels work out well. They have high-speed data lines and they are self-contained, so you don't worry about food or beds.""
Creating desk space basically overnight was a large challenge for companies after 9/11, says Rybak from Arc Partners. ""Some companies really had to scramble. One bank had its entire trading office in a ballroom in one of the hotels in midtown Manhattan. They were only able to get six communication lines into the ballroom. Four went to the trading systems, two lines went to confirming trades. Most of the traders were working on their cell phones. I was really surprised at the ability of people to create work-arounds.""
In fact, the human factor was both underestimated and overestimated after the WTC disaster. ""Many of us under-called the psychological state of the employees that were involved in the attacks,"" says Ed Broderick, principal analyst, Robert Francis Group (RFG, Westport, CT), a consulting and research services provider. ""Nobody could have planned for the psychological implications of September 11. How can you motivate 500 employees to get back to work after an event like that? Most are just worried about their families and friends.""
Empire's Snow says that after the initial shock, adrenaline took over. ""The people were incredible,"" he says. ""People were at their best when there was an immediate need. The adrenaline was so powerful that you wanted to work. There is probably more post-traumatic stress now that we are getting back to normal.""
MassMutual's Modie says she saw similar qualities in OppenheimerFunds' employees. ""I have never seen people band together the way people did after September 11,"" she says. ""People just wanted to help and get things done,"" Modie adds.
Once the systems were up and running again, the next priority for OppenheimerFunds was to get employees back to one location, says Modie. ""The psychology of having everyone in different offices was not good,"" she says. OppenheimerFunds employees are temporarily at a single location in Manhattan until a new office can be found.
Part of Life of the South's disaster recovery plan involves helping employees get their personal lives in order before they return to work, says Fullington. ""People are worried about their home and family,"" he says. ""If our area is ever hit hard by a hurricane, part of the disaster recovery plan is to assist employees with cash. ATMs will probably be down. We would hand out cash to get people going again. That is part of the disaster response effort.""
Another part of the human factor involves making sure that the proper people are available, says SunGard's Grogan. ""Companies are really taking a long look at the people side,"" he says. ""There are many parts of a disaster recovery plan that rely on key individuals. Sometimes there are not qualified alternates to implement the plan if the primary individual cannot work. Now companies want another qualified back-up person that also knows how to do the job.""
At MetLife, there is a renewed focus on documenting the disaster plan. ""We learned from 9/11that there needs to be attention to detail,"" says Sheinheit. ""This is not new. Everything has to be documented. Who is responsible for what? How do we keep track of our employees?""
The potential for a situation to go from bad to worse was definitely present on September 11, says Arc Partner's Rybak. ""What would have happened if one of the 'mission critical' people was not able to do their job because they were injured?"" she asks. ""Is the documentation available so someone else can pick up the slack?""
-----------------------------------------------------------------------
All Lines Are Busy, Please Try Your Call Again Later...
One problem companies faced in the New York area following the 9/11 attacks, and something for which most disaster recovery (DR) plans were not prepared, was the breakdown in communications. ""I couldn't contact people on 9/11,"" says Heidi Rybak, senior manager and head of Business Continuity, Arc Partners (New York), a financial services consulting provider. ""The phone systems were jammed. I ended up using e-mail. DR plans have to stress communications. The first priority is to find out if people are safe. But how can company know if its staff is safe if there is no phone system?""
Jim Grogan, vice president of alliances at SunGard Availability Services (Wayne, PA) says establishing a process to locate employees should be a core part of DR plans. ""Many companies lost days trying to locate all of their employees,"" he says. ""Alternate forms of communication, such as e-mail, are very important. It's important not just for members of the disaster recovery team, but also for communicating information to staff.
""When simple communication becomes a challenge, that is very unsettling,"" Grogan adds. ""E-mail became the most reliable form of communication. Other companies used their Web sites where employees could report on their status.""
Realizing communications inside New York were a challenge, OppenheimerFunds set up a Web site where WTC employees could report that they were safe, says Christine Modie, executive vice president and CIO at MassMutual, OppenheimerFunds parent. ""Once they were found, we could tell them what to do. But we had to find them first.""
Greg MacSweeney is editorial director of InformationWeek Financial Services, whose brands include Wall Street & Technology, Bank Systems & Technology, Advanced Trading, and Insurance & Technology. View Full Bio
Tweet This