September 11: One Year Later
Q: What changes were inspired by the experience of 9/11?
A: David Snow, Empire: We were very pleased with our disaster recovery plan. We had surprisingly few problems, given the magnitude of the event. However, one change we made is that now we digitize everything in the company. We had paper credentialing, agreements with hospitals and vendors, in the World Trade Center, and they had to be replaced because they were all destroyed. Replacing the documents is not hard to do, because most of the documents are from vendors, who want to get paid. They simply send us a copy, but it was a lot of work.
Maybe this is a psychological thing, but we are building space in the Metrotech center in Brooklyn, and when we signed the lease we insisted that we have the bottom 10 to 12 floors. We are also making certain we stress redundancy and fail-over in all of our technology—which helped us get back up and running so quickly after September 11.
A: Andrew Cole, St. Paul Re: Our disaster recovery plan held up extremely well and we had the operation fullyfunctioning across several disaster recovery locations within three business days—which is the target goal for our plan. Our bigger challenges came when we attempted to move back into our downtown location. The main issues centered on the significant loss of telecommunications capabilities by the major providers due to the destruction of their central offices in lower Manhattan and severe damage to the lines beneath the streets leading to our building. Essentially, our carriers had allowed a single point of failure into their architectures, which knocked out most of lower Manhattan. Thus, we were dependent on the carriers re-establishing their communications infrastructure before we could move back in. To prevent this issue going forward, we have enhanced our disaster recovery strategy to include the addition of a "shadow" telecommunications network. This has been established using different carriers with central offices far removed from our primary network providers' locations.
A: Ron Roecker, Kemper Casualty: Our disaster recovery plans proved to be extremely effective. E-mail was restored by midnight of the 11th and electronic data and applications were recovered and back online in a little more than 24 hours. September 11 showed the value of a strong disaster recovery plan.
Q: What leadership issues em-erged? If someone is hurt, killed or simply unreachable, who takes charge?
A: Cole, St. Paul Re: We were very fortunate to have no physical injuries to any of our employees. Our disaster recovery plan has a clearly defined chain of command with primary, secondary, and tertiary designations for all major functions. Our emergency notification strategy worked well and we had all key players assembled by 9 a.m. the following day to plan and coordinate all of the recovery activities.
A: Lowenthal, Hartford Financial Products: Fortunately, all our employees were able to get safely out of the area. What still resonates for me is how in a time of crisis everyone rises to the occasion. We reached all our managers by end-of-day 9/11, and the next morning most of us met face to face in a suburban location at one of our business partners' offices. During that meeting we worked as a team planning for everything from employee phone-calling chains to recreating lost paper files. For weeks we worked long hours to support our staff so they could keep customer service levels high. At the same time we quickly set up the new location, working with an emergency response team from headquarters. The teamwork and the caring were remarkable. Though we had no real leadership problem in this crisis, we have put additional procedures in place to make it easier to keep track of people. Each manager now has the home address, phone, cell phone and home e-mail address of all other managers. Managers have the same information for each of their direct employees. Also, an electronic copy of this information is accessible to the whole group.
A: Roecker, Kemper Casualty: Kemper was very fortunate that all of our employees made it out of the World Trade Center safely. Our senior management team was fully operational in temporary offices with full communications, including e-mail, by 7:30 a.m. on Sept. 12. However, we have since implemented a formal succession plan for all key individuals and positions. In the unfortunate circumstance that a key leader is lost or unable to perform his or her duties, the succession process will be invoked.
Q: Did your company have to relocate employees after 9/11? What were the technology challenges?
A: Snow, Empire: Empire lost 480,000 square feet of space that day. That dislocated 1,900 people and required a whole bunch of short-term moves. In the short term, we used other buildings, including our offices in Melville, Middletown and Yorktown, NY, and we moved some people to the data center. However, that was not enough space, so we prioritized and brought people back to work as we needed them and found space. Currently, we have a permanent headquarters on 42nd Street in Manhattan for about 200 people. And we have temporary space around Manhattan, including 500 people at 1440 Broadway and 500 people on Part Avenue. We anticipate that our permanent space in Metrotech will be ready in the fall of 2003. Luckily, since we were already fairly high-tech, we were able to quickly install servers in the temporary spaces and put the bandwidth in very fast.
A: Cole, St. Paul Re: Our offices are half a block from Ground Zero and we were displaced until the second week of January. Due to the intense demands on our disaster recovery vendor after 9/11 we were unable to secure our primary disaster recovery workspace in Jersey City for over six weeks. Instead, we used our own branch office, in Morristown, NJ, as our temporary headquarters. We needed to augment the Morristown physical infrastructure on the fly to accommodate the large influx of New York City-based employees. We worked around the clock installing new hubs, routers, jacks, phones, PCs, printers and all of the trimmings. We also had to establish new telecommunication lines to link our disaster recovery data center in Philadelphia to the Morristown location. These were quite a lot of changes to react to, but my staff really came through in the clutch. We also beefed up our remote access architecture to allow for a much greater level of telecommuting while we were displaced. The key lesson is to not assume your disaster recovery vendor is going to be able to deliver in the event of a major catastrophe, and have a contingency plan that includes back-up floor space and, if possible, back-up communication lines.
A: Lowenthal, Hartford Financial Products: Since the Hartford Financial Products home office was based in 7 World Trade Center, along with other employees of The Hartford, we needed to relocate all New York City-based employees. The bulk were temporarily relocated to existing Hartford offices in Hauppauge, NY (on Long Island), and Rockaway, NJ. Others worked from home using VPN connections. Within seven weeks of the attack, we moved to our new, permanent location at 2 Park Avenue in New York City. The most significant technology challenges centered on data and voice communications. While we were fortunate to have existing connections to The Hartford's enterprise WAN from existing offices, having new connections installed at our new location became the critical path to our final relocation. Also, we worked tirelessly with our AT&T partners to re-route our landline corporate phone numbers, first to individual cell phones, and then to the phone system—either in Hauppauge, New Jersey or home phone.
Greg MacSweeney is editorial director of InformationWeek Financial Services, whose brands include Wall Street & Technology, Bank Systems & Technology, Advanced Trading, and Insurance & Technology. View Full Bio
Tweet This