Disaster Recovery Planning..a 9/11 CIO story with lessons learned

1. Disaster Recovery
White Feather Solutions
Disaster Recovery Planning

2. Summary
 First Six Months As CIO of organization in
close proximity to twin towers
 What occurred during and after 9/11
 Post Mortem
 Planning for a Disaster
 BCP recommendations
 RIO proposed roadmap of DR facilities
and services

3. First Six Months As CIO
with Boutique Financial Firm
left three ominous letters from the previous CIO
1. Evaluate departmental requirements and
assess overhead
2. Cut Unnecessary Overhead
3. Cut Extraneous Servers and Systems
4. Cut Personnel

4. First Six Months As CIO At Financial Firm
 Discovered report from the previous CIO who
was in place during the 1993 bombing of the
World Trade Center
 Enterprise was shutdown for 10 days
 Losses totaled over 100M
 No insurance
 No Business Continuity Plan

5. New Mission
 Developed disaster recovery plan
 Had expertise with this type of firm prior to
becoming CIO and wrote several BCP’s for
similar firms engaged in Y2K projects
 Utilized a BCP I developed which was
accepted by Deloitte auditors for Y2K and
updated it to reflect the present IT
procedures

6. New Mission
 Interviewed all the departments in order to add their
segments of the enterprise to the BCP with the latest
business practices
 Queried department heads as to what were their most
critical functions and what needed to be protected and
running during a disaster
 Ascertained what functional interdependencies currently
existed between the departments and if they needed to
be protected and running

7. Presentation of the DR Plan to Management
 Reviewed the outcome of the 1993 bombing and the
significant financial impact it had on the organization which,
like many in NYC, had been caught unprepared on many
levels.
 Outlined the Business Continuity Plan
 Reviewed what was needed in terms of resources for BCP
 Demonstrated graphically how each process would be
addressed during an actual disaster
 Reviewed SLAs negotiated with the vendors
 Proposed new processes to be incorporated in the Enterprise
 Reviewed the costs involved in the implementation of the
plan, approximated $100,000.

8. Results
 Management balked at the business processes and their
associated costs.
 A common roadblock to BCP success: management did not
buy into the ongoing cost for backup servers that, as they saw
it, were not being used.
 BCP presentation took place on February 9th, 2001
 With zero budget, over the next 5 months I created,
implemented and tested a DR plan in my own department. I
shared the plan with all department heads.
 A few of the back office departments implemented the plan.
 THEN CAME 9/11...

9. Workday visuals
 This is what many NYC residents who
walked to work everyday routinely saw

10. September 11, 2001
 reports came into the office that a small plane was off
course and had hit one of the twin towers.
 Office talk was buzzing as to whether business should
close
 15 minutes later the second plane hit . A call came in
from management asking for direction. As CIO, every
facet of the business touched the data network whose
operation was my responsibility. My response was clear.
The business had to shut down.
 Background as Marine fighter pilot post Viet Nam served
myself and my company well.

11. September 11 2001
 About an hour or so after that the collapse:

12. September 11 2001
 And finally the aftermath:

13. September 11 2001

14. Spring in to action
 Our communications plan was immediately implemented.
 Emailed a standard evacuation list to department heads and instructed them to vacate. For those
departments who had the plan invoke it and vacate.
 SMT (Survival Management Teams – 5 two person teams) went through 5 floors of offices making sure
everyone was leaving and giving instruction on where to meet, well away from the disaster,
which had been predetermined.
 Ordered a shutdown of all systems and the evacuation of all personnel, and collection of all
software backups.
 Attempted to call vendors, however land and wireless lines were all tied up. I sent emails
ordering lines, systems, software to be brought to a second facility we had in Queens, some 6
miles away.
 Once all the employees were evacuated and the doors locked, we proceeded to walk to the
United Nations Building, the predetermined meet place. We took a head count, everyone had
made it.

15. Spring into action
Management request
“PICK 5 MEN AND GO IN AND RETRIEVE FINANCIAL SERVERS”
Response
I WILL GO IN IF PROVIDED WITH OXYGEN, PROTECTIVE SUITS AND
CASH TO GET PAST THE SECURITY ROADBLOCKS. THESE MEN
HAVE FAMILIES AND HAVE NOT EXPERIENCED WAR .
Management
CASH ONLY

16. Aftermath
 set up a hotline using voice mail. The city was completely shutdown.
I experienced difficulty updating the voice mail
 Receive call from management. They attempted to perform a rescue
mission of the servers and the financial data. Men that went in, are
still receiving mental health therapy.
 The following day, along with a NYC police officer, two of my people
and myself, a cart, dressed in fire retardant suits, with oxygen
masks, axes, crowbars, we go get the servers
 We had to strap them on our backs and take them, one by one,
down 5 flights of stairs.

17. POST MORTEM
 Proposed DR plan -$100K
 Cost to get basic functionality back - $500k+
 Replacement of lost equipment - $1.2M
 Insurance picked up around 25%
 It took around 21 days to get back on line at
a loss of approximately $160M
 Risk to employees, having to recover the
servers

18. PLANNING FOR A DISASTER
Types Of Disasters
 Fire / Explosions
 Natural Disasters
- Tropical Storms and Hurricanes
- Tornado and Thunderstorms
- Earthquakes
- Floods
 Utilities Disruption
- Electric
- Gas
- Water
- Phone
- Heat / Air Conditioning
- Loss Of Elevators
- Computer System Failure

19. PLANNING FOR A DISASTER
Types Of Disasters
 Security Emergency
- Bomb Threats
- Criminal Acts on Individuals
- Civil Disturbance
 Hazardous Materials Spills/Releases
 Food Borne Illness
 Visitors Incidents
 Employee Incidents
- Employee Serious Injury/Death
- Workplace Violence
 Media Inquiries
 Evacuation
- Site Evacuation Planning Factors
- Community Wide Evacuation
 Shelter-In-Place

20. PLANNING FOR A DISASTER
Components of a Disaster Recovery Plan
 Human Continuity Plan
 Create Survival Management Teams
 Define a scope of what the plan will cover
 Business Continuity Plan
 Define each departments role in the
enterprise and determine how critical
their functions are to the business
 Develop contingencies to critical business processes
 Facility Response Plan
 Create Facility Response Teams
 Describe Facilities & Locations
 Define a scope of what the plan will cover

21. PLANNING FOR A DISASTER
Components of a Disaster Recovery Plan
 Event Management
 Create Crisis Management Teams
 Detection of events
 Incident Detection
 Incident Assessment and
 Classification
 Automatic Detection of Events
 Fire Alarms
 Intrusion Alarms
 Event Response
 Notification of proper authorities

22. PLANNING FOR A DISASTER
Components of a Disaster Recovery Plan
 Crisis Communication Plan
 Determine how the enterprise
communicates with it’s employees
 Post Disaster Review
 Disaster Testing Plan
 Frequency

23. PLANNING FOR A DISASTER
Examples of Components in a Disaster Recovery Plan
Human Continuity Plan
 Survival Management Team Member
 Company will send volunteers for EMT training
 Volunteers will be trained in
light extrication

24. PLANNING FOR A DISASTER
Examples of Components in a Disaster Recovery Plan
Business Continuity Plan