Not the Unthinkable, But What We Didn’t Think Of: Preparing For and Recovering From Disaster

Harvard Art Museums April 11, 2006
Not the Unthinkable, But What We Didn’t
Think Of
Preparing For and Recovering From Disaster
Charles Patch
Director of Systems
The Historic New Orleans Collection

Hurricane Katrina was the most destructive natural disaster in
U.S. history. The overall destruction wrought by
Hurricane Katrina, which was both a large and powerful hurricane
as well as a catastrophic flood, vastly exceeded
that of any other major disaster, such as the Chicago Fire of
1871, the San Francisco Earthquake and Fire of 1906,
and Hurricane Andrew in 1992.
– The Federal Response to Hurricane Katrina, Lessons Learned.
Department of Homeland Security, February 2006
Disaster in the Real World
The Official Word

Living in a post disaster world

Living in a post disaster world
 City population < half of pre-Katrina levels
 > 60% of city uninhabited
 20 public schools in operation
 15% of student body
 5 of 14 branches of public library open
 No second class mail delivery
 Most museums and archives not open until late February

Living in a Post Disaster World
Condition of Public Libraries in LA Post-Katrina

Responding to Disaster in the Abstract
 TERMS:
 Business Activities
 Business Continuity Plan
 Business Impact Analysis (BIA)
 Business Interruption Event
 Business Process
 Business Resource
 Business Resumption Plan
 Control
 Key Business Process
 Maximum Acceptable Outage (MAO)
 Outage
 Procedures
 Resources
 Risk Event
 Risk Management Plan
 Service Area Contingency Plan

Responding to Disaster in the Abstract
 Business Continuity Management (BCM)
is part of a larger Risk Management Plan
 Disaster Recovery is a subset of BCM
 Risk Management considers both
negative and positive risk
 The Prime Directive: Plan for the best,
prepare for the worst
 Proactive design and implementation of
controls to prevent risks from occurring
 Reactive design of controls to mitigate effects
of adverse events that occur.

Responding to Disaster in the Abstract:
Overview of Risk Management Process
Review operation and continuing
suitability of controls
Monitor and Review
Establish plan. Implement controlsImplement Controls
Identify, analyze, rate and prioritize
risks
Evaluate design of existing controls.
Redesign controls if necessary
Identify and Assess
Risks
Determine key business objectives,
processes and resources
Establish Context

Responding to Disaster in the Abstract: Organizational Context
Organizational Objectives
Output Group Output Group
Output Group
key business
process
key business
process
key business
process
key business
process
key business
process
Business Support
Process
Business Support
Process
Business Support
Process

Responding to Disaster in the Abstract: Organizational Context
Collect, Interpret, Display
Cultural Materials for General
Public
Administration
Collections
Management
Information
Systems
Payroll
Facilities
management
Accounts
Payable
Acquisitions
Data storage
File hours with
Paychecks Inc.
Create loan records in
Collection Management
System
Create semi-weekly
backups of all data

Responding to Disaster in the Abstract: Risk Management Process
No
Yes
No
Yes
Determine
Possible Risk
Events using risk
framework
Determine
likelihood and
consequence
without control in
place
Determine risk
level and
compare with
acceptable risk
Acceptable
?
Evaluate design
of existing
controls
Determine
likelihood and
consequence
with control in
place
Determine risk
level and
compare with
acceptable risk
Acceptable
?
Redesign
controls
Record in Risk
Register
Identify
Analyze
Evaluate
Control
Document

Responding to Disaster in the Abstract: Risk Identification

Responding to Disaster in the Abstract: Analysis & Evaluation
 Is a risk minor (acceptable) or major (unacceptable)?
 What is “acceptable”?
 What are the consequences in terms of resources?
 Do they have a detrimental impact on
 Staff
 Facilities
 Collections
 Telecommunications
 Information Systems
 Rank the consequences on the degree of impact on these resources
 What is it worth to you?
 A consequence can be minor if its fullest possible impact will not be
detrimental to institutional resources.
 Last question: will a consequence have an impact on normal
business operations?
 Events which have detrimental impacts on resources are likely to
impact business operations <- remember this!

Responding to Disaster in the Abstract: Controls
 Preventative Controls
 Stop the risk from occurring in the first place
 E.g.: require passwords to access the computer system
 Ban flammable solvents from processing areas
 Corrective Controls
 Minimize the consequence of a risk event once it has occurred
 Emergency conservation supplies
 Backup tapes for computer system
 Evaluate current control mechanisms
 If the consequences remain unacceptable, redesign them

Responding to Disaster in the Abstract: Monitor and Review
 Incorporate risk management into normal business operations
 Strategic planning
 Budget process
 Facilities Maintenance
 Make risk management part of operational management
 Supervisors and key workers participate in planning and analysis
 Sign-offs by managers on risk controls
 Practice procedures
 Structured walk-through
 Review for validity
 Confirmation of supplies needed

Responding to Disaster in the Abstract: The BCM
 A Business Continuity Plan is a Corrective Control
 An interruption in a Business Process is an adverse consequence
of a risk event
 A complete interruption of this sort is an Outage.
 How long can an Outage be tolerated before the viability of the
organization is threatened?
 This duration is the Maximum Acceptable Outage (MAO)
Risk
Management
Conrols
Preventative
Corrective
Disaster
Recovery
Business
Continuity

Responding to Disaster in the Abstract: The BCM
 Business Continuity in 6 arduous steps
 Initiate the project:
 Identify Key Processes:
 Conduct a Business Impact Analysis:
 Design Continuity Controls
 Implement the controls
 Test and Maintain the plan

Responding to Disaster in the Abstract: The Controls (but only for
the collections)
1. Design Continuity Controls
 Review existing controls
 Identify and evaluate options and alternatives
 Select the alternative activities and resources
2. Implement the controls
 Establish recovery teams
 Document “Service Area” action steps
 Establish event escalation process
 Obtain and consolidate contact and inventory lists
 Document recovery management process
3. Test and Maintain the plan
 Paper test
 Manual Verification
 Supply validation
 Supply, Service and equipment availability
 Structured walkthrough
 Unannounced team assembly

Responding to Disaster in the Real World:
 Background: The Yearly Drill
 Street flooding is common
 Hurricane / Tropical Storm alerts are yearly events in New Orleans

Cindy – Wednesday 7/6/05

Dennis –Sunday 7/10/05

Katrina – Monday 8/29/05

Living With the Past

Responding to Disaster in the Real World: What we did right
Routine procedures
 Keep Buildings maintained and up to code
 Fire alarms, fire extinguishers checked on semi-yearly basis
 Routine roof inspections
 Plumbing and electrical services kept to code or better
 Lighting and signage maintained properly
 Given the choice, try not to put things where they could
be harmed
 Follow best practices for data management and storage
 Make duplicates or copies of important documentation
and store them offsite
 E.g.: All accession records and documentation microfilmed
at the end of each year, including updates to active
collections
 Stored in two difference places, one offsite

Responding to Disaster in the Real World: What we did right
Routine procedures
File plans with local fire and police departments
Be sure that room and floor designations are the
same in all documentation, including insurance,
collections management systems and fire
department plans.
Consider requiring third-person contact numbers
Keep current contact lists for vendors, lenders,
staff and emergency services

What We Did Right: Disaster Preparation
 Disaster preparedness planTHE HISTORIC NEW ORLEANS COLLECTION
DISASTER PREPAREDNESS PLAN
TABLE OF CONTENTS
I. Mission Statement/Staff Assignments Page 4
II. Emergency Prevention Strategies Page 8
III. Communication in the Event of Page 10
an Emergency
IV. Visitor Safety
A. Fire Page 11
B. Illness
Page 11
C. Accidents Page 12
D. Robbery Page 12
E. Vandalism Page 12
F. Bomb Threat Page 13
V. Emergency Resources Page 15
Building Resources Page 16
Collections Resources Page 17
VI. Supplies
Page 20
VII. Fire Prevention Page 21
VIII. Hurricane Preparedness Page 22
IX. Disaster Preparedness
A. 533 Royal Street
1. General Page 24
2. Fire Page 24
3. Hurricane Page 25
(General, fire, hurricane are addressed for each building)
B. 714 Toulouse
C. 718 Toulouse
D. 722 Toulouse
E. 726-728 Toulouse
F. 410 Chartres
G. 521 Tchoupitoulas
X. After a Disaster Page 41
Appendix A - Maintenance Checklist Page 42
Appendix B - Information Flow Table Page 43
Telephone Lists Page 44
Appendix C - Bomb Threat Page 46
Appendix D - Williams Residence Priority List Page 48
Appendix E - Chartres Street Receptionist Guidelines Page 52
Recovery assistance
Collections Kid Gloves, Inc. 733-6765
Ron Grose (chandeliers) 831-1669
Ellis S. Joubert (metals) 899-1746
Conservators
Book/paper Christine Smith (703) 836-7757
Northwest Document Conservation Center (508) 470-1010
Alan Balicki (212) 873-3400 ex.287
home (718) 855-1723
Jim Stroud (512) 471-9117
Chris Young (615) 227-0538
Painting Richard White 822-4567
Cynthia Stow (615) 269-3868
Barry Bauman (312) 944-5401
Perry Huston (817) 595-4131
Claire Barry (212) 737-4786
Louise Beeson (and frames) 241-2587

What We Did Right: Staff Preparation
 Regular testing of alarm system and drills
 Training of docent staff in emergency procedures
 All areas kept stocked with emergency supplies

Responding to Disaster in the Real World
What We Did Right 3: Disaster Preparation
 Prioritization of materials

Responding to Disaster in the Real World: What We Did Right
 Storm monitored by Disaster Coordinator from first appearance
early in the week when still a tropical storm
 Emergency supplies checked, calls made to confirm that all
staff has copy of Disaster manual
 Preparation Meeting held Friday morning, August 26th
in
directors office
 Emergency message activated
 Staff put on alert
 Head count of who is available this day and the next
 Initial clean-up of processing areas completed
 Storm monitored throughout day
 Saturday, escalation of probable hit, staff called in at 7:30 AM,
16 people respond.

What We Did Right: Set Up
 Saturday 7:45AM:
 Fresh computer backup tapes produced
 Collection Management System data and
Membership/Development data compressed and sent via FTP
to vendors
 Computer system shut down and disconnected from power
source
 Major servers moved to second floor of buildings and covered
with plastic sheeting
 Phone system shut down and disconnected from power source
 Art work removed from exterior walls
 Sunday:
 Remaining staff either evacuates city or takes protective
measures at their homes
 Hurricane raised to Category 5, then Category 4 status; Direct
hit procedures invoked. Five staff return to carry out
remaining preparations
 Removal of all work remaining materials in processing from
floors and surface areas
 Relocation of all art works on first to second floors

What We Did Right: Set Up
 Building
Preparation

August 28, Sunday: The Evacuation
 Mayor declares mandatory evacuation on
Sunday morning
 Extremely difficult travel conditions
prevail
 8 to 12 hours to reach Baton Rouge (80
mile)
 Director contacts all department heads by
phone to establish whereabouts
 Director and several other staff members
stay to ride out storm

8/29 – 9/3 : Katrina “misses” New Orleans
 Hurricane tracks to the East of the city
 Power and phone service throughout
Southern part of state lost during most
of Monday
 Monday night, most believe New
Orleans has “dodged the bullet” again
and plan return to the city on Tuesday
morning.
 Tuesday morning television reports
80% of city is under water
 Phone service dysfunctional, cable and
Internet access extremely rare
 By Wednesday there are 100,000 extra
people in Baton Rouge
 Wednesday NO School board
announces that school is suspended for
the year
 Initial reports suggest it will be months
before the city can be inhabited again
 Most evacuees begin looking for long-
term living arrangements

What we did right: The Evacuation
 We learned to work the damaged phone system
 Lesson: cell phones worked via text message in the New Orleans area and
with voice when calling outside the New Orleans area code.
 First Contact: Tuesday night, August 30
 Staff members in the area organize rescue of valuable materials.
 Initially planned for September 2, delayed because of conditions until
September 8
 “High Priority” materials were clearly marked and were stored in easily
accessed locations
 Offsite location for materials established upstate
 Movers retained on special contract called in
 Google Group created 9/1 after contact with Minisis Inc.
 Staff assured of continued employment, salary and benefits
 This is the first message posted to the list that is not a response to the “Where are you?”
query.
 Staff web page created 9/5
 Most valuable collections rescued 9/8
 Network gear rescued 9/8
 Personnel Database created 9/13

What Went Right: The Evacuation
 Staff stayed of their own volition
 Our vendor becomes our go-between
 Staff members found the Google Group
 Our web master took the site with him on his laptop
 Our network manager was close enough to join the
convoy
 Our vendor anticipated our need for a personnel database
 The French Quarter was spared and sufficient staff were
on hand to open the facility on October 3.

What We Could Have Done Better
 No designated essential personnel
 Compounded by timing issues
 A failure to communicate
 Dependence on phone system for
communications
 No working email addresses
 No common contact point for reaching
individual staff members
 Administration chose to stay close to the
physical institution rather than
communication hubs.
 Most were in rural areas with poor or
no Internet access
 Nearly all communication from the
administration was by cell phone,
which worked only sporadically
 Administration did not take advantage of
communications channels
 Tended to wait until staff contacted
them rather than contacting staff
 Rarely posted information on the staff
web page or the Personnel DB bulletin
board
 No alternative admin site
 And no policy for defining how to recognize
a workable alternative
 No back-up network site
 Inadequate conservation supplies on site
 Most suppliers for recovery materials listed
in handbook were local

What We Could Have Done Better
 Some things we didn’t think of (that might have been caught in
a risk analysis)
 Book Trucks
 The Real Truck:
 Who was going to drive this?
 The giant image files that weren’t backed up
 Network “single point of failure” (actually 4 potential failure points)
 Magnetic Locks (no electricity, no security)
 Time lost to making ad hoc arrangements
 No pre-arranged alternative “home” for rescued collections
 No pre-arrangements for care of “essential staff” before or after
disaster
 No plan for setting up network in alternative location while staff
was disbursed
 (as opposed to securing backups)

What Didn’t Go Well?

Responding to Disaster in the Future
Lessons Learned
 Increases in scope and intensity increase impact of a disaster.
 Impact can vary in regard to timing:
 As intensity increases, the greater the need for fast recovery
measures
 The recovery of materials may be time-dependent
 Severely damaged physical plants need immediate stabilization
 As scope increases, recovery becomes more difficult to execute
 Common services and infrastructure may be unavailable
 Scope and intensity can affect how you recover
 Materials
 Human Resources
 Property
 Both scope and intensity hamper the execution of business
processes
 A widespread disaster can prevent access to facilities as much as
an intense but localized disaster

Lessons Learned
 Roles:
 Disaster Recovery Coordinator
 Service Area Teams / leaders
 Vendors / Service Providers
 Chosen on an “as available” basis

Responding to Disaster in the Future: Lessons Learned
The Inconvenience of the personal lives of personnel
 “Normal” personnel issues
 Employees with dependents
 Employees with physical disabilities
 Extended absences of employees with critical knowledge
 Disaster issues that exacerbate personnel problems
 People with dependents need more time to prepare for an
evacuation
 People with dependents cannot respond as quickly to emergency
calls as those without dependents
 People with dependents may not be able to return as early as
others
 E.g.: no school in New Orleans until January 2006
 E.g.: inadequate medical care

Lessons Learned?
 Most forms of communication will fail
 No single source will provide adequate information concerning real
events
 Individuals or small groups working independently were most effective
 Data redundancy is highly desirable
 Professionals involved in activities unrelated to what authorities regard
as the central economic interests of the city and region will be
marginalized

Lessons Learned?
 Communications 101
 Mechanics:
 Create multiple forms of asynchronous communication
 Out-of-Region phone bulletin board
 Web discussion groups/web pages/staff directories
 “Real” bulletin board on site for leaving paper messages to
other staff
 Develop policies for choosing administrative sites that
provide for access to the disaster area AND good
communications

Lessons Learned?
 Communications 201
 Semantics:
 Create a protocol for communicating with staff
 Procedures for posting staff supplied information
 Scheduled administrative updates and news
 “Test the channel”
 No news is bad news

Information
 Weigh all information carefully and
whenever possible, rely on trusted
sources in the area
If the images were to be reduced to a sentence in the minds of
Uptown New Orleans, that sentence would be: Crazy black people with
automatic weapons are out hunting white people, and there's no bag limit!
"The perspective you are getting from me," one of Fort Huger's foot
soldiers said, as he walked around the living room with an M-16, "is the
perspective of the guy who is getting disinformation and reacting
accordingly.“
-Michael Lewis
Wading Toward Home
New York Times Magazine, Oct 9, 2005
“We heard stories about helicopters being shot. But you’ve been in
helicopters, and you know how noisy they are. The only way you
know you have been shot at is if there’s a bullet hole,” Landreneau
said. “There were no shots fired at our helicopters.”
-Maj. Gen. Benny Landreneau, Louisiana National Guard commander , quoted in
“What We Signed Up For” by John Hill, Louisiana Life Magazine, March 2006
The media’s willingness to report thinly attributed rumors
may . . . have contributed to a cultural wreckage that will not
clean up easily. . . . Victims, officials and reporters all took
one of the most horrific events in American history and made
it worse than it actually was,”
Lecture by New York Times media reporter David Carr, September 2005
“Four weeks after the storm, few of the widely reported
atrocities have been backed with evidence,” - Times-Picayune
September 26, 2005
The New York Times has confirmed that one person was
murdered at the Convention Center and one at the
Superdome, and the Times-Picayune has confirmed that a
National Guardsman was attacked by an assailant wielding a
metal rod in the darkened Dome. The coroner’s early report
implies that the murder rate among those stranded in
Katrina’s aftermath was at least five times New Orleans’s
normal murder rate. This real, not imagined, violence
prevented New Orleans from getting the level of volunteer
and professional help it needed after Katrina.
-“Who’s Killing New Orleans?” Nicole Gelinas, City Journal, Autumn 2005,
http://www.city-journal.org/index.html
 Information: Who To Believe?

Information: Local Discussion Lists

Facilitate Independent Agency
 Provide materials for short term survival as
well as collections care.
 Leverage the actions of individuals by
providing some method of coordinating
among disparate players
“Response to Katrina is less dependent on traditional disaster
plans than on improvised actions as conditions permit.”
-Report of Hurricane Katrina Damage Assessment:
Debra Hess Norris (Heritage Preservation)
Richard Pearce-Moses (Society of American Archivists)
David Carmichael (Council of State Archivists)
21 September 2005

Have Have More More Than One One
 Off-site storage of data
 Meta-data tagging of individual digital
assets
 Cooperative arrangements for
network operations with sister
institutions
 Cooperative arrangements for
physical storage with sister
institutions

Have Have More More Than One One
Just a few miles west of Pass Christian, the Hancock County Historical
Society in Bay St. Louis fared much better with very little water
damage and a vault that held, protecting thousands of documents,
including family diaries and thousands of local photographs.
Charles Harry Gray, the executive director, was prepared in case
disaster struck. Over the years he had been making copies of all of the
group's most treasured documents, including 30,000 pictures. Not one
single photograph or record was lost.
They are the pieces of Bay St. Louis' 306-year history that made the
town of 8,230 what it is today, he said. Many of the copies were on
computer disks and hard drives, others were sent to the University of
Southern Mississippi, two hours north in Hattiesburg.
"It is imperative that you have copies in other locations because you
never know what's going to happen, what the next catastrophe is
going to be, and there certainly will be one," Gray said.

Disenfranchisement
Though archivists began asking for re-entry into the
city on August 31…(all) requests for attention to
historic paper records were denied…The
Department of Culture, Recreation and Tourism
failed to include archivists on planned
reconnaissance trips into the city or to include
anyone from the archival community in its planning
meetings
Susan Tucker
Curator of Archives at the Newcomb College Center for Research on Women
Email, 10/14/2005

If this ever happens again…
 All staff should have a web-based email address .
 Publish and update as part of the recovery documentation
 All staff should provide a contact phone number outside of the region
 We should establish a telephone bulletin board outside of the institution
 We should consider a subscription to a satellite phone service
 All department heads should have lap-top computers
 Online forum should be set up permanently with a single instructional
message
 At least one institutional email address that is hosted on an off-site
server
 Staff-only emergency web page should be set up permanently
 Policy for alternative administrative site
 Not necessarily choose an actual location but develop criteria for
choosing one
 Develop cooperative arrangements with other institutions for storing
collections
 Develop cooperative arrangements with other institutions to host our
computing equipment and run our network
 Contract for archiving of digital assets that are difficult to transmit or
store easily to tape

Web Sites of Interest
 Business Continuity Management
 NEDRIX - New England Disaster Recovery Information X-change
 http://www.nedrix.com/

Risk Management Standards
 http://www.incom.com.au/enterprise-risk-management-standards.htm
 Better Practice Guide - Business Continuity Management - Keeping
 http://www.anao.gov.au/WebSite.nsf/Publications/4A256AE90015F69B
4A2568EE0010062B
 National Archives of Australia - Business continuity planning for di
 http://www.naa.gov.au/recordkeeping/er/guidelines/9-continuity.html
 Business Continuity Planning & Disaster Recovery Planning World
 http://www.disasterrecoveryworld.com/

Not the Unthinkable, But What We Didn’t Think Of: Preparing For and Recovering From Disaster

Not the Unthinkable, But What We Didn’t Think Of: Preparing For and Recovering From Disaster

Recommended

Recommended

More Related Content

Similar to Not the Unthinkable, But What We Didn’t Think Of: Preparing For and Recovering From Disaster

Similar to Not the Unthinkable, But What We Didn’t Think Of: Preparing For and Recovering From Disaster (20)

Recently uploaded

Recently uploaded (20)

Not the Unthinkable, But What We Didn’t Think Of: Preparing For and Recovering From Disaster

Editor's Notes