1. Designing and Implementing a
Business Continuity Architecture
Breakout Session #2543
Christopher Janoch
Sr. Network Engineer / Architect, Powell Goldstein, LLP
September 18, 2008
Christopher.Janoch@pogolaw.com
2. Designing and Implementing a
Business Continuity Architecture
A Case Study of:
Powell Goldstein, LLP
Christopher.Janoch@pogolaw.com
3. Christopher Janoch
Senior Network Engineer / Infrastructure Architect
VMware Certified Professional
DoubleTake Certified Engineer
Zantaz Certified Engineer
ITIL Certified
Experience in designing DR and BCP infrastructure in
Legal Industries
Financial Industries
Construction Industries
Christopher.Janoch@pogolaw.com
4. Powell Goldstein, LLP
Business View
200 Lawyer Firm (600 Users)
Offices in Atlanta, Washington DC,
Dallas, and Charlotte
Technical View
Primarily Microsoft-based Technology
200+ Servers
Christopher.Janoch@pogolaw.com
5. High Availability (HA)
A system that can provide a continuous service by detecting
hardware, node or application failures and automatically
reconfiguring the system appropriately.
Fault-Tolerant Disk Array
Redundant Power Source / UPS
Redundant Network Connections
Multiple Endpoint Service Clusters
Failover Clusters
Christopher.Janoch@pogolaw.com
6. Disaster Recovery (DR)
A system to aid in the process of restoring operations critical
to the resumption of business (communications, data,
workspace) after a natural or man-made disaster.
Backup / Restore
4-Hr Replacement Service Contracts
Alternate Staging Site for Servers & Workstations
Rebuild and Reinstall Affected Systems
Christopher.Janoch@pogolaw.com
7. Business Continuity Process (BCP)
A system aimed at allowing an organization to continue
functioning after (and ideally, during) a disaster, rather than
simply being able to recover after a disaster.
Christopher.Janoch@pogolaw.com
8. Why do you need a plan?
Christopher.Janoch@pogolaw.com
9. Protection Strategies are Insurance
“Fast, Best, or Cheap – Choose any two!”
$$$ vs. SPEED vs. RISK – Choose any two!
You get what you pay for, But don’t pay too much!
Christopher.Janoch@pogolaw.com
10. YOU need to be the one with a complete PLAN
No one solution or vendor will adequately cover all needs.
No one methodology will cover all situations.
Christopher.Janoch@pogolaw.com
11. How do you make a plan?
Christopher.Janoch@pogolaw.com
12. A Team-Oriented Approach is Needed
Too many people involved and nothing gets done.
Committee Effect
Christopher.Janoch@pogolaw.com
13. A Team-Oriented Approach is Needed
Not enough people and nothing gets done thoroughly.
Limited Focus and Few Opportunities
Christopher.Janoch@pogolaw.com
14. A Team-Oriented Approach is Needed
Department Management
Project Management
System Engineers
Support Teams
Business Representatives
Vendors / Consultants
Christopher.Janoch@pogolaw.com
15. A Team-Oriented Approach is Needed
Department Management
Project Management
BCP
System Engineers DESIGN
TEAM
Support Teams
Business Representatives
Vendors / Consultants
Christopher.Janoch@pogolaw.com
16. Ideas & Solutions are Directed to a Core Design Team
Business Function Application
Staff Users Representatives Engineers
User Support Application Manager
BCP Design Team
Contributions are added by users that know the
Applications and Business Functions the best.
Christopher.Janoch@pogolaw.com
17. BCP Compass Where are We?
Maintenance Analysis
Implementation Solution Design
Organizational Vendor
Acceptance Selection
Christopher.Janoch@pogolaw.com
18. BCP MAP Where Are We Going?
User Access
Data
Application Services
Infrastructure Services
Communications
Environment
Christopher.Janoch@pogolaw.com
19. BCP MAP Where Are We Going?
User Access
Data
Application Services
Infrastructure Services
Communications
Environment
Christopher.Janoch@pogolaw.com
20. BCP MAP Where Are We Going?
User Access
Technology Stuff
Christopher.Janoch@pogolaw.com
21. BCP MAP Where Are We Going?
User Access
Data
Most Design Work Application Services
Infrastructure Services
Most Communications
Troublesome
Environment
Christopher.Janoch@pogolaw.com
22. Determine your Recovery Objectives Maintenance Analysis
Implementation Solution Design
Organizational Vendor
Acceptance Selection
In some cases, the SERVICE is top priority
In others the DATA is more critical
Don’t Rely on IT’s judgment alone!
Christopher.Janoch@pogolaw.com
36. Protection Strategy Decisions Maintenance Analysis
Implementation Solution Design
Organizational Vendor
Acceptance Selection
Centralized Services vs. Autonomous Offices
Centralizing proved to be more affordable, easier to design, and
much easier to maintain
Automation vs. Manual Processes
Automation simplifies crisis management, but adds new risks
Christopher.Janoch@pogolaw.com
37. Costs must be Contained, Predicted, & Controlled
Remember to account for Passive Infrastructure for *every*
Replicated System.
Beware the cost of adding too much redundancy.
Christopher.Janoch@pogolaw.com
38. The Solution MUST Survive in the Real World
“The more they over think the plumbing, the easier it is
to stop up the sink” – Scotty (Star Trek)
Christopher.Janoch@pogolaw.com
39. Understand Service & Application Interdependencies
Test each system before declaring it PROTECTED
Christopher.Janoch@pogolaw.com
40. Understand Service & Application Interdependencies
Test each system before declaring it PROTECTED
Christopher.Janoch@pogolaw.com
41. Understand Service & Application Interdependencies
Test each system before declaring it PROTECTED
Christopher.Janoch@pogolaw.com
42. Understand Service & Application Interdependencies
Test each system before declaring it PROTECTED
Christopher.Janoch@pogolaw.com
43. Understand Service & Application Interdependencies
Test each system before declaring it PROTECTED
Document Dependencies for Future Growth and Design Changes
Christopher.Janoch@pogolaw.com
44. BCP MAP Powell Goldstein’s Map
User Access
Survivable Remote Access
Data Data Replication
Application Services Virtualization
Infrastructure Services Redundant Servers
Alternate Service Providers with
Communications
Diverse Paths
Geographically Separate
Environment
Datacenters
Christopher.Janoch@pogolaw.com
54. Testing & Vendor Selection Maintenance Analysis
Implementation Solution Design
Organizational Vendor
Acceptance Selection
Manufacturer designs may not apply in your environment
“Your System” will *always* be an exception!
Christopher.Janoch@pogolaw.com
55. Testing & Vendor Selection
Products that look similar may not perform the same way.
Don’t be afraid of the “little guy”
Support Statements to cover your implementation and environment
are the Key to reducing future problems.
Take the time to compare alternative solutions!
Christopher.Janoch@pogolaw.com
56. You can’t learn “Everything about Everything”
Vendor implementation and “Health Checks”
Experienced Consultants
Continuing Support Contracts
Recognize when you need to hire assistance!
Christopher.Janoch@pogolaw.com
57. Organizational Acceptance Maintenance Analysis
Implementation Solution Design
Organizational Vendor
Acceptance Selection
A Business Continuity Plan is an Investment
A Business Continuity Plan is a Marketable Commodity
When Projects are embraced at the top of an organization,
they are more easily accepted at the bottom.
Christopher.Janoch@pogolaw.com
58. The Slowest Adaptors may be those closest to you!!
The IT Department:
Non BCP-Compliant Projects
Hesitancy to trust the System
“Reasonable” Conversion Delays
The “Local” Pilot Group
The Blur between Development & Production
Christopher.Janoch@pogolaw.com
59. Implementation Maintenance Analysis
Implementation Solution Design
Organizational Vendor
Acceptance Selection
Work your Map from both the Top and Bottom.
Starting with the Root Dependencies, protect each service
completely before moving to the next.
Your next Outage will not wait for you to be ready!
Christopher.Janoch@pogolaw.com
60. Take advantage of Redundant Systems
Buy New
The new servers become your test lab and
allow you to isolate the systems during implementation.
Avoid the “Re-wiring the House Live” syndrome
Christopher.Janoch@pogolaw.com
61. The System, Design, and Plan will continue to Change
New services will be added to the system.
Assumed RTO’s will be proved Incorrect.
Technologies will be updated.
Designed Solutions won’t work as Planned.
Don’t upgrade BCP Key Components without Testing!
Christopher.Janoch@pogolaw.com
62. Implement Change Management
NOTHING changes without knowledge and approval
EVERYTHING that changes gets documented
Identify who will be responsible for the
Implementation and Testing of which systems.
Standardize Quality Control Checks and
Officially Scheduled Tests.
Clear Processes for Updates, Changes, and Re-Builds must be
Documented and easily available.
Christopher.Janoch@pogolaw.com
63. Ongoing Maintenance Maintenance Analysis
Implementation Solution Design
Organizational Vendor
A New Mindset: Acceptance Selection
Business Continuity: Compliant or Not?
Regular Testing
Don’t just TEST….. USE!
Christopher.Janoch@pogolaw.com
64. BCP Navigational Tools
User Access
Maintenance Analysis
Data
Application Services
Implementation Solution Design
Infrastructure Services
Communications
Organizational Vendor
Acceptance Selection
Environment
!!!! OUTAGE !!!!
RPO RTO
MONTHS DAYS HOURS MINUTES SECONDS SECONDS MINUTES HOURS DAYS MONTHS
Christopher.Janoch@pogolaw.com