Many organizations have taken comprehensive steps to design, implement and test disaster recovery plans. The elements of a real disaster event can introduce many unexpected twists, turns and detours in the recovery path to resuming normal business operations.
In this webcast, Richard Dolewski, VP Business Development and Disaster Recovery at Velocity Technology Solutions will explore:
- Why downtime is a business issue, not just a technology issue;
- Why even the most comprehensive, well-contemplated disaster recovery plans often fall short when they’re put into real-life practice;
- The importance of eliminating single points of failure at the levels of people, technology and geography; and
- The benefits of more modernized approaches including cloud-based disaster recovery.
This webcast will draw on real-world customer examples and show how disasters can unhinge even the most iron-clad disaster recovery plans. Don’t wait for an actual disaster to take you down. Attend this webcast to glean valuable lessons learned from others, and better understand how you can protect your business.
How to Uninstall a Module in Odoo 17 Using Command Line
This is a disaster declaration! This is not a test.
1. This Is a Disaster Declaration!
This Is Not a Test.
Richard Dolewski
VP of Business Continuity & Business Development
Velocity Technology Solutions, Inc.
2. Velocity
Technology
Solutions
Velocity – the ERP in the cloud
company – hosts and manages
ERP, related applications, and disaster
recovery services within its virtual
private cloud environment. Aim is to
lower operational costs, increasing
service levels and improving software
performance while reducing the
burden on IT staff and enabling
deployment flexibility. Our team has
exceptional expertise managing
enterprise software 24/7. Combined
with our proprietary technology and
superfast infrastructure, we increase
application availability, security, and
control. As a result, our customers are
better positioned to accelerate
enterprise growth and agility, attract
and retain customers, and reduce
cost, turning their ERP platforms into a
strategic asset. Velocity is
headquartered in New York City.
www.velocity.cc
2 March 27, 2013 Velocity Proprietary and Confidential
3. Case Study:
Disaster Declaration
The Storyline:
• A real-world disaster and how events unfolded
• Company is in the Manufacturing business
– Corporate data center supports several sales
offices, distribution facilities and manufacturing plants
throughout the U.S.
• I will refer to this organization as The Zed Company
3
3 March 27, 2013 Velocity Proprietary and Confidential
4. What is a Disaster ?
ANYTHING !!!!
That stops your business from functioning & that cannot be
corrected within an acceptable amount of time….
4
5. What is Disaster Recovery?
Reaction to a sudden, unplanned, event that enables an
organization to continue critical business functions until
normal business operations resume.
“…It is not enough to arrange for hardware replacement;…
planning must address continuation of business
operations, or business continuation.”
5 March 27, 2013 Velocity Proprietary and Confidential
6. Companies Regularly Deal with Natural and
Man-Made Disasters
Risk is increasing Disasters happen
Forrester Research
8. What Constitutes a Disaster in Your
Organization?
Disaster Recovery deliverables are based on:
RPO - Recovery Point Objective
RTO - Recovery Time Objective
8 March 27, 2013 Velocity Proprietary and Confidential
9. The Products of a DR Plan
Who will execute recovery actions
What is needed to continue, resume, recover or restore
business functions
When business functions and operations must resume
Where to go to resume corporate, business & operational
functions
How detailed procedures for continuity, resumption, recovery or
restoration
CLASSIC: WHO-WHAT-WHERE-WHEN-HOW
9 March 27, 2013 Velocity Proprietary and Confidential
10. Disaster Scenario
THE ZED COMPANY
10 March 27, 2013 Velocity Proprietary and Confidential
11. Profile: The Zed Company
• Last DR test 8 months ago
• 100% successful - 17 hours
• A special save to facilitate DR test audit
• Have never tested a mid week recovery
• Recently switched to a new software backup
solution
• No Generator (Not in budget)
11 March 27, 2013 Velocity Proprietary and Confidential
12. Profile: The Zed Company
• Primary Technical Owner of DR
conducted last two DR tests in Phoenix
• Alternate person ran operations back
at the office
• Tapes sent offsite daily
• RTO 48 hours
• RPO 24 hours
12 March 27, 2013 Velocity Proprietary and Confidential
13. The Disaster:
The largest power failure in California history
14. Startup Issues: The Zed Company
• Person declaring disaster “Not authorized“ and
unfamiliar with password activation procedures.
• Hot-site begins preparation of hot site, but requests
authorized individual to call within the next hour
14 March 27, 2013 Velocity Proprietary and Confidential
15. Startup Issues: The Zed Company
• Hot-site equipment is available 4 hours after initial
call
• Zed Company still trying to locate authorized staff
(Alternates not available or no longer employed )
• Disaster Declaration fee of $7500.00. What’s this?
• Usage Charge ? What’s this
15 March 27, 2013 Velocity Proprietary and Confidential
16. Startup Issues: The ZED Company
• Staff unable to recall tapes en-route
to hot-site
• Offsite Storage Vendor will not
accept phone requests “No phone requests.
You must sign on to
website to recall tape
media”
No App for retrieving your tapes!
16 March 27, 2013 Velocity Proprietary and Confidential
17. Startup Issues: Zed Company
• Declared a disaster, but no one
shows up ! Somewhere en-route
from SD
• Waiting on staff to arrive – Unable to
locate client. Voice mail. 8 Hours
(Dead cell Phone)
• Client returns call 2 hours later!
• Power is slowly coming on in parts of
the city
• He has heard his Power may come on
shortly
10 Hours
17 March 27, 2013 Velocity Proprietary and Confidential
18. Startup Issues: The Zed Company
• Hot-site suggests tapes get shipped to start recovery
just in case. Client agrees.
• Client does not know which tapes are required.
• All Backup Software recovery reports are kept on
the system
• Suggest sending last full & all of this weeks dailies.
Hot-site staff try to recover
12 Hours
18 March 27, 2013 Velocity Proprietary and Confidential
19. Startup Issues: The Zed Company
• Tape media arrives - a box of 14
• Backup tapes in NO particular order
• No media report arrived with tapes
• Which one contains the catalog?
• Staff loads all tapes into library, inventories using
backup software
• DR Plan missing at hot site - Alternate person
going back to site to obtain the plan
13.5 Hours
19 March 27, 2013 Velocity Proprietary and Confidential
20. Startup Issues: The Zed Company
• Signed into Offsite Tape Vendor website to
determine what backup sets arrived at hot site
• Found tape catalog as labeled in offsite tape vendor
inventory
• Unable to restore tape catalog – “invalid catalog
error”
• Manually cataloged each tape
• 6.5 hours wasted!
20 Hours
20 March 27, 2013 Velocity Proprietary and Confidential
21. Startup Issues: The Zed Company
• Primary Technical Staff person now en-route to hot-
site to assist
• DR Procedures arrive with Primary person
24 Hours
21 March 27, 2013 Velocity Proprietary and Confidential
22. Startup Issues: The Zed Company
• Management are very uptight…Name calling ……
• Every action / or decision is now being questioned
• Pressure is probably at a peak as Plant locations are
asking when ???
22 March 27, 2013 Velocity Proprietary and Confidential
23. Startup Issues: The Zed Company
• Tapes were cataloged, fortunately the data was found
• Start recovery of rest of the system
32 Hours
23 March 27, 2013 Velocity Proprietary and Confidential
24. Startup Issues: The Zed Company
• Senior management (not IT) summons
lead Technical person back as power is
now coming on across the street. Power
is expected anytime now.
• Declaration is HALTED & tapes are
requested back. Against
recommendations!
34 hours
24 March 27, 2013 Velocity Proprietary and Confidential
25. Back on Again: The Zed Company
• Client calls back 4 hours later requesting the
restore to continue. Power is still an issue.
• Hot site informs client the system has been scratched
and they must start all over again.
• Client will bring tapes to begin reload.
38 Hours
25 March 27, 2013 Velocity Proprietary and Confidential
26. Back on Again: The Zed Company
• No staff are available. Everyone exhausted at this
point.
40 Hours
26 March 27, 2013 Velocity Proprietary and Confidential
27. Back on Again: The Zed Company
• Business wants a guarantee the
system will be up in 8 hours.
Business orders severely affected
• Issues - Client didn’t maintain
their backup environment
• First test since new backup
software installed
27 March 27, 2013 Velocity Proprietary and Confidential
28. Back on Again: The Zed Company
• Restore starts immediately, this time with tapes
already cataloged and DR documentation in hand
• System is restored in 17 hours
• Startup procedures incomplete, but manage to get
ERP services up.
62 Hours
28 March 27, 2013 Velocity Proprietary and Confidential
29. System Ready: The Zed Company
Testers have given the green light.
We are back in business.
29 March 27, 2013 Velocity Proprietary and Confidential
30. Report Card: The Zed Company
• Client declares and stops communicating with team
• Indecisive
• Backup policies not documented. A mystery to all!
• New Backup Software recovery never tested
• Backup Software reporting not sent offsite
• Tapes shipped. Documentation incomplete
30 March 27, 2013 Velocity Proprietary and Confidential
31. Report Card: The Zed Company
• Wrong person sent to recover system
• DR Plan unavailable
• Senior Management Pressure
• Staff in a state of flux - Coming & going
• Halted recovery
• Restart all over again
• Lack of staff delegation & training
• No Generator (Now in budget)
31 March 27, 2013 Velocity Proprietary and Confidential
32. Report Card: The ZED Company
Does your business have a say?
Do the bean counters know what
IT can deliver ?
32 March 27, 2013 Velocity Proprietary and Confidential
33. Report Card: The Zed Company
• Management shocked that the IT folks lost all orders
• 24 Hour RPO - Who agreed to this?
• System was not available for 2 days
• 48 Hour RTO - Who agreed to this?
33 March 27, 2013 Velocity Proprietary and Confidential
34. What about Your Email? The Zed Company
• Email has become Mission
Critical!
• Email is how you communicate
with customers, partners, and
employees.
• In a disaster situation, this may
be a key component of your
communications strategy.
• Is Email listed as a critical
application for your organization
and included in your disaster
recovery plan?
34 March 27, 2013 Velocity Proprietary and Confidential
35. Modernize
Your Recovery
35 March 27, 2013 Velocity Proprietary and Confidential
36. No one wants to be
in a position of
explaining why your
organization lost its
data.
36 March 27, 2013 Velocity Proprietary and Confidential
37. Thank you for attending today:
THIS IS A DISASTER DECLARATION.
THIS IS NOT A TEST!
37 March 27, 2013 Velocity Proprietary and Confidential
38. Presenter:
Richard Dolewski
VP of Managed Disaster Recovery
Services & Business Development
richard.dolewski@velocity.cc
206- 436 -3321 mcpressonline.com
velocity.cc
866-638-2779
38 March 27, 2013 Velocity Proprietary and Confidential
Notas del editor
A little bit about who we are. Managed DR services is one of three main, complementary business focuses at Velocity. For many companies, DR planning is not just an insurance policy: it’s an imperative. It is serious business to protect operating revenue and the hard earned trust of your customers. Our managed DR services experts deliver recovery services through the application layer, not just for your data — so you are truly up and running with minimal effort — with scalable tape and electronic backup, data vaulting and high availability solutions in our cloud DR environment.
A disaster is anything that stops your business, whether the event is a man-made event or a storm. The truth is, little “storms” happen all the time. A disaster can be anything as big as Hurricane Katrina, to a localized power outage. Think of the chaos that can ensue when someone hits a telephone pole in the middle of your town, causing a light to go out and traffic to be re-directed. Now think of that in the context of your business, where the “drivers” are your customers, suppliers and partners. The reality is that no one is immune to the sheer randomness of disasters and the havoc they can create. And it’s this sheer randomness that you have to be prepared for.
Businesses regularly deal with disasters of all kinds and they’re more common that we’d like to think. According to a survey conducted by Forrester Research, 61 percent of organizations have enacted their business continuity plan at least once within the past five years, with 28 percent invoking such plans three times or more.