The document discusses the challenges that retail companies face with database downtime and changes. It notes that downtime can result in significant lost revenue for retailers. Common database development practices like using scripts are outlined as being difficult to maintain and not providing adequate change control. The solution, DBmaestro TeamWork, is presented as providing database version control, enforcement of best practices for database changes, automated deployments between environments, and overall improved productivity and quality.
2. Before We Begin
• You will be on mute for the duration of the event
• We are now talking so please type a message
in the Questions box in the Control Panel if you can’t
hear us (please check your speakers and
GoToWebinar audio settings first)
• If you have questions during the session, please submit
them on the Q&A bar on your gotowebinar dashboard
• A recording of the full webinar will be put up online
2
4. About DBmaestro
• Founded in 2008, product launched in 2010
• Founded by Yariv Tabac and Yaniv Yehuda
• Headquartered in Israel, Global Operations
5. • Retail challenges
• The database uniqueness
• Real life tales
• Common approaches analysis
• The solution from DBmaestro TeamWork
5
Agenda
6. Retail’s Unique Challenges
Retailers are smack in the middle of three to five
years of major changes that will redefine the industry.
I would say the last time there was this big [of] a change in
the industry was in 1960, when many of the big boxes
opened up and really opened the huge discount department
store. The biggest change before that - the invention of
modern retailing in postwar America.
Alison Paul, vice chairman and U.S. retail and distribution leader at
consultancy Deloitte
6
“
7. Retail’s Unique Challenges
7
• Business
• Downtime = Loss revenue
• Heavily regulated – large amount of customer data
stored across multiple locations
• Migration from physical stores to web-based online
stores using Augmented Reality experiences
• Freeze toward year-end changes for fear of download
and delaying agility because of lack of confidence in
process
• Rapidly change & Agile market needs in a highly
seasonal environment – downtime is critical
• Streamline development & operations team
8. Online Gaming Unique Challenges
8
• Technical
• Complex architecture and internal dependencies
• Responding quickly to business changes
• 3rd party integrations
• Same metadata across many environments
• Heterogenic environments
• Changing applications structures from packaged
applications to manage physical spaces and stock
to more customized/home grown web-facing
applications
• Streamline geographically diverse development &
operational teams
9. Why Manage Change?
More than
50% errors
9
Copyright@2008, Juniper Networks, Inc.
50%
of unplanned
downtime is
due to
Change
80%
of this, half is
due human
40% of changes fail
10. DECM & ALM
• DECM - Database Enforced Change Management
• Part of ALM (Application Lifecycle Management)
10
solutions
11. Database is a Key Component
• The database holds your essential
information
• Changes can impact the entire system
• Need to be synchronized with other changes
• Often overlooked
11
12. Reaching Inside the Database
• There is more to a database than SQL scripts
• Schema structure
• Code
• Content and meta-content
• Internal dependencies
• …
• Ensure that changes are not made without
authorization
• Ensure no out-of-process changes
12
13. The Weakest Link In a Chain
• Old adage but true
• The database is often “forgotten” and
therefore can become the weakest link
• Essential from a compliance point of view
• Should be the strongest link
13
14. “
“
The Database is a constraint
It was difficult to track who made a change to a
database object and what change they made.
(working-around and failing to enforce usage of file based version control)
Sr. DBA @ Large USA Bank
It took hours to get releases working. some changes
were not documented and left out. we actually preferred
crashes in integration. It is much worse when something
works, but works wrong. in production…
(manual and error prone releases. Change control & change scope issues)
Sr. R&D Manager @ Credit Card company
15. The Database is a constraint
We recently had a disaster - the script in the version
“
control was not updated and when executed in production,
ran the wrong revision. That cost tens of thousands of $
(an out-of-process update to QA that was not properly tracked. Same for integ/stage/prod…)
DBA @ Algo Trading company
We had multiple releases to production every day. That
“
is one release a week with multiple follow up fixes, and yet
more fixes
(code overrides, partial versions, wrong versions – all pushed to production)
CTO @ Credit Card company
16. The Database is a constraint
We had an incident where a trigger was not correctly
“
implemented during a code release. a trigger from a
previous build was used instead which was only detected
on Tuesday morning on the first business day after our code
release. this was a customer-facing application and made
our team look, and feel, bad about the release.
we realized that we needed to bring more discipline and
rigor to our database changes.
(manual process are hard to repeat over and over without errors)
Sr. DBA @ Large USA Bank
17. Common DB Development Methods
• Treating database scripts as native code
17
• Small scripts
• One large script
• Automatically generated scripts (Compare & Sync)
18. Two isolated Processes
Version Control Process
A
18
(file based)
Development Process
(real database)
Check-Out
Script
Modify Script
Get updated
Script from DB
Check-In
Script
Compile
Script
in DB
Debug Script
in DB
?
?
?
?
A’
Version control repository is out of sync from the tested
database and cannot acts as Single Source of Truth
19. Scripts & Version Control
19
• Challenges…
• Working on the wrong revisions
• Code-overrides
• Scripts do not always find their way to the version control
solution
• Out of process updates go unnoticed
• Hard to locate outdated update scripts
• Playing safe? what we really need (manual steps):
• The actual code of the object
• The upgrade script
• A roll-back script
20. Scripts are static…
• Scripts, unless super sophisticated:
• Unaware of changes made in the target environment
• Time passed from their coding to the time they are run
• Potentially overriding production hot-fixes or work done
in parallel by another team
• Content changes are very hard to manage
• Metadata & lookup content does not practically fit into
20
the VC
• In most cases they are simply not managed
21. Testing Scripts
• Single object based scripts
• Hard to test in their entirely (holistically)
• Hard to test due to colliding dependencies
• Need to run in a specific order…
• Large multi object based script
• Represents the entire update - can deal with
21
dependencies
• Much harder to deal with project scope changes
• Hard to mange – a big list of commands
22. Build Once Deploy Many (Scripts)
Int QA Pre Prod Prod
22
X
1.1234567 1.1.1
Database Deploy Script
Re-Base (due to defects)
Environment
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1..147 1.1
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
Out of Process
Change
X
X
X
X
X
? 1.1.1
X
23. Scripts – Bottom Line
• Overhead time
• No trust in the version control repository
• Difficult and very complex to maintain
• No agility to business changes
23
24. Compare & Sync
24
• Challenges…
• Handling out of process changes
• Handling parallel development
• Handling conflicts
• Not an integral part in development phase - version
control repository is NOT the single source of truth
Source vs.
Target
Action
= No Action
≠ ? We do not have all of
the information
25. Build & Deploy On Demand
(Compare & Sync)
Int QA Pre Prod Prod
27
1.1234567
1..147 1.1 1.1
*
Database Deploy Script
* Execute the same script Environment
being executed at the
Stage environment
Re-Base (due to defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1.1 1.7
File Based
Version Control
Out of Process
Change
1..17.1 1.17.1
26. Deployment Automation Safety Net
An index exists in Target (Production) but not in
Source (QA). What should we do? Drop the index or not?
28
Baseline Aware Analysis
Source vs.
Target
Action
= No Action
≠ ?
Source vs.
Baseline
Target vs.
Baseline
Action
= = No Action
≠ = Deploy Changes
= ≠ Protect Target
≠ ≠ Merge Changes
You do not have all
of the information
With Baselines and 3 way
analysis the unknown is now
known
Simple Comp & Sync
30. Safety Net Deployment Automation
• Raise red flags on conflicts
• Support out-of-process changes
• Utilize baseline aware analysis
• Understand the nature of the changes
32
Source vs.
Baseline
Target vs.
Baseline
Action
= = No Action
≠ = Deploy Changes
= ≠ Protect Target
≠ ≠ Merge Changes
31. Benefits – By Role
33
• Development
• Database change repository
• Follow SCM best practices (Check-Out/Check-In)
• Documentation of who did what, where, when & why
• Operation
• Integrated deployment engine
• Business level audit
• Roles & responsibilities enforcement
• Management
• Complete visibility into changes in progress
• Management reports
• No silos
32. Other Benefits
34
• Productivity improvements
• No manual work and less reworking
• No time spent in testing deployment scripts
• Supports parallel development
• Quality improvements
• No code overrides
• No promoting wrong code
• Supports partial releases
• Less deployment failures
• Accurate & complete deploy script
• Supports task based development
• Auditing & Regulations
• A complete documentation of who did what,
where, when & why
33. Testimonials - Efficiency
Focusing on changes rather than managing
changes and dealing with re-work, boosted overall
productivity of 200-300 developers. We estimate
we were able to do 15% more with the same
resources.
We went from several fix-centric deployments a
day, to one feature-centric deployment a week.
The amount of incidents in production has declined
as well. We had 20% less incidents.
CIO @ Credit Card company
35
“
34. Regulation requirement - SOX & derived 357
Auditing - change management approach with
change management auditing. You have to do it
anyway, do it automatically and efficiently.
Change management in production itself is
regulatory required (ITIL). But you cannot ensure it
without managing the whole process starting at
Dev.
We have to comply to regulation - but the business
benefits from it.
CIO @ Credit Card company
36
“
Testimonials - Regulation
36. Customer Testimonials
38
“Massachusetts Department of Education uses
DBmaestro TeamWork to reduce database
change deployment time by up to 95%”
Mr. Dan Boyle, Supervisor of Database
Administration & Development
“Isracard has increased the throughput of our
development team immensely and along with that
our team is more motivated and satisfied on the job.
With the TeamWork™ proprietary Automation
Module we are able to deploy changes between
environments automatically and this reduces the
number of human errors and increases the
production system's stability”
Ofer Krichman, Head of Infrastructure Software
Department , Isracard LTD
“DBmaestro TeamWork facilitated the database
development process in the Ministry of Education in
the current IT development projects. The ROI of
DBmaestro TeamWork is doubtless...”
Mrs. Sofia Mintz, CIO
“By implementing the DBmaestro solution, We can
finally take control of our databases. By
Integrating the application alongside our source
control and continuous integration paradigm, We are
able to deploy our application within minutes.”
Mr. Jonathan Bar-Sela, Development Manager,
Clalbit systems
“DBmaestro TeamWork, provides us full control on
our databases environments. Having our schemas
sync between all environments made a big change in
the DBA’s and developers efficiency. TeamWork
integration to MS-TFS, provides us End-to-End
products development life cycle”
Mr. Shahaf Azriely, Director of IT, IncrediMail Ltd.
“By implementing TeamWork™, Sura finally has the
ability to apply the SCM best practices on
database development…Finally, Sura's database
deployment will be quick and efficient with an ideal
rollback option.”
Juan David Perez Chief Architect, Sura Insurance
37. 39
Summary
• Enforce best practices for database development
• Closely manage your database development
• Know who can do what, where, when & why
• Build a safety net to catch problems early
• Reduce deployment issues
• Enable automated deployments
39. 41
Integrations
• ALM Solutions
• TFS
• RTC
• Task Management Systems
• Clear Quest
• Perforce
• JiRA
• TeamForge
• Any tool, using open API
• Continuous Integration
• Jenkins
• Bamboo
• Any tool, using web-services
• Release Automation
• IBM UrbanCode Deploy
• CA Release Automation
• Serena
• BMC DevOps
• Any tool, using web-services
• IDEs
• Oracle SQL Developer
• SQL Server Management Studio
• Visual Studio
• JDeveloper
• Other Database Tools
• Delphix (Virtual Branches)
• DM Suite (Content Masking)
• Oracle SQL Developer (Unit Test
& Code Coverage)
40. Merge, Rebase, Fork, …
42
Merge to dev1 Dev1
Dev2
Trunk
DB
VC
Fork
Fork
Fork
Fork
41. Use of Baseline
4 5 6 9 10
Before MS 2.1 MS 2.2
43
Dev1
Int / QA
Ver 1.0
1
Ver 1.0
Label/Tag/Baseline
Backup Label/Tag
Merge
MS 1.1
Merged - B
Before MS 1.1
2 3
MS 1.1
Merged - M
Branch
Merging
MS 1.1
MS 1.1
MS 1.2
Merged - B
Before MS 1.2
Branch
Merging
7 8
MS 1.2 MS 1.2
Merged - M
MS 1.2
Dev2 Ver 2.0 MS 2.1
Merged - B
Merged - B
Before MS 2.2
MS 2.1
Branch Merged - M
Merging
MS 2.1
Branch
Merging
MS 2.2
MS 2.2
Merged - M
MS 2.1 MS 2.2
1 Int / QA (Live) Dev1 …
2 Int / QA (MS 1.1) Dev1 (Live) Dev1 (Ver 1.0)
3 Dev1 (MS 1.1 Merged - B) Int / QA (Line) Int / QA (MS 1.1)
…
Notas del editor
80% of outages impacting mission-critical services caused by people and process issues thru 2015, with the majority of those outages (50%+) caused by change/configuration/release integration and hand-off issues (Gartner RAS Core Research Note G00208328 Ronni J. Colville, George Spafford [October 27, 2010] – Strategic Planning Assumption(s) “Top Seven Considerations for Configuration Management for Virtual and Cloud )