presentation in IBM Innovate 2013 in Orlando FL

1. WebMD Adopts Automated Deployment in Support of
Continuous Integration to Transform their SDLC
Teresa Dietrich, Vice President Technical Operations
Derek Chang, Senior Director

2. We were here…
2

3. So we assembled a team…
3

4. To fix our SDLC…
4

5. And now everyone is happy.
5

6. Technology @ WebMD
WebMD, Medscape, MedicineNet, eMedicine, UK cobrand
Serving nearly 1 Billion Pageviews a month, 132 million unique
Running over 200 separate applications, vast majority in-house developed
Environments: Dev/Devint, QA01/02, QA00, Production/DR
Two main data centers, geographically diverse
OS: mix of Linux and Windows
Datastores: Sql Server, Oracle, Mongo, Vertica and mysql
Web: mix of Apache and IIS
App: mix of Tomcat, ASP, .Net 3.x and 4.x
Service: ActiveMQ, Memcache
6

7. Before - State of Deployments
Build repository – a free for all NAS share; manual upload
Package management – HP proprietary package library; manual upload/download
Configuration management – HP proprietary CML
Deployment module – HP software policy
Deployment initiation – SBM ticketing system
Pre/post deployment tasks – custom scripts and manual steps
Dev
Sends
Email
QA
Opens
Ticket
Lead
Assign
Ticket
Engineer
Opens SubTickets
DBA
Runs
Script
Engineer
deploys
build
QA
Smoke
Tests
Ticket
Closed
7

8. Before – Deployment Metrics
year
total
# of work days
average
daily max
date
2011
2650
251
10.5
34
7/1/2011
2012
3174
251
12.6
50
9/20/2012
Average time to deploy: 55 minutes
8% of tickets rejected thus required clarification
Average build deployment per day: 11.6
Daily Max: 50
8

9. Before - Pain Points
Require multiple systems
Relies heavily on email for communication
Inconsistent environment setup
Only Ops did deployments in QA and Prod
Error prone
– Wrong information provided in ticketing system
– Tons of customized scripts
– Complexity of deployment procedures
– Manual steps involved in pre and post deployment
High maintenance cost
– Configuration management
– Host inventory
– Deployment scripts
9

10. Becoming Agile
Hired an Agile consulting group
Signed up for Software to manage Agile Dev
Trained Scrum Masters
Learned to write user stories, plan sprints
and used colored Post-Its
Scheduled daily
Stand-ups.
Woohoo, now we are officially
Agile!
10

11. Agile Implications for Deployments
11

12. Transform our SDLC
12

13. Waterfall
vs
Agile
13

14. Transform our SDLC
14

15. Promote Once
vs
Promote Continuously
15

16. Transform our SDLC
16

17. Static
vs
Elastic
17

18. Continuous Integration Team
Formed a CI Team w/ representatives from:
–
–
–
–
–
Dev
WebOps
QA
DBA
Platform Ops
Each team member had to have day to day hands-on
responsibilities.
Each team member had to have the authority to make
decisions on behalf of the organization they represented.
Each team member had to be able to commit to at least 4
hours a week .
Team reported back to Sr Mgmt at specified milestones.
18

19. Measuring Success
Automated Unit Testing
Build Ready to Build Deployed (by env)
Successful Build Deployment (by env)
Successful Automated Smoke test (by env)
Build Deployment Execution time (by env)
Build from Dev to Production time
19

20. Automated Deployment
Sr Management created a document with 1 page on
assumptions and 1 page on milestones to kick off the team on
their first assignment.
Assumptions included:
– Standards developed for: build, packaging, config, naming
and monitoring
– Roles and Responsibilities by Environment
– Deployment success and Environment Exit criteria
20

21. Evaluation Process
#
Procedure
Working
group
Management
P1
principles and assumptions
P2
define evaluation criteria
X
P3
define baseline functionalities and
features
collect list of products to be
evaluated
finalize product evaluation form
and list to be evaluated
X
research products based on
criteria
review/rank products
X
X
X
P9
P10
review and finalize list of products
for POC
vendor demo
In house POC
X
X
X
P11
review POC results
X
Description
X
P4
P5
P6
P7
P8
X
X
X
referrals, forums
X
5-7 products
vendor documentation, forums and
user community
X
POC should include must-have
functionalities defined in F11 below
2-3 vendors
1 product at a time
21

22. Most Important Features
Efficient configuration management
Host inventory
Capability of easy code promotion, backup and rollback
Comparison – artifact/environment/configuration
Orchestration engine – reusability and visibility
Capability to integrate with other systems
Support
22

23. Developing the Requirements
Guidelines from Sr Management
Review and understand the tools we were using.
In-depth analysis of current deployment process
Understand our current pain points.
Learn from what products are available on the market.
Learn from what other people are using and doing right.
23

24. Functional Requirements
#
Functionality
Category
importance
F1
F2
start application
stop application
operation
operation
must-have
must-have
F3
application configuration management - administration
operation
must-have
F4
F5
F6
F7
F8
F9
F10
application configuration management - versioning
application configuration management - comparison
application build deployment
application build rollback
fail-safe protection
customizable post-deployment operations
single interface for deployment-related operations
administration
administration
operation
operation
usability
operation
usability
must-have
must-have
must-have
must-have
must-have
must-have
must-have
F11 log/report for troubleshooting and auditing
administration
must-have
F12
F13
F14
F15
F16
integration
administration
administration
Operation
integration
must-have
must-have
must-have
must-have
must-have
integration with build server
Visibility into deployment operation
Event and hand-off Notification
Scheduled Deployment
Integration with configuration management solution
24

25. Functional Requirements
#
Functionality
Category
importance
F17 integration with ticketing system
F18 integration with revision control system
integration
integration
nice-to-have
nice-to-have
F19 integration with software builder
integration
must-have
F20
F21
F22
F23
F24
F25
F26
integration
integration
integration
operation
administration
administration
usability
nice-to-have
nice-to-have
nice-to-have
nice-to-have
must-have
nice-to-have
must-have
F27 deploy unbundled artifacts from multiple sources
usability
must-have
F28
F29
F30
F31
usability
integration
administration
usability
nice-to-have
must-have
must-have
must-have
Integration with testing automation solution
Integration with defect tracking solution
Integration with availability monitoring
database build deployment
security and access control mechanism
agentless architecture
native support for scripting language
mobile app deployment
clustering database connectivity awareness
resource inventory
Support both windows/Linux platform
25

26. Evaluation Score Card
Danny
3
Carrie
4
Matt
4
SRE
3
Ab
Raj
Yong
total
average
Vendor support
Vito
3
3
3
5
28
3.5
User community
5
4
4
5
4
4
3
5
34
4.25
Documentation
4
5
5
5
5
4
4
5
37
4.625
Cost for license*
1
2
2
1
1
1
2
2
12
1.5?
Cost for infrastructure
2
2
3
3
2
2
1
2
17
2.125
Effort for maintenance
3
4
5
5
3
3
1
1
25
3.125
Cost for development and
implementation
3
4
3
3
4
3
4
3
27
3.375
Interfaces such as API or
web services
5
5
5
3
37
**4.625
Functionality
Refer to table 1
5
4
5
4
5
5
5
5
38
4.75
Usability
Ease of use
4
4
4
4
4
4
5
4
33
4.125
Learnability
4
4
3
3
4
3
5
4
30
3.75
Product support
Cost (lower)
Integrability
5
5
5
4
* including derived maintenance charge and yearly support
** scale will be adjusted prior to vendor evaluation phase
26

27. Necessary Standards and Adjustments
Change of roles and responsibilities
All configurations go into uDeploy and standards across apps.
Standardize environment setup and rebuild if necessary.
Standardize naming
– Application/component/environment naming
– Versioning and release number
– Configuration key/value pairs
– Agent/resource naming
Automatic smoke testing
27

28. Identifying All Possibilities
Source: referrals, communities, and conferences
Up to 5 were proposed by each member
28 were identified as potential candidates
3 were given to each team member for initial screening
18 were eliminated for not meeting critical requirements
10 possible candidates moved to the second round
28

29. Choosing the Contenders
Evaluation criteria: function checklist and vendor documents
Priority ranked 10 remaining products
Top 5 for on-site demo and discussion
Each vendor was given criteria and use cases
Use scorecard to rate and major pros/cons comparison
Top 2 to participate on-site POC
29

30. Proof of Concept
POC designed by WebMD
Selected 2 most representative applications
Success criteria – use cases and functional requirements
Timeline – 5 days on-site and 2 weeks afterward
Participated in installation, design and implementation
Daily working session
30

31. Final Selection
Use case verification and documentation
In-depth review and discussion
Pros and cons comparison
The team voted unanimously and chose uDeploy!
Evaluation result was communicated to Sr. Mgmt for final
review
On-site training for the entire Technology organization
Demonstration along with highlights of the selection process
31

32. Demo
32

33. Key Benefits of uDeploy
Usability, everything accessible in one tool
Integration capabilities to existing SDLC tools
Application Configuration Management
Ease of promoting builds through environments
Orchestration of 3rd party tools and reuse of templates
Visibility – process, troubleshooting, reporting, approval and
notification
33

34. After – Deployment Metrics
Duration March/4/2013-June/3/2013
75 components developed – around 30% implemented
419 agents in use
1070 build deployments
– 65 work days
– 15.1 per day
– Daily max: 43 on May/29/2013
Average - End to end time – 3 minutes 11 seconds
34

35. Lessons Learned
Change is hard - redefining roles and responsibilities
Bottleneck just shifted forward, on to the next challenge.
DevOps culture - collaboration between Dev, Ops and QA.
Ease of use, less technological expertise required
When a piece of software is the centerpiece of your SDLC, it’s
worth the work to find the right solution.
Cross functional teams work well when they have clear
direction and authority from management.
Integration with other tools is key.
Defining the process, policy and standards are as important as
selecting the right tool.
35

36. And now how here we are….
36

37. Appendix 1 – roles and responsibilities
#
SDLC environment
Devint
QA01/QA02
QA00
Production
t1
t2
Environment Prep initiation (ticket or email to identify need)
capacity evaluation/deployment target designation for new application
Dev
Dev/Ops
QA
Dev/Ops/QA-PERF
QA
Dev/Ops/QA-PERF
Ops
Dev/Ops/QA-PERF
t3-1
t3-2
Dev
Dev
ProdOps
ProdOps
ProdOps
ProdOps
ProdOps
ProdOps
t3-3
t3-4
t3-5
t4-1
t4-2
t4-3
t4-4
t4-5
Server provisioning request
database request
mssql
storage request
DNS/VServer provisioning request
middleware installation
setup new uDeploy component template for applications
update existing uDeploy component template for applications
application related config variables/definitions
app related config values*****
middleware related config variables/values*****
Dev
Dev
HPSA - ProdOps and dev self service
Dev; signed off by Ops/QA
Dev; signed off by Ops/QA
Dev; signed off by Ops/QA
Dev
Dev
ProdOps
ProdOps
ProdOps
n/a
n/a
n/a
QA
ProdOps
ProdOps
ProdOps
ProdOps
n/a
n/a
n/a
ProdOps
ProdOps
ProdOps
ProdOps
ProdOps
n/a
n/a
n/a
ProdOps
ProdOps
t4-6
t5
t6
t6-1
t6-2
t6-3
t7
snapshot
approval for deployment
people who push Deployment button
Troubleshoot if deployment fails
Automatic smoke testing
Troubleshoot if smoke testing fails
Testing and Sign-off
snapshot implementation will be based
on discussion with professional service
Dev
Dev
Dev
uDeploy/Selenium
n/a
Dev
QA
QA
QA
uDeploy/Selenium
QA
QA
QA
ProdOps
ProdOps****
uDeploy/Selenium
ProdOps****
QA
Ops management
ProdOps
ProdOps****
uDeploy/Selenium
ProdOps****
QA
*template includes component and process templates, config key/value pairs
**e.g. functional testing for QA01/02; UAT for QA00
***based on promotion approach; will revisit during on-site training as well as professional service engagement
****party initiates communication and troubleshooting, which will involve Dev/Ops/QA
***** values will be contributed by all parties
37

38. Appendix 2 – provisioning process
#
1 Setup uDeploy access** and perform uDeploy upgrade
Step
initiator
CI
2 Setup devint based on Ops standard
Dev
2.1 Provision new VMs - devint
Deploy infrastructure software and servers
2.2
(IIS/Tomcat/Memcached/ActiveMQ) -devint
2.3 Provision DNS and Netscaler rules - devint
owner
PlatformOps
ticket
firstaid ticket#0 platformOps uDeploy ticket
Dev
SiteOps
firstaid ticket#1 sysops host provisioning
Dev
ProdOps
firstaid ticket#2 ProdOps general
Dev
ProdOps
firstaid ticket#3 ProdOps general
2.4 Provision Storage - devint
Dev
StorageOps
firstaid ticket#4 sysops storage provisioning
2.5 Provision DB - devint
Dev
DBOps
firstaid ticket#5 DBOps general
Dev
Dev/ProdOps
3.1 naming standardization
Dev/ProdOps
Dev/ProdOps
3.2 Component mapping
Dev/ProdOps
Dev/ProdOps
3.3 Server inventory
Dev
ProdOps
3.4 Review/confirm server inventory
Dev/ProdOps
Dev/ProdOps
3.5 Install uDeploy agents
Dev/ProdOps
Sysadmin
3.6 Component/environment mapping
Dev/ProdOps
Dev/ProdOps
3 Application/component/environment population
Dev/ProdOps
Dev/ProdOps
4.1 Collect/Review/consolidate application config
4 Application config
Dev/ProdOps
Dev/ProdOps
4.2 Designate Config namespace
Dev/ProdOps
Dev/ProdOps
4.3 Import config
Dev/ProdOps
Dev/ProdOps
4.4 Build server configuration
Dev
Dev
Dev
Dev/ProdOps
Dev
Dev/ProdOps
Dev
Dev/ProdOps/DBOps
Dev/QA/DBOps/ProdOps
Dev/QA/DBOps/ProdOps
QA
firstaid ticket#6 sysops general
Dev/ProdOps
5 Process Development
5.1 Develop Application Process if necessary
Develop or Reuse component template (IIS/Tomcat/Windows
5.2
service/DB deployment,…,etc)
Designate roles for each environment (configuration, deployment,
6
approval)***
7
Set up additional notification if necessary; with product email alias
** provision user accounts and assign members to each privileged groups as shown in table 2 below
*** admin for each environment will be responsible to designate roles
38

39. Appendix 3 - guidelines
Creation of automation templates should be done at the rollout of the tool and not for each product or
application. Types of templates may include Applications, Configurations and DB changes. Before a build
is automated for the first time through the tool in DevInt, an existing template must be identified or a new
template must be developed.
If a new template is needed, there should an agreement that none of the existing templates are
appropriate and involvement from a cross functional team to develop a new one.
The Automated Deployment Process begins in the DevInt environment.
The exit criteria from each environment should be at least one successful automated deployment using
the prescribed tool. Success is defined as a successful unit test in DevInt or a successful smoke test in QA.
Until the tool is identified, assumption is that the configuration lives with the build. When QA is
referenced below, the assumption is that it is a member of the new Deployment Management team not a
general QA analyst.
We need to define, review and agree to a set of standards with regards to builds, build packaging,
configurations, naming and monitoring to ensure the success of the automation deployment process.
The groups listed below each environment have the responsibility of creating and executing the
deployment and first level troubleshooting within that environment.
39

40. Appendix 4 – demo summary
Application: https://deploy.webmd.net/#application/73d7afd9-ca16-41d8-b67c-5ec00857d0c2
– Quick summary of state of component/environment and configuration
Component: https://deploy.webmd.net/#component/ac0dbc11-ce86-4b87-9623-661575815d2f
– History view of who did what where.
Environment: https://deploy.webmd.net/#environment/ac5bcdf6-c412-4ab4-82b9-94c5df514ee4
– Where you see version history and request details
Host inventory https://deploy.webmd.net/#environment/ac5bcdf6-c412-4ab4-82b9-94c5df514ee4/inventory
Application property https://deploy.webmd.net/#application/d7de8f89-d861-458d-81c5-1778f7360dcb/properties
Component property https://deploy.webmd.net/#component/ac0dbc11-ce86-4b87-9623-661575815d2f/properties
Environment and environment-component property https://deploy.webmd.net/#environment/ac5bcdf6-c412-4ab4-82b994c5df514ee4/properties
Global/system property: https://deploy.webmd.net/#system/properties
More to come – process, resource, version - properties/configuration management
Application process https://deploy.webmd.net/#applicationProcess/222e0a13-874d-4df1-b5ac-87e69af6f360/13
– Manage component operations during deployment – sequence, lock, decision point and approval
Component process https://deploy.webmd.net/#componentProcess/9b7208f5-3190-486c-994e-0922075f7d28/43
– Where deployment actions take place – stop iis, check application pool, download artifact, search and replacement,
start iis
– Component process template
Environment specific process https://deploy.webmd.net/#environment/08dd45de-a528-46ee-b07e5bbf527616c4/approvals
Plugins
Rest API
Everything is versioned!
Snapshot https://deploy.webmd.net/#application/d7de8f89-d861-458d-81c5-1778f7360dcb/snapshots
40

41. Appendix 5-1: application view
41

42. Appendix 5-2: process template
42

43. Appendix 5-3: component view
43

44. Appendix 5-4: component process
44

45. Appendix 5-5: Component property
45

46. Appendix 5-6: component-environment view
46

47. Appendix 5-7: environment property
47

48. Appendix 5-8: environment inventory
48

49. Appendix 5-9: artifact view
49