https://info.dynatrace.com/apm_dtm_ops_17q4_wc_accelerate_user_driven_innovation_en_registration.html Accelerate User Driven Innovation [Webinar] DevOps adopters are more agile, more reliable and more successful but, only 2% of companies worldwide have adopted DevOps best practices. We know it’s more difficult for enterprises companies with legacy systems and processes to get started but it isn’t impossible. To help you accelerate your own DevOps journey & realise some of the benefits, we’re thrilled to be hosting international DevOps experts Andreas Grabner, Mark Tomlinson and James Pulley. With combined experience across hundreds of DevOps deployments they have some remarkable use cases to share including Verizon, and even our own story of transforming from on premise six month waterfall deployment to a cloud native one hour continuous delivery model. Don’t miss these amazing insights. Register today!

1. Andreas Grabner
DevOps Activist
@grabnerndi
Accelerate User Driven Innovation
February 2017
Mark Tomlinson
Performance Sherpa and CEO
@mark_on_task
James Pulley
Performance Evangelist and CTO
NewCOE LLC, @perfbytes

2. confidential
User Driven Innovation?

3. confidential
24 “Features in a Box” Ship the whole box!
Very late feedback 

4. confidential
Accelerated User Driven Innovation
1 “Feature at a Time”
Optimize Before DeployImmediate Customer Feedback

5. confidential
DevOps: Faster & Efficient Innovation
Smaller Apps, Micro-Services More Deployments
App-, Service- & End-User Feedback Loops
Happy Users
Lower Costs

6. Total Number of Users
per User Experience
Conversion Rate
Use Case #1: User Feedback Driven Decisions

7. New Features + Day # 1 of Mkt Push
Overall increase of Users!
Jump in Conversion Rate!
Use Case #1: User Feedback Driven Decisions

8. Users keep growing
Increase # of “tolerating” users!
Lower Conversion as Day #1
Day #2 of Marketing Campaign
Use Case #1: User Feedback Driven Decisions

9. Drop in Conversion Rate
Spikes in FRUSTRATED Users!
Hotfix Deployment was rolled out
Use Case #1: User Feedback Driven Decisions

10. User Experience Back to Normal
Jump in Conversion Rate!
Fix of the Hotfix was rolled out
Use Case #1: User Feedback Driven Decisions

11. 4x $$$ to IaaS
Use Case #2: Resource / Cost Driven Decisions

12. Use Case #3: User Behavior Driven Decisions

13. confidential
Use Case: Accelerated User
Driven Innovation in Finance

14. confidential
Defining Performance Engineering
 You must model the system for
performance, before any code is
written
 You must test the model or idea for
performance risk
 You must understand the big picture to
write small components of code
 You must collaborate with others:
developers and system architects
 You must adapt your thinking and your
relationships
“If you can impact the
[performance of the] code before it
is written, then and only then, can
you consider your work to be
performance engineering.
Otherwise, you are just doing
testing.”
- Jim Duggan, Gartner

15. confidential
Purpose-
Built
Jobs
Capture-
Playback
LoadRunner
LoadRunner
+ others
Open
Source
Integrated
COE Suites
Open
Source
Cloud Tools
Open
Source

16. confidential
DEV:
DIY Scala Tool
Yourkit, JProfiler
Local log parsing
TEST:
SmartBear/SoapUI
DIY Selenium FX
Stand-alone JMeter on
Laptop
LT01:
Manual Setup and
Execution
DIY Scala Tool
Curl & Perl
Shared NFS log parsing
PROD:
Solaris tools - prstat
Dynatrace 4.2
Traverse
Nagios
AWR & Ignite DB
Monitoring
Tools, Teams and Environments: 2012

17. confidential
DEV:
Dynatrace 4.2
Local log parsing
JProfiler
Some JMeter + Plugins
TEST:
Functional Test
Automation
LT01:
Dynatrace 4.2
Local log parsing
JMeter + Plugins in the lab
Automated prstat capture
Automated Loadosophia
Upload
Ignite DB Monitoring
Integrated Scenarios
PROD:
Dynatrace 4.2
Solaris tools - prstat
Traverse
Nagios
Ignite
Tools, Teams and Environments: 2013-14

18. confidential
DEV:
Dynatrace 6.3
Splunk
JProfiler
Rundeck + JMeter + Plugins
Real-time InfluxDB &
Grafana
JMeter + Plugins
TEST:
Functional Test
Automation
Exploratory Testing
LT01:
Dynatrace 6.3
One-off Logs to Splunk
Rundeck + JMeter + Plugins
Real-time InfluxDB &
Grafana
Ignite DB Monitoring
Integrated Scenarios
Notification to Wiki/Jira/Slack
PROD:
Dynatrace 6.3
Splunk
Solaris tools - prstat
Traverse
Nagios
Ignite
Tools, Teams and Environments: 2015-16

19. confidential
DEV:
Self-Service
Measurements
Dynatrace 7
Splunk
PTaaS & ECD
K8 & Prometheus
App Dynamics
TEST:
Functional Test
Automation
Exploratory Testing
LT01:
Continuous Execution
Dynatrace 7
Splunk
PTaaS & ECD
K8 & Prometheus
Ignite DB Monitoring
PROD:
Dynatrace 7
Splunk
Solaris tools - prstat
Traverse
Nagios
Ignite
Tools, Teams and Environments: 2017+

20. confidential
Performance Practice
(production-like)
Production
Performance
(“the Real World”)
Development
(very unreal world)
Fast Performance Flows
PROMOTIONAL FLOW
FEEDBACK FLOW
operate
invent

21. confidential
Performance Practice
(production-like)
Production
Performance
(“the Real World”)
Development
(very unreal world)
Fast Performance Feedback
6
5
4
1
2
3
CI CD
PROMOTIONAL FLOW
FEEDBACK FLOW
unit perf
essential perf
new releasestrategic tests
triage
operate
invent
design

22. confidential
Make Performance Continuous
IN DEVELOPMENT
 Enable Self-Service Measurements – on
demand execution of a small test
 Every engineer on the team can build,
submit and execute a test
 Test environments may be built on-the-
fly
 Connected to CI system for unattended
executions after nightly builds on on-
demand pushes
IN LOADTEST
 Enable Non-stop schedule for
performance test executions
 Trending test results on measurement
thresholds, compared to baseline
 Use a split-schedule for high-priority
test execution - nightly continuous
execution, self-service during the day
 Connected to CD system to automate
release feedback
In all environments use common tooling for monitoring, profiling and
logging – on by default, automated and accessible to all.

23. confidential
Use Case: Accelerated User
Driven Innovation at
Dynatrace

24. confidential
2011: APM about to be disrupted!
 Migrate from On-Prem to VM, Cloud, Containers and PaaS
 Architectures include micro-services, on-demand scaling,
self-healing
 ”Cloud Natives“ demand SaaS based solutions
 Digital Transformers demand Analytics for Biz, Dev, Ops &
Sec
 Many new players on the market

25. confidential
Challenges to master!
 Bridging the gap between ”New Stack“ and “Enterprise Stack“
 Deploying the same way our customers do: Continuously!
 Not disrupting current operations and slower moving customers
 Aligning 300+ engineers across 3 different geos
 Solution: Innovation through Incubation!

26. 2 major releases/year
customers deploy & operate on-prem
26 major releases/year
170 prod deployments/day
self-service online sales
SaaS &
Managed
2011 2016

27. 29 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
believe in the mission impossible
6months
major/minor release
+ intermediate fix-packs
+ weeks to months
rollout delay
sprint releases (continuous-delivery)
1h : code to production

28. NOC lessons learnt

29. Shift-Left Quality
Quality/Performance matters in Dev/Staging as well!
Make Dev/CSA/PM dependent from Quality in trunk!
DevOps = start thinking like an Ops before Commit
Shift-Right Metrics
enable DEVs defining quality metrics
make DEVs to the primary consumers of their metrics

30. acting as
Engineers
Role of Dynatrace DevOps Team
Dynatrace Managed/SaaS
Orchestration Layer
DynatracePipeline Visualization
Deployment Timeline
Log Overview
using Dynatrace Log APIJIRA Integrations
&
Product Managers

31. confidential
https://github.com/Dynatrace/ufo
Raising Awareness of Pipeline Quality

32. ruxit theme 2014.05.15
Be proud of your feature!
DevOps  NoOps

33. Dynatrace Transformation by the numbers
26
170
Releases / Year
Deployments / Day
31000 60h
Unit & Int Tests / hour UI Tests per Build
More Quality
~200 340
Code commits / day Stories per sprint
More Agile
93%
Production bugs found by Dev
More Stability 450 99.998%
Global EC2 Instances Global Availability

34. confidential
Use Case: Accelerated User
Driven Observations at
eCommerce

35. confidential
Trends across eCommerce
▪ Development taking ownership of all quality issues, except
Security
▪ Orchestration and deployment is moving to the public
cloud (Jenkins, AWS, CloudAzure)
▪ CI is the norm
▪ Serverless computing is becoming second nature
▪ Rules based analytics are replacing first tier diagnostics,
moving human performance analysis to tougher issues

36. confidential
Cautionary Notes
▪ Developer Architecture Myopia
▪ Loose Gates on Quality and Performance
▪ Developer ownership on quality, but the need for
performance training
▪ Broken links between the business and Dev/Ops -
particularly marketing
▪ Pattern feedback loop to the code analysis level remains
immature
▪ With rules based analysis, the feeder tier for new
performance engineers is compromised.

37. confidential
Final Thoughts …

38. confidential
% 20%
organization & culture technology

39. Confidential, Dynatrace, LLC
Develop
Ship
Deploy
Run
Scale
Compute
nodejs mongo db netty cassandra redis
ansible jenkins puppet chef
docker cloudfoundry rh openshift rh atomic rocket
core os rancher kvm busybox
mesos marathon kubernetes swarm
amazon azure openstack mesosphere calico weave
eureka/hystrix
A whole new technology stack & polyglot development

40. confidential
Ecosystem …

41. confidential
Dev/Test: Bad Code
Changes
Ops/Biz: Unhappy End
Users
Performance:
Overloaded COEs

42. confidential
Dev/Test: Check-In
Better Code
Ops/Biz: Use Real-Time
End User Feedback
CI/CD/Performance:
Stop Bad Builds Faster

43. confidential

44. confidential
AI-Driven Digital Operation Ecosystem
DAVIS / REST

45. Andreas Grabner
DevOps Activist
@grabnerndi
Accelerate User Driven Innovation
February 2017
Mark Tomlinson
Performance Sherpa and CEO
@mark_on_task
James Pulley
Performance Evangelist and CTO
NewCOE LLC, @perfbytes