5 Steps for Identifying Application Development Deficiencies and Fixing Problems FAST Are your software development practices and IT systems negatively impacting your overall business? How do you adopt processes that will drive customer and business value quickly, and then extend that throughout your organization? In this webinar featuring Gary Carr, Software Architect at American Fidelity Assurance, you’ll learn how AFA changed from a process where troubleshooting performance issues involved several iterations between development, QA and production to an environment where top-notch performance of their applications comes easily. Understand how changes to the business, such as regulatory controls, customer expectations, and development process drove their need to identify deficiencies and fix problems faster so that technology truly supports the goals of their organization and their customers. • Learn metrics-based techniques to build applications faster and with more confidence, while practically eliminating their defect remediation cycle. • Find out why separation of duty for the development team no longer serves customers well. • Discover how seeing all the data for every transaction helps developers connect to users (and one another) with less effort, and helps IT solve problems faster.

1. Confidential, Dynatrace LLC
5 Steps for Identifying Deficiencies
and Fixing Problems FAST
Gary Carr
Software Architect
American Fidelity
Asad Ali
Product Specialist
Dynatrace
1

2. 1. Know what’s important to ALL key stakeholders
• Who identifies deficiencies?
• Who discovers problems?
• Who wants resolution?
2. Create specific criteria for PoC
• Review current systems, know your gaps
• Stay open minded
3. Work on Deployment
4. Always look for the metrics
5. Share the results
Confidential, Dynatrace LLC
The 5 Steps
2

3. Nine times on Forbes “Top 100 Places to Work”
Focus on 24x7 service to Customers, Colleagues, Community
Technology supports relationships, rather than replacing
Partner support is key to growth
1.9 million policies, 1,800 employees
Who is American Fidelity?
Confidential, Dynatrace LLC3

4. In-house tools provided the basics
Confidential, Dynatrace LLC
Before Dynatrace
• API, ESB, F5 URL calls
• DNS related problems
• No visibility into
exceptions
4

5. • Two kinds of problems
• Visible, but lots of time and resources
• Invisible
Confidential, Dynatrace LLC
Before Dynatrace
“It was like being in a dark room”
Traci Tabyanan, Applications Manager, American Fidelity
5

6. • Business
• More accurately measure user
satisfaction
• Provide excellent customer
experience
• Release applications more often
• Reduce technical debt
Confidential, Dynatrace LLC
Reasons for Enterprise-ready APM
“We’re heading toward being a technology company that supplies insurance”
Diana Bittle, VP of Software Solutions, American Fidelity
43k Users
6

7. Confidential, Dynatrace LLC
Reasons for Enterprise-ready APM
• Technical
• Reproduce production problems
accurately in dev
• Understand how 3rd party tools operate
• Excessive database calls
• Visual topology of complex delivery chain
• Between tiers and technology
platforms
• Inconsistent performance between test
environments for same build
• Code-level diagnostics
7

8. • Use of guesswork to identify root cause
• Lack of visibility into the end user response time
• Visibility gaps between application tiers
• Need to reproduce problems to identify root cause
• Necessity for war rooms for major problems
Confidential, Dynatrace LLC
How to identify deficiencies in your APM
8

9. Confidential, Dynatrace LLC
Don’t be like Michael Scott…
“the office” is a trademark of NBC
9

10. Confidential, Dynatrace LLC
Prepare for Successful PoC
• Focused on enterprise wide applications
• Invited developers, testers, DevOps, security, and project managers
• Shared applications and issues with APM experts
• Set up before PoC began with specialists from our company and Dynatrace
• Very specific requirements for specific tests
• Found a need in our company and used the PoC to solve
• Visibility from individual’s browser to deepest calls in the system
• In-depth analysis of resource consumption per transaction
• Dashboards specific to each stakeholder
• Showed enterprise level
• AND provided easy drill-down
10

11. • Define concrete and specific PoC success criteria
• Use most significant application(s)
• Stress test your environment with the APM tool
• Invite all stakeholders to the PoC
• Look for value DURING
• Ensure apples-to-apples
Confidential, Dynatrace LLC
Lessons Learned from 500+
11

12. • Problem #1: Most of
the execution time for
main app was spent in
.NET remoting and ADO
• + ID where most of time
was spent
• Focus on reducing
technical debt in those
areas
Confidential, Dynatrace LLC
AF’s PoC: Key Findings
12

13. • Problem #2: Able to isolate
remoting issue to one particular
server
• Been looking for a month
• Saw because capture all transaction,
including exceptions
• Seamless fit: infrastructure and
processes
• Helped id specific code changes
Confidential, Dynatrace LLC
AF’s PoC: Key Findings
13

14. • Simple view of
complex delivery chain
• Automatically created
• Auto-discovery
removes guesswork
Confidential, Dynatrace LLC
PoC: Key Findings
14

15. • Database connectivity issues
• Configuration settings
• Remoting calls => $40M savings
• Architecture anti-patterns
• Memory leaks
• N+1 query patterns
Confidential, Dynatrace LLC
Other Key Findings
15

16. • Involve DevOps early!
• Share successes
• Early example:
• 2-day MIRT right after deployment, so were
not using Dynatrace much yet…
• Let’s check one thing  easier than thought
• Solution in 20 minutes
• Benefit grows exponentially
• Now Dynatrace involved early in MIRT
16 Confidential, Dynatrace LLC
Deployment Success

17. Confidential, Dynatrace LLC
Metrics help build better apps, faster
More enrollments.
Yeah!
17

18. Confidential, Dynatrace LLC
Metrics help build better apps, faster
More processing. Boo!
• From 20 minutes to
2 hours
• Errors and
unexpected timeouts
• Tried increasing
planned timeouts
18

19. Confidential, Dynatrace LLC
Metrics help build better apps, faster
• Quickly saw exact
problem
• Database query
taking 7 minutes
 Called 443 times
• Down to 56 calls
• Straightforward code
change
19

20. Confidential, Dynatrace LLC
Gasps + Clapping = Performance Hero
Even as enrollments continue to grow, processing time remains low
20

21. Confidential, Dynatrace LLC
Take-aways from a developer
• Look at System Profiles> YourApp> Diagnose Events> Exceptions
• Customize your timeframe (i.e. 7PM – 9PM)
• Use PurePath™ = “Intense Tracing”
• Look for method names
• Look for long execution times
• Keep drilling down
• Sampling would not have found this issue
• Don’t be afraid to ask for help
21

22. • Coincidences bring nightmares
• Sometimes the state of the system is the problem
• Gap-free APM provides
• All the data leading up to issue, from every tier
• Plus inter-tier stitching
• Prioritize and reduce technical debt
• Faster fixes
• “Customer cannot log in, it happened at 8:45 a.m.”
• Just grab all the data from 8:44 a.m. to 8:46 a.m.
Confidential, Dynatrace LLC
It’s all about the visibility
22

23. • Ability to compare slow transactions
with fast transactions
• Reduces the need to reproduce
problems
• Easily help identify root cause of pool
exhaustion issues
• Enables conversion of application data
to business data
• Eliminates guesswork
Confidential, Dynatrace LLC
Benefits of gap-free data
23

24. Confidential, Dynatrace LLC
User Satisfaction
• Now developers have a closer
connection to customers
• APM shows impact of coding
methods
• Developers can see real-time user
satisfaction
24

25. • Promotes DevOps culture
• Reduces Ops anxiety over newer code
release
• Enables frequent code releases
• Make Dev an equal partner in problem
resolution
• Promotes innovation
Confidential, Dynatrace LLC
Dev and Prod synch serves customers
25

26. • War room calls directly show the power
• Town Hall meetings: App Development, Business, Marketing, and more
• TV and tablets with HTML5
• Results
• Challenges!
Confidential, Dynatrace LLC
How AF spreads the use of APM
26

27. • Promote key improvements to
upper management
• Use APM tool to make the
developers the “hero”
• Translate APM success to
business $$$
• Display APM dashboards in public
areas
• Use APM tool to reduce schism
between Dev and Ops
Confidential, Dynatrace LLC
Insights from other customers on spreading APM
27

28. • Continue to improve relationship with the business
• Using dashboards
• See their applications
• Dynatrace Community
• Sharing information
• Training
• User Groups and Conferences
• New release strategy
Confidential, Dynatrace LLC
Extra notes
28

29. • Define concrete PoC criteria
• Use significant application(s) for PoC
• Look for value DURING
• Share success of APM results
• Promote DevOps culture
• Make Dev an equal partner in problem resolution
Confidential, Dynatrace LLC
Summary
29

30. 1. Know what’s important to ALL key stakeholders
• Who identifies deficiencies?
• Who discovers problems?
• Who wants resolution?
2. Create specific criteria for PoC
• Review current systems, know your gaps
• Stay open minded
3. Work on Deployment
4. Always look for the metrics
5. Share the results
Confidential, Dynatrace LLC
In Review: The 5 Steps
30

31. Copyright 2016, Dynatrace LLC
THANK YOU!
Confidential, Dynatrace LLC 31