From Splunklive! San Francisco

1. Copyright © 2014 Splunk Inc.
Wells Fargo
Digital Channels
Splunk Journey

2. Nick Gordon
Monitoring Strategies &
Engineering
Online & Mobile Banking

3. My Background and Role
Moved from Sydney, Australia in 2001
12 years at Wells Fargo
Leading Monitoring Strategies and
Engineering team
Backcountry skier
My favorite Splunk T-Shirt:

4. Agenda
Background & Role
Company Overview
How we got started
Where are we going
Wins with Splunk
Best Practices for Splunk

5. Wells Fargo Overview
1852

6. Wells Fargo Overview
22 Million desktop &
13 Million mobile
customers
#1 Most valuable
banking brand in the
world
$21 Billion revenue
per quarter
First US bank to offer
online account
access 1995
#2 Largest
philanthropic giver in
the US
#1 Largest bank in the
world by market cap
Learn more about Wells Fargo:
https://www08.wellsfargomedia.com/downloads/pdf/about/wellsfargotoday.pdf

7. Software Development at Wells Fargo

8. Software Development at Wells Fargo

9. One Use Case – Many Systems

10. How We Got Started
Homegrown tools
(nothing)

11. Application Performance Management
at Wells Fargo
Availability
Response
time
Capacity MTTR
APM

12. Business Case for Splunk
• Complex Operational
support process
• Customer interaction
across many systems
• Difficulty to grant
appropriate access
across all systems
• Unable to centralize and
correlate data
• Centralized all our data
in one place
• Navigated application
data efficiently to trace
customer flow through
our applications
• Improved transition
between between
development and
production
A single system which
could unite the data
from multiple systems
for correlation
Before Splunk After SplunkNeeds
14x faster analysis and reporting of logs vs. gunzip+grep

13. The Internal Sell
After SplunkBefore Splunk
“When can I get it?”“Why do I have to?”

14. Splunk Enterprise Scalability
Enterprise-class Scale, Resilience and Interoperability
Send data from thousands of servers using any combination of Splunk forwarders
Auto load-balanced forwarding to Splunk Indexers
Offload search load to Splunk Search Heads
• 2 job servers
• 2 search heads
• 15 indexers
• 475TB of SAN/NAS
• 150 forwarders
• 365 days data retention
• 1TB per day

15. Splunk Geolocation

16. Splunk Application Performance Tuning

17. Splunk Application Performance Tuning

18. Splunk Decision Analytics

19. Best Practices for Splunk
Need a Splunk COE or as we call it community practice.
Balance between logging too much and not enough.
Personally Identifiable Information is confidential and must be dealt
with appropriately.
Log standards before you begin.
Logs
Cost

20. A few points to remember
Concept
Prove the
concept, get
data in front
of people,
buy-in
Data
Get as much
data into
Splunk as
possible
Dashboard
Build
dashboards
to
summarize
your data
Share
Information,
lessons
learned,
wins and
best
practices
Creativity
Provide
flexibility to
engineers to
come up
with unique
solutions

21. Splunk for Wells Fargo
“Splunk helps us answer questions that
we didn’t even know we had.”

22. Thank You