This document discusses how Splunk was used in integration testing for a large program at a cable TV and internet company in the Netherlands. Splunk was introduced to index message traffic and system logs. This provided testers insight into the overall flow and helped solve integration issues. It allowed issues to be assigned to the right team and prevented problems in production. The document outlines benefits of using Splunk in testing such as speeding up test phases, fact-based reporting on quality, and reducing time to market.
3. 3
Leading Cable TV and internet company of the Netherlands
Recently acquired by Liberty Global
Internet, Cable TV, mobile network, Telephony
4. 4
My background
Programme test manager (contracter)
Experience in both test and operations
Author of a book on performance testing (available on Amazon)
Joined a running programme in the integration phase
My favorite Splunk tee-shirt tag line: Put that in your | and Splunk it
7. 7
Exploratory testing
Testing isn’t just creating and running some scripts
Simultaneous: learn / design execute
No up front scripts
Fast, effective
Relies on feedback what about logs?
8. 8
Testing not the same
Testing or checking?
Changed scene
– Agile / devops / continuous delivery
– SOA: chain or web?
– Change based specs vs system based
9. 9
Testers
Spider in the web (integration)
Manage RCA
Mediate solutions
Looking for trouble
Trouble shooting...
Ring a bell?
10. 10
The situation
Large programme
Multiple vendors
Internal and off shore development
Internal and off shore testing
Many new components
Multiple teams and suppliers (off shore)
Delays in integration (ticket ping pong)
Splunk guru in the team
11. 11
Introduced Splunk
Tibco messages indexed in Splunk
Core systems logs indexed in Splunk
Order tables (rising column)
Created dashboards
Screenshot here
17. 17
Results
Solved the integration issues
Testers very happy:
Insight in the total flow
Self service on test data issues
Issues assigned to the right team
Find the overlooked issues
Jump started the integration phase
After care was cancelled due to NO reported issues
Team rule: no issue logged that hasn't been Splunked
1
19. 19
Data’s story
1
DateTime Status CustI CorID Product Unit
09/30/2015 10:45:00 Success B254788 CorID12548687 Apple 1
09/30/2015 10:45:01 Pending B356489 CorID12548688 Carrot 5
09/30/2015 10:45:02 Failed B366547 CorID12548689 Banana 5
CO12548689 2015-30-9 10:43:58.001 Order placed 5 Bananas
CO12548689 2015-30-9 10:44:20.001 Reply from logistics Bananas not by piece
CO12548689 2015-30-9 10:45:02 Order registered in Order DB Status failed
20. 20
Why we do it
Testers: access, insight and first time right
(test) Management: facts and figures on quality
and performance
Company: prevent embarrassing incidents, lower
costs, faster time to market
2
21. 21
Do’s
Get business process information in Splunk
SOA messages
Order tables (rising column)
Logs, logs, logs
Get a Splunk expert on board.
Create dashboards for test teams (detail)
Create dashboards for management (pretty pictures)
Train testers and admins
2
22. 22
Splunk in test
Speeds up test phases
Prevents incidents in production
An answer to today’s test challenges
Facts and figures on quality
Increase time to market
Benefits increase exponentially with complexity
2
23. 23
So why in test?
first time right
handle agile development
handle complexity
speed up integration
2
26. 26
Splunk Icons
Search Bar chart Lock Cloud Cloud – alt Folder Envelope
Storage - 3Storage
Server Indexer Forwarder Search head
Datacenters
Splunk serverFirewall
Desktop Laptop
Failed server
Hadoop storage Datacenter
27. 27
Splunk Icons
Application - alt Virtual machine Virtual server Network www or Global Tools
Log file Router Load balancer Script
Pie chart Gears/Settings Gear Messaging Tag/ticket
Document
Application
Analyze
Network Switch
Shield
Active Directory
28. 28
Splunk Icons
Checkmark InfoAlert StopiPhoneiPadAndroid
Twitter Facebook LinkedIn RSS You Tube Shopping cartGPS Tower
Healthcare Hospital Office building VoIP Phone Support POS Card Reader RFID
29. 29
Splunk Icons
Forwarder - AIX Forwarder- Datastore Forwarder- Free BSD Forwarder- Linux Forwarder- Windows Forwarder- Web Forwarder- OSX
Forwarder- Solaris Splunk server - AIX Splunk server -
Datastore
Splunk server -
Gear
Splunk server -
Linux
Splunk server -
Network
Splunk server -
Web
Splunk server - OSX Splunk server –
Free BSD
Splunk server –
Solaris Splunk search Failed Splunk server
35. 35
Splunk Object Style and Color
Hardware Product
Business/Corporat
e
Highlight OnlyGenericVirtualization
Generic
These are suggested
uses for colors only.
36. 36
Applying Splunk Object Style
To apply the Splunk object style to any shape:
1. Select the shape (Object A) with the desired style
2. Click on Format Painter (paintbrush) tool in toolbar
3. Click on new shape (Object B) to apply style
41. 41
Splunk Enterprise Overview
AlertsMessages Metrics ChangesScriptsConfiguration
s
Log Files
Indexes Any Data from Any Source
DatabasesNetworks Servers
Virtual
Machines
Smartphones
and Devices
Custom
Applications
Security
Tickets
Web
Server
Sensors
42. 42
Splunk Enterprise Scalability
Enterprise-class Scale, Resilience and Interoperability
Send data from thousands of servers using any combination of Splunk forwarders
Auto load-balanced forwarding to Splunk Indexers
Offload search load to Splunk Search Heads
43. 43
Integrated Analytics Platform for Hadoop Data
4
Full-featured,
Integrated
Product
Insights for
Everyone
Works with
What You
Have Today
Explore Visualize Dashboards ShareAnalyze
Hadoop
(MapReduce &
HDFS)
NoSQL
Data Stores
44. 44
Industry Leading Platform for Machine Data
Any Machine Data Operational Intelligence
HA Indexes
and Storage
Search and
Investigation
Proactive
Monitoring
Operational
Visibility
Real-time
Business
Insights
Commodity
Servers
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Applications
Custom
ApplicationsMessaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID
45. 45
Table Example
Column Title Column Title Column Title Column Title
Text Text Text Text
Text Text Text Text
Text Text Text Text
Text Text Text Text
Text Text Text Text
Splunk is the leading platform for machine data analytics with over 5,600 organizations using Splunk (as of 8/1/13) – for data volumes ranging from tens of GBs to tens of TBs to over 100 TBs of data PER DAY.
Splunk software reliably collects and indexes all the streaming data from IT systems, technology devices and the Internet of Things in real-time - tens of thousands of sources in unpredictable formats and types. Splunk software is optimized for real-time, low latency and interactivity.
Organizations use Splunk software and their data the following ways:
1. Find and fix problems dramatically faster
2. Automatically monitor to identify issues, problems and attacks
3. Gain end-to-end visibility to track and deliver on IT KPIs and make better-informed IT decisions
4. Gain real-time insight from operational data to make better-informed business decisions
This is described as Operational Intelligence: visibility, insights and intelligence from operational data.
Hunk (Splunk Analytics for Hadoop) is a full-featured, integrated product offering – that delivers interactive data exploration, analysis and visualization for Hadoop.
Full-featured, integrated product:
Insights for everyone:
Works with what you have today:
Splunk is the leading platform for machine data analytics with over 5,600 organizations using Splunk (as of 8/1/13) – for data volumes ranging from tens of GBs to tens of TBs to over 100 TBs of data PER DAY.
Splunk software reliably collects and indexes all the streaming data from IT systems, technology devices and the Internet of Things in real-time - tens of thousands of sources in unpredictable formats and types. Splunk software is optimized for real-time, low latency and interactivity.
Organizations use Splunk software and their data the following ways:
1. Find and fix problems dramatically faster
2. Automatically monitor to identify issues, problems and attacks
3. Gain end-to-end visibility to track and deliver on IT KPIs and make better-informed IT decisions
4. Gain real-time insight from operational data to make better-informed business decisions
This is described as Operational Intelligence: visibility, insights and intelligence from operational data.