TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development Some interesting talks about using TSTATS and the internal Splunk logs, have a Splunk Trainer share his journey with Splunk and how he's managed to achieve every possible Splunk certification (over 10!), and a short discussion about emerging thoughts of using development/release frameworks with Splunk deployments.

1. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Splunk User Group Edinburgh

2. © 2018 SPLUNK INC.
Harry McLaren
● Alumnus of Edinburgh Napier (Active Mentor)
● Managing Consultant at ECS
● Leader of the Splunk User Group Edinburgh

3. © 2018 SPLUNK INC.
Introduction to ECS
Splunk Partner - UK
– Type: Security / IT Operations / Managed Services (SOC / Splunk)
– Awards: Splunk Revolution Award & Splunk Partner of the Year

4. © 2018 SPLUNK INC.
Agenda
• Housekeeping: Event Overview & House Rules
• Tstats, _internals and Me (Andrew McManus)
• Journey of a Splunk Trainer (Tom Wise)
• Development & Release Life-cycles with Splunk (Harry McLaren)

5. © 2018 SPLUNK INC.
Splunk [Official] User Group
“The overall goal is to create an authentic, ongoing
user group experience for our users, where
they contribute and get involved”
● Technical Discussions
● Sharing Environment
● Build Trust
● No Sales!

6. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
| tstats, _internals and Me
Or How I Stopped Worrying About Expensive Search
Queries, and Learned to Love Interrogating the tsidx
Andrew McManus

7. © 2018 SPLUNK INC.
About Myself
● Associate Security Consultant at ECS
● Prior - Senior/Security Operations Center Analyst at ECS
● Credentials: Power User (Admin this weekend, hopefully)
● Current tasks at <redacted>: Troubleshooting configuration and
investigation.

8. © 2018 SPLUNK INC.
Why talk about this?
● No (Splunk) Environment is perfect.
– Scheduling, search run times, missing data sources
● Insight into environment required
● Internal logs provides insight into performance
● | tstats provides insight into data
● Combined, issues can be detected and squashed.

9. © 2018 SPLUNK INC.
What Splunk logs about itself?
● Splunk logs and indexes everything about itself
● _audit – user activity
– Login attempts, user searches, configuration changes
– Everything thrown together in one sourcetype, audittrail
● _internal – many sourcetypes containing various logs
– Scheduler logs, CPU/Mem usage, license usage and more.
– Sourcetype per log type.
– Events can have log_levels (INFO, WARN, ERROR)
● _introspection – system metrics.
– Per Process metrics
● _fishbucket – checkpoints for ingested files

10. © 2018 SPLUNK INC.
Scenario 1 – Licenser Issues
● Alert came in the prior day regarding your Licenser
● Don’t have access to DMC but _internal
● Use licenser logs to determine what data blew the license:
● Demo 1):
– index=_internal source="/opt/splunk/var/log/splunk/license_usage.log" st=* | bucket _time span=1d | eventstats sum(b)
as total_used, values(poolsz) as limit by _time | eval over_limit=if(total_used>limit, "YES", "NO") | stats
sum(eval(b/1024/1024)) as mb values(over_limit) by st, _time
● (Can also check per_index_thruput metrics for indexers)
– index=_internal metrics kb series!=_* group="per_index_thruput" | eval indexed_GB = (kb / 1024 / 1024) | timechart
eval(round((sum(indexed_GB)),2)) as TotalGB fixedrange=t span=1d

11. © 2018 SPLUNK INC.
Scenario 2 – Missing Data
● Problem:
● Scheduled search gets data from number of logs w/ UTC timestamps
● Analyst notices that log source never shows up in search
● Search re-ran over same period the next day:
● Missing logs present when manually searched later than scheduled.

12. © 2018 SPLUNK INC.
Scenario 2 – Missing Data (cont)
● Investigation:
● Use _internal index to investigate scheduler logs to see
when search runs and what time-frame it scans.
– Or _audit
● Find events that aren’t indexed in time for search with tstats

13. © 2018 SPLUNK INC.
| tstats
● Uses tsidx files to report on:
– Indexed values
– Accelerated data models
● If field is not indexed, cannot be used in search
● Fast
● Seriously fast.

14. © 2018 SPLUNK INC.
Tsidx file
● Can use walklex to investigate what’s in a file.
– Returns unique id (term ID), how many times the term occurs and the term.
● Example:

15. © 2018 SPLUNK INC.
| tstats
● Can find event counts per index, sourcetype or source for
example.
● Metrics with indexing times can be determined.
● Fast stats on Accelerated Data Models can be determined in
seconds.

16. © 2018 SPLUNK INC.
Example – TSTATS count events by host
| stats countindex = main by host| tstats count where|tstats count where index=main by host, _time span=1h

17. © 2018 SPLUNK INC.
How Fast?
Without Tstats: 9.002s. With Tstats: 0.183s
48x decrease in time taken.

18. © 2018 SPLUNK INC.
Revisiting Scenario 2
● Using tstats, you can see every hour, events from source
comes in up to a hour later.
– | tstats earliest(_indextime) as it, latest(_indextime) as lit where index=<index>
sourcetype=<sourcetype> by source, _time span=1h | convert ctime(it) ctime(lit)
● This source is logging later than expected
– Another more complex search:
– | tstats earliest(_indextime) as earliest_indexed, latest(_indextime) as latest_indexed where index=<index>
sourcetype=<sourcetype> by source, _time span=1h | rename _time as time_window | join time_window [ search
index=_audit info=completed savedsearch_name=<savedsearch_name> | eval time_window=round(api_et,0) |
rename _time as search_time | table search_time time_window savedsearch_name] | eval data_missing =
if(latest_indexed>=search_time, "YES", "NO") | table savedsearch_name search_time time_window source
data_missing earliest_indexed latest_indexed | sort search_time source | convert ctime(search_time)
ctime(time_window) ctime(earliest_indexed) ctime(latest_indexed)
● Solution
● Change Scheduling for alert until after all logs come in.

19. © 2018 SPLUNK INC.
Other examples:
● Quick Glass pane view of event counts per index/host/source etc…

20. © 2018 SPLUNK INC.
Compare Event Counts to Run Times

21. © 2018 SPLUNK INC.
Scenario 1 in tstats?
● Internals by default appear to be put into a datamodel.
● index=_internal source="/opt/splunk/var/log/splunk/license_usage.log" st=* | timechart
span=1h sum(eval(b/1024/1024/1024)) as gb by st
● Link: Search
● Or:
● | tstats sum(server.licenser.daily_usage.gb) as usage from datamodel=internal_server
where nodename=server.licenser.quota groupby server.st, _time span=1h
● Link: Search

22. © 2018 SPLUNK INC.
Further Reading
▶ .conf2017: Searching FAST: How to Start Using tstats and Other Acceleration Techniques -
David Veuve
• https://conf.splunk.com/files/2017/slides/searching-fast-how-to-start-using-tstats-and-other-
acceleration-techniques.pdf
• Covers Data Acceleration Models in greater detail.
▶ .conf2017: Worst Practices... And How To Fix Them - Jeff Champagne
• https://conf.splunk.com/files/2017/slides/worst-practicesand-how-to-fix-them.pdf
• (2016 the same content, except talks about Virtualisation)

23. © 2018 SPLUNK INC.
Any Questions?
● Have you made anything cool with tstats or _internal indexes?
Contact Me:
Andrew McManus – andrew.mcmanus@ecs.co.uk
Slack – On Splunk and SecurityScotland Slack Channels

24. My Journey

25. $ whoami
 Tom Wise
 Security Consultant & Splunk Specialist @ ECS
 Splunk Consultant II
 Splunk Architect II
 Splunk Trainer

27. Certifications….so far!

28. Accreditations…so far!
Splunk Accredited Sales Rep I
Splunk Accredited Sales Rep II
Splunk Accredited IT & App Sales Rep
Splunk Accredited Sales Engineer I
Splunk Accredited Implementation Fundamentals
Splunk Accredited Core Implementation
Splunk Accredited ES Implementation

29. Origin Story
 Every Splunk hero has one!
 Spanned 2 Continents
 Varsity Baby!
 Boom…Injury
 Return to roots.
 Man up & save the world one event at a time!

30. 1st Foray into IT Wilderness

31. Space…the next frontier!

32. Personal Achievements

33. UK Challenge 2015
Lake District
6th / 60
Most Sporting Team
2015

34. Put that in your | and Splunk it!
 Started with ECS in June 2016
 Consultant II by November 2016
 ITSI Specialist since 2017
 Enterprise Security Implementation gained in 2018
 Splunk Trainer since January 2018

35. The Padawan to Master
 2 x Interviews with Heads of Splunk Training (US)
 Competence and Personality tested
 1 x Train the Trainer session
 Show the logistic tooling & processes
 Present a module to the Trainer

36. A Day in the Life…
 Access to the training center to confirm numbers and attendees.
 All pre-class actions are automated:
 Links to user watermarked documents sent to individuals & full list to trainer 2
days before the course.
 Lab credentials created and sent to instructor.
 WebEx session automatically configured ready to be joined.
 Question etiquette
 Best to save questions, unless pertinent to continuing, until at least after the 2nd
lab. Generally a lot of time is spent before and after first lab answering access
issues.
 Where possible ask the question to all participants as it will help in the learning
and is not as easy for the trainer to miss.

37.  Labs should be done in order as some activities rely on previous labs
being completed.
 However don’t rush! The labs do not need to be completed in the allotted
time, only before the entire course finishes.
 If in an office location, ensure that access over HTTPS and SSH is
available out of the network.
 If not then this can be worked around, but will take some time for the
instructor to implement.
 Test your access before the class (if possible)

38. Enablement
 This position allows me to:
 Offer more internal training along side the standard path.
 Provide Splunk training to ECS clients.
 Keep in touch with any changes coming in.

39. Tips
 Don’t eat yellow snow
 Install, Install, Install
 Until it is muscle memory
 Where possible script
 Helps develop logic and defensive coding practices.
 Always Best Practice in Labs & Exams!
 Implementing Best Practice even if not specifically asked can gain
additional points/kudos.

40. Questions?

41. © 2018 SPLUNK INC.
Development &
Release Cycles
Using SDLC for Controlled
Splunk-based Success

42. © 2018 SPLUNK INC.
Software Development Life Cycle

43. © 2018 SPLUNK INC.
Continuous Development, Deployment & Integration
“DevOps is a culture, and not a collection of technology or role”

44. © 2018 SPLUNK INC.
Splunk is “Agile” by Default
Search Processing Language (SPL)
Empowerment to Users
Web User Interface
Creation & Sharing of Knowledge Objects

45. © 2018 SPLUNK INC.
Where DIY Could Harm the Business
Security
• Tuning of Rules / Alerts
• Disabling of Saved Searches
IT
Monitoring
• Manually Setting Thresholds
• Forgetting Tactical Changes
Business
Analytics
• Changes to Business Logic
• Requirements Not Fully Understood

46. © 2018 SPLUNK INC.
▶ Local Work in Progress (WIP)
▶ Environment Specific Branches?
• Local > Dev > Test > Release
▶ Tags
▶ Integration into Workflow
Version Control
Systems (VCS)

47. © 2018 SPLUNK INC.
Configuration Management
● Manual vs. Automatic
● Agent & Agentless
● Actions, Roles, Groups
● Complemented by VCS

48. © 2018 SPLUNK INC.
Best of Both: Splunk Route-to-Live
New
Requirement
Create
Feature
in DevEnv
Commit to
Dev Branch
Validate /
Collect
Features
Merge into
Test Branch
Testing
Process in
TestEnv
Change /
Governance
Merge into
Prod Branch
Release into
ProdEnv

49. © 2018 SPLUNK INC.
Resources
● A Visual Guide to Version Control
● Version Control using Git and GitLab
● Deploying Splunk Securely with Ansible Config Management
● Configuration Management 101: Writing Ansible Playbooks
● What is DevOps?
● DevOps is a culture, not a role!

50. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Thank You

51. © 2018 SPLUNK INC.
Get Involved!
● Splunk User Group Edinburgh
– https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html
– https://www.linkedin.com/groups/12013212
● Splunk’s Slack Group
– Register via http://splunk-usergroups.signup.team/
– Channel: #edinburgh
● Present & Share at the User Group?
Connect:
‣ Harry McLaren | harry.mclaren@ecs.co.uk | @cyberharibu | harrymclaren.co.uk
‣ ECS | enquiries@ecs.co.uk | @ECS_IT | ecs.co.uk

52. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Thank You