In addition to seeing the latest features in Splunk Enterprise, learn some of the top commands that will solve most search and analytics needs. Ninja’s can use these blindfolded. New features will be demonstrated in the following areas: TCO and Performance Improvements, Platform Management and New Interactive Visualizations.
4. 4
Agenda
What’s new in 6.4 (and a few goodies from 6.3!)
– TCO & Performance Improvements
– Platform Security and Management
– New Interactive Visualizations
Harness the power of search
– The 5 Search Commands That Can Solve Most Problems
6. 6
Safe Harbor Statement
During the course of this presentation,we may make forward looking statements regarding future events
or the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC. The forward-looking statements
made in this presentation are being made as of the time and date of its live presentation. If reviewed
after its live presentation, this presentation may not contain current or accurate information. We do not
assume any obligation to update any forward looking statements we may make. In addition, any
information about our roadmap outlines our general product direction and is subject to change at any
time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described orto includeany suchfeatureor functionalityina futurerelease.
7. 7
Splunk Enterprise & Cloud 6.4
Storage TCO
Reduction
- TSIDX Reduction
reduces historical data
storage TCO by 40%+
Platform Security &
Management
New Interactive
Visualizations
- Improved DMC
- New SSO Options
- Improved Event Collector
- New Pre-built Visualizations
- Open Community Library
- Event Sampling and Predict
8. 8
TSIDX Reduction
Provides up to 40-80% storage reduction
Retention Policy on TSIDX Files
Creates “mini” TSIDX
Performance trade-off between
storage costs and performance
– Rare vs Dense Searches
No functionality loss
Can restore original TSIDX files if
needed
8
9. 9
Splunk Enterprise & Cloud 6.4
Storage TCO
Reduction
- TSIDX Reduction
reduces historical data
storage TCO by 40%+
Platform Security &
Management
New Interactive
Visualizations
- Improved DMC
- New SSO Options
- Improved Event Collector
- New Pre-built Visualizations
- Open Community Library
- Event Sampling and Predict
10. 10
Management & Platform Enhancements
Management
– Distributed Management Console
New monitoring views for scheduler,
Event Collector, system I/O performance
– Delegated Admin roles
HTTP Event Collector
– Unrestricted data for payloads
– Data Indexing acknowledgement
SAML Identity Provider Support
– OKTA, Azure AD, ADFS
1
SAML Support
OKTA
Azure AD
ADFS
Ping FederateAWS IoT
Event Collector
11. 11
Splunk Enterprise & Cloud 6.4
Storage TCO
Reduction
- TSIDX Reduction
reduces historical data
storage TCO by 40%+
Platform Security &
Management
New Interactive
Visualizations
- Improved DMC
- New SSO Options
- Improved Event Collector
- New Pre-built Visualizations
- Open Community Library
- Event Sampling and Predict
12. 12
Custom Visualizations
Unlimited new ways to visualize your data
15 new interactive visualizations useful
for IT, security, IoT, business analysis
Open framework to create or customize
any visual
Visuals shared via Splunkbase library
Available for any use: search, dashboards,
reports…
1
13. 13
New Custom Visualizations
1
Treemap
Sankey
Diagram
Punchcard Calendar
Heat Map
Parallel
Coordinates
Bullet GraphLocation
Tracker
Horseshoe
Meter
Machine Learning
Charts
Timeline
Horizon
Chart
Multiple use cases across IT, security, IoT, and business analytics
14. 14
Event Sampling
• Powerful search option provides
unbiased sample results
• Useful to quickly determine dataset
characteristics
• Speeds large-scale data investigation
and discovery
14
Optimizes query performance for big data analysis
15. 15
Predict Command Enhancements
• Time-series forecasting
• New algorithms:
• Support bivariate time series
with covariance
• Predict multiple series independently
• Predict missing values within series
• 80-100X performance improvement
15
Forecast Trends and Predict Missing Values
19. 19
search and filter | munge | report | cleanup
Search Processing Language
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) dc(clientip)
| rename sum(KB) AS "Total MB" dc(clientip) AS "Unique Customers"
20. 20
Five Commands That Will Solve Most Data Questions
eval - Modify or Create New Fields and Values
stats - Calculate Statistics Based on Field Values
eventstats - Add Summary Statistics to Search Results
streamstats - Cumulative Statistics for Each Event
transaction - Group Related Events Spanning Time
24. 26
stats – Calculate Statistics Based on Field Values
Examples
• Calculate stats and rename
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) AS “Total KB”
• Multiple statistics
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB)
• By another field
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB) by clientip
25. 27
stats – Calculate Statistics Based on Field Values
Examples
• Calculate stats and rename
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) as “Total KB”
• Multiple statistics
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB)
• By another field
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB) by clientip
26. 28
stats – Calculate Statistics Based on Field Values
Examples
• Calculate statistics
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) AS "Total KB”
• Multiple statistics
sourcetype=access*
| eval KB=bytes/1024
| stats avg(KB) sum(KB)
• By another field
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB) by clientip
27. 30
eventstats – Add Summary Statistics to Search Results
Examples
• Overlay Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes
| timechart latest(avg_bytes) avg(bytes)
• Moving Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes by date_hour
| timechart latest(avg_bytes) avg(bytes)
• By created field
sourcetype=access*
| eval http_response = if(status == 200, "OK", "Error”)
| eventstats avg(bytes) AS avg_bytes by http_response
| timechart latest(avg_bytes) avg(bytes) by http_response
28. 31
Examples
• Overlay Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes
| timechart latest(avg_bytes) avg(bytes)
• Moving Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes by date_hour
| timechart latest(avg_bytes) avg(bytes)
• By created field
sourcetype=access*
| eval http_response = if(status == 200, "OK", "Error”)
| eventstats avg(bytes) AS avg_bytes by http_response
| timechart latest(avg_bytes) avg(bytes) by http_response
eventstats – Add Summary Statistics to Search Results
29. 32
eventstats – Add Summary Statistics to Search Results
Examples
• Overlay Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes
| timechart latest(avg_bytes) avg(bytes)
• Moving Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes by date_hour
| timechart latest(avg_bytes) avg(bytes)
• By created field
sourcetype=access*
| eval http_response = if(status == 200, "OK", "Error”)
| eventstats avg(bytes) AS avg_bytes by http_response
| timechart latest(avg_bytes) avg(bytes) by http_response
30. 34
streamstats – Cumulative Statistics for Each Event
Examples
• Cumulative Sum
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total
| timechart max(bytes_total)
• Cumulative Sum by Field
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total by status
| timechart max(bytes_total) by status
• Moving Average
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| streamstats avg(avg_bytes) AS moving_avg_bytes window=10
| timechart latest(moving_avg_bytes) latest(avg_bytes)
31. 35
streamstats – Cumulative Statistics for Each Event
Examples
• Cumulative Sum
sourcetype=access*
| timechart sum(bytes) as bytes
| streamstats sum(bytes) as cumulative_bytes
| timechart max(cumulative_bytes)
• Cumulative Sum by Field
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total by status
| timechart max(bytes_total) by status
• Moving Average
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| streamstats avg(avg_bytes) AS moving_avg_bytes window=10
| timechart latest(moving_avg_bytes) latest(avg_bytes)
32. 36
streamstats – Cumulative Statistics for Each Event
Examples
• Cumulative Sum
sourcetype=access*
| timechart sum(bytes) as bytes
| streamstats sum(bytes) as cumulative_bytes
| timechart max(cumulative_bytes)
• Cumulative Sum by Field
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total by status
| timechart max(bytes_total) by status
• Moving Average
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| streamstats avg(avg_bytes) AS moving_avg_bytes
window=10
| timechart latest(moving_avg_bytes) latest(avg_bytes)
33. 38
transaction – Group Related Events Spanning Time
Examples
• Group by Session ID
sourcetype=access*
| transaction JSESSIONID
• Calculate Session Durations
sourcetype=access*
| transaction JSESSIONID
| stats min(duration) max(duration) avg(duration)
• Stats is Better
sourcetype=access*
| stats min(_time) AS earliest max(_time) AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration) avg(duration)
34. 39
transaction – Group Related Events Spanning Time
Examples
• Group by Session ID
sourcetype=access*
| transaction JSESSIONID
• Calculate Session Durations
sourcetype=access*
| transaction JSESSIONID
| stats min(duration) max(duration) avg(duration)
• Stats is Better
sourcetype=access*
| stats min(_time) AS earliest max(_time) AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration) avg(duration)
35. 40
transaction – Group Related Events Spanning Time
Examples
• Group by Session ID
sourcetype=access*
| transaction JSESSIONID
• Calculate Session Durations
sourcetype=access*
| transaction JSESSIONID
| stats min(duration) max(duration) avg(duration)
• Stats is Better
sourcetype=access*
| stats min(_time) AS earliest max(_time) AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration) avg(duration)
36. 41
Learn Them Well and Become a Ninja
eval - Modify or Create New Fields and Values
stats - Calculate Statistics Based on Field Values
eventstats - Add Summary Statistics to Search Results
streamstats - Cumulative Statistics for Each Event
transaction - Group Related Events Spanning Time
See many more examples and neat tricks at docs.splunk.com and answers.splunk.com
43. 48
cluster – Find Common and/or Rare Events
Examples
• Find the most common events
*
| cluster showcount=t t=0.1
| table cluster_count, _raw
| sort - cluster_count
• Select a field to cluster on
sourcetype=access*
| cluster field=bc_uri showcount=t
| table cluster_count bc_uri _raw
| sort -cluster_count
• Most or least common errors
index=_internal source=*splunkd.log* log_level!=info
| cluster showcount=t
| table cluster_count _raw
| sort -cluster_count
44. 49
cluster – Find Common and/or Rare Events
Examples
• Find the most common events
*
| cluster showcount=t t=0.1
| table cluster_count, _raw
| sort - cluster_count
• Select a field to cluster on
sourcetype=access*
| cluster field=bc_uri showcount=t
| table cluster_count bc_uri _raw
| sort -cluster_count
• Most or least common errors
index=_internal source=*splunkd.log* log_level!=info
| cluster showcount=t
| table cluster_count _raw
| sort -cluster_count
45. 50
cluster – Find Common and/or Rare Events
Examples
• Find the most common events
*
| cluster showcount=t t=0.1
| table cluster_count, _raw
| sort - cluster_count
• Select a field to cluster on
sourcetype=access*
| cluster field=bc_uri showcount=t
| table cluster_count bc_uri _raw
| sort -cluster_count
• Most or least common errors
index=_internal source=*splunkd.log* log_level!=info
| cluster showcount=t
| table cluster_count _raw
| sort -cluster_count
Editor's Notes
Here is what you need for this presentation:
Link to videos on box: <coming soon>
You should have the following installed:
6.4 Overview
OI Demo 3.2 – Note this is not on Enablement yet. Please request this from sluedtke@splunk.com. The enablement link will be placed here once availabile.
NOTE: Configure your role to search the oidemo index by default, otherwise you will have to type “index=oidemo” for the examples later on.
There is a lot to cover in this presentation! Try to go quickly and at a pretty high level. When you get through the presentation judge the audience’s interest and go deeper in whichever section. For example, if they want to know more about Choropleths and polygons spend some time there, or if they want to go deeper on the search commands talk through the extra examples.
Objective: We want to help you change from this..
Today I’m going to show you some of the new features available in Splunk 6.4.
For TCO & Performance Improvements we’ve created new options to reduce your storage footprint as well as a new event sampling feature to optimize query performance and help you answer questions faster.
For Platform Security and Management we have added new single sign-on capabilities, new features to the HTTP Event Collector and finally new views and dashboards to the Distributed Management Console.
Then for my favorite part, the new Interactive Visualizations. Not only did we double the amount of visualizations available in Splunk, but we’ve provided a way for developers, partners and the community to create their own and integrate with the Splunk interface natively.
Lastly we will go through some of the most commonly used search commands and how they are used so you can become a Splunk Ninja in 6.4!
To this…
Splunk safe harbor statement.
Let’s start with TCO & Performance Improvements.
Extra Material:
Q: How does it affect performance? Can I still search the data?
A: You can access the data in all of the normal ways, and for many search and reporting activities there is little impact. But for “needle in the haystack” ad-hoc searches, the performance will no longer be optimal. For “dense” searches (searches whose results return most of the data for the time range searched), the performance impact will be minimal. For “sparse” or “needle in the haystack” searches (searches that return very few results), searches that typically return in seconds will now return in minutes. Note: This feature can be selectively applied to any index to provide the greatest amount of flexibility to our customers.
The goal is to apply this feature to data that is less frequently accessed – data for which you are willing to sacrifice some performance in order to gain a very significant cost savings. Splunk specialists can help you set the right policies for the right data.
Q: Do apps and Premium Solutions still work?
A: Yes. Apps and Premium Solutions will work.
Q: How does it affect performance? Can I still search the data?
A: You can access the data in all of the normal ways, and for many search and reporting activities there is little impact. But for “needle in the haystack” ad-hoc searches, the performance will no longer be optimal. For “dense” searches (searches whose results return most of the data for the time range searched), the performance impact will be minimal. For “sparse” or “needle in the haystack” searches (searches that return very few results), searches that typically return in seconds will now return in minutes. Note: This feature can be selectively applied to any index to provide the greatest amount of flexibility to our customers.
The goal is to apply this feature to data that is less frequently accessed – data for which you are willing to sacrifice some performance in order to gain a very significant cost savings. Splunk specialists can help you set the right policies for the right data.
Q: How do I control what data is minimized? Can I bring data back to the standard state?
A: You set policy by data age and by the type of data (index). Different data can have different time criteria for minimization. You can return data to the original state if needed. Splunk specialists can help you set the right policies for the right data.
Q: Why does your optimization data take up so much space?
A: Even including the optimization data, Splunk compression techniques have already reduced the customer’s storage requirements by over 50% during indexing. The optimization metadata (TSIDX – time-series index) is what enables the customer to ask any question of their data and handle any type of investigation or use case in real time.
By keeping data in its original unstructured state, Splunk offers the flexibility to ask any question of the data, handling any type of investigation or use case. Splunk structures the answer to each query on the fly, rather than forcing the customer to create a fixed data structure that limits the questions that can be asked. The TSIDX data enables us to deliver this unique flexibility with real-time speed.
Q: Why is the savings range so large (40-80%)?
A: The storage used by TSIDX varies depending on the nature and cardinality (uniqueness) of the data indexed. So the savings will vary as well across data types. Repetitive data fields will have a lower savings while unique (high cardinality) data will see a higher savings. Typical syslog data, for example will fall in the middle – about 60-70%.
High cardinality data returns a higher savings because it requires more index entries to describe it. When the TSIDX is reduced, the savings are larger. We expect most customers will see an overall benefit of 60% or more. We expect the average savings to be 60% or more.
Platform Security & Management
DMC
In 6.3 we re-worked the Distributed Management Console. In 6.4 we enhanced it even more adding new views and monitoring capabilities for things such as:
- HTTP Event Collector Views - Performance tracking for the HTTP Event Collector feature including breakdowns by authorization token.
- TCP Inputs - A partner to the Forwarder performance views in DMC tracking TCP queue health and other TCP input statistics.
Deployment Wide Search Statistics - Identify top Search Users across a multi-Search Head deployment including frequent and long running searches.
- Distributed Search View - A dashboard dedicated to tracking metrics for search in distributed deployments. Includes views for bundle replication performance and dispatch directory statistics.
- Resource Usage, I/O - In addition to useful data on CPU and Memory consumption, now also see I/O bandwidth utilization for any Splunk host or across hosts.
- Index Performance, Multi-pipeline - Updated views in the Deployment-wide and Instance-scoped Indexing Performance pages to accommodate multi-pipeline indexing.
- Threshold Control - Fine-grain controls for visual thresholds for DMC displays containing CPU, Memory, Indexing Rate, Search Concurrency, and Up/Down Status.
HTTP Event Collector
In 6.3 we added the HTTP Event Collector. Now we’ve improved it by enabling unrestricted data for payloads (besides JSON) and data indexing acknowledgements so customers can verify data was received.
SAML
And finally we’ve added additional Single Sign On Options for added flexibility
Platform Security & Management
Release 6.4 delivers an array of new pre-built visualizations, a visualization developer framework, and an open library to make it simple for customers to access, develop and share interactive visualizations
15 new pre-built visualizations help customers analyze and interact with data sets commonly found in IT, security, and machine learning analysis
A new developer framework allows customers and partners to easily create or customize any visualization to suit their needs
Splunkbase now contains a growing library of visualizations provided by Splunk, our partners and our community
Doubles the visualizations in Splunk today and creates an open environment for the unlimited creation and sharing of new visualizations
Once a visual is imported from SplunkBase it is treated the same as any native Splunk feature, and is available for general use in the Visualizations dropdown.
15 new pre-built visualizations help customers analyze and interact with data sets commonly found in IT, security, and machine learning analysis. We survey out customer and field to choose an initial set that would meet many common needs.
The new Event Sampling feature makes it faster to characterize very large datasets and focus your investigations. It is an integrated option of Search, offering a dropdown menu to control sampling 1 per 10, per 100, 1000, 10,000 etc.
Of course the performance is equally as fast – a 1 per 1000 search runs 1000x faster.
Main algo used – Kalgan filter
Algorithmic improvements
Support bi-variate time series by taking covariance between the individual time series into account.
Predict for multiple time series at the same time - this treats individual time series independently, i.e. without computing covariance
Predicting missing values in time series and accounting for that during prediction via missing value imputation methods (i.e., “No value was recorded, but it was most likely 5”)
Use Splunk Ninja App and Demo Instructions
For more information, or to try out the features yourself. Check out the overview app which explains each of the features and includes code samples and examples where applicable.
<This section should take ~15 minutes>Search is the most powerful part of Splunk.
The Splunk search language is very expressive and can perform a wide variety of tasks ranging from filtering to data, to munging, and reporting. The results can be used to answer questions, visualize results, or even send to a third party application in whatever format they require.
Although there are 135 documented search commands; however, most questions can be answered by using just a handful.
These are the five commands you should get very familiar with. If you know how to use these well, you will be able to solve most data questions that come your way. Let’s take a quick look at each of these.
<Walk through the examples with a demo. Hidden slides are available as backup. NOTE: Each of the grey boxes is clickable. If you are running Splunk on port 8000 you won’t have to type in the searches, this will save time.>
Note: Chart is just stats visualized. Timechart is just stats by _time visualized.
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) AS "Sum of KB"
sourcetype=access*
| stats values(useragent) avg(bytes) max(bytes) by clientip
sourcetype=access*
| stats values(useragent) avg(bytes) max(bytes) by clientip
Eventstats let’s you add statistics about the entire search results and makes the statistics available as fields on each event.
<Walk through the examples with a demo. Hidden slides are available as backup>
Eventstats let’s you add statistics about the entire search results and makes the statistics available as fields on each event.
Let’s use eventstats to create a timechart of the average bytes on top of the overall average.
index=* sourcetype=access*
| eventstats avg(bytes) AS avg_bytes
| timechart latest(avg_bytes) avg(bytes)
We can turn this into a moving average simply by adding “by date_hour” to calculate the average per hour instead of the overall average.
index=* sourcetype=access*
| eventstats avg(bytes) AS avg_bytes by date_hour
| timechart latest(avg_bytes) avg(bytes)
Streamstats calculates statistics for each event at the time the event is seen. So for example, if I had an event with a temperature reading I could use streamstats to create a new field to tell me the temperature difference between the event and one or more previous events. Similar to the delta command, but more powerful. In this example, I’m going to take the bytes field of my access logs and see how much total data is being transferred code over time.
To create a cumulative sum:
sourcetype=access*
| timechart sum(bytes) as bytes
| streamstats sum(bytes) as cumulative_bytes
| timechart max(cumulative_bytes)
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total by status
| timechart max(bytes_total) by status
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| streamstats avg(avg_bytes) AS moving_avg_bytes window=10
| timechart latest(moving_avg_bytes) latest(avg_bytes)
Bonus: This could also be completed using the trendline command with the simple moving average (sma) parameter:
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| trendline sma10(avg_bytes) as moving_average_bytes
| timechart latest(avg_bytes) latest(moving_average_bytes)
Double Bonus: Cumulative sum by period
sourcetype=access*
| timechart span=15m sum(bytes) as cumulative_bytes by status
| streamstats global=f sum(cumulative_bytes) as bytes_total
A transaction is any group of related events that span time. It’s quite useful for finding overall durations. For example, how long did it take a user to complete a transaction. This really shows the power of Splunk. Think about it, if you are sending all your data to splunk then you have data from multiple subsystems (think database, webserver, and app server), you can see the overall time it’s taking AND how long each subsystem is taking. So many customers are using this to quickly pinpoint whether slowness is because of the network, database, or app server.
NOTE: Many transactions can be re-created using stats. Transaction is easy but stats is way more efficient and it’s a mapable command (more work will be distributed to the indexers).
sourcetype=access*
| stats min(_time) AS earliest max(_time) AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration) avg(duration)
There is much more each of these commands can be used for. Check out answers.splunk.com and docs.splunk.com for many more examples.
Android coming soon!
Now go do this Fu in your own environment!
But don’t just say you know the “Fu”…
<If you have time, feel free to show one of your favorite commands or a neat use case of a command. The cluster command is provided here as an example >
“There are over 135 splunk commands, the five you have just seen are incredibly powerful. Here is another to add to your arsenal.”
You can use the cluster command to learn more about your data and to find common and/or rare events in your data. For example, if you are investigating an IT problem and you don't know specifically what to look for, use the cluster command to find anomalies. In this case, anomalous events are those that aren't grouped into big clusters or clusters that contain few events. Or, if you are searching for errors, use the cluster command to see approximately how many different types of errors there are and what types of errors are common in your data.
Decrease the threshold of similarity and see the change in results
sourcetype=access* | cluster field=bc_uri showcount=t t=0.1| table cluster_count bc_uri _raw | sort -cluster_count