4. What Does Machine Data Look Like?
Sources
Order Processing
Middleware
Error
Database
Error
Virtual Host
Failure
4
5. Machine Data Contains Critical Insights
Sources
Order Processing
Middleware
Error
Database
Error
Virtual Host
Failure
5
6. Machine Data Contains Critical Insights
Sources
Order Processing
Middleware
Error
Database
Error
Virtual Host
Failure
6
7. Splunk : Index and Analyze Any Data, Any Amount, Any Source
Powerful, end-to-end, real-time platform for Machine Data
Customer
Facing Data
Outside the
Datacenter
Click-stream data
Shopping cart data
Online transaction data
Logfiles
Windows
Registry
Event logs
File system
sysinternals
Linux/Unix
Configuration
s
syslog
File system
ps, iostat, top
Configs Messages
Traps
Alerts
Metrics
Virtualization
& Cloud
Scripts
Applications
Web logs
Log4J, JMS, JMX
.NET events
Code and scripts
Hypervisor
Guest OS, Apps
Cloud
7
Changes
Tickets
Databases
Configurations
Audit/query
logs
Tables
Schemas
Manufacturing,
logistics…
CDRs & IPDRs
Power consumption
RFID data
GPS data
Networking
Configurations
syslog
SNMP
netflow
8. Splunk : Index and Analyze Any Data, Any Amount, Any Source
Powerful, end-to-end, real-time platform for Machine Data
Customer
Facing Data
Outside the
Datacenter
Click-stream data
Shopping cart data
Online transaction data
Manufacturing,
logistics…
CDRs & IPDRs
Power consumption
RFID data
GPS data
Any amount, any location, any source.
Logfiles
Windows
Registry
Event logs
File system
sysinternals
Linux/Unix
Configuration
s
syslog
File system
ps, iostat, top
Configs Messages
Traps
Alerts
Metrics
Scripts
Changes
Tickets
No upfront schema
No custom connectors
Virtualization
Databases
No RDBMS Applications
& Cloud
Web logs
Configurations
Log4J, JMS, JMX
Hypervisor to
No needApps filter/forward Audit/query
.NET events
logs
Guest OS,
Code and scripts
Cloud
8
Tables
Schemas
Networking
Configurations
syslog
SNMP
netflow
9. Splunk Enables the Connected Datacenter
Business Insights
Gain real-time insight from your machine data to
make better-informed business decisions.
Cloud Services
Operational Visibility
Gain operational visibility to make betterinformed IT decisions.
Custom
Applications
Packaged
Applications
Proactive Monitoring
Monitor infrastructure to identify issues, problems
and attacks before they impact your customers
and services.
Infrastructure
Applications
Virtualization
Search and Investigation
Find and fix problems across the organization using
machine data.
Server, Storage,
Networking
9
10. Splunk : Platform For IT Operational Intelligence
Plug-Ins, Templates and Apps Accelerate Value From Machine Data
XenApp
XenDesktop
Web Intelligence
Server, Storage,
Network
Server
Virtualization
Operating
Systems
Infrastructure
Applications
SDKs
Business
Applications
Cloud
Services
Custom
Applications
UI
API
Other
Monitoring
Ticketing/Help
Desk
No rigid schemas– Add in data from any other source.
10
12. The Virtual Datacenter Challenge
Too much complexity and too little visibility
Not enough data about virtualization
• Most tools retain or report on summarized metrics that obfuscate real problems
• Most tools don’t proactively monitor logs
Virtualization data alone doesn't solve problems
• Solving end user or application level problems requires visibility at every
technology tier
Point solutions offer inadequate analyses
• Complete operational reporting for capacity planning, security reporting, end to
end performance and change impact analyses is missing
12
13. Key Considerations For Monitoring VMware
Environments
Provide access to underlying machine data to quickly identify problem spots
and troubleshoot issues in real-time
Persist data over time to determine performance and utilization trends for
planning, analytics and optimization
Gain holistic visibility across diverse infrastructures and heterogeneous
technologies
13
15. The Splunk App For VMware
Proactive
Monitoring
Proactive Identification of Problem Spots and Health Issues
Comprehensive Performance, Capacity, Security And Change Analyses
Analytics
Big Data
Solution
Scale And Correlate Across All Tiers Of Your Technology Stack
15
16. How It Works
Splunk
UF/LF
Provides: Dashboards,
Views, Field Extractions
Splunk Add-on
for vCenter
>
Splunk App
for VMware
VMware ESXi
VMware ESXi
From VC:
VC Logs
vCenter
server
>
Data Collection
Node (DCN)
Splunk
UF/LF
>
From VC:
Performance Metrics*,
Inventory, Hierarchy, Tasks,
and Events Data
From ESXi:
ESXi Logs
* Performance data at 20 s granularity
16
17. What’s New in v3.0?
Fast Time To Value
UI-based setup for fast and
easy installation,
management and
monitoring
Effortless
Scale-out
Provide analytics for large-scale
VMware deployments with
fewer data collectors and
reduced data volumes
17
Accelerated
Reporting
Dramatically improved
performance for search
and reporting
19. End-to-end Visibility
“ We have deep visibility and
correlation across all tiers of our
cloud infrastructure – giving us not
only ongoing monitoring of key
datacenter statistics, but also giving
us business visibility into customer
experience and usage.
”
Elad Gotfrid,
Manager of IT
Splunk used to correlate the business data
(users, usage) with the IT/Infrastructure data
Understand resource/usage and cost per customer
Monitor the entire environment from server, storage,
network, hypervisors, custom cloud back-end for
possible SLA issues, trouble spots and more
19
20. One Splunk – Many Uses
“ Using Splunk for VMware gets
us our data in one place, for
many uses: capacity planning,
event monitoring,
performance analysis, security
monitoring and more.
”
Peter Cole
Technical Lead, ITS Operations
A definitive record of what happened in our environment
Analyze and trend performance as well as user
activities very easily
Useful for both operational monitoring, capacity usage,
performance metrics and for security monitoring
20
21. Detailed History For Analysis &
Troubleshooting
“ I love that I can track virtual
machines in my environment as
they move from host to host. I
can now identify the root cause
of issues or errors.
”
Matthew Cluver
Network Operations Analyst
Splunk already used for operating system and
applications event monitoring & analysis
For the first time, they have insight into granular
virtualization layer data – helps solve problems
immediately
21
22. Easy Access To A Variety Of Data
“ With all our data stored centrally in
Splunk, it helps us to dive straight
into the source of problems by
looking at the context of the error
rather than manually digging
through multi-gigabyte log files
”
Delivered end-to-end visibility across the
infrastructure
Enabled 100% up-time with a 50% increase in
transactions
Reduced troubleshooting times from 1.5 hours per
log file to 5 minutes across VMware infrastructure
-- Big premium retail chain
22
23. Centralized Monitoring Across IT Operations
“ Splunk has become a critical
part of our operations;
everything funnels through
Splunk. It provides central
visibility to our various
teams and business units
”
-- Major Healthcare Management
Company
Cross correlate data across technologies to accurately
detect problem spots in business critical claims systems
Significantly reduced MTTR from 7-8 hours to less than
5 minutes per issue
Gain end-to-end insights across multiple types of web
servers, operating systems and storage on complex
VMware deployments
23
25. Why Splunk Over Everyone Else!
You don’t know what data you will need till you need it
– Every other tool only has access to 5 min summaries of data
– Most don’t even incorporate log data
Most other tools find it hard to collect & retain all the data
– Splunk scales to the largest datacenters; and not just for virtualization data
– Can be used for any use case – capacity, configuration monitoring, security,
change and asset tracking and more...
Splunk isn't JUST for virtualization – it is for everything
25
26. Operational Intelligence for IT and Business Users
IT Operations Management
Web Intelligence
Application Management
Business Analytics
Security and Compliance
Customer
Support
LOB Owners/
Executives
Operations
Teams
Website/Business
Analysts
System
Administrator
Application
Developers
Security
Analysts
26
Auditors
IT
Executives
27. Proven at 6,400+ Customers in 90+ Countries
Over 60 of the Fortune 100
Cloud and Online Services
Education
Energy and Utilities
Financial Services and Insurance
Government
Healthcare
Manufacturing
Media
Retail
Technology
Telecommunications
Travel and Leisure
27
28. A Growing, Global Community of Users
1,000+ unique
visitors per week
to dev.splunk.com
Local User Groups
and
SplunkLive events
320+ Apps and
20,000+ questions –
and answers
28
Annual
Users’ Conference
1,800+ users
29. Easy to Get Started
Download and install in minutes
1. Download
2. Eat your Machine Data
29
3. Start Splunking
32. Do I Really Need The Splunk App For VMware?
I already have vCOPS, how will the Splunk App for VMware help me?
The Splunk App for VMware provides unique insights into VMware environments that complements the
vCOps solution. Splunk differentiators include the ability to:
-
Collect and persist performance metrics at 20s granularity for troubleshooting, trending and
analytics
-
Analyze and monitor log and event data from ESX/i hosts and VCs, with a topology overlay
-
Correlate virtualization metrics with events, logs and performance metrics from applications, OSes, storage, networking or any other virtualization, software and hardware technologies
-
Scale to monitor, analyze and report the largest VMware deployments
-
Provide a range of analytics like capacity, security, change tracking without needing additional
software purchases
32
33. How Is Splunk Different From Log Insight?
VMware integrates Log Insight with vCOPS – how is Splunk different?
•
Log Insight is for (VMware) logs only: Splunk is far beyond just logs and individual technology layers. It’s more
about building a broad scope of insight and operational intelligence across an enterprise, in IT and the business
•
Log Insight and vCOPS are silo’ed tools with limited integrations: The Splunk App for VMware incorporates
and support analytics on VMware logs, performance metrics, topology, tasks, and events in one console. It supports
multiple use cases such as security, operational health, capacity planning, etc. Equivalent functionality on the
VMware stack requires 4-5 different products, additional licenses and more investment
•
Log Insight & vCenter Ops do not support cross‐tier correlation or analytics: Splunk has a very powerful
query language with over 200 commands for advanced analytics, reporting and correlation
•
Log Insight is yet to prove itself, particularly with large data volumes: Splunk is a proven solution with
over 5600 paying customers and tens of thousands of users of our free offering, with vibrant a community that has
built more than 400 Apps, most of them for free. Our largest customer implementation indexes over 100 TBs a day
and reports off petabytes of data at rest proving it’s scalability over enterprise-class IT environments
33
35. Immediate Visibility into the Overall Health
Identify overall health of your hosts and
determine if too much memory is being
reclaimed or swapped, if the CPU consumption is
high and drill down for specifics
Quickly visualize VM CPU consumption,
memory usage and CPY Wait times to
understand overall VM health across your
environment
Drill down for additional details on specific issues from anywhere on this report
Determine datastore over/under
consumption quickly for
optimization of memory usage
Gain insights into any system
alarms in the environment that
may need immediate attention
35
36. Visualize Multiple vCenters Instantly
Visualize the topology of the VMware
implementation in a tree-like view across
multiple vCenters in a single console
36
37. Threshold Based Reports On VM Performance
Report on each performance
counter based on pre-defined
thresholds for immediate
insights into any problems in the
environment
Compare performance of a single VM in
relation to the rest of the VMs in the
environment
37
38. Report on Virtual Machine Performance
Get dynamically notified on any issues in the VM
immediately
Drill down into a report to gain insights
into the VM
Track VMs as they move from one
host to the other
38
39. Chart Performance Baselines
Identify performance abnormalities on
vCenters/hosts/clusters/VMs by
comparing performance metrics on a
single node with the rest of the virtualized
environment.
39
40. Get Detailed Visibility Into the Hosts
Get notified on the abnormalities in the
hosts immediately
Identify host
configuration …
…and the connected datastores
…and the VMs and status of these VMs
…and audit trail of all tasks and events
40
…and system errors from
host logs
41. Drilldown for Memory Consumption on
Datastores
Get insights into the datastores
Drill down into the datastore to
understand which files are consuming
most space and memory with a detailed
list of all files and memory consumption
41
43. Get Capacity Insights
Choose the performance type, threshold and frequency for
a defined time period
Identify VCs and ESX/i hosts that meet the filter criteria
Drill down for trend over time
43
44. Monitor The Security Posture
Access reports on user, config changes, harmful logins, repeated login
attempts outside of permissions and more and gain insights into
security vulnerabilities
44
45. Track Changes and Audit Tasks and Events
View any tasks performed/changes made to the host or
VMs
Filter specific hosts or VMs of
interest in a folder like view that
retains the virtual infrastructure
hierarchy
45
46. Browse Logs Easily With Intelligent Filters
Identify vCenter Requests
Add additional filters
Filter specific hosts or VMs of
interest in a folder like view that
retains the virtual infrastructure
hierarchy
Browse through
service consolde,
vmkernel, hostd,
agent... logs
46
Notas del editor
Unlike traditional structured data or multi-dimensional data– for example data stored in a traditional relational database for batch reporting – machine data is non-standard, highly diverse, dynamic and high volume. You will notice that machine data events are also typically time-stamped – it is time-series data. Take the example of purchasing a product on your tablet or smartphone: the purchase transaction fails, you call the call center and then tweet about your experience. All these events are captured - as they occur - in the machine data generated by the different systems supporting these different interactions. Each of the underlying systems can generate millions of machine data events daily. Here we see small excerpts from just some of them.
When we look more closely at the data we see that it contains valuable information – customer id, order id, time waiting on hold, twitter id … what was tweeted. What’s important is first of all the ability to actually see across all these disparate data sources, but then to correlate related events across disparate sources, to deliver meaningful insight.
When we look more closely at the data we see that it contains valuable information – customer id, order id, time waiting on hold, twitter id … what was tweeted. What’s important is first of all the ability to actually see across all these disparate data sources, but then to correlate related events across disparate sources, to deliver meaningful insight.
Over the last 7 years, Splunk has grown from being a search engine for your underlying logs and analogous to google for IT data to an engine for machine data to a platform for operational intelligence. What do we mean by that? We have extended our solution to incorporate data from various data sources. Splunkbase has 300+ Apps, most of them being free Apps. The purpose of these Apps is to put context around the data (say from your firewalls or storage or network and such) and these Apps comes with a pre-built understanding of that data. The Apps are step1 to accelerating your value from the data. However, you’re not limited to what is available. Splunk’s capability to integrate with existing IT solutions and other monitoring solutions make us a platform to get visibility and intelligence on your IT operatipons. The Splunk SDKs empower developers to customize and extend the power of Splunk, establishing Splunk as the platform for machine data. We have partnered with other monitoring vendors to ingest data from their solutions into Splunk thus provding you complete and holistic visibilty. We hope that this is just the beginning and expect to open up a whole new world of enterprise apps. What have developers been building using Splunk Enterprise? Examples include the following:Run searches and retrieve Splunk data from existing Customer Service/Call Center applications (Comcast use case) Integrate Splunk data into existing BI tools and dashboard (Tableau, MS Excel)Build mobile applications with KPI dashboards and alerts powered by Splunk (Otto Group use case)Log directly to Splunk from remote devices (Bosch use cases)Build customer-facing dashboards powered by user-specific data in Splunk (Socialize, Hurricane Labs use cases)Programmatically extract data from Splunk for long-term data warehousingWe hope this is just the beginning. We hope to open up a whole new world of enterprise apps.
Understand how much resources each customer consume (CPU, Memory, Network, etc …) and when.Customer can have more then 1 VM or environment , splunk help us aggregate the date easily and look at the customer level usageSLA DashboardsMeasure service level Analyze and present statistics according to business guidelines
Peter Cole from melbourne IT cant wait to get Splunk App for VMware deployed across his environment. Some of the big benefits he gets from it:Find where storage is way over provisioned, clean up snapshots where they are taking up space, find errors in logs related to storageFind out what happened when in the environment, for troubleshooting, issue diagnosis, security reporting and moreUnderstand service levels of virtual machines in detail during performance/load testing
Rapid Troubleshooting and AnalysisDiscovery Communications, the world's largest non-fiction media company, uses Splunk to monitor application and operating system logs and events. The Splunk App for VMware enhances their operational visibility by giving them access to their virtualization layer data. With Splunk Discovery Communications gets an immediate understanding of virtualization layer failures and receives alerts before there is a full-blown impact on operations."I love that I can track virtual machines in my environment as they move from host to host.I can now identify the root cause of issues or errors" -Matthew Cluver, Network Operations Analyst, Discovery Communications.When asked which views of the app he likes – he liked them all!
Consolidate VMware, Network, storage, operating system and applications data
Customers start by using Splunk Enterprise to address one specific solution area. Then they leverage it and their machine data to solve other pressing problems over time.Consequently, Splunk Enterprise has many critical uses across IT and the business: Application Management: provide end-to-end visibility across distributed infrastructures; troubleshoot across application environments; monitor for performance degradation; trace transactions across distributed systems and infrastructure.Development: accelerate development and test cycles; support advanced development methodologies like agile, continuous; integrate enterprise applications with SDKs and a robust API; build enterprise applications that leverage Splunk software.Infrastructure and Operations Management: proactively monitor across IT silos to ensure uptime; rapidly pinpoint and resolve problems; report on SLAs/track SLAs of service providers.Security and Compliance: provide rapid incident response, real-time correlation and in-depth monitoring across data sources; statistical analysis for advance pattern detection and threat defense.Web and Business Analytics: gain visibility and intelligence on customers, services and transactions; identify trends and patterns in real time; fully understand the impact of new product features on back-end services.Both IT and business professionals can analyze machine data to get real-time visibility and operational intelligence.With our data engine and our customers' machine data, organizations can meaningfully improve their performance in a wide range of areas e.g. meet service levels, reduce costs, mitigate security risks, maintain compliance and gain insights.
More than 5,600 users in over 90 countries have purchased the enterprise license of Splunk. This includes a majority of the Fortune 100. Enterprises, service providers and government agencies in 90 countries use Splunk to improve service levels, reduce IT operations costs, mitigate security risks and drive new levels of operational visibility.As they gain new visibility into their real-time and historical machine data, Splunk’s customers are finding answers and solving the most challenging issues facing IT and the business.
With thousands of enterprise customers and an order of magnitude more actual users, we have a thriving community.We launched a dev portal a few months back and already have over 1,000 unique visitors per week.We have over 300 apps contributed by ourselves, our partners and our community.Our knowledge exchange Answers site has over 20,000+ questions answered.And in August 2012 we ran our 3rd users’ conference with over 1,000 users in attendance, over 100 sessions of content, customers presenting.Best of all, this community demands more from Splunk and gives us incredible feedback.
Splunk Enterprise is simple to deploy, scales from a single server deployment to global large-scale operations and delivers fast payback. Download Splunk Enterprise for free, install it in 5 minutes on your laptop or on any commodity server, point it at any machine data and start using it. Splunk software is often deployed for the first time while under fire. A serious service outage or security incident in progress is stressful, but with Splunk Enterprise, you can complete your investigation in a few minutes versus hours or days.