The document outlines the agenda for a virtual SplunkLive! event for higher education on January 28, 2015. It includes an overview of Splunk, presentations from various universities on their Splunk implementations, and breakout sessions on getting started with Splunk, security, and IT operations. It also provides information on Splunk products and capabilities for IT operations, security, application delivery, business analytics, industrial data, and the Internet of Things.
3. (ALL
TIMES
EASTERN
US
TIME
ZONE)
1:00
Welcome
1:10
Splunk
Overview
[Monzy
Merza,
Splunk]
1:45
Internet2
NET+
Splunk
Offering
[Andrew
Kea_ng,
I2]
2:00
Ohio
State
University
[Mark
Runals]
2:30
Baylor
University
[Jon
Allen,
Keith
Schonenfield]
3:00
University
of
Washington
[S.
De
Vight,
P.
Michaud]
3:30
Splunk
Cloud
[Nick
Pavlovich,
Splunk]
3:50
10
minute
break
4:00
Breakout
Sessions
Gecng
Started
Security
IT
Opera_ons
TODAY’S
AGENDA
5. 5
Safe
Harbor
Statement
During
the
course
of
this
presenta_on,
we
may
make
forward
looking
statements
regarding
future
events
or
the
expected
performance
of
the
company.
We
cau_on
you
that
such
statements
reflect
our
current
expecta_ons
and
es_mates
based
on
factors
currently
known
to
us
and
that
actual
events
or
results
could
differ
materially.
For
important
factors
that
may
cause
actual
results
to
differ
from
those
contained
in
our
forward-‐looking
statements,
please
review
our
filings
with
the
SEC.
The
forward-‐looking
statements
made
in
this
presenta_on
are
being
made
as
of
the
_me
and
date
of
its
live
presenta_on.
If
reviewed
ager
its
live
presenta_on,
this
presenta_on
may
not
contain
current
or
accurate
informa_on.
We
do
not
assume
any
obliga_on
to
update
any
forward
looking
statements
we
may
make.
In
addi_on,
any
informa_on
about
our
roadmap
outlines
our
general
product
direc_on
and
is
subject
to
change
at
any
_me
without
no_ce.
It
is
for
informa_onal
purposes
only
and
shall
not
be
incorporated
into
any
contract
or
other
commitment.
Splunk
undertakes
no
obliga_on
either
to
develop
the
features
or
func_onality
described
or
to
include
any
such
feature
or
func_onality
in
a
future
release.
6. Disrup;ve
Approach
to
Unstructured
Data
Structured
RDBMS
SQL
Search
Schema
at
Write
Schema
at
Read
1980-‐2010
2010+
ETL
Universal
Indexing
Unstructured
Volume
|
Velocity
|
Variety
7. 7
Make
machine
data
accessible,
usable
and
valuable
to
everyone.
7
7
7
8. COLLECT
DATA
FROM
ANYWHERE
SEARCH
AND
ANALYZE
EVERYTHING
GAIN
REAL-‐TIME
OPERATIONAL
INTELLIGENCE
The
Power
of
Splunk
8
9. 9
Why
Splunk?
FAST
TIME-‐TO-‐VALUE
ONE
PLATFORM,
MULTIPLE
USE
CASES
VISIBILITY
ACROSS
STACK,
NOT
JUST
SILOS
ASK
ANY
QUESTION
OF
DATA
ANY
DATA,
ANY
SOURCE
OR
DEPLOYMENT
MODEL
10. 10
Turning
Machine
Data
Into
Business
Value
Index
Untapped
Data:
Any
Source,
Type,
Volume
Online
Services
Web
Services
Servers
Security
GPS
Loca_on
Storage
Desktops
Networks
Packaged
Applica_ons
Custom
Applica_ons
Messaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call
Detail
Records
Smartphones
and
Devices
RFID
On-‐
Premises
Private
Cloud
Public
Cloud
Ask
Any
Ques;on
Applica;on
Delivery
Security,
Compliance
and
Fraud
IT
Opera;ons
Business
Analy;cs
Industrial
Data
and
the
Internet
of
Things
11. Phases
of
Opera;onal
Intelligence
Reac;ve
Search
and
Inves_gate
Proac_ve
Monitoring
and
Aler_ng
Opera_onal
Visibility
Proac;ve
Real-‐_me
Business
Insight
12. IT
Opera_ons
Applica_on
Delivery
Developer
Plamorm
(REST
API,
SDKs)
Business
Analy_cs
Industrial
Data
and
Internet
of
Things
12
Delivers
Value
Across
IT
and
the
Business
Security,
Compliance,
and
Fraud
13. Why
Domino’s
uses
Splunk
for
Applica;on
Management
and
Business
Analy;cs
Understand
device
and
app
usage
trends
for
orders
Real-‐;me
revenue
insights
from
store
data
Visibility
into
online
and
mobile
coupon
redemp;on
Refine
campaigns
for
higher
conversion
13
14. 14
Apps
&
Capabili;es
for
Business
Analy;cs
Apps,
Features
&
Partners
• DB
Connect
• Stream
• ODBC
Driver
• Data
Models
• Pivot
15. IT
Opera_ons
Security,
Compliance,
and
Fraud
Applica_on
Delivery
Developer
Plamorm
(REST
API,
SDKs)
Business
Analy_cs
Industrial
Data
and
Internet
of
Things
15
Delivers
Value
Across
IT
and
the
Business
16. Building
Smarter
Transporta;on
Improving
Safety
Reducing
Fuel
Costs
Improving
On-‐Time
Opera_ons
Over
$1
Billion
in
Poten;al
Savings
16
17. 17
Apps
&
Capabili;es
for
Industrial
Data
&
Internet
of
Things
• DBConnect
• REST
API
and
SNMP
Modular
Inputs
• Universal
Forwarder
for
Raspberry
Pi
Apps,
Features
&
Partners
REST
19. 19
What’s
New
in
Splunk
Enterprise
6.2
Gecng
Data
In
Advanced
Field
Extractor
Instant
Pivot
Event
Paqern
Detec_on
Prebuilt
Panels
Search
Head
Clustering
Distributed
Management
Console
Powerful
Analy;cs
for
Broader
Number
of
Users
Faster
Data
Onboarding
Breakthrough
Scalability
and
Centralized
Mgmt.
20. Unparalleled
Cloud
Service
for
Machine
Data
100%
Up;me
SLA
Hybrid
Plaform
Secure
and
Reliable
Instant
Access
20
21. 21
What’s
New
in
Hunk
6.2
Hunk
Sandbox
Data
Explorer
Faster
to
Deploy
and
Gain
Value
Instant
Pivot
Event
Paqern
Detec_on
Prebuilt
Panels
More
Powerful
Analy;cs
for
Everyone
AWS
Hunk
Service
Hunk
Apps
Extend
Exploratory
Analy;cs
22. Extending
Opera;onal
Intelligence
to
Mobile
Apps
Deliver
Beqer
Performing,
More
Reliable
Apps
Deliver
Real-‐Time
Omni-‐Channel
Analy_cs
End-‐to-‐End
Performance
and
Capacity
Insights
22
23. New
Data
Sources
Universal
Forwarder
on
z/Linux
Syncsort
Ironstream
on
z/OS
Mainframe
Kepware
Industrial
Data
23
Splunk
App
for
Stream
Wire
Data
24. Mainframe
Data
VMware
Plamorm
for
Machine
Data
Easy
to
Adopt
Splunk
Exchange
PCI
Security
DB
Connect
Mobile
Forwarders
Syslog
/
TCP
/
Other
Sensors
&
Control
Systems
Rich
Ecosystem
of
Apps
Across
Data
Sources,
Use
Cases
&
Consump;on
Models
Stream
24
26. Educa;on
Healthcare
Technology
Energy
and
U;li;es
Manufacturing
Telecommunica;ons
Cloud
and
Online
Services
Government
Retail
Financial
Services
and
Insurance
Media
Travel
and
Leisure
26
Proven
at
8,400+
Customers
in
100
Countries
Over
3/4
the
Fortune
100
27. FREE
ONLINE
SANDBOX
FREE
DOWNLOAD
FREE
AMAZON
MACHINE
IMAGES
(AMI)
27
Easy
to
Try
&
Get
Started
1
3
2
36. 36
About Me
IT Security in some fashion for 12+ years
At OSU for 2 ½ years
Using Splunk for 2 ½ years (direct correlation)
Other LM/SIEM Space
• Managed a medium size ArcSight deployment
• Used Symantec’s MSSP
Splunk Apps:
• Data Curator, Forwarder Health, Change Tracker/Config Mgmt
37. 37
Large Place
64k Students; 43k Staff; 175 Undergraduate Programs; ~200k IPs
Distributed
100+ IT groups; 30 CIOs; 7 Campuses; 1,245 Buildings; own zip code
Technology
You name it we probably have it (somewhere)
OSU Environment
38. 38
1.7 TB data per day
430B events in the system
10k+ Devices
12 types of firewalls
Multiple OS
90+ teams with data in Splunk
700+ different types of data
350+ users
Splunk After 2+ Years
39. 39
Lessons Learned
Don’t boil the ocean
• Have a data rollon / data definition process
• Start leveraging a Common Information Model (CIM)
Check out Splunk’s
There are different work streams
• Data Management – getting data in
• Knowledge Management – getting data out
Data Curator app
• Designed to help with previous point
40. 40
Splunk – First Steps
1. If you have firewall data make an interactive dashboard that helps
teams identify blocks.
2. Go out and buy a 30” or 40” TV and display something on it
• Splunk v6.x embedded reports
• Huge ROI
41. 41
Don’t Display…
Top 5 Countries Attacking Us
1. China
2. US
3. Romania
4. Somewhere
5. Somewhere Else
Top 5 Authentication Locations
1. Columbus, OH
2. Ohio (other)
3. US
4. etc
5. etc
42. 42
IDS – Last 24hrs
Use built in Splunk map if you must; doesn’t display numbers /sigh
49. 49
Accounts Sending Spam
sourcetype = snort [sourcetype = msexchange_data sender=
$user$ original_client_ip=* | dedup original_client_ip | rename
original_client_ip as src_ip | fields src_ip] | …
Pass the user name token (red) to the subsearch (blue) which pulls out the
associated IPs and renames them according to the field snort uses
50. 50
Grade Change
• Investigation kickoff evidence – lockpick stuck in lock
• Many logs useful
• Learning Management System
• Various authentication logs
• Wireless
51. 51
Library Proxy Abuse
OSU pays for
online resources
Student falls for
phishing
Malicious site
leverages account
creds and library proxy
Notification by vendor that there was an issue
• Had user name – how can we identify malicious behavior?
52. 52
Recent Security Work Leveraging Splunk
User Agent string looks interesting!
Often the malicious actors will setup a website that leverages the compromised creds.
The number of source IPs will be very low.
53. 53
Cheating on LMS Tests
Online test taking will only grow
What can we use to spot anomalies?
Ø Multiple tests from same IP
Ø Time elements from tests (ie time taken vs avg time)
56. 56
Summary
Going from a data repository to an engine takes time
You have a data lake full of black swans
• Use use cases to drive your efforts / start somewhere
• Don’t wait for perfect
59. 59
About
Baylor
• Private
faith
based
ins_tu_on
• Founded
in
1845
• 16,260
students
• Over
2,900
faculty/staff
60. 60
Jon
Allen
• Over
15
years
at
Baylor
University
• Started
the
informa_on
security
group
• M.S.
Computer
Science
61. 61
Keith
Schoenefeld
• 15
Years
in
Higher
Educa_on
Informa_on
Security
• Vulnerability
Management
• Log
Management
(ng-‐syslog,
rsyslog,
Splunk)
• Splunk
Cer_fied
Architect
by
the
end
of
February.
62. 62
Enhancing
Security
Infrastructure
• PCI
compliance
• Gaining
vision
into
high
volume
log
sources
– Ac_ve
Directory
– Firewalls
– IDS/IPS
• Build
a
new
service
within
IT
that
has
security
advantages
63. 63
Ini_a_ve
Buy
In
• Great
security
wants
us
to
do
what
• Push
the
opera_onal
benefits
• Find
one
or
two
early
wins
64. 64
Cluster
Master
Cluster
Members
Dedicated
Search
Head
Splunk
Forwarders
.
.
.
65. 65
Technical
Specifica_ons
• Dedicated
Search
Head
(x1)
– 48
cores
– 64G
RAM
• Cluster
Members
(x3)
– Clustered
for
High
Availability
and
Faster
Searching
– Each
has:
ê 3.3
TB
local
storage,
configured
in
RAID
10
(~2000
iops)
ê 10
TB
SAN
storage
(~
700
iops)
ê 32
cores
ê 64G
RAM
66. 66
Networking
Group
• Firewall
• IPS
• IAS
• DHCP
• Networking
Devices
• Windows
Servers
• Linux
Servers
Servers
• Ac_ve
Directory
• Exchange
• Linux
Servers
PCI
• Firewall
• IPS
• Ac_ve
Directory
Client
Services
• AV
Items
in
RED
are
logs
we
could
not
previously
access
effec_vely.
67. 67
Proven
Effec_veness
• Servers
ê User
Login
troubleshoo_ng
– Cuts
troubleshoo_ng
_me
from
3
hours
to
10
minutes
each
ê Email
flow
troubleshoo_ng
– Cuts
troubleshoo_ng
_me
from
1
hour
to
10
minutes
each
ê Server
Performance
sta_s_cs
– Exchange
Volumes
69. 69
Robust
Toolset
• Raw
logs
to
knowledge
in
minutes
• Use
visuals
to
explain
complex
issues
• Link
disparate
data
sources
70. 70
Shellshock
Time
Ac;on
Device
Source
IP
Dest
IP
Dest
Port
Dest
Net
Tue
Oct
21
04:33:56
2014
ids
bro
89.121.161.232
129.62.aa.bb
80
DC
Tue
Oct
21
04:34:02
2014
reset-‐both
PAN
89.121.161.232
129.62.aa.bb
80
DC
Tue
Oct
21
04:40:05
2014
ids
bro
188.10.85.113
129.62.cc.dd
80
Dept.
A
Tue
Oct
21
04:40:11
2014
reset-‐both
PAN
188.10.85.113
129.62.cc.dd
80
Dept.
A
Tue
Oct
21
04:40:23
2014
ids
bro
188.10.85.113
129.62.cc.ee
80
Dept.
A
Tue
Oct
21
04:40:28
2014
reset-‐both
PAN
188.10.85.113
129.62.cc.ee
80
Dept.
A
Tue
Oct
21
04:40:30
2014
ids
bro
188.10.85.113
129.62.cc.ff
80
Dept.
A
Tue
Oct
21
04:40:35
2014
reset-‐both
PAN
188.10.85.113
129.62.cc.ff
80
Dept.
A
83. 83
Agenda
" About
us
" Splunk
at
the
University
of
Washington
" Suppor_ng
an
exis_ng
service
" Providing
data
to
UX
with
client-‐side
instrumenta_on
84. 84
Academic
and
Collabora_ve
Applica_ons
" A
division
within
UW-‐IT
focused
on
building
student
facing
Web
applica_ons
" Must
develop
new
applica_ons
while
maintaining
legacy
applica_ons
with
limited
resources
" Facts
and
figures
– Small
team
of
6
engineers
– Maintain
~15
applica_ons
– Support
over
140,000
users
across
3
campuses
– Support
9
groups
on
campus
running
their
own
Splunk
instances
via
our
license
master
86. 86
My
Background
and
Role
" Stephen
De
Vight
– With
the
UW
since
2006
– Current
Role:
Web
Applica_on
Engineer,
2011
– Mission:
To
support
teaching
and
learning
on
campus
through
the
development
of
interac_ve
Web
and
mobile
applica_ons
89. 89
Suppor_ng
an
Exis_ng
Service
• Homegrown
suite
of
academic
applica_ons
• Currently
consists
of
8
dis_nct
tools
• Released
in
1999
90. 90
Our
Needs
– Situa;on:
Legacy
database
logging
system
reached
end
of
life,
was
not
scaling
well,
and
was
too
costly
to
directly
replace
– Struggling
with:
Finding
a
solu_on
that
is
both
easy
to
build
and
maintain
as
well
as
being
able
to
scale
to
our
needs
– Wanted:
An
easy
to
use,
UI-‐driven,
applica_on
to
search
our
log
data
– Enter
Splunk:
Splunk
Enterprise
allowed
us
to
build
a
custom
searching
app
as
well
as
a
dashboard
for
monitoring
service
status
91. 91
Catalyst
Log
Search
• Advanced
XML
view
• Search
form
negates
the
need
for
users
to
learn
Splunk
search
language
or
understand
our
log
formacng
and
structure
• Support
can
analyze
user
ac_vity
to
provide
insight
into
incident
reports
Screenshot
here
92. 92
Catalyst
Dashboard
• Gauge
current
level
of
ac_vity
at
a
glance
• Examine
last
day
of
ac_vity
for
anomalous
usage
• Targets
slowest
loading
URLs
for
performance
improvement
93. 93
Data
Driven
User
Experience
• Mobile
Web
version
of
our
student
portal
• Focused
on
providing
_mely,
ac_onable
informa_on
to
our
students
• Based
on
a
student's
situa_on
and
the
_me
of
the
quarter
we
dynamically
display,
hide,
move,
and
reorder
content
94. 94
Our
Needs
– Situa;on:
UX
needs
a
way
to
validate
their
assump_ons
around
what
content
is
relevant
to
a
student
at
various
points
in
the
quarter
– Struggling
with:
Correla_ng
user
ac_vity
with
ins_tu_onal
data
(e.g.
class
standing,
campus,
etc.)
– Wanted:
A
self-‐driven
means
for
UX
and
business
analysts
to
analyze
log
data
– Enter
Splunk:
Splunk,
along
with
our
client-‐side
logging
solu_on,
allows
us
to
correlate
user
ac_vity
with
certain
ins_tu_onal
aqributes
we
log
95. 95
Client-‐Side
logging
• Google
Analy_cs
did
not
get
us
everything
we
needed
• Using
logger4javascript
to
collate
events
and
POST
to
a
REST
interface
• Events
are
bundled
to
reduce
network
overhead
• Events
are
wriqen
to
file
by
REST
server
hlp://www.log4javascript.org/
99. 99
Evenqypes
and
Transac_ons
index=myuw_production
(sourcetype=myuw_link_log
OR sourcetype=myuw_session_log)
Build
an
evenqype
that
contains
both
link
and
session
logs
100. 10
0
Session
Ac_vity
with
Transac_ons
index=myuw_production
eventtype=link_event
|transaction fields=session_key
maxspan=8h
|search target_url=*dars.asp
AND action=click
|stats count by target_url
• Create
a
transac_on
based
on
session_key
• Find
transac_ons
that
contain
a
link
click
to
‘*dars.asp’
• Get
count
of
other
URL
targets
clicked
within
that
transac_on
101. 10
1
Combining
Logs
with
Transac_ons
index=myuw_production eventtype=link_event
|transaction fields=session_key maxspan=8h
|search action=click
|stats count by class_level
• Create
a
transac_on
based
on
session_key
• Find
link
events
that
have
a
click
ac_on
• Using
the
session
log,
determine
how
many
link
clicks
were
made
by
each
class
level
102. 10
2
What’s
Next
" Add
more
of
our
applica_on’s
logs
to
Splunk
– Deploying
forwarders
via
Ansible
to
our
hosts
" Get
addi_onal
people
up
to
speed
with
querying
in
Splunk
" Reach
out
to
addi_onal
campus
partners
who
want
to
buy
into
the
license
103. 10
3
Top
Takeaways
" Building
a
search
form
makes
Splunk
simple
to
use
" Determine
your
analysis
needs
before
crea_ng
your
logging
scheme
" Client
side
logging
can
provide
valuable
insight
into
user
behavior
" Transac_ons
make
combining
logs
easy