1. Copyright © 2013 Splunk Inc.
Splunk… on a Plane?
Ken Bantoft
VP Satcom Technology & Development
Satcom Direct

2. About Me
Spent 12 years doing Networking, Linux, High Performance
Computing in Finance, Bio-Technology and other sectors
Left IT in 2007 to focus on product development
Did a 1 week contract fixing Avionics Networking code, and haven’t
left Aviation since.
Now responsible for Product & Services Development at Satcom
Direct

3. About My Company
Satcom Direct provides connectivity and communications for
Aviation, Maritime and Land Mobile customers. Built around a
core focus of support and service, we now serve thousands of
customers world wide, including the Fortune 500, NATO & Allied
Forces, and various Heads of State.

4. Agenda
Splunk – not really on a plane (yet)
Data Sources
How we use Splunk
– Support – Monitoring & Alerting
– Business Analytics
Tracking Planes
– The technican’s flight tracker
Splunk Tips

5. Copyright © 2013 Splunk Inc.
Data Sources

6. Data Sources
We feed Splunk pretty much anything we can get our hands
on, both standard IT data, and some more esoteric data
–
–
–
–
–
CDRs for Phone Calls (AudioCodes, Asterisk)
Syslogs from network appliances & servers
Radius accounting data
Logs from Satcom Systems (via email, or mobile apps)
Aircraft Position + Status Reports
We normalize Aircraft Position reports before feeding them to
Splunk
– Fields are extremely complex, often missing, sometimes delayed, and come
from at least 5 different sources. And they are all totally inconsistent.

7. Data Sources - AudioCodes
Max-Forwards: 70
User-Agent: AeroV-Gateway
CSeq: 102 OPTIONS
Call-ID: 66bac96862403ef05c1aac9922e3d3d2@63.###.###.238
Contact: <sip:AeroV-Gateway@63.###.###.238>
To: <sip:63.###.###.241>
From: "AeroV-Gateway" <sip:AeroV-Gateway@63.###.###.238>;tag=as7a930744
Via: SIP/2.0/UDP 63.###.###.238:5060;branch=z9hG4bK47c1eef2;rport
Sep 14 14:50:02 63.###.###.241 OPTIONS sip:63.###.###.241 SIP/2.0
Sep 14 14:50:02 63.###.###.241 (
lgr_flow)(658474 ) ---- Incoming SIP Message from 63.###.###.238:5060 to SIPInterface #0 UdpTransportObject[#3343] --- [Time: 09-14-2013@14:50:02]
Sep 14 14:50:02 63.###.###.241 ( sip_stack)(658473 ) New SIPMessage created - #15 [Time: 09-14-2013@14:50:02]
Sep 14 14:49:58 63.###.###.241 ( sip_stack)(658472 ) SIPDialog(#138) changes state from DialogDisconnected to DialogIdle [Time: 09-14-2013@14:49:58]
Sep 14 14:49:58 63.###.###.241 (
lgr_flow)(658471 ) |
| TransactionUserMngr::ReturnDialog - #138 [Time: 09-14-2013@14:49:58]
Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658470 ) Resource SIPMessage deleted - #12 [Time: 09-14-2013@14:49:53]
Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658469 ) AcSIPStackAPI::FreeDialogAPI - #34 [Time: 09-14-2013@14:49:53]
Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658468 ) SIPDialog(#138) changes state from DialogConnected to DialogDisconnected [Time: 09-142013@14:49:53]
Sep 14 14:49:53 63.###.###.241 (
lgr_flow)(658467 ) |
|(SIPTU#138)DIALOG_DISCONNECT_REQ
State:DialogConnected(370678c35bed1a1c1d2f36a20e0b0fd0@63.###.###.248) [Time: 09-14-2013@14:49:53]
Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658466 ) Resource SIPMessage deleted - #70 [Time: 09-14-2013@14:49:53]

8. Data Sources - Expand
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop
Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22.
Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated

9. Copyright © 2013 Splunk Inc.
Monitoring & Alerting

10. Support: Monitoring and Alerting
•
Splunk provides a real-time dashboard in our NOC about the status
of several key services
•
Previously, support techs would need to login to 3-5 different
systems to look for faults or errors. Each system had a different
UI, different formats and different data. Techs learned, but over
long periods as errors were often infrequent and obscure
•
Now data is in one system, one interface, with intelligence ‘coded
in’ by our senior techs

11. Support: Monitoring and Alerting
•
We merge log data with our Configuration Management database
so we can display aircraft Tail Numbers, Phone Numbers and
relevant data directly on the dashboard.
– Allows our support team to see customers as their aircraft logon to the
satellites and move data or make voice calls
– Support techs can verify while still on the phone with the customer (data is
~60-90 seconds delayed)
CSA Data Entry
CM Servers
Feed Splunk CSV tables for Lookups
indexer

12. Support: Monitoring and Alerting
•
We can be proactive – Splunk alerting allows us to capture issues
immediately – customers unable to connect (incorrect
passwords, or invalid settings). We know we’ll get a call, or we can
call the customer directly.

13. Support: Monitoring & Alerting
•
.conf 2013 Stump the Experts Report – counting in-flight (Literally!)
transactions over time to gauge volumes

14. Support: Monitoring and Alerting
•
Alerts help capture out of the ordinary situations
•
More that # occurrences in a given timespan alerts take 60 seconds
to setup – use them
•
Now when something spirals out of control, you’ll know!

15. Copyright © 2013 Splunk Inc.
Business Analytics

16. Business Analytics
•
We’ve always been a data driven organization – we focus heavily on
configuration management for customer avionics
•
Using Splunk to analyze the data helps us make smart decisions
•
Each time we deep dive into the data, we learn new things

17. Business Analytics
•
We used Splunk to determine how to size our new DNS
infrastructure
•
Fed DNS stats (Bind + script + syslog) into Splunk for a few weeks,
visualized the results and then were able to do capacity planning

18. Business Analytics – VoIP Call Rates
•
We can monitor the Country Codes dialed for our Satellite Voice
calls in aggregate, so we know what countries our customers call
most often. We then push our telecom & VoIP providers to
negotiate better rates.
•
Splunk tells us what countries we need to focus on, so we ignore
the long rate cards and get right down to the ones we care about.

19. Business Analytics – VoIP Call Rates
•
We can then route outbound calls based on destination country
code to a different provider, reducing our direct cost per second for
call terminations

20. Copyright © 2013 Splunk Inc.
Flight Tracking

21. Flight Tracking
Where the plane is coming or going isn’t what is important
Common problems with Satellite communications are handovers –
where you change which satellite you are talking to while in flight
Historically it’s hard to correlate events with location visually
Google Earth/Google Maps were a major leap, but not automated
Enter Splunk w/Google Maps plugin – now we can put all the data
in a consistent visual format.

22. Flight Tracking Data
FAA ASDI
users
Other Apps
Sat. Provider 1
FT Server
Process & Normalize All Data
Sat. Provider 2
Satcom
Terminal
forwarder
indexer

23. FAA ASDI Data
<trackInformation><nxcm:aircraftId>ACA117</nxcm:aircraftId><nxcm:speed>280</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce:simpleAltitud
e>103</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="43" minutes="51"
direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="079" minutes="50"
direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:31Z"
trigger="TZ"><trackInformation><nxcm:aircraftId>MES3455</nxcm:aircraftId><nxcm:speed>400</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxc
e:simpleAltitude>360</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42"
minutes="12" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="076" minutes="16"
direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:31Z"
trigger="TZ"><trackInformation><nxcm:aircraftId>ACA114</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce:
simpleAltitude>262</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="53"
minutes="10" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="111" minutes="54"
direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z"
trigger="TZ"><trackInformation><nxcm:aircraftId>UAL801</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce:
simpleAltitude>340</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42"
minutes="59" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="082" minutes="52"
direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z"
trigger="TZ"><trackInformation><nxcm:aircraftId>EJA802</nxcm:aircraftId><nxcm:speed>370</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce:
simpleAltitude>400</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42"
minutes="15" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="078" minutes="52"
direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:32Z" trigger="UZ"><boundaryCrossingUpdate><nxcm:aircraftId>PAG113</nxcm:aircraftId><nxcm:flightAircraftSpecs specialAircraftQualifier="B757_TCAS"
equipmentQualifier="G">BE99</nxcm:flightAircraftSpecs>
* http://www.fly.faa.gov/ASDI/asdi.html

24. Flight Tracker – Post Normalization
TimeOfReport
9/8/13 20:21
9/8/13 20:20
9/8/13 20:19
9/8/13 20:19
9/8/13 20:18
9/8/13 20:18
9/8/13 20:17
9/8/13 20:17
9/8/13 20:17
9/8/13 20:16
9/8/13 20:07
9/8/13 19:57
9/8/13 19:47
9/8/13 17:21
9/6/13 19:59
9/6/13 19:49
9/6/13 19:41
Source
FaaAsdiFAA
FaaAsdiFAA
FaaAsdiFAA
FaaAsdiFAA
FaaAsdiFAA
SbbGps
FaaAsdiFAA
FlightDeckFusion
FaaAsdiFAA
FaaAsdiFAA
FlightDeckFusion
FlightDeckFusion
SbbGps
SbbGps
FlightDeckFusion
FlightDeckFusion
SbbGps
Received
9/8/13 20:26
9/8/13 20:25
9/8/13 20:24
9/8/13 20:24
9/8/13 20:23
9/8/13 20:22
9/8/13 20:23
9/8/13 20:17
9/8/13 20:22
9/8/13 20:21
9/8/13 20:07
9/8/13 19:57
9/8/13 19:52
9/8/13 17:28
9/6/13 19:59
9/6/13 19:49
9/6/13 19:47
MessageId
Latitude
Longitude Altitude
Heading
FaaAsdi132839420
35.8889 -115.0775
15100
FaaAsdi132839201
35.8986 -115.1664
11800
FaaAsdi132839013
35.9114 -115.2625
9200
FaaAsdi132838985
35.9264 -115.2839
8600
FaaAsdi132838854
35.9797 -115.2719
7200
SbbGps20130908201801000000N651SD
35.9907
-115.253
FaaAsdi132838737
35.9942 -115.2483
7000
SD20130908201716976007N651SD
36.02
-115.2
5900
FaaAsdi132838595
36.0314 -115.1908
5300
FaaAsdi132838463
36.0681 -115.1708
3100
SD20130908200716316162N651SD
36.0967 -115.1517
2000
SD20130908195716125081N651SD
36.0983
-115.16
2000
SbbGps20130908194757000000N651SD
36.0997 -115.1603
SbbGps20130908172106000000N651SD
36.0995 -115.1603
SD20130906195946601934N651SD
36.1 -115.1583
2100
SD20130906194946395228N651SD
36.0983 -115.1583
2100
SbbGps20130906194144000000N651SD
36.0999 -115.1595
Speed
272
285
284
295
272
246
218
204
195
14
0
0
0

25. Flight Tracking

26. Copyright © 2013 Splunk Inc.
Splunk Tips

27. Transactions
Insanely powerful for gathering statistics.
tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State
startswith="negotiating to accelerating" endswith="accelerating to drop"
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed
Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop
Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22.
Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated

28. Transactions
Run against a few hours of data, and we see lots of transactions
occurring. So we know how long each Aircraft is ‘in session’ for.

29. Transactions
Now what? Let’s do some math and get some stats!
tag="Expand" "status changed" |rex
"s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State
startswith="negotiating to accelerating" endswith="accelerating
to drop" | eval ConnectedFor(Mins)=round(duration/60) | lookup
taillookup ip as AircraftIP OUTPUT subnet_name as Tail|stats
sum(ConnectedFor(Mins)) as TimeOnline by Tail| sort TimeOnline

30. Transaction - Visualizations
Once you have the data, visualizations on the dashboard allow us
to know at a glance if a service is performing within limits
We adjust the gauge colors – in this case, higher is better

31. Don’t Fear CSV
KISS – and CSV is certainly that
Great for mapping things like IP/Subnets to Customers
Easier to manipulate text files to clean them up
Great for things that don’t change too often
# Sort by IP address so searches are easier
sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ip-customers.in > ip-customers.csv
cp ip-customers.csv /opt/splunk/etc/system/lookups/ip-customers.csv
CIDR Lookup Scripts: http://answers.splunk.com/answers/5916/using-cidr-in-a-lookup-table

32. Summary
Alerting based on frequency of events within a timeframe can be
extremely powerful to detect anomalies
Sometimes you need to clean up your data before you send it into
Splunk – Garbage in, garbage out
Adding external lookups can be as simple as CSV files – don’t
overthink it
’transaction’ helps make sense of time & duration based data
Use Splunk to guide your choices with real data – embrace
Empiricism to make good business decisions

33. Q & A Time

34. Copyright © 2013 Splunk Inc.
Thank You!