Splunk for Developers

1. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Copyright © 2015 Splunk Inc.
Splunk for Developers
Scott Haskell
Principal Sales Engineer
San Francisco

2. Grigori Melnik, Principal Product Manager – Splunk Developer Platform2
EMPOWERING DEVELOPERS
Gain
Application
Intelligence
Build Splunk
Apps
Integrate &
Extend
Splunk

3. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Copyright © 2015 Splunk Inc.
Splunk for Application
Development

4. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Build
Unit Testing
Code
Check-in Integration
Testing Deploy
Staging
Application Development Challenges
4

5. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Build
Unit Testing
Code
Check-in Integration
Testing Deploy
Staging
Lack of visibility across the product
development lifecycle
Pressure to increase velocity and
agility with DevOps
Limited insights into behavior and
performance from application logs
Application Development Challenges
5

6. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Quickly trace and identify errors anywhere
in the codebase with real-time search
and monitoring
Instrument your app logs to gain
application intelligence
Break down dev tool silos with real-time
insights from machine data
GAIN END-TO-END VISIBILITY
ACROSS THE DEV TOOL CHAIN
FIND AND FIX
ISSUES FASTER
PUSH BETTER CODE
USING ANALYTICS
Splunk for Application Lifecycle Intelligence
6

7. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Real-time dashboards show error rate
in production and impact of pushing
new builds
Developers can search and visualize
web logs, Java logs, eventlogs etc;
trace tx without complex
instrumentation
Alerts notify developers as soon as a
problem arises
Find and Fix Issues Faster
7

8. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Gain end-to-end visibility to make
informed decisions
Analytics insights without the need for
additional analytics tools
Ask questions while exploring and
collecting data
Push Better Code Using Analytics
8

9. Grigori Melnik, Principal Product Manager – Splunk Developer Platform9 9
CI / Build
Servers
Project and Issue
Tracking
Code
Repository
QA / Testing
Tools
End-To-End Visibility Across The Dev Tool Chain
Deployment Servers /
Automation

10. Grigori Melnik, Principal Product Manager – Splunk Developer Platform10

11. Grigori Melnik, Principal Product Manager – Splunk Developer Platform11
CI / Build
Server
Code
Review
Task
Tracking
What Data Can You Splunk?
Logs – Which code has already been reviewed for this release/sprint? Who has
completed the most code reviews? What code has NOT been reviewed?
Logs/API – Who is changing files? What kinds of files are being changed? What
branches are most active? What types of activities are occurring for a branch?
Version
Control
Logs/API – How many builds completed today/this week/this month? Which
check-in kicked off this build? Which tests ran against this failed build?
Logs – Which tasks are assigned to which developers? What progress is being
made to complete assigned tasks? What tasks remain for this release/sprint?
1

12. Grigori Melnik, Principal Product Manager – Splunk Developer Platform12
Key Benefits of Application Lifecycle Intelligence
Reduced Time
to Market
Shrink the time it takes
to get code through
dev/test to market
through faster issue
identification and
resolution
Increased
Agility
With real-time visibility
into processes like code
check-ins, builds and
tests to support
DevOps practices like
continuous integration
“Our devs are now able to
find and fix issues five to ten
times faster.”
“We can monitor all the
automation and handoffs it
takes to deploy 5-10 times
a day”
Application
Insights
Instrument customer
application logs to
capture critical
business events and
user behavior
“My code isn’t ready until it’s
Splunk-ready”
1

13. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Copyright © 2015 Splunk Inc.
Demo:
ADLC

14. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Copyright © 2015 Splunk Inc.
Touring the Splunk
Development Platform

15. Grigori Melnik, Principal Product Manager – Splunk Developer Platform15
Evolving the Splunk Platform
Collection
Indexing
Search Processing Language
Core Functions
Inputs, Apps, Other
Content
SDKs & plug-ins
Operational Intelligence Platform
Content
Core Engine
User and Developer Interfaces
Web Framework
REST API

16. Grigori Melnik, Principal Product Manager – Splunk Developer Platform16
Powerful Platform for Enterprise Developers
1
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript/CSS Extensions C#
JavaScript
Python
Ruby
Java
PHP
Data Models
Search Extensibility
Modular Inputs
SDKs
KV Store

17. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Log directly to
Splunk via TCP,
UDP, HTTP
Integrate search
results with other
applications using
custom
visualizations
Create and run
searches from
other applications
The REST API and SDKs
17
VisualizeSearch Manage
Add/Delete Users
Manage Inputs
Index

18. Grigori Melnik, Principal Product Manager – Splunk Developer Platform18
The Splunk REST API
Exposes an API method for every feature in the product
– Whatever you can do in the UI – you can do through the API
– Index, Search, Visualize, Manage
API is RESTful
– Endpoints are served by splunkd
– Requests are GET, POST, and DELETE HTTP methods
– Responses are Atom XML & JSON
– Versioning as of Splunk 5.0
– Search results can be output in CSV/JSON/XML
1

19. Grigori Melnik, Principal Product Manager – Splunk Developer Platform19
SDKs Overview
19
• Stay true to the semantics of the particular language
• E.g. Keep Python “pythonic”
• E.g. C#: Fully async , PCL, support for Rx
• Provide implementation that feels natural to the developer
• E.g. Project, build, IDE (where applicable) support
• Cover REST API endpoints based on use cases of language
• Namespaces
• owner: splunk username (defaults to current user)
• app: app context (defaults to default app)
• sharing: user | app | global | system

20. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
A Developer’s Smörgåsbord
 Data ingestion
– Input
 Scripted inputs
 Modular inputs
 HTTP Event Collector
 Custom (trained) source
types
 Custom sources
– Data ingestion pipeline
 Field extractions
 Field transformations
– Indexing
 Custom indexes
 Searching
– Search authoring
 Custom search commands
 Macros (basic, parametrized)
 Saved searches
– Data classification
 Event types
 Transactions
– Data enrichment
 Lookups
 KV store collections
 Workflow actions
– Data normalization
 Tags
 Aliases
– Data mining
 cluster & dedup
 anomalousvalue
 kmeans
 predict commands …
 Processing & reporting
– Search-time mapping
 Data models
– CIM extensions
– Custom UI/visualizations
 Pages, views & dashboards
 JS Extensions
 CSS Extensions
 Custom setup screens
– Scheduled processing
 Scheduled reports
– Alerting
 Scripted alerts
– Branding & navigation
 Custom app navigation &
branding
– Manageability
 Custom splunkweb
controllers
 Custom splunkd endpoints

21. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Copyright © 2015 Splunk Inc.
Building Splunk Apps

22. Splunk Developer Guidance


Splunk Reference Apps
Complete, working real-world Splunk solutions
built together with partners (Conducive; Auth0)
̶ 2 (pseudo-) production releases
̶ entire code & test repos on GitHub
̶ under Apache 2.0
Associated Guidance
I. Start-to-Finish Journey Documentary
II. Essentials
dev.splunk.com/goto/devguide

23. 1. Started with a Questions BacklogArchitecture
– What does a typical Splunk application reference architecture look like?
– What common paradigms are applicable to Splunk app development?
– What are the typical deployment topologies? Why should I choose a specific one? What are the confounding factors
on the choice of my topology?
– How do I partition my Splunk solutions?
– What are the tradeoffs of various types of inputs?
– How do I architect my Splunk solution and deployment for a very large scale?
– How do I architect my Splunk solution for the cloud? What are specific considerations for deploying to AWS or Azure?
– What’s the landscape of Splunk extension points?
– How do I integrate data from Splunk into existing applications and systems?
– How do I plan and design a robust alerting and monitoring subsystem on top of Splunk?
– What should I consider for my sizing requirements?
– What are recommended configurations of Splunk deployment to meet my sizing requirements?
– Should I architect my solution to index my data in local data center (zone) or centrally?
– What are things we can automatically degrade so we can make sure our core experience is working?
– When something happens, how do I effectively propagate the info and react to it?
– How are other solutions on Splunk built? What were the challenges? How have they been addressed?
Packaging and Deployment
– How do I piece together various parts of a Splunk app (custom search commands, mod inputs etc.)?
– How do I package a Splunk solution with a single install that automatically rolls out all the necessary dependencies?
– How do I manage my Splunk solution versioning, backward and future compat?
– What's the best way to split up custom apps for deployment?
Development
– How should I set up my development environment to be productive with Splunk?
– What are different ways of how I develop my Splunk app ? Pros and cons of using specific SDK vs REST APIs?
Pros and cons of using SimpleXML vs Advanced XML vs Web Framework …
– How do I analyze a data source for a TA?
– What are the different ways of enriching the data in Splunk? What are their tradeoffs?
– When should I use event types and transactions for data classification?
– How do I extend Splunk to define a custom input capability?
– When should I use modular inputs vs scripted inputs vs..?
– What are streaming vs non-streaming outputs considerations?
– How do I deal with long-running scripts? Handling shutdown/restart of Splunk? Concurrency? State persistence etc.
– Why should I not use transactions?
– When should I use pivot vs tstats?
– Why should I use data models?
– When my data source touches on many data models, should I assume complete separation or heavy inheritance?
– How do I extend an existing data model?
– What does CIM offer and why should I build CIM-compliant apps?
– In the context of CIM, what are the tradeoffs of using my props.conf and transforms.conf and rewriting them on
indexing, completely discarding the vendor supplied field names? How do I reconcile the advantages of a clean
interface & normalisation, but at the cost of losing alignment with published vendor documentation, and a learning
curve for existing users?
– How do I manage my solution declarative configuration? How do I detect/troubleshoot bad config?
– How do I log and analyze data that is not event driven (certain web feeds, html parsing, image meta data)?
– Compare and contrast ad-hoc searching vs background searching
– How do I handle transient faults?
– How do I effectively manage credentials?
– What’s the effect of search head location on my app and the overall user experience?
– How do I develop an integrated mechanism to let me connect Splunk to my MOM (messaging middleware) and index
my messages?
– How do I handle the requirement that app configs must be different across different server types in a distributed
environment (e.g. apps on search heads shouldn't have inputs enabled)?
Quality/Compliance
– What quality gates should I consider? What kind of para-functional characteristics are important to consider?
– What heuristics do I use to bless/block a release?
– How do I test a data model?
– How do I prepare event generation when building/testing an app?
– What kind of perf testing should I do and how?
– How do I test UI?
– How do I security certify my solution?
– How do I design to satisfy my retention and compliance policies?
– How do I architect to design my availability requirements?
– How do I handle geographic disaster recovery / fault tolerance?
– How do I properly instrument my solution so that I know what’s happening?
Sustained Engineering
– How do I maintain/service/support Splunk apps?
– How do my customers handle updating their customized configs once new versions of my app come out?
Business
– Why should I build on Splunk?
– What kind of skill do I need my devs to have to build a Splunk solution?
– What is the community building? How are current devs creating unique experiences using Splunk – I typically want to
see some marketplace success
– Cost and pricing are very important to me as a entrepreneur developer. If I am coming in to build a tool that will be
commercialized I need to know that the cost structure of Splunk won’t cause my service to be economically
unprofitable.
What does a typical Splunk application architecture look like?
How should I set up my dev environment to be productive with
Splunk?
How do I integrate Splunk into existing systems?
How do I prepare my event generation when developing &
testing an app?
How do I package an app? deal with app versioning and updates?

24. 2. Mined business requirements with partner
3. Formulated learning objectives
4. Reconciled 2 & 3 with our designs
…

25.  Data
 Search language
 Aggregating siloed metrics into
meaningful KPIs
 Data manipulation
 Data normalization
 Sub-searches
 Config-driven
 Persistence with KV store
 Macros
 Viz:
 Dynamic scaling
 Customizing in-the box viz
controls
 General search patterns
 Search optimizations
 Ux Prototyping
 Adapting 3rd party viz library
 Composite charts with interactions
 Dealing with high-volume data sets
 Troubleshooting perf issues
 Post-process or not-post-process –
deployment implications
 Automated UI testing (w.Selenium)
 Setting the stage
 Overall Splunk app structure
 UI technology selection:
Simple XML vs SplunkJS
 Modularity
 Dev & test env
 Dev workflow
 Modularity
 Data onboarding
 CIM compliance
 Tools
 Post-processing
 Integrating with 3rd party
component
 Unit testing (w.Mocha)
 Persisting state (per user)
 Data modeling
 Using lookups
 Building a baseline lookup table
 Windows of time/Custom time ranges
 Overlaying time data
 Using sub-searches to correlate data
 Troubleshooting searches
 Custom nav
 Ux activities permeating all dev
 Data mining:
 Exploration
 Preparation: filtering/deduping/
bucketing
 Using advanced statistics functions
 Threshold-based anomaly detection
 Evaluating goodness /accuracy
Plus non-functional topics:  App versioning
 Packaging Installation
 Security review
 Deployment
 Publishing to splunkbase
 App certification

26. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Copyright © 2015 Splunk Inc.
Demo:
Building solutions with
Splunk Reference App

27. Copyright © 2015 Splunk Inc.27
Splunk Reference App comes preinstalled in the Cloud
Sandbox - www.splunk.com/goto/cloud

28. Grigori Melnik, Principal Product Manager – Splunk Developer Platform
Copyright © 2015 Splunk Inc.
Resources

29. Grigori Melnik, Principal Product Manager – Splunk Developer Platform29
Splunk Developer License
2

30. Grigori Melnik, Principal Product Manager – Splunk Developer Platform30
Where to go for more Info
• Tutorials, Code Samples, Getting Started, Downloads
– http://dev.splunk.com
• Splunk Developer Guidance
– http://dev.splunk.com/goto/devguide
• Splunk Base (Apps)
– https://splunkbase.splunk.com
• GitHub
– https://github.com/splunk
• Twitter
– https://twitter.com/splunkdev
• Blogs
– http://blogs.splunk.com/dev
30

31. Copyright © 2015 Splunk Inc.31
Takeaways
Application development intelligence
Platform, not just an engine
Open & extensible
On-prem and cloud
Developer Guidance : learn and reuse for the win!
Reach out to us (devinfo@splunk.com) and tell us about
your experience

32. Copyright © 2015 Splunk Inc.32
Northern Cal Tech Talks!
Monthly WebEx Sessions
• Ted Talk style presentation
• Q&A Chat forum
So what’s next on the agenda?
• March 23rd @ 10AM PST - Building &
Deploying Apps.
• April 20th @ 10AM PST - Top 5 most useful
search commands.
See more at:
http://live.splunk.com/NorCalTechTalks

33. Copyright © 2015 Splunk Inc.33
SEPT 26-29, 2016
WALT DISNEY WORLD, ORLANDO
SWAN AND DOLPHIN RESORTS
• 5000+ IT & Business Professionals
• 3 days of technical content
• 165+ sessions
• 80+ Customer Speakers
• 35+ Apps in Splunk Apps Showcase
• 75+ Technology Partners
• 1:1 networking: Ask The Experts and Security
Experts, Birds of a Feather and Chalk Talks
• NEW hands-on labs!
• Expanded show floor, Dashboards Control
Room & Clinic, and MORE!
The 7th Annual Splunk Worldwide Users’ Conference
PLUS Splunk University
• Three days: Sept 24-26, 2016
• Get Splunk Certified for FREE!
• Get CPE credits for CISSP, CAP, SSCP
• Save thousands on Splunk education!