How much visibility do you really have into your Spring applications? How effectively are you capturing,harnessing and correlating the logs, metrics, & messages from your Spring applications that can be used to deliver this visibility ? What tools and techniques are you providing your Spring developers with to better create and utilize this mass of machine data ? In this session I'll answer these questions and show how Splunk can be used to not only provide historical and realtime visibility into your Spring applications , but also as a platform that developers can use to become more "devops effective" & easily create custom big data integrations and standalone solutions.I'll discuss and demonstrate many of Splunk's Java apps,frameworks and SDK and also cover the Spring Integration Adaptors for Splunk.
11. Splunk is a Platform for Machine Data
11
Developer
Platform
Data collection
and indexing
Report
and
analyze
Custom
dashboards
Monitor
and alert
Ad hoc
search
HA Indexes and
Storage
Commodity
Servers
12. What Does Machine Data Look Like?
12
Twitter
Care IVR
Middleware
Error
Order Processing
13. Machine Data Contains Critical Insights
13
Customer ID Order ID
Customer’s Tweet
Time Waiting On Hold
Twitter ID
Product ID
Company’s Twitter ID
Twitter
Care IVR
Middleware
Error
Order Processing
Customer IDOrder ID
Customer ID
14. Machine Data Contains Critical Insights
14
Order ID
Customer’s Tweet
Time Waiting On Hold
Product ID
Company’s Twitter ID
Twitter
Care IVR
Middleware
Error
Order Processing
Order ID
Customer ID
Twitter ID
Customer ID
Customer ID
19. How can Splunk help out the Spring Developer ?
• During Dev/Test
– Use Splunk to deliver deeper insights , hook in more
thorough test case assertions
– Aggregate data from your apps in development
• Integrate the Data you have collected with Splunk
– Use Spring as the EAI backbone to build integrated data
solutions , correlate data form numerous sources
• Build standalone Big Data apps
– Let Splunk do the hard yards on the data and searching side
20. What are some of the hooks?
• REST API
• Splunk SDK for Java
• Spring Integration Adaptors
• Logging
• JMS Messaging
• JMX MBeans
• REST
• BCI(byte code injection) tracing
22. Splunk SDK for Java
• Open sourced under the Apache v2.0 license
• git clone https://github.com/splunk/splunk-sdk-java.git
• JRE 6+
• Maven/Gradle Repository
• Code Examples :
– Connect
– Hit your first endpoint
– Send data in
– Search for data , Simple and Realtime
– Scala and Groovy can play too
25. When Spring and Splunk collide
• Developers like tools & frameworks that increase productivity
• An SDK makes it easier to use a REST API
• A declarative Enterprise Integration framework makes it easier to build
solutions that need to integrate, transform, filter and route data from
heterogeneous channels and data sources
• We now have the Spring Integration Splunk Adaptors to make it easier for
Java Developers to integrate Splunk into their solutions utilizing a
semantic they are most familiar with in the Spring framework.
26. Spring Integration And Splunk
Inbound Adapter
– Used to execute Splunk searches and produce messages containing results
– Search modes:
BLOCKING, NORMAL, REALTIME, EXPORT, SAVEDSEARCH
– Date/Time Ranges
Outbound Adapter
– Write system or application events to Splunk
– Write to a named index, submit a REST request, write to a data input bound to
a server TCP port
Message payload for Splunk I/O adapters is SplunkEvent
27. Spring
Integration
Twitter adaptor
polls for tweets
Spring
Integration
Splunk
outbound
adaptor sends
events to
Splunk via
HTTP REST
Realtime or
historical search
from SplunkWeb
Raw events from Twitter are transformed into best practice logging
format
Splunk Java
SDK
Tweets
29. SplunkJavaLogging
• A logging framework to allow developers to as seamlessly as possible
integrate Splunk best practice logging semantics into their code
• Transport log events to Splunk directly from your code
• Custom handler/appender implementations(REST and Raw TCP) for
common Java logging frameworks .
• LogBack
• Log4j (Log4j2 coming also)
• java.util.logging
• Utility classes for formatting log events
• Configurable in memory buffer to handle network outages
29
30. Developers just log as they are used to
30
2012-08-07 15:54:06:644+1200 name="Failed Login" event_id="someID" app="myapp" user="jane" somefieldname="foobar"
Better
A-HA
31. Semantic Logging
Log anything that can add value when aggregated, charted or further analyzed
Example Bogus Pseudo-Code:
void submitPurchase(purchaseId)
{
log.info("action=submitPurchaseStart, purchaseId=%d", purchaseId)
//these calls throw an exception on error
submitToCreditCard(...)
generateInvoice(...)
generateFullfillmentOrder(...)
log.info("action=submitPurchaseCompleted, purchaseId=%d", purchaseId)
}
• Create Human Readable Events
• Clearly Timestamp Events
• Use Key-Value Pairs (JSON Logging)
• Separate Multi-Value Events
• Log Unique Identifiers
37. JMS Messaging Splunk Input
37
• JMS is an interface that abstracts your underlying MOM provider implementation
• Send messages to parallel queues or topics in Spring that Splunk can tap into
• Index messages from :
• MQ Series / Websphere MQ
• Tibco EMS
• ActiveMQ
• HornetQ
• RabbitMQ
• SonicMQ
• JBoss Messaging
• Weblogic JMS
• Native JMS
• StormMQ
• Note : Non-JMS inputs also available (Stomp , ZeroMQ)
43. Expose Mbeans in your Spring App
• 3 tiers can be exposed
– JVM (java.lang domain)
– Framework / Container (Spring , Tomcat etc…)
– Application (whatever you have coded)
• Attributes , Operations and Notifications
• Use Splunk for JMX to monitor the internals of your
running Spring apps
44. Splunk for JMX
• Multiple connectivity options
– rmi/iiop ,direct process attachment, mx4j http connectors
• Works with all JVM variants
• Scales out to monitor large scale JVM infrastructures
46. REST is easy with Spring
• Create an endpoint with Spring Integration
• Splunk can poll the REST API endpoint
• Multiple authentication mechanisms
• Custom response handling / pre-processing
• Send responses in XML , JSON
• Splunk can natively index the responses and then simply
search over the auto extracted fields
47. REST : The Data Potential
• Twitter
• Foursquare
• LinkedIn
• Facebook
• Fitbit
• Amazon
• Yahoo
• Reddit
• YouTube
• Flickr
• Wikipedia
• GNIP
• Box
• Okta
• Datasift
• Google APIs
• Weather Services
• Seismic monitoring
• Publicly available socio-economic data
• Traffic data
• Stock monitoring
• Security service providers
• Proprietary systems and platforms
• Other “data related” software products
• The REST “dataverse” is vast , but I
think you get the point.
47
There is a world of data out there available via REST that can be brought into Splunk, correlated
and enriched against your existing data, or used for entirely new uses cases that you might
conceive of once you see what is available and where your data might take you.
49. Splunk Java Agent
49
An instrumentation agent for tracing code level metrics via bytecode injection, JMX
attributes/operations/notification and decoded HPROF records and streaming these
events directly into Splunk
https://github.com/damiendallimore/SplunkJavaAgent
• class loading
• method execution
• method timings (cumulative, min, avg, max, std deviation)
• method call tracing(count of calls, group by app/app node(for clustered systems)/thread/class/package)
• method parameter and return value capture (in progress)
• application/thread stalls , thread dumps and stacktraces
• errors/exceptions/throwables
• JVM heap analysis, object/array allocation count/size,class dumps, leak detection, stack traces, frames
• JMX attributes/operations/notifications from the JVM or Application layer MBean Domains
50. Design goals
50
• Just pull out the raw metrics , then let Splunk perform the crunching
• Format events in best practice semantic , well defined key value pairs , tagged
events help correlation across distributed environment
• Low impact to the instrumented application
• No code changes required
• Flexible configuration
• Extensible
• Generic open source agent , I may have used some Splunk terms in the naming
conventions, but it is still completely generic , anyone want to collaborate !
• Not a full blown APM solution , just pulling raw data.
• Incorporate into your Spring apps during Dev/Test to get deeper insights
51. Setup should be as simple as possible
51
This is all you pass to the JVM at startup :
-javaagent:splunkagent.jar
Everything required by the agent is built into the one single jar file
We also have a new Eclipse plugin that incorporates this functionality if you don’t
want to setup the JVM command line argument manually.
57. Android SDK project
57
• Cousin to the Splunk SDK for Java, has all the same functionality and code examples are
the same
• Utility classes to make common tasks easier
• logging to Splunk
• searching Splunk
• pulling system/device/sensor metrics from Android and logging to Splunk
• Send Android data to Splunk and then use the Spring Integration Splunk Inbound adaptor
to integrate this into your Spring applications.
• Community preview currently published to Github
59. HUNK (Splunk Analytics for Hadoop)
• A new product offering from Splunk , currently in Beta preview
• Allows you to use the power and simplicity of Splunk to search over
data locked away in HDFS
• Sits on top of HDFS as if it was a native Splunk Index
• Virtual Indexes
• So you can use the Spring Integration Splunk Inbound Adaptor to
search over data in HDFS, or correlate your HDFS data with other
data you have indexed and integrate it into your Spring Applications.
59
60. Splunk sits on top of HDFS
Hadoop
Storage
Immediately
start
exploring, anal
yzing and
visualizing raw
data in Hadoop
1 2Point
Splunk at
Hadoop
Cluster
Explore Analyz
e
Visualiz
e
Dashboard
s
Share
62. Where to Go for More Info
62
Twitter
@splunkdev
Blog
http://blogs.splunk.com/dev
Demos
http://demos.splunk.com
Email
devinfo@splunk.com
Portal
http://dev.splunk.com
Github
https://github.com/splunk
63. Easy to Get Started
Download and install in minutes
3. Start Splunking1. Download 2. Eat your Machine Data
64. splunk.com/goto/conf
4th Annual Event
3+ days | 100+ sessions | 30+ customer sessions
1,500+ IT Pros | 20+ partners
2 days of Splunk University
Specialist Tracks: CIO, Big Data, Executive
Sept 30 – Oct 3
Las Vegas
65. Links
Github Gists for SDK code examples : https://gist.github.com/damiendallimore
SDK docs at dev.splunk.com : http://dev.splunk.com/view/SP-CAAAECN
Splunk SDK for Java Github repository : https://github.com/splunk/splunk-sdk-java
Maven/Gradle/Ivy Repository : http://splunk.artifactoryonline.com/splunk/ext-releases-
local
Splunk Spring Integration repository on Github : https://github.com/SpringSource/spring-
integration-extensions/tree/master/spring-integration-splunk
Splunk Spring Integration demo on Github : https://github.com/damiendallimore/spring-
integration-splunk-webex-demo
Splunk Eclipse plugin : http://dev.splunk.com/view/splunk-plugin-eclipse/SP-CAAAEQP
Splunk Java Logging on Github : https://github.com/splunk/splunk-library-javalogging
65
66. Links cont….
Splunk Java Agent on Github :
https://github.com/damiendallimore/SplunkJavaAgent
Splunk Android SDK on Github : https://github.com/damiendallimore/splunk-sdk-
android
Splunk REST API reference :
http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTcontents
Free Splunk download : http://www.splunk.com/get?r=header
Best practice logging overview : http://dev.splunk.com/view/logging-best-
practices/SP-CAAADP6
Splunk SDK for Java videos : http://dev.splunk.com/view/get-started/SP-
CAAAECH
HUNK Beta video : http://www.splunk.com/view/SP-CAAAH2F
Developer Evangelist at SplunkBuild various apps,add-ons & tools across the Splunk developer landscape , talk about it , get other developers knowledgeable and interested in our various developer hooksSplunk architect/admin and community collaboratorI was once a customer and was spending a lot of time on Splunkbase (still do !)CoderMostly in the Java ecosystem, but have several languages in my batbelt, generally interested in coding anything that’s useful to someone, use the right language for the job.Born and raised in Aotearoa (New Zealand)
Tee shirtDividiedloyaltysKudos to elison for a great spectacle
Massive amounts of “data” are being generated everydayWe are often blind to where that data is and what that data is telling us , it lives buried away in a dark caveWe can navigate our way through the murk and make discoveries if we have the right equipment and know how (big data tools and techniques )Not a teeshirt company , not a twitter analysis company , not a twitter competition platform
Where we started founders , where we areIntegrated platform for data collection, searching & correlation and visualizationReal time visibility of events, Real time alertingNo need to write Map Reduce jobsOpen and extensibleNumerous ways to get your data into Splunk (TCP,UDP,REST,File monitoring, custom inputs etc..)“Schema on the fly”, no need to define/enforce structure up frontHighly available distributed architecture allows Splunk to scale out to TB’s per day , Index replicationStream in data directly from your application codeProvides comprehensive controls for data security, retention and integritySplunkbase community , heaps of free apps and add-ons, community supportInternet of things , elevators , SCADA , cars, medical devices , sensors, wifi , camera , mics , Mobile , rfid, network packets , SIEM quadrant , vending machines, fitbitetc..Splunk’sflagship product is Splunk Enterprise. Splunk Enterprise is a fully featured, powerful platform for collecting, searching, monitoring and analyzing machine data.Splunk collects machine data securely and reliably from wherever it’s generated. It stores and indexes the data in real time in a centralized location and protects it with role-based access controls. You can even leverage other data stores. Splunk lets you search, monitor, report and analyze your real-time and historical data. Now you have the ability to quickly visualize and share your data, no matter how unstructured, large or diverse it may be. Troubleshoot problems and investigate security incidents in minutes (not hours or days). Monitor your end-to-end infrastructure to avoid service degradation or outages. Gain real-time visibility and critical insights into customer experience, transactions and behavior. Use Splunk and make your data accessible, usable and valuable across the enterprise.
Unlike traditional structured data or multi-dimensional data– for example data stored in a traditional relational database for batch reporting – machine data is non-standard, highly diverse, dynamic and high volume. You will notice that machine data events are also typically time-stamped – it is time-series data. Take the example of purchasing a product on your tablet or smartphone: the purchase transaction fails, you call the call center and then tweet about your experience. All these events are captured - as they occur - in the machine data generated by the different systems supporting these different interactions. Each of the underlying systems can generate millions of machine data events daily. Here we see small excerpts from just some of them. Volume , velocity , variety , veracity ,valueStitchSingle pain of glass
When we look more closely at the data we see that it contains critical information – customer id, order id, time waiting on hold, twitter id … what was tweeted. What’s important is first of all the ability to actually see across all these disparate data sources, but then to correlate related events across disparate sources, to deliver meaningful insight.
What have developers been building using Splunk Enterprise? Examples include the following:Run searches and retrieve Splunk data from existing Customer Service/Call Center applications (Comcast use case) Integrate Splunk data into existing BI tools and dashboard (Tableau, MS Excel)Build mobile applications with KPI dashboards and alerts powered by Splunk (Otto Group use case)Log directly to Splunk from remote devices (Bosch use cases)Build customer-facing dashboards powered by user-specific data in Splunk (Socialize, Hurricane Labs use cases)Programmatically extract data from Splunk for long-term data warehousing
Code examples from Eclipse
Service is a wrapper that facilitates access to all Splunk REST endpointsCollections use a common mechanism to create and remove entitiesEntities use a common mechanism to retrieve and update property values, and access entity metadataArgs provide a POJO getter/setter paradigm fro REST API endpoint arguments
Writing to file it nice , buffers things , but sometimes there are constraints that forbid this , disk space etc..Bureaucratic constraints to depoing collectors / forwarders.
Demo : Splunk java logging code example + Splunk search
Demo : Spring integration code + active MQ with Splunk search
Splunk Java Agent demo from Eclipse with Splunk searches
Demo :Show shome data off my Nexus going into splunkcowboy.comSimple search exampleHaversine
Splunk Enterprise is simple to deploy, scales from a single server deployment to global large-scale operations and delivers fast payback. Download Splunk Enterprise for free, install it in 5 minutes on your laptop or on any commodity server, point it at any machine data and start using it. Splunk software is often deployed for the first time while under fire. A serious service outage or security incident in progress is stressful, but with Splunk Enterprise, you can complete your investigation in a few minutes versus hours or days.
Developer track , hackathon , and vegas !.conf2013 war our 3rd annual conferenceHeld in Las Vegas at The Cosmopolitan Hotel in September.Goal here is to make our customers smarter, because smarter customers find new ways to use Splunk and tell their colleagues to use Splunk. Specific conference goals:Help customers answer: Where will your data take you?Empower customers with knowledgeFoster deep, supportive relationships within the Splunk communityGarner rich feedback and input to create a better SplunkReinforce Splunk CommunityEquip Customers and Partners with skills for successCreate channel for sharing best practices—expanding use casesLive, in-person venue for trainingFoundation for everything Splunk--future Users’ Conferences, regional user groups, fueling Splunkbase and Splunk Answers…