Splunk is a software platform that allows users to gain real-time insights from industrial machine data. It collects, indexes, enriches, and analyzes data from sensors and industrial assets. Splunk helps users monitor equipment performance, detect anomalies, avoid downtime, and optimize manufacturing processes. The presentation demonstrates how Splunk has helped a semiconductor manufacturer improve yields, increase uptime, expand reporting capabilities, and decrease operating expenses by analyzing data from their fabrication facilities.
Splunk for Monitoring and Diagnostics in the Industrial Environment
1. Splunk for Monitoring and Diagnostics
Gaining real-time insights into industrial operations
Manish Jiandani
Director Solutions Marketing
2. 2
Safe Harbor Statement
During the course of this presentation,we may make forward looking statements regarding future events
or the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC. The forward-looking statements
made in this presentation are being made as of the time and date of its live presentation. If reviewed
after its live presentation, this presentation may not contain current or accurate information. We do not
assume any obligation to update any forward looking statements we may make. In addition, any
information about our roadmap outlines our general product direction and is subject to change at any
time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described orto includeany suchfeatureor functionalityina futurerelease.
3. 3
A World of Connected Assets
Internet of ThingsIndustrial Data
Transportation | Energy | Utilities | Building Management
Oil and Gas | Manufacturing
Wearables, Home Appliances, Consumer
Electronics, Gaming Systems, Personal
Security, Set-Top Boxes, Vending
Machines, Mobile Point of
Sale, ATMs,
Personal Vehicles
Sensors, Pumps, GPS, Valves, Vats, Conveyors,
Pipelines, Drills, Transformers, RTUs, PLCs,
HMIs, Lighting, HVAC, Traffic
Management, Turbines,
Windmills, Generators,
Fuel Cells,
UPS
3
Retail | Home | Consumer
Telemedicine | Connected Cars
4. 4
Challenges in Industrial Landscape
Data Collection &
Analytics
Batch Oriented/
Rear-View Approach
Security and Privacy
IT/OT Convergence
Ad hoc Analysis of OT
Data
Correlate Data Across
Application/Infrastructur
e Silos
CHALLENGES
5. 5 5
Make machine data accessible,
usable and valuable to everyone.
5
6. 6
HA/DR Admin Data Security Apps SDKs/APIsScale
Collect
Data
Index
Data
Enrich
Data
Search &
Explore
Analyze
& Predict
Report &
Visualize
Alert &
Action
6
Fully Integrated Enterprise Platform
7. 7
Turning Machine Data Into Business Value
7
Platform for Machine Data
Application
Delivery
IT
Operations
Security,
Compliance
and Fraud
Business
Analytics Industrial
Data and
Internet of
Things
10. Improving SCADA Operations and Security
95%Improvement in
Incident Response Time
Analyze 51K miles of pipeline data
from servers and OT networks
Improved pipeline safety and
availability through higher
application uptime
Increase regulatory
compliance
11. 11
Splunk For Monitoring and Diagnostics of Industrial
Assets
Ensure
equipment
in the field
is operating
as intended
Monitor
and avoid
unplanned
downtime
Perform
better
root-cause
analysis
Reduce
costs and
optimize
processes
12. 12
IoT and Industrial Machine Data
DevelopVisualize PredictAlertSearch
Engineers Data
Analysts
Security
Analysts
Business
Users
Native Inputs
TCP, UDP, Logs, Scripts, Wire, Mobile
SDKs and APIs
Java, JS, C#, Python, Ruby, PHP
Modular Inputs
MQTT, AMQP, COAP, REST, JMS
HTTP Event Collector
Token Authenticated JSON
Real-time
Technology Partnerships
Kepware, ThingWorx, Cisco, Palo Alto
Maintenance
Info
Asset
Info
Data
Stores
External Lookups/Enrichment
OT
Industrial Assets
IT
Consumer and
Mobile Devices
13. 13
Splunk’s IoT and Industrial Partner Ecosystem
SDKs UI
Ingest and Platforms
IoT and ICS SecurityAdvanced Analytics and ML Custom User Interfaces
Services and Delivery
14. 14
Splunk and Kepware
Exploration and
Production
Operations
Enterprise Data Environment
- Splunk > Enterprise
- Splunk > Cloud
OPC DA
OPC UA
OPC HDA
Splunk
Universal
Forwarder
Local Data Collection
- SCADA
- HMI
15. 15
Best Practices
Build Baselines of Asset Performance
Find Seasonality in Your Operations
Monitor Trends and KPIs
Identify Anomalies and Outliers
Enrich Operational Data with External Sources
18. 18
About “The Company”
Global electronics manufacturer
Fortune 100 company
2015 Revenues - 150B+ USD
2015 Profit – 20B+ USD
Employees – 100K+
19. 19
From Wafer to Chip - A Complex Process
Silicon Cleaning Texturing Etching
Ion
Implantation
Doping Deposition Patterning Dicing Chip
Sub
process-
1
Sub
process-
2
Sub
process-
3
Sub
process-
1
Sub
process-
2
Sub
process-
3
27. 27
Get Started
More Info: Splunk.com/IoT
Download Splunk
Download Kepware or Modular Inputs
Download other Splunk Apps (MQTT, COAP, Kepware Explorer)
Visit Splunk Answers
Download the case study -
http://www.splunk.com/en_us/solutions/solution-areas/internet-of-
things/monitoring-diagnostics.html
The form factor of compute is changing.
The internet of things is the convergence of embedded computing, networks everywhere, and powerful platforms for coordinating devices and understanding the data that is produced by those devices. Splunk is an excellent solution for understanding the machine data that is emitted by both industrial environments and the broader internet of things.
In the industrial world, challenges around managing the complex networks of machines, applications, and people are not that different from those in traditional IT.
The big difference is that Industrial environments are cyber-physical – poor operation has the ability to affect the physical environment in very impactful ways.
The teams that manage these systems have many drivers to develop all levels of operational intelligence from the machine data generated by their environments.
At Splunk, our mission is to make machine data accessible, usable and valuable to everyone. And this overarching mission is what drives our company and product priorities.
Splunk provides an open, fully integrated platform. That means you can collect, index, analyze, report and predict on machine-generated data from a single product. It’s enterprise-ready with high availability and disaster recovery features, role-based access control and scales to index hundreds of terabytes per day. It’s an open platform with over 500 Splunk Apps available and allows for custom development.
Splunk products are being used for data volumes ranging from gigabytes to hundreds of terabytes per day. Splunk software and cloud services reliably collects and indexes machine data, from a single source to tens of thousands of sources. All in real time. Once data is in Splunk Enterprise, you can search, analyze, report on and share insights form your data. The Splunk Enterprise platform is optimized for real-time, low-latency and interactivity, making it easy to explore, analyze and visualize your data. This is described as Operational Intelligence.
The insights gained from machine data support a number of use cases and can drive value across your organization.
New York Air Brake’s Train Dynamic Systems Division is using Splunk to manage inter-train forces, the “slinky factor” inherent in large freight trains with 6 inches of flex between cars. With splunk, they are able to produce insight and reports allowing the owners of the locomotives they manage to better train the engineers, and better manage the acceleration and braking of the trains throughout thousand mile journeys. Managing this data with Splunk, they can produce 5-10% fuel savings for customers. For their largest customers this can mean a billion dollars in savings a year.
Enterprise Product Partners is using Splunk to monitor and manage their critical Industrial Control System infrastructure. This infrastructure powers 51000 miles of some of the most critical hardware in the world – oil pipelines.
By using Splunk enterprise and partner solutions from companies like Palo Alto Networks, EPP is able to better monitor and manage the availability of the applications and hardware in their environment, and are able to react more quickly to the unexpected but inevitable downtime in a system this large and complex.
PHIMSA regulations require that you react to critical application downtime almost immediately – and EPP is using Splunk to satisfy this requirement. Since starting with Splunk, they have seen tremendous improvement in their response time.
There are many free add-ons and Apps for Splunk software that simplify the connection and collection of data from both industrial systems and the Internet of Things. These include:
Rest API Modular Input: Poll local and remote REST APIs and index the responses.
Amazon Kinesis Modular Input: Index data from Amazon Kinesis, a fully managed service for real-time streaming data.
Apache Kafka Modular Input: Index messages from Apache Kafka messaging brokers, including clusters managed by Zookeeper.
DB Connect 2: Integrate structured data sources with your Splunk real-time machine data collection.
Universal Forwarder for Linux (ARM – Raspberry Pi): Dedicated Splunk package for Linux and ARM based systems where data needs to be collected directly from embedded devices such as the Raspberry Pi.
MQTT Modular Input: Index messages from MQTT, a machine-to-machine connectivity protocol, by subscribing Splunk software to MQTT Broker Topics.
AMQP Modular Input: Index data from message queues provided by AMQP brokers.
JMS Modular Input: Poll and index message queues and topics from messaging queues and topics, including MQTT messages, provided by message providers, including TibcoEMS, Weblogic JMS and ActiveMQ.
Protocol Data Inputs: Recieve data via a number of different data protocols such as TCP , TCP(s) ,HTTP(s) PUT/POST/File Upload , UDP , Websockets , SockJS.
Splunk App for Stream: Capture, filter and index real-time streaming wire data and network events.
COAP Modular Input: Index messages from a COAP (Constrained Application Protocol) Server.
SNMP Modular Input: Collect data by polling SNMP attributes and catching SNMP traps from datacenter infrastructure devices providing cooling and power distribution.
In addition, Splunk has a powerful ecosystem of technology partners.
Kepware Technologies – Connects Splunk software with thousands of industrial devices communicating on over a hundred proprietary industrial protocols. Stream real-time data to Splunk from industrial control systems, including SCADA.
Carvoyant – Connected car platform, integration with Splunk software allows enterprises to monitor their automobile fleets, including geo-location, engine parameters and diagnostics.
B&B SmartWorx – Intelligent sensors and gateways. Integration with Splunk (Splunk App) will include sensor data collection (via MQTT), and gateway and sensor network diagnostics and cyber security.
Bluvision– Intelligent beacons. Integration with Splunk (Splunk App) will include beacon data collection (via Websockets). Powerful retail applications.
ThingWorx (PTC) – The leading IoT Application Development Platform. Seamless data exchange between ThingWorx applications and Splunk Enterprise and Splunk Cloud, and ThingWorx customers can access Splunk search and analytics through the ThingWorx mashup builder.
Buddy.com – Cloud services for connected devices. Integration (Splunk App) will allow Splunk to stream data from any device connected to the buddy platform.
Octoblu (Citrix) – IoT developer platform. Has created libraries that allow any Octoblu-enabled device to stream its data to Splunk software and allows those same devices to use Splunk search and analytics to inform their own decisions and logic
Red Balloon Security – Security platform for the defense of embedded systems in the enterprise (IP Phones, Printers, switches and routers, etc). Uses proprietary firmware level protection and appliance-based endpoint monitoring, and is integrating (Splunk TA/ES Compliant CIM) with Splunk software to allow Enterprise Security monitoring of threats to embedded enterprise devices.
Bayshore Networks – Content-aware cyber security platform for industrial networks. Is integrating (Splunk TA/ES Compliant CIM) with Splunk software to allow Enterprise Security monitoring of threats to SCADA and other industrial networks.
Foxguard Solutions– Cyber security and compliance solutions for industrial networks built with Splunk. NERC-CIP compliance specialists.
UltraElectronics-3eti – Cyber security platform for industrial networks. Building ES compliant TA to allow collection and analysis of security relevant ICS data in Splunk.
Distrix – Software defined networking for industrial networks and the internet of things. Simplifies connectivity and delivers and enhances data over extremely complex networks. Distrix’s SDN supports Splunk to Splunk communication, and can enhance other data, including timestamping and meta-data enrichment, for ingestion in Splunk.
Prelert – Anomaly detection app for Splunk Enterprise. Valuable app for management of sensors and devices where rapid identification of anomalies in sensor readings or operations are critical.
Predikto – Leverages the power of Predictive Analytics enabling organizations to use their data to predict future asset failures.
N3N – Custom, advanced user interfaces for Splunk specializing in isometric views of industrial facilities.
R Project App – harness the power of R statistical processing language directly from Splunk interfaces and search processing language.
D3.js – Data driven documents for powerful user experiences.
HTML5 – Advanced web interfaces and applications for browser and mobile based user experience.
Wafers are typically small, thin, shiny silicon disks, created from highly specialized processes. The disks, which resemble DVDs, are about 1 inch to 18 inches in diameter. Wafers are the canvas on which computer chips are drawn, etched and chemically processed. The larger the wafer, the more chips can be created at one time. A run of wafers can produce chips worth millions of dollars
A wafer might go through dozens of steps on its path to becoming a series of chips. Several processes take weeks and are so complex that precision is measured in terms of atoms, yet length of etched pathways are measured in kilometers. Automated carriers move the wafers from station to station. While certain processes have to happen in a set order, a set order isn’t critical to all processes. Intelligent decision algorithms enacted in real time change the order of processing depending on bottlenecks, downed machines and related factors. In the past, all of this machine data was critical and important, but it took weeks to get log data and longer to get meaning from it. The manufacturer wanted to decrease downtime to ensure profitability and eliminate delays in its customers’ finely tuned processes as well.
Unfortunately, many things can go wrong in the process of creating wafers. Here are just a few examples:
• Processes can get stuck in an infinite loop, repeating the same steps or series of steps
• A defect can occur on a line or in a process
• A critical process step can be stopped or broken
• A carrier can be stopped, backing up traffic
• A set of wafers can be flawed
Each of these processes can significantly affect yield, which affects profits. Previously, it was difficult for operations management to keep up with automated processes, let alone optimize them.
The company was challenged to:
Reduce high operating costs due to
Process inefficiencies
High defect rates – requiring production reruns and waste
MTTR – Mean time to resolution
Catch the equipment downtime in time and also understand the root cause to prevent the issue from happening again.
Catch defects early in the process this was an arduous process , to feed in data from various sources, understand what went wrong took days or weeks
Real-time Machine Data Platform
Collect, store and analyze machine data at scale
Correlate data across multiple systems
Fast time-to-value, easy integration, fewer resources needed
Full stack, no need to invest in multiple tools
Advanced Statistical Analysis
Analytics capabilities to spot anomalies and trends
Predictive modeling
Industrial Visualization
Prioritize data collection and analysis
Baseline, correlate, verify, optimize
To get the operational visibility it required, the customer determined that it needed four solutions, including a Complex Event Processer (CEP) system, a big data platform, a statistical management platform and industrial visualization software. A team of 20 specialists who were in the process of weaving these systems together for a working prototype estimated it would take a full year. Then, one of the operations managers happened to attend a meeting in another part of the semiconductor facility where Splunk software had been rolled out to meet security requirements. Further research revealed that not only could Splunk software be applied to the machine data coming in from the manufacturing floor, but it could also handle all four of the major functions required in the prototype
Splunk software makes it possible to tune manufacturing systems to create immediate corrective actions and rules, causing issues to be fixed in minutes rather than hours, or hours rather than weeks, depending on the complexity. For example, after team members used Splunk software to understand processes and error conditions with more insight, they created Splunk alerts to interact with manufacturing software to catch problems early. Figure 1 shows what happens when an issue occurs between the third and fourth process step. (Note: In the illustration and those that follow, hundreds of steps are condensed into eight in order to make complex scenarios understandable. In actuality, Splunk software has been used to tailor more than 200 rules and corrective actions for hundreds of possible scenarios.)
The customer designed its manufacturing software to respond to conditions, backups and defects in the most efficient way possible, so that when feasible, a wafer might undergo Process 4 prior to undergoing Process 2, for example. And if there is a defect after a process, a wafer might repeat a prior process or the same process, until the defect is fixed. But things can go wrong. For example, a wafer can get into an “infinite loop” and cause bottlenecks or other issues on the manufacturing floor. Figure 2 illustrates “normal” and alternative paths, including some paths that are indicative of defects or problems in the manufacturing process.
Before Splunk software, it was nearly impossible to gain insight into which alternative paths were associated with later defect issues. But when the customer applied the Splunk solution to the problem, it was finally possible to see the big picture and find opportunities to tune the processes even further