It is not enough to build a mesh of sensors or embedded devices to obtain more insights about the surrounding environment and optimize your production systems. Usually, your IoT solution needs to be capable of transferring enormous amounts of data to storage or the cloud where the data have to be processed further. Quite often, the processing of the endless streams of data has to be done in real-time so that you can react on the IoT subsystem's state accordingly. This session will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources. In particular, attendees will learn about data streaming to an Apache Ignite cluster from embedded devices and real-time data processing with Apache Spark.

1. © 2017 GridGain Systems, Inc.
Where Fast Data Meets the IoT
Apache Ignite and Apache Spark
Denis Magda
Ignite PMC Chair
GridGain PM

2. © 2017 GridGain Systems, Inc.
• IoT Demands to Software
• IoT Software Stack
• Device OS/RTOS
• Data Collection and Enrichment
• HTAP Platform
• Application APIs
• Demo
Agenda

3. © 2017 GridGain Systems, Inc.
IoT Demands to Software
Real-time Processing
SQL, Geo-Spatial
Analytics (BI, ML)
High-Availability
Simple Scalability

4. © 2017 GridGain Systems, Inc.
IoT Software Stack
Device OS/Real-Time OS
Data Collection and Enrichment
HTAP Platform
Application APIs

5. © 2017 GridGain Systems, Inc.
Apache IoT Software Stack
Device OS/Real-Time OS
Data Collection and Enrichment
HTAP Platform
Application APIs

6. © 2017 GridGain Systems, Inc.
Apache MyNewt
Open Source RTOS
Cortex M, MIPS Bluetooth, Wifi,
TCP/IP
Secure Bootloader
Remote Firmware
Upgrade

7. © 2017 GridGain Systems, Inc.
Data Collection and Enrichment
DURABLE MEMORY
DURABLE MEMORY
Ignite Cluster

8. © 2017 GridGain Systems, Inc.
Apache Ignite In-Memory Computing Platform
Memory-Centric Storage
Ignite Native Persistence
(Flash, SSD, Intel 3D XPoint)
Third-Party Persistence
(RDBMS, HDFS, NoSQL)
SQL Transactions Compute Services MLStreaming
Applications
Key/Value
IoTFinancial
Services
Pharma &
Healthcare
E-CommerceTravel &
Logistics
Telco

9. © 2017 GridGain Systems, Inc.
Data Grid
JCache Transactions Compute SQL
RDBMS
NoSQL
HDFS
Server Node
Distributed Key-Value Store
Dynamic
Scaling
Distributed
partitioned
hash map
ACID TransactionJCache & SQL
Server Node Server Node
3rd party storage caching
DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY

10. © 2017 GridGain Systems, Inc.
Distributed SQL
JDBC ODBC SQL API
Java .NET C++ BI
SELECT, UPDATE,
INSERT, MERGE,
DELETE, CREATE
and ALTER
DDL, DML Support
Cross-platform
Compatibility
Indexes in
RAM or Disk
Dynamic
Scaling
Server Node Server NodeServer Node
Apache Ignite Cluster
DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY
Tools

11. © 2017 GridGain Systems, Inc.
Compute Grid
DURABLE MEMORY
DURABLE MEMORY
Ignite Cluster
C1
R1
C2
R2
C = C1 + C2
R = R1 + R2
C = Compute
R = Result
in T/2 time
Automatic Failover
Load Balancing
Zero Deployment

12. © 2017 GridGain Systems, Inc.
Ignite and Spark Integration
Spark Application
Spark Worker
Spark
Job
Spark
Job
Yarn Mesos Docker HDFS
Spark Worker
Spark
Job
Spark
Job
Spark Worker
Spark
Job
Spark
Job
In-Memory Shared RDD or DataFrame
Share RDD
across jobs on
the host
In-Memory
Indexes
SQL on top of
RDDs
Share RDD
Globally
Ignite Node Ignite Node Ignite Node

13. © 2017 GridGain Systems, Inc.
Machine Learning Grid
K-Means Regressions Decision Trees
R C++ Python Java
Server Node Server NodeServer Node
Distributed Core Algebra
DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY
Scala REST
Random ForestDistributed Algorithms
Dense and Sparse
Algebra
Collocated Processing
Multi-Language
Support

14. © 2017 GridGain Systems, Inc.
Apache Ignite Users
FinTech
Financial Services Software Logistics & Travel
E-commerce
Telco
IoT
Pharma & Healthcare
Adtech

15. © 2017 GridGain Systems, Inc.
SilverSpring Networks develops IoT solutions that
transmit energy consumption data between meters,
consumers and utilities in real time.
Problem
• Could not meet latency and throughput SLAs
• Missing scalability and elasticity
GridGain Solution
• 50 millions meters stream the data back in real-time
• Collocated in-memory processing
• Advanced security and multi-tenancy
SQL
Smart Meters
GridGain Ignite Cluster
DB
IN-MEMORY IN-MEMORY IN-MEMORY IN-MEMORY
GridGain Advanced Security
- Smart Metering and Utilities
Compute Transactions
SilverSpring Platform

16. © 2017 GridGain Systems, Inc.
Demo

17. © 2017 GridGain Systems, Inc.
Enter our raffle: 2 winners each week! http://bit.ly/IMC_raffle

18. © 2017 GridGain Systems, Inc.
Thank you for joining us. Follow the conversation.
http://ignite.apache.org
Any Questions?
#apacheignite
#denismagda