Internet of Things (IoT) devices are becoming more ubiquitous in consumer, business and industrial landscapes. They are being widely used in applications ranging from home automation to the industrial internet. They pose a unique challenge in terms of the volume of data they produce, and the velocity with which they produce it, and the variety of sources they need to handle. The challenge is to ingest and process this data at the speed at which it is being produced in a real-time and fault tolerant fashion. Apache Apex is an industrial grade, scalable and fault tolerant big data processing platform that runs natively on Hadoop. In this deck, you will see how Apex is being used in IoT applications and also see how the enterprise features such as dimensional analytics, real-time dashboards and monitoring play a key role. Presented by Pramod Immaneni, Principal Architect at DataTorrent and PPMC member Apache Apex, on BrightTALK webinar on Apr 6th, 2016

1. Pramod Immaneni <pramod@datatorrent.com>
PPMC Member, Architect @DataTorrent Inc
Apr 6th, 2016
A native Hadoop platform
IoT Ingestion & Analytics using Apache Apex

2. © 2015 DataTorrent
IoT (Internet of Things)
2
• Network of objects in physical world with electronic sensors
• Collecting and sending data
•Consumer
• Smart home – Smart bulbs, thermostats, locks
• Connected vehicles
• Wearables
• Industrial
• Manufacturing
• Retail
• Energy

3. © 2015 DataTorrent
Challenges
3
• Velocity of data
• Machines generate data at a much higher pace than humans
• They generate data continuously without pause posing processing
challenges
• Certain aspects of data may become stale if not processed in time
• Volume
• Large volumes of data for processing and storage
• Volume keep increasing because of the exponential growth in devices
• Research estimates zettabytes by 2020
• Variety
• Types of data – sensor, location, kpi
• Formats – csv, json
• Delivery mechanisms – files, message queues

4. © 2015 DataTorrent
What should a solution have
4
• Ingest and perform analytics on data in motion
• Handle the incoming throughput while maintaining SLAs
• Scale with the volume of data
• Fault tolerant so that no data is lost
• Flexible to handle data variety and allow custom business
logic
• Easy to use tools for the user

5. © 2015 DataTorrent
What is Apex
5
• Platform and runtime engine to enable scalable and fault-
tolerant distributed applications
• Hadoop native
• Process streaming or batch big data
• High throughput and low latency
• Library of commonly needed business logic
• Build any custom business logic in your application

6. © 2015 DataTorrent
Applications
6
• Distributed processing
• Application logic broken into components called operators that run in a distributed fashion
across your cluster
• Scalable
• Operators can be partitioned into multiple instances to handle large volumes
• Fault tolerant
• Automatically recover from node outages without having to reprocess from beginning
• State is preserved
• Long running applications
• Operators
• Use library to build applications quickly
• Write your own in Java using the API
• Operational insight – DataTorrent RTS
• See how each operator is performing and even record data

7. © 2015 DataTorrent
Apex Platform Overview
7

8. © 2015 DataTorrent
Apache Malhar Library
8

9. © 2015 DataTorrent
Application Development Model
9
 A Stream is a sequence of data tuples
 A typical Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library
• Operator has many instances that run in parallel and each instance is single-threaded
 Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Output
Stream
Tupl
e
Tupl
e
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator

10. © 2015 DataTorrent
Native Hadoop Integration
10
• YARN is
the
resource
manager
• HDFS used
for storing
any
persistent
state

11. © 2015 DataTorrent
Advanced Windowing Support
11
 Application window
 Sliding window and tumbling window
 Checkpoint window
 No artificial latency

12. © 2015 DataTorrent
Scalability & Partitioning
12
NxM PartitionsUnifier
0 1 2 3
Logical DAG
0 1 2
1
1 Unifier
1
20
Logical Diagram
Physical Diagram with operator 1 with 3 partitions
0
Unifier
1a
1b
1c
2a
2b
Unifier 3
Physical DAG with (1a, 1b, 1c) and (2a, 2b): No bottleneck
Unifier
Unifier0
1a
1b
1c
2a
2b
Unifier 3
Physical DAG with (1a, 1b, 1c) and (2a, 2b): Bottleneck on intermediate Unifier

13. © 2015 DataTorrent
Advanced Partitioning
13
0
1a
1b
2 3 4Unifier
Physical DAG
0 4
3a2a1a
1b 2b 3b
Unifier
Physical DAG with Parallel Partition
Parallel Partition
Container
uopr
uopr1
uopr2
uopr3
uopr4
uopr1
uopr2
uopr3
uopr4
dopr
dopr
doprunifier
unifier
unifier
unifier
Container
Container
NICNIC
NICNIC
NIC
Container
NIC
Logical Plan
Execution Plan, for N = 4; M = 1
Execution Plan, for N = 4; M = 1, K = 2 with cascading unifiers
Cascading Unifiers
0 1 2 3 4
Logical DAG

14. © 2015 DataTorrent
Dynamic Partitioning
14
• Partitioning change while application is running
ᵒ Change number of partitions at runtime based on stats
ᵒ Determine initial number of partitions dynamically
• Kafka operators scale according to number of kafka partitions
ᵒ Supports re-distribution of state when number of partitions change
ᵒ API for custom scaler or partitioner
2b
2c
3
2a
2d
1b
1a1a 2a
1b 2b
3
1a 2b
1b 2c 3b
2a
2d
3a
Unifiers not shown

15. © 2015 DataTorrent
Fault Tolerance
15
• Operator state is checkpointed to a persistent store
ᵒ Automatically performed by engine, no additional work needed by operator
ᵒ In case of failure operators are restarted from checkpoint state
ᵒ Frequency configurable per operator
ᵒ Asynchronous and distributed by default
ᵒ Default store is HDFS
• Automatic detection and recovery of failed operators
ᵒ Heartbeat mechanism
• Buffering mechanism to ensure replay of data from recovered point so
that there is no loss of data
• Application master state checkpointed

16. © 2015 DataTorrent
Processing Guarantees
16
Atleast once
• On recovery data will be replayed from a previous checkpoint
ᵒ Messages will not be lost
ᵒ Default mechanism and is suitable for most applications
• Can be used in conjunction with following mechanisms to achieve
exactly-once behavior in fault recovery scenarios
ᵒ Transactions with meta information, Rewinding output, Feedback from
external entity, Idempotent operations
Atmost once
• On recovery the latest data is made available to operator
ᵒ Useful in use cases where some data loss is acceptable and latest data is
sufficient
Exactly once
• At least once + state recovery + operator logic to achieve end-to-end
exactly once

17. © 2015 DataTorrent
Ingestion Operators
17
• NFS/FTP/S3/HDFS Readers and Writers
• Kafka readers and writers
• Cassandra, HBase, Hive, JDBC
• CSV, JSON, Avro, XML parsers
• Filtering and enrichment
• Dedup
• Dimension computation
• Stream Joins
• …

18. © 2015 DataTorrent
Pipeline builder
18

19. © 2015 DataTorrent
Monitoring Console
Logical View
19

20. © 2015 DataTorrent
Monitoring Console
Physical View
20

21. © 2015 DataTorrent
Real-Time Dashboards
Real Time Visualization
21

22. © 2015 DataTorrent
IoT solutions
22

23. © 2015 DataTorrent
Resources
23
• Ingest application – http://www.datatorrent.com/download
• Solutions – http://www.datatorrent.com/solutions
• Subscribe - http://apex.incubator.apache.org/community.html
• Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex
• Free Enterprise License for Startups - https://www.datatorrent.com/product/startup-
accelerator/
• Cloud Trial - http://datatorrent.com/cloudtrial.html

24. © 2015 DataTorrent
We Are Hiring
24
• jobs@datatorrent.com
• Developers/Architects
• QA Automation Developers
• Information Developers
• Build and Release
• Community Leaders

25. © 2015 DataTorrent
Upcoming Events
25
• April 14th 6pm PDT – IOT Big Data Ingestion and Processing in
Hadoop by Silver Spring Networks
• May 4th 6pm PDT – Fault Tolerant HDFS r/w with Apache Apex and
Apex (native Hadoop) benchmarks

26. End
26