In this session, we are going to create a Lenses DataOps hub for IoT data with Apache Kafka and InfluxDB flows over Kubernetes. We will demonstrate how to create streaming flows and securely explore and monitor real-time data. We will use Kubernetes to spin up scalable flows and go through how we can simply provision such flows with secret management and monitoring end to end out capabilities.
9. #DataOps IoT Challenges
Enormous Volumes of Data
“500 billion connected devices by 2025”
Its Management and Analysis will become
harder and continue to break traditional tools
11. How to implement a Streaming
Architecture for IoT Time-Series
data and Real-Time insights?
12. Infrastructure Layer
Where the Data lives
Self-service Data Access,
Multi-tenancy,
Security, Governance to
Accessibility & Visibility
for ALL
13.
14. IoT & kafka
High Volumes, N devices &
irregular intervals
Real Time Analytics &
Microservices
Multiple sources of data &
long term storage
An open source streaming framework
with messaging semantics where
records are key-value pairs
➔ Unlimited streams of data,
➔ Producers & Consumers
(pub/sub)
➔ Processing and analysing data
in motion
➔ Connect API to move data with
pluggable reusable components
15. The flow implementation
➔ Sensor produces data to Kafka
➔ Create my Data Flows:
◆ Kafka - to - Kafka
processing
◆ Route to InfluxDB
19. Lenses SQL Processors
● Simply Filter, Enrich, Split & Bind your data
● Manipulate Live Streams of Data
● Scalable: Cloud / Kubernetes Native
PROCESS
20. Connect Kafka to InfluxDB
Real Time Ingestion
Distributed
Fault tolerance
Scalability
Error Handling
Monitoring & Alerting
Governance & Security
Easy Data Manipulation
...
INGEST
30. Lenses Box
● FREE for Developers
● Single Broker setup
● All ecosystem services
● 25+ Connectors
● Synthetic Data generators
● Live examples
● Lenses intuitive UI
● The powerful Lenses SQL
● Works on your Laptop
● Works on Cloudhttps://lenses.io/lenses-box