Many cities are investing in technologies to transform their cities into smart city- environments in which data collection and analysis is utilized to manage assets and resources efficiently. Modern technology can help connect the right data, at the right time, to the right people, processes and systems. Innovations around smart cities and the Internet of Things give cities the ability to improve motor safety, unify and manage transportation systems and traffic, save energy and provide a better experience for the residents. By utilizing an event streaming platform, like Confluent, cities are able to process data in real-time from thousands of sources, such as sensors. By aggregating that data and analyzing real-time data streams, more informed decisions can be made and fine-tuned operations developed for a positive impact on everyday challenges faced by cities. Learn how to: -Overcome challenges for building a smarter city -Build a real time infrastructure to correlate relevant events -Connect thousands of devices, machines, and people -Leverage open source and fully managed solutions from the Apache Kafka ecosystem

1. 1
Kai Waehner | Technology Evangelist, Confluent
contact@kai-waehner.de | LinkedIn | @KaiWaehner | www.confluent.io | www.kai-waehner.de
Enabling Smarter Cities and
Connected Vehicles with an
Event Streaming Platform
Slides created together
with Robert Cowart

2. 2
Agenda
1. Goals for Creating a Better World
2. Challenges for Building a Smarter City
3. The Smarter City Nervous System
4. Architecture Patterns for Edge, Hybrid and Global Infrastructures
5. Streaming Connectivity to Devices, Machines and People
6. Data Correlation in Real Time
7. Integration and Correlation between 100000 Connected Cars
www.kai-waehner.de | @KaiWaehner

3. 3
Agenda
1. Goals for Creating a Better World
2. Challenges for Building a Smarter City
3. The Smarter City Nervous System
4. Architecture Patterns for Edge, Hybrid and Global Infrastructures
5. Streaming Connectivity to Devices, Machines and People
6. Data Correlation in Real Time
7. Integration and Correlation between 100000 Connected Cars
www.kai-waehner.de | @KaiWaehner

4. 4
Goals for The Smarter City
Improve Pedestrian Safety
Improve Vehicle Safety
Proactively Engage First Responders
Reduce Traffic Congestion
Enable Connected/Autonomous Vehicles
Improve Customer Experience
Automate Business Processes
www.kai-waehner.de | @KaiWaehner

5. 5
Innovative new business models emerging…
https://www.wejo.com/
https://parknowgroup.com/on-street-cashless-mobile-parking-payments/
https://www.scheidt-bachmann.de/en/article/news/ticketless-parking-management-system-the-
future-has-begun-motorists-can-now-park-and-pay-without/
www.kai-waehner.de | @KaiWaehner

6. 6
Virtual Singapore:
A Digital Twin of
the (Smart) City
Possible Uses of Virtual Singapore
• Urban Planning (e.g. Crowd
Simulation)
• Collaboration and Decision-Making
• Communication and Visualisation
• Improved Accessibility
• Analysis on Potential for Solar
Energy Production
• … https://www.nrf.gov.sg/programmes/virtual-singapore
www.kai-waehner.de | @KaiWaehner

7. 7
Goals for The Smarter City
The right insights
(enriched and analyzed)
At the right time
(increasingly “real-time”)
To the right people,
processes and systems
www.kai-waehner.de | @KaiWaehner

8. 8
Agenda
1. Goals for Creating a Better World
2. Challenges for Building a Smarter City
3. The Smarter City Nervous System
4. Architecture Patterns for Edge, Hybrid and Global Infrastructures
5. Streaming Connectivity to Devices, Machines and People
6. Data Correlation in Real Time
7. Integration and Correlation between 100000 Connected Cars
www.kai-waehner.de | @KaiWaehner

9. 9
Integration with different data sources and technologies…
Traffic Cameras (video & metrics)
LIDAR
Real-time Traffic Services
Traffic Signals
Other sensors
• MetroTech IntelliSection with RTT
• Swarm Analytics Perception Box
• Quanergy
• Velodyne
• HERE
• Bing Maps
• Tom Tom
• Automated Traffic Signal Performance Measures (ATSPM)
• SAE J2735 (DSRC/WAVE) via Roadside Unit (RSU)
• Surface Temperature
• Pressure
• Induction
www.kai-waehner.de | @KaiWaehner

10. 10
Integration with different data sources and technologies…
Traffic Cameras (video & metrics)
LIDAR
Real-time Traffic Services
Traffic Signals
Other sensors
• MetroTech IntelliSection with RTT
• Swarm Analytics Perception Box
• Quanergy
• Velodyne
• HERE
• Bing Maps
• Tom Tom
• Automated Traffic Signal Performance Measures (ATSPM)
• SAE J2735 (DSRC/WAVE) via Roadside Unit (RSU)
• Surface Temperature
• Pressure
• Induction
And that is just
some of the traffic
related data!
www.kai-waehner.de | @KaiWaehner

11. 111111
The
requirement
for multiple
perspectives
…
www.kai-waehner.de | @KaiWaehner

12. 12
The need for transformation and correlation…
2019-09-30 00:00:00.500,80,82,52
2019-09-30 00:00:00.600,80,43,4
2019-09-30 00:00:00.700,80,2,6
2019-09-30 00:00:00.700,80,2,2
2019-09-30 00:00:00.900,80,82,6
2019-09-30 00:00:01.000,80,43,6
2019-09-30 00:00:02.473,80,400,0
2019-09-30 00:00:03.000,80,82,9
2019-09-30 00:00:03.900,80,81,9
2019-09-30 00:00:04.200,80,82,9
2019-09-30 00:00:04.400,80,8,6
2019-09-30 00:00:04.400,80,81,6
2019-09-30 00:00:04.400,80,4,6
2019-09-30 00:00:04.400,80,7,6
2019-09-30 00:00:04.400,80,8,2
2019-09-30 00:00:04.400,80,4,2
2019-09-30 00:00:04.400,80,7,2
2019-09-30 00:00:04.500,80,81,9
2019-09-30 00:00:04.500,80,44,6
2019-09-30 00:00:08.500,80,9,2
GAP OUT
GREEN TERMINATION
BEGIN YELLOW CLEARANCE
www.kai-waehner.de | @KaiWaehner
Traffic light station
sensor information

13. 13
Why the “right time” is “real time”…
www.kai-waehner.de | @KaiWaehner

14. 14
Why the “right time” is “real time”…
https://www.ntsb.gov/investigations/AccidentReports/Pages/HWY18MH010-prelim.aspx
www.kai-waehner.de | @KaiWaehner

15. 15
Why data correlation is important…
www.kai-waehner.de | @KaiWaehner

16. 16
Agenda
1. Goals for Creating a Better World
2. Challenges for Building a Smarter City
3. The Smarter City Nervous System
4. Architecture Patterns for Edge, Hybrid and Global Infrastructures
5. Streaming Connectivity to Devices, Machines and People
6. Data Correlation in Real Time
7. Integration and Correlation between 100000 Connected Cars
www.kai-waehner.de | @KaiWaehner

17. 17
A Streaming Platform is the Underpinning of an
Event-driven Architecture
Sensors
Cameras
CRM
Mobile
Real-time routing
Cross selling
Data warehouse
Producers Consumers
Object
detection
Sensor
event
CRM
data
Customer
experiences
Streams of real time events
Stream processing
apps
Connectors Connectors
Stream processing
apps

18. 18
Apache Kafka – The Commit Log
Time
P
C1 C2
C3
www.kai-waehner.de | @KaiWaehner

19. 19
Apache Kafka – A Distributed System
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4
www.kai-waehner.de | @KaiWaehner

20. 20
Apache Kafka (kafka.apache.org) includes
Kafka Connect and Kafka Streams
Kafka Streams
Your app
sinksource
Kafka ConnectKafka Connect
www.kai-waehner.de | @KaiWaehner

21. 21
Building the Smarter City Nervous System with Confluent
• Middleware
• Streaming ETL (Transform,
Enrichment, Multi-Stream)
• Business Applicationswww.kai-waehner.de | @KaiWaehner

22. 22
Agenda
1. Goals for Creating a Better World
2. Challenges for Building a Smarter City
3. The Smarter City Nervous System
4. Architecture Patterns for Edge, Hybrid and Global Infrastructures
5. Streaming Connectivity to Devices, Machines and People
6. Data Correlation in Real Time
7. Integration and Correlation between 100000 Connected Cars
www.kai-waehner.de | @KaiWaehner

23. 23
A Kafka Cluster
Zookeeper Zookeeper Zookeeper
Kafka Broker Kafka Broker Kafka Broker
Schema Registry
Schema Registry
Producer Consumer
Kafka Connect
Kafka Connect
www.kai-waehner.de | @KaiWaehner

24. 24
Disaster Recovery
with 2 Kafka Clusters
Zookeeper Zookeeper Zookeeper
Kafka Broker Kafka Broker Kafka Broker
Schema Registry
Schema Registry
Producer Consumer
Kafka Connect
Kafka Connect
Zookeeper Zookeeper Zookeeper
Kafka Broker Kafka Broker Kafka Broker
Schema Registry
Schema Registry
Producer Consumer
Kafka Connect
Kafka Connect
Data Center
City-North
Data Center
City-South
Streaming Replication
www.kai-waehner.de | @KaiWaehner

25. 25
Aggregation of Kafka Clusters
Zookeeper Zookeeper Zookeeper
Kafka Broker Kafka Broker Kafka Broker
Schema Registry
Schema Registry
Producer Consumer
Kafka Connect
Kafka Connect
Zookeeper Zookeeper Zookeeper
Kafka Broker Kafka Broker Kafka Broker
Schema Registry
Schema Registry
Producer Consumer
Kafka Connect
Kafka Connect
Zookeeper Zookeeper Zookeeper
Kafka Broker Kafka Broker Kafka Broker
Schema Registry
Schema Registry
Producer Consumer
Kafka Connect
Kafka Connect
Zookeeper Zookeeper Zookeeper
Kafka Broker Kafka Broker Kafka Broker
Schema Registry
Schema Registry
Producer Consumer
Kafka Connect
Kafka Connect
Zookeeper Zookeeper Zookeeper
Kafka Broker Kafka Broker Kafka Broker
Schema Registry
Schema Registry
Kafka Connect
Kafka Connect
Kafka Broker Kafka Broker Kafka Broker
www.kai-waehner.de | @KaiWaehner
Analytics
Data Center / Cloud
Data Collection
Data Center
City-North
Data Collection
Data Center
City-East
Data Collection
Data Center
City-South
Data Collection
Data Center
City-West

26. 26
Regional Edge Processing
with Kafka Clusters
Zookeeper
Kafka Broker
Schema Registry
OPC-UA
MQTT
PLC4X
KSQL
Grafana
Postgres
Kafka Connect
Zookeeper
Kafka Broker
Schema Registry
OPC-UA
MQTT
PLC4X
KSQL
Grafana
Postgres
Kafka Connect
Zookeeper
Kafka Broker
Schema Registry
OPC-UA
MQTT
PLC4X
KSQL
Grafana
Postgres
Kafka Connect
Zookeeper
Kafka Broker
Schema Registry
OPC-UA
MQTT
PLC4X
KSQL
Grafana
Postgres
Kafka Connect
Zookeeper Zookeeper Zookeeper
Kafka Broker Kafka Broker Kafka Broker
Schema Registry
Schema Registry
Kafka Connect
Kafka Connect
Kafka Broker Kafka Broker Kafka Broker
Real Time Correlations
DC North
Real Time Correlations
DC East
Real Time Correlations
DC South
Real Time Correlations
DC West
www.kai-waehner.de | @KaiWaehner
Synchronization
Data Center / Cloud

27. 27
Architecture patterns for distributed, hybrid, edge
and global Apache Kafka deployments
https://www.kai-waehner.de/blog/2020/01/29/
deployment-patterns-distributed-hybrid-edge-global-multi-data-center-kafka-architecture/
www.kai-waehner.de | @KaiWaehner

28. 28
Agenda
1. Goals for Creating a Better World
2. Challenges for Building a Smarter City
3. The Smarter City Nervous System
4. Architecture Patterns for Edge, Hybrid and Global Infrastructures
5. Streaming Connectivity to Devices, Machines and People
6. Data Correlation in Real Time
7. Integration and Correlation between 100000 Connected Cars
www.kai-waehner.de | @KaiWaehner

29. 29
Ingestion
Edge
Collection
Edge to
Cloud
Cloud
Ingress
Kafka
Ingress
dslink-scala-
kafka10
toketi-kafka-
connect-iothub
kafka-connect-
gcp-pubsub
www.kai-waehner.de | @KaiWaehner

30. 30
Data Processing
Producers Consumers• Client Libraries
• Kafka Connectors
• Proxies
• Client Libraries
• Kafka Connectors
• Proxies
X
Consumer Groups
www.kai-waehner.de | @KaiWaehner

31. 31
Kafka Connect
Elasticsearch Sink
Connector
Kafka Connect
InfluxDB Sink
Connector
Kafka Clients
JavaScript, Golang,
C, C++, Python,
REST, etc.
Stream
Services
User Facing
Dashboards
Infrastructure
Health
Egress
www.kai-waehner.de | @KaiWaehner

32. 32
Kafka Connect - Data Sources and Sinks
Data Diode
Pre-built
Connectors
Hundreds of open source and commercial connectors available

33. 33
Event Streaming and IoT Platforms are Complementary
Kafka
Cluster
Siemens
MindSphere
Kafka Client
(Java, .NET, Go, Python)
Kafka Streams
ksqlDB
Sensors
REST
Proxy
MQTT
Broker
MQTT
Connector
Kafka Connect
Azure
IoT Hub
Mobile App

34. 34
Agenda
1. Goals for Creating a Better World
2. Challenges for Building a Smarter City
3. The Smarter City Nervous System
4. Architecture Patterns for Edge, Hybrid and Global Infrastructures
5. Streaming Connectivity to Devices, Machines and People
6. Data Correlation in Real Time
7. Integration and Correlation between 100000 Connected Cars
www.kai-waehner.de | @KaiWaehner

35. 35
Data Processing and Correlation
Topic (observation-raw)
www.kai-waehner.de | @KaiWaehner

36. 36
Data Processing and Correlation
Topic (observation-raw)
Metadata and Geo
(lat/long) Enrichment
www.kai-waehner.de | @KaiWaehner

37. 37
Traditional
Database
Event Streaming
Process
SELECT * FROM
DB_TABLE
CREATE TABLE T
AS SELECT * FROM
EVENT_STREAM
Active Query: Passive Data:
DB Table
Active Data: Passive Query:
Event Stream
www.kai-waehner.de | @KaiWaehner

38. 38
Data Processing and Correlation
Topic (observation-raw)
Metadata and Geo
(lat/long) Enrichment
Topic (observation-meta)
www.kai-waehner.de | @KaiWaehner

39. 39
Data Processing and Correlation
Topic (observation-raw)
Metadata and Geo
(lat/long) Enrichment
Topic (observation-meta)
Streams
Health Score &
Incident Detection
www.kai-waehner.de | @KaiWaehner

40. 4040
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
www.kai-waehner.de | @KaiWaehner

41. 41
Data Processing and Correlation
Topic (observation-raw)
Metadata and Geo
(lat/long) Enrichment
Topic (observation-meta)
Streams
Health Score &
Incident Detection
Topic (observation-out)
www.kai-waehner.de | @KaiWaehner

42. 42
Data Processing and Correlation
Topic (observation-raw)
Metadata and Geo
(lat/long) Enrichment
Topic (observation-meta)
Streams
Health Score &
Incident Detection
Topic (observation-out)
Elasticsearch
Kafka Connect
Elasticsearch Sink
Connector
www.kai-waehner.de | @KaiWaehner

43. 43
Agenda
1. Goals for Creating a Better World
2. Challenges for Building a Smarter City
3. The Smarter City Nervous System
4. Architecture Patterns for Edge, Hybrid and Global Infrastructures
5. Streaming Connectivity to Devices, Machines and People
6. Data Correlation in Real Time
7. Integration and Correlation between 100000 Connected Cars
www.kai-waehner.de | @KaiWaehner

44. 44
Streaming Analytics with Kafka and TensorFlow
MQTT
Proxy
Elastic
Search
Grafana
Kafka
Cluster
Kafka
Connect
Car Sensors
Kafka Ecosystem
TensorFlow
Other Components
Kafka
Streams
(Java)
All
Data
Critical
Data
Ingest
Data
Potential
Detect
KSQL
TensorFlow
Train
Analytic
Model
Consume
Data
Preprocess
Data
Analytic
Model
Deploy Analytic
Model
Python
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
www.kai-waehner.de | @KaiWaehner

45. 45
Architecture for 100000 Connected Cars
Kafka + KSQL + MQTT + TensorFlow + Kubernetes
https://www.youtube.com/watch?v=7oVSLt0AZ3M
www.kai-waehner.de | @KaiWaehner

46. 46
Questions?
Let’s connect...
Kai Waehner
Technology Evangelist
kai.waehner@confluent.io
@KaiWaehner
www.confluent.io
www.kai-waehner.de
LinkedIn