2019 Confluent Streaming Event

1. Why we 💗 Kafka @ Bosch PT
Ralph Debusmann &
IoT Solution Architect
Robert Bosch Power Tools
Lars Dülfer
Distinguished Software Engineer
Novatec
1

2. WHY WE LOVE
KAFKA @ BOSCH
PT – PART 1

3. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Agenda
1. RefinemySite – The Digitization of Lean Construction
2. How we use Apache Kafka and the Confluent Platform (on Confluent Cloud Enterprise)
3. Use Cases driven by (Immutable) Event Logs in Apache Kafka
4. Learnings / Pitfalls
3

4. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Digital Construction Site Ecosystem
„RefinemySite“

5. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
5

6. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Welcome to the world of Lean Construction
RefinemySite is inspired by the Last Planner ® System
6

7. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Many challenges – one solution
Last Planner with RefinemySite
7
Communication &
Documentation
Continuous
Improvement
Collaborative
Planning

8. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
by continously combining infos from planning and production
RefinemySite supports the continues improvement process
8

9. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
RefinemySite development fundamentals:
9

10. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
RefinemySite development fundamentals:
10

11. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
RefinemySite development fundamentals:
11

12. HOW WE USE
APACHE KAFKA
AND
THE CONFLUENT
PLATFORM

13. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Components / Usages
How we use Kafka
 Confluent Cloud Enterprise (Broker, Schema Registry, ACL‘s)
 Spring Boot with Spring Kafka (Consumer and Producer API)
 Topics with infinite retention -> (Immutable) Event Logs
 Topics with time-based retention
 Multiple Schemas per Topic
 Log compaction will be used shortly for some topics
 Write-Through Approach (Current State in Data Store + Events in Kafka)
13

14. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
14

15. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
15
Core (Producer) Service
Custom Kafka Connector
Downstream (Consuming) Services
- some are producing events again ...

16. USE CASES DRIVEN BY
THE CONCEPT OF
(IMMUTABLE) EVENT
LOGS IN APACHE KAFKA

17. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Activity Streams
RefineMySite Use Cases Driven by Kafka
 The Activity Stream is just a human readable view on the
project event stream
 A dedicated service transforms the events into a human readable
format and provides a REST API for the activities
 Activites are stored in Azure Cosmos DB with sharded collections
17

18. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Statistics
RefineMySite Use Cases Driven by Kafka
 Statistics are calculated based on the stream
of project events
 Easy to develop new statistics / KPI‘s by re-reading
the event stream and creating new views
18

19. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Notifications & News
RefineMySite Use Cases Driven by Kafka
 Notification Service subscribes to events,
determines the recipients for each event
(fan-out) and creates notifications based
on the event
 Updates about the „last seen“ status are sent
to the clients via Server Sent Events
19
Deliver all relevant information for the
right people at the right time.

20. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Live Updates
RefineMySite Use Cases Driven by Kafka
 Live Updates allow clients to react on
server events and update their state
 This enhances the collaborative user experience
 Clients receive events using Server Sent Events
 A service subscribes to all project events and
determines to which online client an event has
to be sent
User A
User B
20

21. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Business Activity Monitoring
RefineMySite Use Cases Driven by Kafka
21

22. LEARNINGS AND
PITFALLS
(AT LEAST A FEW OF
THOSE ... )

23. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Event design decides about applicability of certain features
Learnings and Pitfalls
Snapshots (event-based state transfer / asynch state transfer)
... ...Project
Created
Identifier: ...
Name: ...
Participant
Added
Project: ...
Employee: ...
Role: ...
Task Created
Identifier: ...
Project: ...
Name: ...
Start: ...
End: ...
Task Assigned
Identifier: ...
Participant: ...
Task
Started
Identifier:
Task
Postponed
Start: ...
... ...Project
Created
Identifier: ...
Name: ...
Participant
Created
Project: ...
Employee: ...
Role: ...
Task Created
Identifier: T1
Project: ...
Name: ...
Start: 01.09.
End: ...
Status: Draft
Task Updated
Identifier: T1
Project: ...
Name: ...
Start: ...
End: ...
Assignee: ...
Status: Open
Task Updated
Identifier: T1
Project: ...
Name: ...
Start: ...
End: ...
Assignee: ...
Status:
Started
Task Updated
Identifier: T1
Project: ...
Name: ...
Start: 01.10.
End: ...
Assignee: ...
Status:
Started
„Real Events“ (with Deltas)
23

24. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
A field is added with a default value – this is backward and
forward compatible
Schema Evolution can be tricky
24
{
"namespace": "example.avro",
"type": "record",
"name": "user",
"fields": [
{"name": "name", "type": "string“},
{"name": "favorite_number", "type": "int"},
{"name": "favorite_color", "type": "string", "default": "green"}
]
}
{
"namespace": "example.avro",
"type": "record",
"name": "user",
"fields": [
{"name": "name", "type": "string“},
{"name": "favorite_number", "type": "int"},
]
}
Version 1
Version 2

25. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
A consumer with an older schema just ignores the favourite color
Schema Evolution can be tricky
25
Mike, 5 Steve, 13 Peter, 8...
James, 33,
blue
Jon, 19, red ...
Topic
Mike, 5 Steve, 13 Peter, 8... James, 33 Jon, 19 ...
Using version 1 of the schema, the consumer receives:
From this time on,
messages were written with version 2

26. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Consumers with updated schema use default favorite color
Schema Evolution can be tricky
26
Mike, 5 Steve, 13 Peter, 8...
James, 33,
blue
Jon, 19, red ...
Topic
From this time on,
messages were written with version 2
Using version 2 of the schema, the consumer receives:
Mike, 5,
green
Steve, 13,
green
Peter, 8,
green
...
James, 33,
blue
Jon, 19, red ...

27. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Depending on the required compatability, only certain schema
changes are allowed
Schema Evolution can be tricky
27
Compatibility Type Changes allowed Check against which schemas Upgrade first
BACKWARD
•Delete fields
•Add optional fields
Last version Consumers
BACKWARD_TRANSITIVE
•Delete fields
•Add optional fields
All previous versions Consumers
FORWARD
•Add fields
•Delete optional fields
Last version Producers
FORWARD_TRANSITIVE
•Add fields
•Delete optional fields
All previous versions Producers
FULL •Modify optional fields Last version Any order
FULL_TRANSITIVE •Modify optional fields All previous versions Any order
NONE •All changes are accepted Compatibility checking disabled Depends
Source: https://docs.confluent.io/current/schema-registry/avro.html
This is required if
you keep events
forever and want
ability to re-read
your event log.

28. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
What we learned from the „write-through“ approach
Learnings and Pitfalls
28
 You can only guarantee „at-least once“ semantics
 Prepare your consumers for duplicated events (Idempotent Consumers)
 Be carefull how you implement retries in the process that transports events from the DB table to
Apache Kafka
 Using Kafka Transactions (almost) eliminates duplicates
DB of User Service
Table A
Table B
Table C
Table
User
Events
Kafka User Topic
Kafka Connector
(Custom Application)

29. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Create a decent set of Testing Tools
Learnings and Pitfalls
29

30. PT/BDO | 2019-11-11
© Robert Bosch Power Tools GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
Building idempotent consumers requires a bit of thinking ...
Learnings and Pitfalls
30
 Think about the kind of duplications that can happen in your event stream
 Will there only be duplicates immediately following each other like -> A B C D E E F G H I J K L ... ?
 Will you also have repeating sequences like -> A B C D E F G H I F G H I J K L ... ?
 Spend time and write proper tests for these szenarios

31. WHY WE LOVE
KAFKA @ BOSCH
PT – PART 2

32. 32
&
Where we are now
Power Tools

33. 33

34. 34

35. 35

36. 36

37. 37

38. 38

39. 39
&
The Future is Databaseless and Serverless
Power Tools

40. 40

41. 41

42. 42

43. 43

44. 44

45. 45

46. 46

47. 47

48. 48

49. 49

50. 50

51. LOOKING FORWARD
TO SEEING AND
DISCUSSING WITH
YOU AT LUNCH!
THANK YOU!
51