Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement? This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019
1. gschmutz
Kafka as an Event Store – is it Good
Enough?
Guido Schmutz
Kafka Summit New York – 2.4.2019
gschmutz http://guidoschmutz.wordpress.com
2. gschmutz
Agenda
1. How do we build applications traditionally?
2. CQRS & Event Sourcing
3. What exactly is an Event Store?
4. Implementing Event Store
5. Summary
Kafka as an Event Store – is it Good Enough?
3. gschmutz
Guido Schmutz
Working at Trivadis for more than 22 years
Oracle Groundbreaker Ambassador & Oracle ACE Director
Consultant, Trainer, Solution Architect for Java, Oracle, Microservices and
Big Data / Fast Data
Head of Trivadis Architecture Board
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
151st edition
Kafka as an Event Store – is it Good Enough?
4. gschmutz
How do we build applications
traditionally?
Kafka as an Event Store – is it Good Enough?
5. gschmutz
Data Access Layer
Monolithic Applications – using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
Kafka as an Event Store – is it Good Enough?
6. gschmutz
Data Access Layer
Monolithic Applications – using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
Object/Relational
Impedance Mismatch
• Traditional approach
to persistence
• Store current state
• CRUD operations
• Coupling between
read & write
• Increased Complexity
Kafka as an Event Store – is it Good Enough?
7. gschmutz
• Traditional approach
to persistence
• Store current state
• CRUD operations
• Coupling between
read & write
• Increased Complexity
Data Access Layer
Monolithic Applications – using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
Object/Relational Mapping
SELECT acc.id, acc.account_number, acc.account_type.id,
acct.name, cus.first_name, cus.last_name,
addr.street, addr.city
FROM account_t acc
, account_type_t acct
, customer_t cus
, cust_adr_t cusa
, address_t addr
WHERE acc.account_type_id = acct.account_type_id
AND adc.customer_id = cus.customer_id
AND cusa.customer_id = cus.customer_id
AND cusa.address_id = addr.address_id
AND cusa.type = 'MAIN'
Kafka as an Event Store – is it Good Enough?
8. gschmutz
Microservices - using Layered Architecture
Microservices …
• are responsible for their data
• might use NoSQL instead of
RDBMS
• often still use traditional approach
to persistence
• “Data silos” do no longer support
database join
• keep synchronous
communication to a minimum
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Account Microservice
{ }
Account API
Account
Account Logic
Product Microservice
{ }
Product API
Product
Product Logic
Finance App
Finance UI UI Logic
GUI
REST
REST
REST
Kafka as an Event Store – is it Good Enough?
sync request/response
async, event pub/sub
9. gschmutz
Domain Driven Design (DDD) – Concepts
Kafka as an Event Store – is it Good Enough?
Domain Objects – hold the state of the
application
Entity – Domain Objects with an identity
Value Object – an immutable type that is
distinguishable only by the state of its
properties and has no identity
Aggregate - A cluster of domain objects that
can be treated as a single unit
Aggregate Root – one object of aggregate is
root object. Any reference from outside goes
through aggregate root
Aggregate Root
Account Aggregate
Customer
Aggregate
Aggregate Root
10. gschmutz
Events
Distribute to all handlers
strong ordering req’s
No results
Queries
Route with load balancing
Sometimes scatter-gather
Provide result
Three mechanisms through which services can interact
Kafka as an Event Store – is it Good Enough?
Commands
Route to single handler
Use consistent hashing
Provide Result
Adapted from Axon IQ
11. gschmutz
Microservices with Event-driven communication
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Account Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
REST
REST
REST
Pub /
Sub
Customer
Mat View
Kafka as an Event Store – is it Good Enough?
sync request/response
async, event pub/sub
Finance App
Finance UI UI Logic
GUI
This is Event Streaming and not
really Event Sourcing
13. gschmutz
Command Query Responsibility Segregation (CQRS)
Optimize for write and read differently
API is split between
• commands - trigger changes in state
• queries - provide read access to the state
Still using CRUD pattern, but separates
”R” from CRUD
Might involve eventual consistency
between write and read model
Data Storage
Write Model
Read Model
(read-only)
Service
Command
API
Query
API
App
UI
Projection Handler
UI Logic
Kafka as an Event Store – is it Good Enough?
publish
command
query
project
read
insert
update
delete
1
2
3
4
14. gschmutz
Event Sourcing – Persist state-changing events and not
state
Kafka as an Event Store – is it Good Enough?
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
10 20 100 2000
15. gschmutz
Event Sourcing
persists the state of an aggregate as a
sequence of state-changing events
Each event describes a state change
that occurred to the aggregate in the
past
new event is appended to the list of
events
an aggregate’s current state is
reconstructed by replaying the events
=> a.k.a ”rehydration”
Rehydration also needed for queries
Kafka as an Event Store – is it Good Enough?
Event Store
ServiceApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
Service
Subscribe
publish
publish
apply (append)
REST
Data Storage
trigger replycommand
command
1 2
3
4
5
5
16. gschmutz
Event Sourcing - ”Rehydrate” State
Kafka as an Event Store – is it Good Enough?
1. Create an empty Aggregate
object
2. Read all events stored for
that Aggregate from event
store
3. Apply each event to the
Aggregate object in the
correct order
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
Account
<empty>
Account
id: 123
accountType: Savings
balance: 0
Account
id: 123
accountType: Savings
balance: 3000
transactions: [+1000, +2000]
Account
id: 123
accountType: Savings
balance: 2500
transactions:[+1000, +2000, -500]
Account
id: 123
accountType: Savings
balance: 1000
transactions: [+1000]
applyTo
applyTo
applyTo
applyTo
17. gschmutz
Event Sourcing – Write Path CreateAccount
command
Kafka as an Event Store – is it Good Enough?
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
AccountCreated
id: 123
accountType: Savings
10
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
Account Aggregate
<empty>
18. gschmutz
Event Sourcing – Write Path DepositMoney
command
Kafka as an Event Store – is it Good Enough?
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
10 20
Account Aggregate
id: 123
accountType: Savings
balance: 0
19. gschmutz
Event Sourcing – Write Path DepositMoney
command
Kafka as an Event Store – is it Good Enough?
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
10 20 100
Account Aggregate
id: 123
accountType: Savings
balance: 1000
20. gschmutz
Event Sourcing – Write Path WithdrawMoney
command
Kafka as an Event Store – is it Good Enough?
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
10 20 100 2000
Account Aggregate
id: 123
accountType: Savings
balance: 3000
21. gschmutz
Event Sourcing - Potential Benefits
Kafka as an Event Store – is it Good Enough?
1. Subscribe to changes from other
Aggregates
2. Examine a historical record of every
change that has ever been applied on
the model
3. Use the event store data for trend,
forcast and other business analytics
4. Consider “what if” questions by
replaying events to Aggregates which
have experimental enhancements
5. Patch errors by adding ”correction”
events (if it is legally allowed)
6. Perform “undo” and “redo”
operations by replying varying sets
of Events
22. gschmutz
Event Sourcing & CQRS
Event sourcing is commonly combined
with the CQRS pattern
Combines best of Event Sourcing and
CQRS
Project events published by Event
Store into Read Model (Materialized
Views)
Write Model and Read Model might
only support eventual consistency
Kafka as an Event Store – is it Good Enough?
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
1
Event Store
publish
apply (append)
trigger reply
2
3
4
5
5
6
23. gschmutz
What is an Event Store?
Kafka as an Event Store – is it Good Enough?
24. gschmutz
Event Store Capabilities
Kafka as an Event Store – is it Good Enough?
1. Append Events efficiently
2. Read aggregate’s events in order
3. Full Sequential Read (over all
aggregates)
4. Consistent writes
5. Event versioning
6. Subscribable event stream
7. Correction events (O)
8. Ingestion & event time, bi-temporal (O)
9. Adhoc-Query on event store (O)
10. Snapshot Optimization (O)
11. High-Availability and Reliability (O)
26. gschmutz
Event Store Implementations
Kafka as an Event Store – is it Good Enough?
• Event Store (https://eventstore.org/) – by Greg Young
• Axon Framework & Relational DB (https://axoniq.io/) - by Axon IQ
• Axon DB (https://axoniq.io/) - by Axon IQ
• Eventuate (https://eventuate.io/) – by Eventuate.io
• Serialized (https://serialized.io/) – by Serialized.io
• Build your own ….
• Apache Kafka ???
28. gschmutz
Kafka as an Event Store
Kafka as an Event Store – is it Good Enough?
1. One, single-partitioned Kafka topic per Aggregate
2. One, partitioned Kafka topic per Aggregate Type
3. One single, highly partitioned Kafka topic for all Aggregate Types
Should you put several Event Types in the same Kafka topic?:
https://www.confluent.io/blog/put-several-event-types-kafka-topic/
29. gschmutz
1) One, single-partitioned Kafka topic per Aggregate
Instance
Kafka as an Event Store – is it Good Enough?
This will guarantee that the events are
stored in order
Reading state of an aggregate is as
simple as reading a topic from offset 0
Not really feasible as there will be just
too many topics needed
Kafka
Customer Aggregate
Account Aggregate
30. gschmutz
2) One, partitioned Kafka topic per Aggregate Type
Kafka as an Event Store – is it Good Enough?
Required number of partitions is dependent
on number of aggregate instances
Events are produced with aggregate-id as
the key
guarantees that events are stored in order
For reading state of an aggregate, all data
of all aggregate instances have to be
scanned => slow
Possible optimization: only read the
partition where aggregate instance is stored
Kafka
Customer Aggregate
Account Aggregate
31. gschmutz
3) One single, highly partitioned Kafka topic for all
Aggregate Types
Kafka as an Event Store – is it Good Enough?
Required number of partitions is dependent
on number of aggregate types * instances
Events are produced with aggregate-id as
the key
guarantees that events are stored in order
For reading state of an aggregate, all data
of all aggregate types & instances have to
be scanned => really slow
Possible optimization: only read the
partition where aggregate instance is stored
Kafka
Customer Aggregate
Account Aggregate
32. gschmutz
Kafka as an Event Store
Kafka as an Event Store – is it Good Enough?
# Capability Kafka Broker
1 Append events efficiently yes
2 Read aggregate’s events in order not efficiently
3 Full sequential Read yes
4 Consistent Writes no
5 Event Versioning yes (if Avro is used)
6 Subscribeable Event Stream yes
7 Correction events (O) no
8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header
9 Snapshot Optimization (O) no
10 Ad-Hoc Query on Events (O) no
11 High-Availability and Reliability (O) yes
33. gschmutz
Event Store
Kafka is not a Database … a Database is not Kafka
We can use Kafka to run part of our own
Event Store implementation
add a database to get missing capabilities
But be careful with Dual Write!
• Would need distributed transactions
• Otherwise no guarantee for both writes
to happen
Application
{ }
API DatabaseBiz Logic
REST
Event Hub
Kafka as an Event Store – is it Good Enough?
Other App
Consumer
34. gschmutz
Event Store
Kafka is not a Database … a Database is not Kafka
We can use Kafka to run our own Event
Store implementation
adding a database to get missing
capabilities
But be careful with Dual Write!
• Would need distributed transactions
• Otherwise no guarantee for both writes
to happen
Application
{ }
API DatabaseBiz Logic
REST
Event Hub
Kafka as an Event Store – is it Good Enough?
Other App
Consumer
35. gschmutz
Event StoreEvent Store
Two solutions for avoiding «dual write»
Write Event first then consume it to write
it to database
Write through database (CDC, outbox
design pattern)
Application
{ }
API
Database
Biz Logic
REST
Kafka as an Event Store – is it Good Enough?
Event Hub
Other App
Biz Logic
Application
{ }
API
Database
REST
Biz Logic
CDC
Event Hub
CDC
Connector
Other App
Biz Logic
Publish
37. gschmutz
Axon
Kafka as an Event Store – is it Good Enough?
• Spring Boot with Axon Framework for
Application
• MongoDB for Event Store
• Kafka Broker for Event Bus
• Kafka Streams or KSQL for
Projection Handler
• Kafka Connect / Spring Boot to
persist in read model
• NoSQL and/or RDBMS for read
model
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply
38. gschmutz
Event Sourcing with Axon
Kafka as an Event Store – is it Good Enough?
Account
Events
Account
Command
Account Aggregate
Account Command
Response
Account App
Event Store
Account
Customer
Projection
Command Handler
Event Handler
Account Query
Projection Handler
Query Handler
Account Query
Account Query
Response
Customer
Event
https://github.com/gschmutz/various-demos/tree/master/event-sourcing
39. gschmutz
Event Sourcing with Axon - Aggregate
Kafka as an Event Store – is it Good Enough?
@Aggregate
public class AccountAggregate{
@AggregateIdentifier
private String id;
private BigDecimal balance;
private String forCustomerId;
private String accountType;
@CommandHandler
...
@EventSourcingHandler
...
40. gschmutz
Event Sourcing with Axon - Command Handler
Kafka as an Event Store – is it Good Enough?
@CommandHandler
public AccountAggregate(AccountCreateCommand command) {
Assert.hasLength(command.getForCustomerId(),
"CustomerId must have a value");
Assert.hasLength(command.getAccountType(),
"AccountType must have a value");
...
apply(new AccountCreatedEvent(command.getId(),
command.getForCustomerId(),
command.getAccountType(),
new BigDecimal("0")));
}
41. gschmutz
Event Sourcing with Axon – Command Handler
Kafka as an Event Store – is it Good Enough?
@CommandHandler
public void on(WithdrawMoneyCommand command) {
Assert.isTrue(command.getAmount() > 0,
"Amount should be a positive number");
if(command.getAmount().compareTo(this.balance) > 0 ) {
throw new InsufficientBalanceException(
"Insufficient balance. Trying to withdraw:" +
command.getAmount() +
", but current balance is: " + this.balance);
}
apply(new MoneyWithdrawnEvent(command.getId(),
command.getAmount()));
}
42. gschmutz
Event Sourcing with Axon – Event Handler
Kafka as an Event Store – is it Good Enough?
@EventSourcingHandler
public void handle(AccountCreatedEvent event) {
id = event.getId();
forCustomerId = event.getForCustomerId();
accountType = event.getAccountType();
balance = event.getBalance();
}
@EventSourcingHandler
public void handle(MoneyWithdrawnEvent event) {
balance = balance.subtract(event.getAmount());
}
43. gschmutz
Event Sourcing with Axon – Projection Handler
Kafka as an Event Store – is it Good Enough?
public class AccountQueryController {
@Autowired
private AccountRepository accRepo;
@EventHandler
public void on(AccountCreatedEvent event,@Timestamp Instant instant) {
Account account = new Account(event.getId(),event.getBalance(),
event.getAccHolder(),event.getAccHolderName(),
instant.toString());
accRepo.insert(account);
}
@EventHandler
public void on(MoneyDepositedEvent event,@Timestamp Instant instant) {
Account account = accRepo.findByAccountNo(event.getId());
account.setBalance(account.getBalance().add(event.getAmount()));
account.setLastUpdated(instant.toString());
accRepo.save(account);
}
44. gschmutz
Axon Fwk with Axon DB
Kafka as an Event Store – is it Good Enough?
• Spring Boot with Axon Framework
for Application
• Axon DB for Event Store and
Event Bus
• Spring Boot for Projection Handler
• Spring Boot to persist in read
model
• NoSQL and/or RDBMS for read
model
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply
45. gschmutz
Axon as an Event Store
Kafka as an Event Store – is it Good Enough?
# Capability Axon Framework Axon Framework & Axon DB
1 Append events efficiently yes yes
2 Read aggregate’s events in order yes yes
3 Full sequential Read yes yes
4 Consistent Writes yes yes
5 Event Versioning yes yes
6 Subscribeable Event Stream yes yes
7 Correction events (O) no no
8 Event time & ingestion time, aka. Bi-temporal (O) no no
9 Snapshot Optimization (O) yes yes
10 Ad-Hoc Query on Events (O) yes yes
11 High-Availability and Reliability (O) possible yes
47. gschmutz
Kafka & Kafka Streams
Kafka as an Event Store – is it Good Enough?
Kafka Streams with State for Event
Store
Kafka Broker for Event Bus
Kafka Streams or KSQL for Projection
Handler
No reply of events, current snapshot is
held in state store
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply
49. gschmutz
Kafka & Kafka Streams as an Event Store
Kafka as an Event Store – is it Good Enough?
# Capability Kafka & Kafka Streams
1 Append events efficiently yes
2 Read aggregate’s events in order no (snapshot state only holds current snapshot)
3 Full sequential Read no
4 Consistent Writes yes (only one event per aggregate in flight)
5 Event Versioning yes (if Avro used)
6 Subscribeable Event Stream yes
7 Correction events (O) no
8 Event time & ingestion time, aka. Bi-temporal (O) no
9 Snapshot Optimization (O) yes (snapshot state only)
10 Ad-Hoc Query on events (O) limited (KSQL, Presto on Kafka, Drill on Kafka, …)
11 High-Availability and Reliability (O) yes
51. gschmutz
Summary
• Event Sourcing and CQRS might be more natural to business people than IT => we
are used to work with “CRUD based persistence”
• Event Sourcing provides history and logging for free
• Kafka Broker alone is really “just” Event Streaming, not Event Sourcing
• Axon Framework supports the implementation of Event Sourcing applications with
Pluggable Event Store and Event Bus implementations
• Axon DB implements an Event Store and an Event Bus
• Kafka and Kafka Streams with State Store supports event sourcing in a ”streaming
fashion”
Kafka as an Event Store – is it Good Enough?
52. gschmutz
Technology on its own won't help you.
You need to know how to use it properly.
Kafka as an Event Store – is it Good Enough?