Implementing event sourcing using Scala in a very lightweight manner. Given at the DUSE (Dutch Scala Enthusiasts) meeting at 2011-10-5.

1. Exploring Lightweight
Event Sourcing
Erik Rozendaal <erozendaal@zilverline.com>
@erikrozendaal

2. Goals
• The problem of data persistence
• Understand event sourcing
• Show why Scala is well-suited as
implementation language
Exploring lightweight event sourcing 2

3. Problem
Exploring lightweight event sourcing 3

4. N-tier
Exploring lightweight event sourcing 4

5. N-tier
• Presentation, Business, and Data Layers
Exploring lightweight event sourcing 4

6. N-tier
• Presentation, Business, and Data Layers
• Shared data model (“domain”)
Exploring lightweight event sourcing 4

7. N-tier
• Presentation, Business, and Data Layers
• Shared data model (“domain”)
• Heavy use of a single, global, mutable
variable (“the database”)
Exploring lightweight event sourcing 4

8. N-tier
• Presentation, Business, and Data Layers
• Shared data model (“domain”)
• Heavy use of a single, global, mutable
variable (“the database”)
• Is it any surprise that we’re struggling to
build modular, maintainable applications?
Exploring lightweight event sourcing 4

9. Domain Driven Design
• Domain-Driven Design An approach to software development that suggests
that (1) For most software projects, the primary focus should be on the
domain and domain logic; and (2) Complex domain designs should be based
on a model.
• Domain Expert A member of a software project whose field is the domain
of the application, rather than software development. Not just any user of
the software, the domain expert has deep knowledge of the subject.
• Ubiquitous Language A language structured around the domain model and
used by all team members to connect all the activities of the team with the
software.
Exploring lightweight event sourcing 5

10. Data must be durable
Exploring lightweight event sourcing 6

11. But current
applications are lossy
Exploring lightweight event sourcing 7

12. Lossy?
Exploring lightweight event sourcing 8

13. Lossy?
• UPDATE invoice WHERE id = 1234
SET total_amount = 230
Exploring lightweight event sourcing 8

14. Lossy?
• UPDATE invoice WHERE id = 1234
SET total_amount = 230
• What happened to the previous order
amount?
Exploring lightweight event sourcing 8

15. Lossy?
• UPDATE invoice WHERE id = 1234
SET total_amount = 230
• What happened to the previous order
amount?
• Why was the order amount changed?
Exploring lightweight event sourcing 8

16. Lossy?
• UPDATE invoice WHERE id = 1234
SET total_amount = 230
• What happened to the previous order
amount?
• Why was the order amount changed?
• Application behavior is not captured!
Exploring lightweight event sourcing 8

17. Behavior?
Exploring lightweight event sourcing 9

18. Behavior?
Status:
"Draft"
Total:
Invoice
0.00
Exploring lightweight event sourcing 9

19. Behavior?
Status: Status: Recipient:
"Draft" "Draft" "Erik"
Total: Total:
Invoice Invoice
0.00 0.00
Exploring lightweight event sourcing 9

20. Behavior?
Status: Status: Recipient: Status: Recipient:
"Draft" "Draft" "Erik" "Draft" "Erik"
Total: Total: Total:
Invoice Invoice Invoice
0.00 0.00 9.95
Item:
Invoice Item
"Food"
Amount:
9.95
Exploring lightweight event sourcing 9

21. It’s just data mutation
Status: Status: Recipient: Status: Recipient:
"Draft" "Draft" "Erik" "Draft" "Erik"
Total: Total: Total:
Invoice Invoice Invoice
0.00 0.00 9.95
Item:
Invoice Item
"Food"
Amount:
9.95
Exploring lightweight event sourcing 10

22. It’s just data mutation
Status: Recipient:
"Draft" "Erik"
Total:
Invoice
9.95
Item:
Invoice Item
"Food"
Amount:
9.95
Exploring lightweight event sourcing 10

23. ORM implementation
“Inexperienced programmers love magic because
it saves their time. Experienced programmers hate
magic because it wastes their time.”
– @natpryce
Exploring lightweight event sourcing 11

24. Event Sourcing
• All state changes are explicitly captured
using domain events
• Capture the intent of the user and the
related data
• Events represent the outcome of application
behavior
Exploring lightweight event sourcing 12

25. Domain Behavior
Exploring lightweight event sourcing 13

26. Domain Behavior
generate
Draft Invoice
Created
Exploring lightweight event sourcing 13

27. Domain Behavior
generate
Draft Invoice
Created
apply
Status:
"Draft"
Total:
Invoice
0.00
Exploring lightweight event sourcing 13

28. Domain Behavior
generate
Draft Invoice Invoice Recipient
Created Changed
Recipient: "Erik"
apply generate
Status:
"Draft"
Total:
Invoice
0.00
Exploring lightweight event sourcing 13

29. Domain Behavior
generate
Draft Invoice Invoice Recipient
Created Changed
Recipient: "Erik"
apply generate apply
Status: Status: Recipient:
"Draft" "Draft" "Erik"
Total: Total:
Invoice Invoice
0.00 0.00
Exploring lightweight event sourcing 13

30. Domain Behavior
generate
Draft Invoice Invoice Recipient Invoice Item
Created Changed Added
Recipient: "Erik" Item: "Food"
Item amount: 9.95
Total amount: 9.95
apply generate apply generate
Status: Status: Recipient:
"Draft" "Draft" "Erik"
Total: Total:
Invoice Invoice
0.00 0.00
Exploring lightweight event sourcing 13

31. Domain Behavior
generate
Draft Invoice Invoice Recipient Invoice Item
Created Changed Added
Recipient: "Erik" Item: "Food"
Item amount: 9.95
Total amount: 9.95
apply generate apply generate apply
Status: Status: Recipient: Status: Recipient:
"Draft" "Draft" "Erik" "Draft" "Erik"
Total: Total: Total:
Invoice Invoice Invoice
0.00 0.00 9.95
Item:
Invoice Item
"Food"
Amount:
9.95
Exploring lightweight event sourcing 13

32. Only the events need
to be stored on disk
Event
Store Draft Invoice Invoice Recipient Invoice Item
.......
Created Changed Added
Recipient: "Erik" Item: "Food"
Item amount: 9.95
Total amount: 9.95
Exploring lightweight event sourcing 14

33. Reloading from history
Exploring lightweight event sourcing 15

34. Reloading from history
Draft Invoice
Created
apply
Status:
"Draft"
Total:
Invoice
0.00
Exploring lightweight event sourcing 15

35. Reloading from history
Draft Invoice Invoice Recipient
Created Changed
Recipient: "Erik"
apply apply
Status: Status: Recipient:
"Draft" "Draft" "Erik"
Total: Total:
Invoice Invoice
0.00 0.00
Exploring lightweight event sourcing 15

36. Reloading from history
Draft Invoice Invoice Recipient Invoice Item
Created Changed Added
Recipient: "Erik" Item: "Food"
Item amount: 9.95
Total amount: 9.95
apply apply apply
Status: Status: Recipient: Status: Recipient:
"Draft" "Draft" "Erik" "Draft" "Erik"
Total: Total: Total:
Invoice Invoice Invoice
0.00 0.00 9.95
Item:
Invoice Item
"Food"
Amount:
9.95
Exploring lightweight event sourcing 15

37. Reloading from history
Draft Invoice Invoice Recipient Invoice Item
Created Changed Added
Recipient: "Erik" Item: "Food"
Item amount: 9.95
Total amount: 9.95
apply apply apply
Status: Status: Recipient: Status: Recipient:
"Draft" "Draft" "Erik" "Draft" "Erik"
Total: Total: Total:
Invoice Invoice Invoice
0.00 0.00 9.95
Item:
Invoice Item
"Food"
Amount:
9.95
Exploring lightweight event sourcing 15

38. YAGNI?
Exploring lightweight event sourcing 16

39. YAGNI?
• On the checkout page we’d like to promote
products that customers previously
removed from their shopping cart.
Exploring lightweight event sourcing 16

40. YAGNI?
• On the checkout page we’d like to promote
products that customers previously
removed from their shopping cart.
• Can you tell us how many people removed
a product but bought it later anyway?
Exploring lightweight event sourcing 16

41. YAGNI?
• On the checkout page we’d like to promote
products that customers previously
removed from their shopping cart.
• Can you tell us how many people removed
a product but bought it later anyway?
• ... over the past 5 years?
... and how much time passed in-between?
Exploring lightweight event sourcing 16

42. YAGNI?
• We have reports of a strange bug, but have
not been able to isolate it. Could you look
at the production data to find out what’s
going on?
Exploring lightweight event sourcing 17

43. Is it worth it?
Exploring lightweight event sourcing 18

44. Is it worth it?
• Make the event sourcing implementation as
simple as possible
Exploring lightweight event sourcing 18

45. Is it worth it?
• Make the event sourcing implementation as
simple as possible
• ... while avoiding the complexities of
databases, ORMs, etc.
Exploring lightweight event sourcing 18

46. Is it worth it?
• Make the event sourcing implementation as
simple as possible
• ... while avoiding the complexities of
databases, ORMs, etc.
• ... unless you want or need them :)
Exploring lightweight event sourcing 18

47. Ingredients
Exploring lightweight event sourcing 19

48. Ingredients
• Scala case classes
Exploring lightweight event sourcing 19

49. Ingredients
• Scala case classes
• Immutable data
structures
Exploring lightweight event sourcing 19

50. Ingredients
• Scala case classes
• Immutable data
structures
• Single Responsibility
Principle (SRP)
Exploring lightweight event sourcing 19

51. Ingredients
• Scala case classes
• Immutable data
structures
• Single Responsibility
Principle (SRP)
• Events for durability
Exploring lightweight event sourcing 19

52. Ingredients
• Scala case classes
• Immutable data
structures
• Single Responsibility
Principle (SRP)
• Events for durability
• Memory Image
Exploring lightweight event sourcing 19

53. “CRUD”
Exploring lightweight event sourcing 20

54. “CRUD”
Create/Update/Delete
Exploring lightweight event sourcing 20

55. “CRUD”
Create/Update/Delete
HTTP POST
Validate input and
generate event
Exploring lightweight event sourcing 20

56. “CRUD”
Create/Update/Delete
HTTP POST Save Event
Validate input and Event
generate event Store
Exploring lightweight event sourcing 20

57. “CRUD”
Create/Update/Delete
HTTP POST Save Event
Validate input and Event
generate event Store
Apply Event
Status: Recipient:
"Draft" "Erik"
Invoice Total: 9.95
Item:
Invoice Item
"Food"
Amount:
9.95
Exploring lightweight event sourcing 20

58. “CRUD”
Create/Update/Delete
HTTP POST Save Event
Validate input and Event
generate event Store
Apply Event
Status: Recipient:
"Draft" "Erik"
Invoice Total: 9.95
Query
Render View
Item:
Invoice Item
"Food"
Amount:
9.95
Exploring lightweight event sourcing 20

59. “CRUD”
Create/Update/Delete
HTTP POST Save Event
Validate input and Event
generate event Store
Apply Event
Read Status:
"Draft"
Recipient:
"Erik"
Invoice Total: 9.95
HTTP GET Query
Render View
Item:
Invoice Item
"Food"
Amount:
9.95
Exploring lightweight event sourcing 20

60. HTTP POST Save Event
Validate input and Event
Controller
generate event Store
Apply Event
Status: Recipient:
"Draft" "Erik"
Invoice Total: 9.95
HTTP GET Query
Render View
Item:
Invoice Item
"Food"
Amount:
9.95
post("/todo") {
val item = field("item", required) map (_.collapseWhitespace)
item.fold(
error => new ToDoView(toDoItems, Some(error)),
value => {
record(Created(ToDoItem(UUID.randomUUID(), value)))
redirect(url("/todo"))
})
}
Exploring lightweight event sourcing 21

61. HTTP POST Save Event
Validate input and Event
Memory Image
generate event Store
Apply Event
Status: Recipient:
"Draft" "Erik"
Invoice Total: 9.95
HTTP GET Query
Render View
Item:
Invoice Item
"Food"
Amount:
9.95
case class ToDoItems (
all : Map[String, ToDoItem] = Map.empty,
recentlyAdded: Vector[String] = Vector.empty) {
def apply(event: Any) = event match {
case Created(item: ToDoItem) =>
copy(all.updated(item.id, item), recentlyAdded :+ item.id)
case Updated(item: ToDoItem) =>
copy(all.updated(item.id, item))
case Deleted(id: String) =>
copy(all - id, recentlyAdded.filter(_ != id))
}
def mostRecent(count: Int) = recentlyAdded.takeRight(count).map(all).reverse
}
Exploring lightweight event sourcing 22

62. HTTP POST Save Event
Validate input and Event
View
generate event Store
Apply Event
Status: Recipient:
"Draft" "Erik"
Invoice Total: 9.95
HTTP GET Query
Render View
Item:
Invoice Item
"Food"
Amount:
9.95
<table class="zebra-striped">
<thead><tr><th>To-Do</th></tr></thead>
<tbody>{
for (item <- toDoItems.mostRecent(20)) yield {
<tr><td>{ item.text }</td></tr>
}
}</tbody>
</table>
Exploring lightweight event sourcing 23

63. Domain Model
Domain Model
2: Save Event
1: Send Command
Validate input and
generate command Event Store
3: Apply Event
3: Apply Event
Status: Recipient:
"Draft" "Erik"
Invoice Total: 9.95
Item:
Invoice Item
"Food"
Amount:
9.95
Presentation Model
Exploring lightweight event sourcing 24

64. Domain Code
sealed trait InvoiceEvent
case class InvoiceCreated() extends InvoiceEvent
case class InvoiceRecipientChanged(recipient: String) extends InvoiceEvent
case class InvoiceItemAdded(item: InvoiceItem, totalAmount: BigDecimal) extends InvoiceEvent
case class InvoiceSent(sentOn: LocalDate, paymentDueOn: LocalDate) extends InvoiceEvent
Exploring lightweight event sourcing 25

65. Domain Code
sealed trait InvoiceEvent
case class InvoiceCreated() extends InvoiceEvent
case class InvoiceRecipientChanged(recipient: String) extends InvoiceEvent
case class InvoiceItemAdded(item: InvoiceItem, totalAmount: BigDecimal) extends InvoiceEvent
case class InvoiceSent(sentOn: LocalDate, paymentDueOn: LocalDate) extends InvoiceEvent
sealed trait Invoice extends AggregateRoot {
protected[this] type Event = InvoiceEvent
}
Exploring lightweight event sourcing 25

66. Domain Code
case class DraftInvoice(
recipient: Option[String] = None,
nextItemId: Int = 1,
items: Map[Int, InvoiceItem] = Map.empty) extends Invoice {
def changeRecipient(recipient: String): Behavior[DraftInvoice] = ...
def addItem(description: String, amount: BigDecimal): Behavior[DraftInvoice] = ...
def send(sentOn: LocalDate): Behavior[Either[String, SentInvoice]] = ...
...
private def recipientChanged = when[InvoiceRecipientChanged] {
event => copy(recipient = Some(event.recipient))
}
private def itemAdded = when[InvoiceItemAdded] {
event => copy(nextItemId = event.item.id + 1, items = items + (event.item.id -> event.item))
}
private def sent = when[InvoiceSent] {
event => SentInvoice(event.sentOn, event.paymentDueOn)
}
}
Exploring lightweight event sourcing 26

67. Domain Code
case class DraftInvoice(
recipient: Option[String] = None,
nextItemId: Int = 1, The “Brains”
items: Map[Int, InvoiceItem] = Map.empty) extends Invoice {
def changeRecipient(recipient: String): Behavior[DraftInvoice] = ...
def addItem(description: String, amount: BigDecimal): Behavior[DraftInvoice] = ...
def send(sentOn: LocalDate): Behavior[Either[String, SentInvoice]] = ...
...
private def recipientChanged = when[InvoiceRecipientChanged] {
event => copy(recipient = Some(event.recipient))
}
private def itemAdded = when[InvoiceItemAdded] {
event => copy(nextItemId = event.item.id + 1, items = items + (event.item.id -> event.item))
}
private def sent = when[InvoiceSent] {
event => SentInvoice(event.sentOn, event.paymentDueOn)
}
}
Exploring lightweight event sourcing 26

68. Domain Code
case class DraftInvoice(
recipient: Option[String] = None,
nextItemId: Int = 1, The “Brains”
items: Map[Int, InvoiceItem] = Map.empty) extends Invoice {
def changeRecipient(recipient: String): Behavior[DraftInvoice] = ...
def addItem(description: String, amount: BigDecimal): Behavior[DraftInvoice] = ...
def send(sentOn: LocalDate): Behavior[Either[String, SentInvoice]] = ...
... The “Muscle”
private def recipientChanged = when[InvoiceRecipientChanged] {
event => copy(recipient = Some(event.recipient))
}
private def itemAdded = when[InvoiceItemAdded] {
event => copy(nextItemId = event.item.id + 1, items = items + (event.item.id -> event.item))
}
private def sent = when[InvoiceSent] {
event => SentInvoice(event.sentOn, event.paymentDueOn)
}
}
Exploring lightweight event sourcing 26

69. Domain Code
case class DraftInvoice(
recipient: Option[String] = None,
nextItemId: Int = 1,
items: Map[Int, InvoiceItem] = Map.empty) extends Invoice {
def changeRecipient(recipient: String): Behavior[DraftInvoice] =
recipientChanged(InvoiceRecipientChanged(recipient))
def addItem(description: String, amount: BigDecimal): Behavior[DraftInvoice] =
itemAdded(InvoiceItemAdded(InvoiceItem(nextItemId, description, amount), totalAmount + amount))
def send(sentOn: LocalDate): Behavior[Either[String, SentInvoice]] =
if (readyToSend_?)
for (updated <- sent(InvoiceSent(sentOn, paymentDueOn = sentOn.plusDays(14))))
yield Right(updated)
else
pure(Left("invoice not ready to send"))
private def totalAmount = items.values.map(_.amount).sum
private def readyToSend_? = recipient.isDefined && items.nonEmpty
...
}
Exploring lightweight event sourcing 27

70. Domain Code
case class DraftInvoice(
recipient: Option[String] = None,
nextItemId: Int = 1,
items: Map[Int, InvoiceItem] = Map.empty) extends Invoice {
...
protected[this] def applyEvent = recipientChanged orElse itemAdded orElse sent
...
}
Exploring lightweight event sourcing 28

71. Event Store
• Stores sequence of events
• Dispatches events when successfully
committed
• Replays events on startup
• It’s your application’s transaction log
Exploring lightweight event sourcing 29

72. Conclusion
• Event Sourcing captures full historical
information
• Fully encapsulated domain code decoupled
from persistence and presentation layers
• Easier to fully understand compared to
traditional ORM based approach
• Need to learn “Event-Driven” thinking
Exploring lightweight event sourcing 30

73. Example Application
• https://github.com/erikrozendaal/scala-
event-sourcing-example
• (soon) https://github.com/zilverline/lessdb-
example
Exploring lightweight event sourcing 31

74. Implementation
Limitations
• Single-threaded event store using disruptor
pattern
• ~ 5.000 commits per second
• Number of events to replay on startup
• ~ 70.000 JSON serialized events per second
• Total number of objects to store in main memory
• JVM can run with large heaps (tens of gigabytes)
Exploring lightweight event sourcing 32

75. But ready to scale
• Advanced event store implementations support
SQL, MongoDB, Amazon SimpleDB, etc.
• Use persisted view model for high volume objects
(fixes startup time and memory usage)
• Easy partitioning of aggregates (consistency
boundary with unique identifier)
• Load aggregates on-demand, use snapshotting
Exploring lightweight event sourcing 33

76. Thanks!

77. References
• Example code https://github.com/erikrozendaal/scala-event-sourcing-example
• CQRS http://cqrsinfo.com/
• Greg Young, “Unshackle Your Domain”
http://www.infoq.com/presentations/greg-young-unshackle-qcon08
• Pat Helland, “Life Beyond Distributed Transactions: An Apostate's Opinion”
http://www.ics.uci.edu/~cs223/papers/cidr07p15.pdf
• Erik Rozendaal, “Towards an immutable domain model” http://
blog.zilverline.com/2011/02/10/towards-an-immutable-domain-model-
monads-part-5/
• Martin Fowler, “Memory Image”, http://martinfowler.com/bliki/
MemoryImage.html
• Martin Fowler, “The LMAX Architecture”, http://martinfowler.com/articles/
lmax.html
Exploring lightweight event sourcing 35