These are the slides I used for my presentation at the Flink Forward conference in Berlin on 2019-10-09. https://europe-2019.flink-forward.org/conference-program#when-ordering-matters Abstract: In the last decades many systems have been used that were described as "queues" (AQ, ActiveMQ, RabbitMQ, etc.), yet from a computer science perspective these are not queues at all. Many of us have learned to work quite effectively with these messaging systems and we all understand that we cannot expect to receive the messages in any particular order and that we get all messages exactly once (which we can expect with a queue). With the arrival of Kafka and Flink a new class of applications became possible. In this talk I will go into several real applications from the bol.com context that all revolve around low latency behavioral analytics. I will talk about the entire end-to-end pipeline from the webbrowser and application server to application and discuss many of the things to think about when creating your analysis application. I will also touch upon using state machines as a way of doing this type of behavioral analysis using very simple software and show example algorithms from our context.

3. Niels Basjes | Corporate Inventor
and you have
100K events/sec
When ordering
matters

4. • Shopping
• Conversation
• Measuring
• Cause and effect
• End-to-end pipeline
• State machines
Agenda

5. ir. drs. Niels Basjes
TU-Delft Computer Science (MSc)
Nyenrode Business School (MSc)
Software Developer, Researcher,
Consultant, Infra/Web/IT Architect
Corporate Inventor
Contributor for Apache Hadoop,
Pig, HBase, Flink, Beam, Parquet, …
Apache Avro Committer & PMC
Speaker & Teacher at
Universities and Conferences
@nielsbasjes
https://github.com/nielsbasjes

7. • Listen
• Understand
• Process
• Respond
Communicate
Communication
requires
“Remembering”
~
Observe
UnderstandProcess
Respond

8. I want something to play with

9. I want something to play with

10. I want something to play with

11. I want something to play with

14. Going online …

16. 22.085.386

19. ~ 22 million
products for sale
> 10 million
active customers
> 8000 million
pageviews/year
Season 2017
~16.000.000 presents

20. To have an online conversation
we need
• Accurate observations
• Don’t miss any response
• Scalable low latency ‘loop’
• Quick enough
• 100K events/second
• State per session
• Remember per visitor
• Ordering guarantees
• Give the right response
~
Observe
UnderstandProcess
Respond

21. Observe
UnderstandProcess
Respond
Online conversation M2
Measuring 2.0
U2
Understanding 2.0
Creating
content
Streaming
DataScience

22. Why M2?
We already have
Omniture/GA…

23. Measuring interaction
• What are we showing (and why)?
• How are our visitors responding?
• Pages
• Products/Offers
• Add to cart
• Purchase
• Advertising
• Inspiration

24. Use cases
• Personalization
• Site optimization
• Fraud prevention
• Dashboards
• Attribution modelling
• Data Science
• …

25. ~ 3K-4K pages/sec
~ 30 events/page
~ 100K events/sec
Q3 2019: Per day
• 1 500 000 000
measurements
• 3TiB data
‘All’ details
Why
Where
What
x

26. JavaScript data is …
• Broken … fundamentally broken
• Measuring a side effect
• Missing & Duplicate orders
• Blockable
• Intelligent Tracking Protection
• Ad blockers
• Spiders
• Hackers
• Fragile
• HTTP 414: URI Too long
• Locked in
• “Boxed” SaaS solutions.

27. Measurements are…
too old.
• Available once every 24 hours.
• So personalization is a ‘day behind’.
Useless inspiration:
I was interested in this YESTERDAY

28. Data relevance decay
Age of the data
Valueofthedata
Days WeeksMinutes

29. Time is not always important
Crowd pattern analysis of website usage
Building a better website for future visitors: (Micro)Batch
Individual pattern analysis of website usage
Supporting and advising the current visitor: Realtime

30. Batch processing

31. Stream processing

32. Measure
accurately

33. Where to measure?
• Measure where “it” happens.
• “In” the responsible “frontend” service!
• Webshop
• App API
• Basket Service
• Order Service
• …
• Usually NOT in the browser

34. Measure pages
• Serverside
• What is in the page
• Clientside (Javascript)
• What part was viewed
• Screen resolution

35. Measure orders
• Listen to Order events!
• with website/app sessionid.
• The “Order confirmation” page.
• Just a viewing of the order.

36. Record everything at the start
• Measure what “really” happens.
• Keep all relevant details
• Product: ProductId, Product type, …
• Offer: OfferId, ProductId, Price, Condition, SellerId, …
• Later joining on productid/offerid is “impossible”.
• Webshop caching
• Data volume / Extra latency

37. Ordering

38. • All single event entities
• No correlations
• No logical ‘cause and effect’
• No conversation
This is easy

39. Our usecases
• Banner optimization
• Look / Search  Next page better banner
• A/B testing
• Show feature  Use  Buy product
• Search Suggestions
• Search  Find  Choose  Buy product
• Attribution modeling
• Show ad  Click  Buy product
• …

40. Behavioral analytics
• Cause and effect
• Action: We show something
• Reaction: To click or not to click

41. Event ordering matters
• Click banner , Buy product
• Banner WAS (possibly) part of reason to buy.
• Buy product, Click banner
• Banner WAS NOT part of reason to buy.
“WAS” or “WAS NOT” is based on
The ordering of the events

42. Finite State machine
• Simple, low latency, pattern detection
• Ordered events

43. Pushdown automaton
• State machine
• with a memory stack
• Simple, low latency,
pattern detection
• Ordered events

44. Event ordering matters
• A fast temperature change is dangerous
• should alert IMMEDIATELY
• Delta stays in bounds
• Expect “Ordered”
while (curr = newEvent()) {
if (tooBig(curr, prev))
sendAlert();
prev = curr;
}
-40
-20
0
20
40
60
80
100
T1 T2 T3 T4 T5 T6 T7 T8 T9
Temperature Delta
This is a simple
pushdown automaton

45. Event ordering matters
• A fast temperature change is dangerous
• should alert IMMEDIATELY
Ordering problems
• Many false positives !
• Many false negatives !
-40
-20
0
20
40
60
80
100
T1 T5 T7 T4 T2 T8 T3 T6 T9
Temperature Delta
!
! !
!
!

46. Repairing event ordering
• Is hard
• Needless complexity
• Takes time
• Buffer for the maximum ‘out-of-orderness’ period.
• Several minutes
• We want really low latency
123 4 56 78 9
Sliding time based
sort buffer

47. Exactly once please
• At least once
• Need data deduplication 
• Is hard
• Large memory buffer
• Idempotent output
• Takes time

48. So we try to
never damage
the ordering

49. Achieving end-to-end ordering...
1. The measuring point
2. Measurement transport
3. Measurement processing

50. The measuring point
• Single entity
•  single measuring instance
• Multiple instances
• Multiple output buffers
• Race conditions
• Ordering problems

51. The measuring point
In IOT:
• One temperature sensor
• one recording device
At bol.com
• One visitor
• Single webshop instance
• Session routing is a MUST have!!
• Not perfect!
• Impact negligible
• “View” measurements
• Orders

52. Transport

53. Message transport
• We need ordering per session: FIFO
• “Queue” or “Partitioned Queue”
• Session pinned to a specific partition
https://en.wikipedia.org/wiki/Queue_(abstract_data_type)
Partitioned Queue

54. Many “Queue” are not a Queue !
https://en.wikipedia.org/wiki/Java_Message_Service
https://stackoverflow.com/questions/16300353/activemq-lifo-ordering

55. Many “Queue” are not a Queue !
https://cloud.google.com/pubsub/docs/ordering

56. High volume partitioned queues
• Apache Kafka
• https://kafka.apache.org/
• Production ready
• Apache Pulsar
• https://pulsar.apache.org/
• Connector for Flink very new.
• Pravega
• http://pravega.io/
• Being beta tested
• MapR Event Store
• https://mapr.com/products/mapr-eventstore/
• Production ready
• Amazon Kinesis
• Sorry, wrong cloud
• Microsoft Event Hubs
• Sorry, wrong cloud Azure Event Hubs

57. Use
Google PubSub
If ordering
does not matter
at-least-once
is Ok.
High IO
‘Distributed Set’

58. If ordering
matters
and/or
exactly-once
is needed
High IO
‘Partitioned Queue’
Use
Apache Kafka

59. Processing

60. Measurement processing
Requirements
• Low latency
• Exactly once
• Ordering guarantees
• A pushdown automaton per
session
• Keyed Stateful processing
• Where the ‘key’ is the ‘session id’
Observe
UnderstandProcess
Respond
~

61. Choosing a Processing toolkit
Apache Beam
• Low latency … except
• Exactly once by deduplication
• NO ordering guarantees
• NO natural keyed stateful processing
• “Dynamic” scaling
• Abstract Java API
• Runs on
• DataFlow
• Flink

62. Choosing a Processing toolkit
Apache Flink
• Low latency
• Exactly once
• Ordering guarantees (Chandy–Lamport)
• Keyed Stateful processing
• “Fixed” scaling
• Easy Java API
• Runs on
• Hadoop
• Kubernetes

63. Changes
happen

64. Applications change!
• New business
• New insights
• New wishes
• New scope
• New …
The records will
•  get new fields
•  have obsolete fields

65. Data producerData producerData producer
Streaming applications
Data producer Streaming Interface
Data consumers
Data consumersData consumers
Data consumers
The real payload is
“byte array”
Multiple Applications
Rolling upgrades
Canary releases
Multiple Applications
Rolling upgrades
Canary releases

66. Kafka persists messages
• A message is retained until the TTL expired.
• So a topic will contain several message versions!
• With different fields
V1 V2
V3 V4

67. So we need something to
• Serialize records into bytes
• Data types
• Nested records
• Bidirectional Schema evolution

68. Apache Avro
is what we use !

69. Apache Avro (IDL Schema)
Code generation

70. Avro Message format
• Single record into bytes encoding
• Designed for evolving streaming applications
• Need schema database:
• Key = 64bit long
• Value = String
The json version of the schema

71. Producing from Flink into Kafka

72. Consume from Kafka into Flink

73. Consume from Kafka into Flink

74. Using
the data

75. Search Suggestions

76. The search funnel
Search
Find
(Product page)
Choose
(Add to cart)
Buy

77. Deeper = more relevant
(Search)
Find
Choose
Buy

78. High level
M2
Stateful
analysis
Relevant events
Suggestions
Event to
scored
suggestions
Suggestion Delta
Is this a valuable event?
Pushdown Automaton
1. Input:
“Harry Potter” Product page
2. Score:
“Harry Potter” 25
3. Suggest:
“H”  “Harry Potter”: 25
“Ha”  “Harry Potter”: 25
“Har”  “Harry Potter”: 25
Ordering is important Ordering is NOT important

79. Search Suggestions
Default
no search
Searched
Found
Chosen
Initial
Search
for “X”
To PDP
Add to Cart
Within PDP
Anythingelse
The PDP is really a set of
pages about the product,
offers, reviews, …
= Send out event that something useful happened
Searched
for “X”
PDP for “X”
ATC for “X”

80. Recommended reading

81. Join us
100

82. https://careers.bol.com

83. @nielsbasjes
https://www.linkedin.com/in/nielsbasjes/
https://niels.basj.es/
https://github.com/nielsbasjes
ir. drs. Niels
Basjes
Thanks for
listening