1.
Real World Akka RecipesJamie AllenBjörn AntonssonPatrik Nordwall

2.
The Fallacy ofGuaranteed DeliveryJamie Allen@jamie_allen

3.
Guaranteed Delivery• From Enterprise Integration Patterns• Messaging system uses built-in storeto persist• ACK everywhere– Producer to sender– Sender to receiver– Receiver to consumer@jamie_allen3

4.
Akka Guarantees• Not much, intentionally• At most once, with no reordering• Pick your poison:– At most once– At least once– Exactly once• You have to add it on top@jamie_allen4

5.
How Do I Do It?• Handle “at least” semantics on receiverto deal with duplicates– Idempotent behavior in receiver– Check message ID• Handle “at most” semantics on thesender via retries– ACK every time message is handled– Cancel repeated send@jamie_allen5

6.
Durable Mailboxes?@jamie_allen6Uh, no.

7.
@jamie_allen• Doesn’t work with future-based messagesending (ask, ?)• No guarantee is there that the messageeven got to the mailbox in distributedsystems• Asking for guarantees in an uncertainworld7Durable Mailboxes

8.
Event Sourcing?• Wonderful pattern for compiling a list oftime-series events• Separation of concerns from actormailboxes• Still lots of things that can go wrong– Disk writing– Replication consistency– Network partitions– Application latency@jamie_allen8

9.
External Durable Message Queue• You still have to ACK• No certainty the message you neededeven got this far• Additional dependencies in yourarchitecture@jamie_allen9

10.
Guaranteed Delivery Doesn’t Exist• We don’t know what we don’t know• Increased effort• Increased complexity• Increased latency• No guarantees of consistency• Doesn’t guarantee ordering@jamie_allen10

11.
So What Do We Do?• This falls outside of actor supervision;nothing the actors know about hasgone wrong• Listen to Roland Kuhn:“Recovery ... should ideally beautomatic in order to restore normalservice as quickly as possible.”@jamie_allen11

12.
Sentinels12@jamie_allen

13.
Sentinels• Supervisors handle failure BELOW them.Sentinels handle failure ABOVE.• Responsible for querying a “source of truth” andgetting latest state• Sends data to supervisor, who resolvesdifferences in which instances of actors shouldexist versus those that do• Supervisor forwards data to instances thatshould exist for them to resolve their internalstate@jamie_allen13

14.
Missed Events@jamie_allenCustomer 2Customer 1Customer SupervisorAdd Customer 3Delete Customer 1Update Customer 2

15.
Sentinels@jamie_allenCustomer 2Customer 1Customer SupervisorCustomer SentinelCustomer 2Customer 3

16.
Sentinels@jamie_allenCustomer 2Customer 1Customer SupervisorCustomer SentinelCustomer 2Customer 3

17.
Sentinels@jamie_allenCustomer 2Customer 1Customer SupervisorCustomer SentinelXCustomer 2Customer 3

18.
Sentinels@jamie_allenCustomer 2Customer SupervisorCustomer SentinelCustomer 2Customer 3Customer 2

19.
Sentinels@jamie_allenCustomer 2Customer SupervisorCustomer SentinelCustomer 2Customer 3Customer 3

20.
Sentinels• Localize them for each kind of datathat must be synchronized in yoursupervisor hierarchy• Do not create one big one and try toresolve the entire tree at once@jamie_allen20

21.
Drawbacks• Doesn’t work well with localized eventsourcing - time series can be lost• Does introduce additional complexityand tunable latency over applicationswith no guarantees• Pattern only works when there is aqueryable source of truth@jamie_allen21

22.
Inconsistent Views?• Using Sentinels at multiple levels of asupervisory hierarchy can lead totemporarily inconsistent views whenchild actors are resolved beforeparents on delete (no atomicity)• But is this necessarily bad?@jamie_allen22

23.
Sentinels in Hierarchy@jamie_allenC2C1CSA2A1ASD2D1DSA2SSS

24.
Sentinels in Hierarchy@jamie_allenC2C1CSA2A1ASD2D1DSA2SSSX

25.
Sentinels in Hierarchy@jamie_allenC2C1CSA2A1ASD2D1DSA2SSSX

26.
Sentinels in Hierarchy@jamie_allenC2C1CSASD2D1DS SSSX

27.
Sentinels in Hierarchy@jamie_allenC2C1CSD2D1DS SSXAS S

28.
Sentinels in Hierarchy@jamie_allenC1CS S

29.
A Huge Win• Your system is resilient to externalfailures• You can tune sentinel updatefrequency to meet changingrequirements• Your system is considerably lesscomplex than attempting to guaranteeno message loss@jamie_allen29

30.
Flow ControlBjörn Antonsson@bantonsson

31.
Pure Push Applications• Often the first Actor application youwrite– Once you start telling and stop asking• Easy to implement and reason about• Fits nicely with short lived jobs thatcome at a fixed rate31@bantonsson

32.
32@bantonsson

33.
Why do you need anything else?• Produce jobs faster than you can finishthem• Jobs are expensive compute/memorywise• External resources impose limits• Unpredictable job patterns33@bantonsson

34.
What can you do instead?• Push with rate limiting– A fixed number of jobs per time unit arepushed• Push with acknowledgment– A fixed number of jobs can be in progress.– New jobs are pushed after old jobs finish• Pull– Jobs are pulled from the master at the ratethat they are completed34@bantonsson

35.
Push with rate limiting• A timer sends the master ticks at fixedintervals• When a tick arrives, the master fills upits token count• If a job arrives and there are no tokens,it gets queued• When the master has tokens, it pullsjobs off the queue and pushes them35@bantonsson

36.
36@bantonsson

37.
Push with acknowledgement• The master push a fixed number of jobsbefore waiting for an acknowledgement• If a job arrives and the master cantpush, it gets queued• To keep workers busy, push more thanone job per worker– You can use a high water mark to stop anda low water mark to start pushing37@bantonsson

38.
38@bantonsson

39.
Pull• The master actor queues incoming jobs• Worker actors ask the master for a joband receives jobs when available• The workers dont need to do activepolling• Can lead to lag if jobs are smallcompared to the time it takes to get anew one– Use batching to counteract lag39@bantonsson

40.
40@bantonsson

41.
References• Push with rate limiting– Kaspar Fischerhttp://letitcrash.com/post/28901663062/throttling-messages-in-akka-2• Pull– Derek Wyatthttp://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2– Michael Pollmeierhttp://www.michaelpollmeier.com/akka-work-pulling-pattern-to-throttle-work/41@bantonsson

42.
DistributedWorkersPatrik Nordwall@patriknw

43.
43

44.
Goal• elastic addition/removal of front endnodes• elastic addition/removal of workers• thousands of workers• jobs should not be lost44

45.
45

46.
46

47.
47

48.
48

49.
49

50.
50

51.
51DistributedPubSubMediator

52.
52

53.
53

54.
54

55.
55

56.
56