Akka, actor systems and reactive programming are new trends in software development with Java and Scala. This talk dives into details of Akka usage with Scala.
We will discuss different models of concurrency handling and the actor model in particular. The particular implementation of the model we will be talking about is Akka, a system that supports writing code with the actor model in Java, Scala and .net. We will cover how actors send messages to each other via mailboxes and dispatchers, will take a look at the supervision hierarchy and error handling with the error kernel pattern.
In addition to the theoretical section we will also discuss a few projects where Planet OS has used Akka and the actor model to achieve high reliability systems:
- an operational dashboard for the world's second largest offshore wind park
- DPipe, the highly configurable data processing pipeline that is doing the heavy lifting behind http://data.planetos.com, our universal data access api
By Kristo Iila
Engineering Manager at Planet OS.
2. PLANET OS - MARCH 2016
The Many Ways of Concurrency
• Multithreading / semaphores / locking / mutexes
• Lock-free data structures
• Software Transactional Memory
• Message passing
• Futures / Async
• Actors
3. PLANET OS - MARCH 2016
What are Actors?
• Lightweight object
• No shared state
• Communication via message passing
• Supervision hierarchies
• Originally implemented in Erlang
(high availability telephone switches)
• Don’t need to think in terms of threads and locks
• Distributable by design
4. PLANET OS - MARCH 2016
An Actor System Implementation - AKKA
• Implements actor model
• Makes remoting seamless
• Works with Java, Scala and .net
• Fits very well with scala
• Automatically scheduled to threads
• Can have multiple thread pools
Actor A Mailbox
Actor B
Message
5. PLANET OS - MARCH 2016
Messages
• Communication between actors
• Tell: Fire and Forget
• Ask: Tell + Future { reply }
• Guaranteed ordering for one
sender and receiver pair
• Access to a mailbox
• Messages can be anything
// Tell example
actorRef ! Notify("hi!")
// Ask example
val future = qaActor ? Question(“What?”)
// Process result
future.onComplete { … }
// Send result to actor
future pipeTo actorRef2
6. PLANET OS - MARCH 2016
Futures
• Future is a way to
Future { socket.read() }
retrieve results …
of concurrent operations …
asynchronously or synchronously.
• Examples
Eg code block:
Or actor ask: qaActor ? Question(“What is the meaning of life?”)
• Futures are composable
7. PLANET OS - MARCH 2016
Actor Creation
• Each actor
val qaActor = system.actorOf(
Props[QuestionAnswerer],
“Oracle”
)
... is an instance of a class with parameters
… has a name
• To create an actor:
8. PLANET OS - MARCH 2016
Actor References
• You can not talk to actors directly
• When you create an actor, you get an ActorRef
• ActorRef is a reference to an actor
… that can be passed around
… also in messages
… is serializable
… possibly remote
… hides local state
10. PLANET OS - MARCH 2016
But What if We Want an Already Existing ActorRef?
val selection = context.actorSelection("/user/AllKnowing/oracle")
• ActorSelection
val allAnswerers = context.actorSelection("/user/AllKnowing/*")
• The result can contain multiple actors:
allAnswerers ! Question(“Why are there petunias?”)
• But you can still send it messages:
• Transparently remote
11. PLANET OS - MARCH 2016
But how do I know the actor path?
• Actors are in a hierarchy (tree)
• Tree root is ActorSystem
• System guardian and user guardian
• All user-created actors live under
/user
• It is a bit more complicated if the
actor is in a remote machine
ActorSystem
/user /system
/user/Allknowing
/user/AllKnowing/Oracle
12. PLANET OS - MARCH 2016
What else can we do with the hierarchy? Supervision!
• Reliability
• Error kernel pattern
• Errors are escalated up the hierarchy
• Each level can decide how to respond
• Escalate error
• Restart child
• Stop child
• Resume child
• Top level guardian will shut down entire actor system
/c1
/b1
/c2
/a1
/b2
/user
/a2
/b3
/c4 /c5
/
/
system
13. PLANET OS - MARCH 2016
Where do messages go? Mailboxes!
• Each actor has access to one
• Possibly shared
• Multiple types
• Bounded / Unbounded
• Priority
• Can be order-preserving, but not always
Actor A Mailbox
Actor B
Message
14. PLANET OS - MARCH 2016
Message Delivery Reliability
• Local message delivery is reliable
Except for vm errors, and it’s better to always code for failure cases anyway.
• Mostly order preserving
Unless you explicitly select non-order-preserving mailboxes of course
• Remote delivery is unreliable
• Should retry messages until acknowledged
Or be ok with losing some, or most, or all, depending on the network.
• If message processing is idempotent, things get a lot easier to manage
15. PLANET OS - MARCH 2016
Who schedules actors to threads? Dispatchers!
• Messages arrive to actor mailboxes
• Dispatcher takes a message
• from actor mailbox …
• and schedules actor …
• to handle it on a thread
• Threads are taken from execution contexts
• … aka thread pools
• Dispatchers are assigned to actors via rules
Actor A Mailbox
Actor B
Message
Dispatcher
18. PLANET OS - MARCH 2016
Websocket handling with actors
• Actor based web server
• One actor that receives connections
• Creates per-connection actors
• Supervises / terminates connection actors
• Connection actors handle data updates
• Better performance than
thread-per-connection model
• Easier and scales better than
… callbacks for data send/receive
… event loops
Listening
Socket
Connection
Manager Actor
Connection
Actor
Connection
Actor
Connection
Actor
19. PLANET OS - MARCH 2016
Data update publish
Kafka
Connection
Actor
Connection
Actor
Connection
Actor
Kafka
receiver actor
Socket
Connection
Socket
Connection
Socket
Connection
20. PLANET OS - MARCH 2016
SCADA
SCADA
SCADA
WS
Kafka
Queue
Process
Turbine 1
Turbine 2
Turbine 3
Power Plant
Dashboard
21. PLANET OS - MARCH 2016
Event processing with actors
• Actors represent domain entities
• wind turbine
• power plant
• alert
• Receive event updates from Kafka
• Produce new events sent to Kafka
• Some events are joins of multiple events (eg alerts)
• Some events go to Web Sockets
22. PLANET OS - MARCH 2016
Challenges
• Code must correctly handle idempotency
• Code must correctly handle crashes / restarts anywhere
• Data updates must not be lost
• Even cascaded data updates must not be lost
• Purely actor based model must mix consistency handling
in business logic
24. PLANET OS - MARCH 2016
DPipe - the magic behind the Data Hub
• Data pipeline:
• Acquisition
• Transformation
• Analysis
• Indexing
• For spatio-temporal data:
• Time series
• Forecasts
• Satellites
• Sensors
• Models
25. PLANET OS - MARCH 2016
DPipe - large scale data processing
• Multiple stages
• Each stage runs a cluster of actor systems
• Actor system has a scheduler that distributes work
to worker actors
• Multiple pipelines also within a stage
• Can scale linearly with data sources
• Can do fork/join in data pipes with parallelisation
• Framework for exactly-once processing at scale
26. As if it wasn’t already quite obvious
We ARE in fact looking for new team members.
There are many interesting challenges to work on.
Interested? Come talk to me or
e-mail kristo@planetos.com