Aggregate programming is a novel paradigm that addresses, at the core, many issues commonly found in the development of large-scale, situated, self-adaptive systems. It is a particular macro-programming approach where a developer expresses the behaviour of the system at the aggregate-level, by targeting the distributed computational machine which is given by the entire set of (possibly mobile and heterogeneous) networked devices pervading the environment. It is the model that takes care of turning a system-level behaviour specification into the concrete, device-centric programs executed locally by each component.
Aggregate computing is formally grounded in the field calculus, a minimal functional language that works with computational fields, i.e., distributed data structures mapping devices (digital representatives of space-time portions) to computational objects. Fields are a useful unifying abstraction for drawing a connection between the physical and the computational world, and between the local and global programming viewpoints. This approach is compositional, allowing to define layers of building blocks of increasing abstraction, and is also amenable to formal analyses.
In this talk, I will present scafi (SCAla with computational FIels), an aggregate computing framework for the Scala programming language which provides (i) an internal DSL for expressing aggregate computations as well as (ii) a library support for the configuration and execution of aggregate systems. There is no need to learn ad-hoc external DSLs anymore: with scafi, Scala programmers can instantaneously start playing with this new, intriguing development approach!
1. Practical Aggregate Programming in Scala
Roberto Casadei
PhD Student in CS&Eng
roby.casadei@unibo.it
Department of Computer Science and Engineering
University of Bologna
Student talk at Scala Symposium, Amsterdam 2016
Slides available at http://www.slideshare.net/RobertoCasadei/presentations
Sample code at https://bitbucket.org/metaphori/scafi-tutorial
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 1 / 30
2. Outline
1 Aggregate Computing: The Basics
2 SCAFI: Practical Aggregate Programming in Scala
3 Conclusion
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 2 / 30
3. Aggregate Computing: The Basics
Outline
1 Aggregate Computing: The Basics
2 SCAFI: Practical Aggregate Programming in Scala
3 Conclusion
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 3 / 30
4. Aggregate Computing: The Basics
Problem: design/programming CASs
Collective/Complex Adaptive Systems (CASs)
Structure: Environment + (Mobile, Large-scale) Networks of { people + devices }
Global interpretation: embedded devices collectively form a “diffused” computational system
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 4 / 30
5. Aggregate Computing: The Basics
An approach to CAS development
Issues ⇒ approach
• Large-scale ⇒ decentralised coordination
• Situatedness + distributed autonomy ⇒ substantial unpredictability ⇒ self-*
• Complex collective behavior ⇒ good abstractions, layered approach, compositionality
Shifting the mindset: from local to global
• Declarativeness and the global viewpoint
• Crowd-aware services
• Failure recovery of enterprise services
• Distributed monitoring and reacting (e.g., temperature, fire)
• Expected global behavior vs. traditional device-centric interface
⇒ Aggregate Programming [BPV15]: a paradigm for programming whole aggregates of devices.
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 5 / 30
6. Aggregate Computing: The Basics
Aggregate programming [BPV15]
From the local/device-centric viewpoint to the global/aggregate viewpoint
Aggregate programming: what
Goal: programming the collective behaviour of aggregates (of devices) ⇒ global-to-local
Aggregate programming: how
Prominent approach (generalizing over several prior approaches and strategies [BDU+
12])
founded on field calculus and self-org patterns
• Computational fields as unifying abstraction of local/global viewpoints
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 6 / 30
7. Aggregate Computing: The Basics
Aggregate Programming Stack
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 7 / 30
8. Aggregate Computing: The Basics
Aggregate (computing) systems & Execution model
Structure ⇒ (network/graph)
• A set of devices (aka nodes/points/things).
• Each device is able to communicate with a subset of devices known as its neighbourhood.
Dynamics
Each device is given the same aggregate program and works at async / partially-sync rounds:
(1) Retrieve context
⇐ Messages from neighbours
⇐ Sensor values
(2) Aggregate program execution
⇒ export (a tree-like repr of computation) + output (result of last expr in body)
(3) Broadcast export to neighbourhood
(4) Execute actuators
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 8 / 30
9. SCAFI: Practical Aggregate Programming in Scala
Outline
1 Aggregate Computing: The Basics
2 SCAFI: Practical Aggregate Programming in Scala
3 Conclusion
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 9 / 30
10. SCAFI: Practical Aggregate Programming in Scala
SCAFI: Scala with Computational Fields
Goal: bring Aggregate Computing to the field of mainstream software development
What
SCAFI [CV16] is an integrated framework for building systems with aggregate programming.
• Scala-internal DSL for expressing aggregate computations.
• Linguistic support + execution support (interpreter/VM)
• Correct, complete, efficient impl. of the Higher-Order Field Calculus
semantics [DVPB15]
• Distributed platform for execution of aggregate systems.
• Support for multiple architectural styles and system configurations.
• Actor-based implementation (based on Akka).
Where
• https://bitbucket.org/scafiteam/scafi
libraryDependencies += "it.unibo.apice.scafiteam" % "scafi-core_2.11" % "0.1.0" // on Maven Central
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 10 / 30
11. SCAFI: Practical Aggregate Programming in Scala
Computational fields [DVB16]
• (Abstract interpretation) Mapping space-time to computational objects
• (Concrete interpretation) Mapping devices to values: φ : δ →
• “Distributed” data structure working as the global abstraction
• The bridge abstraction between local behavior and global behavior
Discrete systems as an approximation of
spacetime
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 11 / 30
12. SCAFI: Practical Aggregate Programming in Scala
Field/aggregate computations
Global viewpoint
• Aggregate interpretation
• Natural/denotational semantics
• Program: computation over whole fields
• Output (at a given time): system-wide
snapshot of a computational field
• Geometric view: properties of collections
of points
Local viewpoint
• Device-centric interpretation
• Operational semantics
• Program: steps of a single device
• Output (at a given time): latest value
yielded by the device
• Geometric view: properties of a single
point
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 12 / 30
13. SCAFI: Practical Aggregate Programming in Scala
So, what is an aggregate program?
• The global program
– "Local programs" obtained via global-to-local mapping
• May take the form of field calculus programs (in the representation given by some PL)
– Actually, the field calculus is like FJ for Java, or the lambda calculus for Haskell
• An aggregate program consists of
1) A set of function definitions
2) A body of expressions.
• Example: an aggregate program in SCAFI
class MyProgram extends AggregateProgram with MyAPI {
def isSource = sense[Boolean]("source")
// Entry point for execution
override def main() = gradient(isSource)
}
– Each device of the aggregate system is given an instance of MyProgram.
– Each device repeatedly runs the main method at async rounds of execution.
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 13 / 30
14. SCAFI: Practical Aggregate Programming in Scala
Computing with fields
Expressing aggregate/field computations in SCAFI
trait Constructs {
def rep[A](init: A)(fun: (A) => A): A
def nbr[A](expr: => A): A
def foldhood[A](init: => A)(acc: (A,A)=>A)(expr: => A): A
def aggregate[A](f: => A): A
// Not primitive, but foundational
def sense[A](name: LSNS): A
def nbrvar[A](name: NSNS): A
def branch[A](cond: => Boolean)(th: => A)(el: => A): A
}
• Mechanisms for context-sensitiveness: nbr, nbrvar, sense
• Mechanisms for field evolution: rep
• Mechanisms for interaction: nbr
• Mechanisms for field domain restriction and partitioning: aggregate, branch
• Reference formal system: field calculus [DVB16, DVPB15]
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 14 / 30
15. SCAFI: Practical Aggregate Programming in Scala
Simple fields
0
(x)=>x+1
true t<0,1>
Constant, uniform field: 5
– Local view: evaluates to 5 in the context of a single device
– Global view: yields a uniform constant field that holds 5 at any point (i.e., at any device)
Constant, non-uniform field: mid()
– mid() is a built-in function that returns the ID of the running device
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 15 / 30
16. SCAFI: Practical Aggregate Programming in Scala
rep: dynamically evolving fields
rep
0
(x)=>x+1
t
v0
t
v1
..
rep(0){(x)=>x+1}
// Signature: def rep[A](init: A)(fun: (A) => A): A
// Initially 0; state is incremented at each round
rep(0){ _+1 }
– Notice: the frequency of computation can vary over time and from device to device
– In general, the resulting field will be heterogeneous in time and space
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 16 / 30
17. SCAFI: Practical Aggregate Programming in Scala
nbr: interaction, communication, observation
nbr de
nbr{e}
φd=[d1→v1,..,dn→vn]
– Local view: nbr returns a field from neighbors to their corresponding value of the given expr e
– Global view: a field of fields
– Needs to be reduced using a *hood operation
– foldhood works by retrieving the value of expr for each neighbour and then folding over the
resulting structure as you’d expect from FP.
// Signature: def nbr[A](expr: => A): A
// Signature: def foldhood[A](init: => A)(acc: (A,A)=>A)(expr: => A): A
foldhood(0)(_+_){ nbr{1} }
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 17 / 30
18. SCAFI: Practical Aggregate Programming in Scala
Context-sensitiveness and sensors
Local context:
1) The export of the previous computation
2) Messages received from neighbours
3) Values perceived from the physical/software environment
Sensing
// Query a local sensor
sense[Double]("temperature")
// Compute the maximum distance from neighbours
foldhood(Double.MinValue)(max(_,_)){ // Also: maxHood {...}
nbrvar[Double](NBR_RANGE_NAME)
}
– nbr queries a local sensor
– nbrvar queries a "neighbouring sensor" (a sort of "environmental probe")
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 18 / 30
19. SCAFI: Practical Aggregate Programming in Scala
Field domain restriction
Alignment
• Aggregate computations can be represented as a trees
• Device exports are "paths" along these trees
• When two devices execute the same tree node, they are said to be aligned
• Interaction is possible only between aligned devices
Use cases for branch
• Partitioning the space into subspaces performing subcomputations
• Regulating admissible interactions (i.e., further restricting the neighbourhood)
branch(sense[Boolean]("flag")){
compute(...) // sub-computation
}{
Double.MaxValue // stable value (i.e., not computing)
}
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 19 / 30
20. SCAFI: Practical Aggregate Programming in Scala
Functions
Two kinds of functions in SCAFI:
1) "Normal" Scala functions: serve as units for encapsulating behavior/logic
def foldhoodMinus[A](init: => A)(acc: (A,A) => A)(ex: => A): A =
foldhood(init)(acc){ mux(mid()==nbr(mid())){ init }{ ex } }
def isSource = sense[Boolean]("source")
2) First-class "aggregate" functions [DVPB15] – which also work as units for alignment
def branch[A](cond: => Boolean)(th: => A)(el: => A): A =
mux(cond, ()=>aggregate{ th }, ()=>aggregate{ el })()
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 20 / 30
22. SCAFI: Practical Aggregate Programming in Scala
Example: the channel I
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 22 / 30
23. SCAFI: Practical Aggregate Programming in Scala
Example: the channel II
Each device is given the same aggregate program:
class ChannelProgram extends AggregateProgram with ChannelAPI {
def main = channel(isSource, isDestination, width)
}
def channel(src: Boolean, dest: Boolean, width: Double) =
distanceTo(src) + distanceTo(dest) <= distBetween(src, dest) + width
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 23 / 30
25. SCAFI: Practical Aggregate Programming in Scala
Scaling with complexity
General coordination operators [VBDP15]
• Gradient-cast: accumulates values “outward” along a gradient starting from source nodes.
def G[V:OB](src: Boolean, init: V,
acc: V=>V, metric: =>Double): V
• Converge-cast: collects data distributed across space “inward” by accumulating values from
edge nodes to sink nodes down a “potential” field.
def C[V:OB](potential: V, acc: (V,V)=>V, local: V, Null: V): V
• Time-decay: supports information summarisation across time.
def T[V:Numeric](initial: V, floor: V, decay: V=>V): V
• Sparse-choice: supports creation of partitions and selection of sparse subsets of devices in
space
def S(grain: Double, metric: => Double): Boolean
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 25 / 30
27. SCAFI: Practical Aggregate Programming in Scala
Quick platform setup
// STEP 1: CHOOSE INCARNATION
import it.unibo.scafi.incarnations.{ BasicActorP2P => Platform }
// STEP 2: DEFINE AGGREGATE PROGRAM SCHEMA
class Demo_AggregateProgram extends Platform.AggregateProgram {
override def main(): Any = foldhood(0){_ + _}(1)
}
// STEP 3: DEFINE MAIN PROGRAM
object Demo_MainProgram extends Platform.CmdLineMain
1) Demo_MainProgram -h 127.0.0.1 -p 9000
-e 1:2,4,5;2;3 --subsystems 127.0.0.1:9500:4:5
--program "demos.Demo_AggregateProgram"
2) Demo_MainProgram -h 127.0.0.1 -p 9500
-e 4;5:4
--program "demos.Demo_AggregateProgram"
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 27 / 30
28. SCAFI: Practical Aggregate Programming in Scala
Manual node setup
// STEP 1: CHOOSE INCARNATION
import scafi.incarnations.{ BasicActorP2P => Platform }
import Platform.{AggregateProgram,Settings,PlatformConfig}
// STEP 2: DEFINE AGGREGATE PROGRAM SCHEMA
class Program extends AggregateProgram with CrowdAPI {
// Specify a "dangerous density" aggregate computation
override def main(): Any = crowdWarning(...)
}
// STEP 3: PLATFORM SETUP
val settings = Settings()
val platform = PlatformConfig.setupPlatform(settings)
// STEP 4: NODE SETUP
val sys = platform.newAggregateApplication()
val dm = sys.newDevice(id = Utils.newId(),
program = Program,
neighbours = Utils.discoverNbrs())
val devActor = dm.actorRef // get underlying actor
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 28 / 30
29. Conclusion
Outline
1 Aggregate Computing: The Basics
2 SCAFI: Practical Aggregate Programming in Scala
3 Conclusion
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 29 / 30
30. Conclusion
Summary: key ideas
Aggregate programming
• A "macro" (programmingengineering) approach to CASs, formally grounded in the Field
Calculus.
• Allows to compose “emergent” phenomena & defines layers of (self-stabilizing) building blocks.
SCAFI: a Scala framework for Aggregate Programming
• Provides an internal DSL for field-based computations
• Provides an actor-based platform for building aggregate systems
Future work
• Evolve SCAFI to support scalable computations in cluster- and cloud-based systems.
• What does it take to set up a framework for adaptive execution strategies?
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 30 / 30
32. Appendix References
References I
[BBVT08] Jacob Beal, Jonathan Bachrach, Dan Vickery, and Mark Tobenkin.
Fast self-healing gradients.
In Proceedings of the 2008 ACM symposium on Applied computing, pages 1969–1975.
ACM, 2008.
[BDU+
12] Jacob Beal, Stefan Dulman, Kyle Usbeck, Mirko Viroli, and Nikolaus Correll.
Organizing the aggregate: Languages for spatial computing.
CoRR, abs/1202.5509, 2012.
[BPV15] Jacob Beal, Danilo Pianini, and Mirko Viroli.
Aggregate Programming for the Internet of Things.
IEEE Computer, 2015.
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 32 / 30
33. Appendix References
References II
[CPV16] Roberto Casadei, Danilo Pianini, and Mirko Viroli.
Simulating large-scale aggregate mass with alchemist and scala.
In Maria Ganzha, Leszek Maciaszek, and Marcin Paprzycki, editors, Proceedings of the
Federated Conference on Computer Science and Information Systems (FedCSIS
2016), Gdansk, Poland, 11-14 September 2016. IEEE Computer Society Press.
To appear.
[CV16] Roberto Casadei and Mirko Viroli.
Towards aggregate programming in Scala.
In First Workshop on Programming Models and Languages for Distributed Computing,
PMLDC ’16, pages 5:1–5:7, New York, NY, USA, 2016. ACM.
[DVB16] Ferruccio Damiani, Mirko Viroli, and Jacob Beal.
A type-sound calculus of computational fields.
Science of Computer Programming, 117:17 – 44, 2016.
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 33 / 30
34. Appendix References
References III
[DVPB15] Ferruccio Damiani, Mirko Viroli, Danilo Pianini, and Jacob Beal.
Code mobility meets self-organisation: A higher-order calculus of computational fields.
volume 9039 of Lecture Notes in Computer Science, pages 113–128. Springer
International Publishing, 2015.
[VBDP15] Mirko Viroli, Jacob Beal, Ferruccio Damiani, and Danilo Pianini.
Efficient engineering of complex self-organising systems by self-stabilising fields.
2015.
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 34 / 30