SPLAY simplifies the prototyping and development of large-scale distributed applications and overlay networks. SPLAY covers the complete chain of distributed system design, development and testing: from coding and local runs to controlled deployment, experiment control and monitoring. Developers specify their applications using Lua, which is executed in a sandboxed environment. The talk will include a short live demo of a simple distributed protocol in Lua deployed on the PlanetLab. SPLAY is the outcome of research and development activities at the Computer Science Department of the University of Neuchatel, Switzerland.

1. !"#$%
Distributed Systems
Made Simple
Pascal Felber, Lorenzo Leonini, Etienne Rivière,
Valerio Schiavoni, José Valerio
Lua Workshop, 8 September 2011
Frick, Switzerland
!"#$%

2. About Me
• 2010-now : PhD student at the University of
Neuchatel, Switzerland (and Lua user since then)
• Topic: large-scale distributed systems, cloud
computing. Daily job:
1. Invent new protocols for cloud applications
2. Do experiments, write papers, go to 1
• 2007-2009: Research engineer at INRIA, France
• BSc and MSc in Computer Engineering at the
Università degli Studi Roma Tre, Italy
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

3. Motivations
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

4. Motivations
• Developing, testing and tuning distributed
applications is hard
• In Computer Science research, fixing the
gap of simplicity between pseudocode
description and implementation is hard
• Using worldwide testbeds is hard
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

5. What is
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

6. What is
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

7. What is
• Machines contributed by universities,
companies,etc.
• 1098 nodes at 531 sites (02/09/2011)
• Shared resources, no privileged access
• University-quality Internet links
• High resource contention
• Faults, churn, packet-loss is the norm
• Challenging conditions
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

8. Daily Job With
Distributed Systems
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

9. Daily Job With
Distributed Systems
1
code
1 • Write (testbed specific) code
• Local tests, in-house cluster, PlanetLab...
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

10. Daily Job With
Distributed Systems
2
1
code debug
2 • Debug (in this context, a nightmare)
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

11. Daily Job With
Distributed Systems
2
1
code debug
2 • Debug (in this context, a nightmare)
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

12. Daily Job With
Distributed Systems
3
1 2
code debug deploy
3 • Deploy, with testbed specific scripts
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

13. Daily Job With
Distributed Systems
3
1 2
code debug deploy
3 • Deploy, with testbed specific scripts
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

14. Daily Job With
Distributed Systems
4
1 2 3
code debug deploy get logs
4 • Get logs, with testbed specific scripts
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

15. Daily Job With
Distributed Systems
4
1 2 3
code debug deploy get logs
4 • Get logs, with testbed specific scripts
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

16. Daily Job With
Distributed Systems
5
1 2 3 4
code debug deploy get logs plots
5 • Produce plots, hopefully
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

17. Daily Job With
Distributed Systems
1 2 3 4 5
code debug deploy get logs plots
5 • Produce plots, hopefully
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

18. !"#$% At Glance
code debug deploy get logs plots
• Supports the development, evaluation,
testing, and tuning of distributed
applications on any testbed:
• In-house cluster, shared testbeds,
emulated environments
• Provides an easy-to-use pseudocode-like
language implemented in Lua
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

19. !"#$% At Glance
code debug deploy get logs plots
• Supports the development, evaluation,
testing, and tuning of distributed
applications on any testbed:
• In-house cluster, shared testbeds,
emulated environments
• Provides an easy-to-use pseudocode-like
language implemented in Lua
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

20. !"#$% At Glance
code debug deploy get logs plots
l ot is
gnup riend
ur f
• Supports the development, evaluation, yo
testing, and tuning of distributed
applications on any testbed:
• In-house cluster, shared testbeds,
emulated environments
• Provides an easy-to-use pseudocode-like
language implemented in Lua
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

21. The Big Picture
require”splay.base”
splayd
splayd
splayd
splayd
Splay
Controller
SQL DB
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

22. The Big Picture
splayd splayd require”splay.base”
splayd
splayd
Splay
Controller
SQL DB
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

23. The Big Picture
splayd splayd
require”splay.base”
splayd
splayd
Splay
Controller
SQL DB
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

24. Why ?
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

25. Why ?
• Light & Fast
• (Very) Close to equivalent code in C
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

26. Why ?
• Light & Fast
• (Very) Close to equivalent code in C
• Concise
• Allowimplementation focus on ideas more
than
developers to
details
• Key for researchers
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

27. Why ?
• Light & Fast
• (Very) Close to equivalent code in C
• Concise
• Allowimplementation focus on ideas more
than
developers to
details
• Key for researchers
• Sandbox thanks to the possibility of easily
redefine (even built-in) functions
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

28. Concise
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

29. Concise
Pseudo code
as published
on original
paper
Executable
code using
SPLAY
libraries
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

30. Sandbox: Motivations
• Experiments should access only their
own resources
• Required for non-dedicated testbeds
• In universities, companies
• Totally available at night/holiday time
• Memory-allocation, filesystem, network
resources are restricted
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

31. Sandboxing LuaSocket
!
!
• Same API as plain LuaSocket
if sock.send then
!
!
new_sock.send = function(self, data)
! local len = #data
• On-demand sandboxed sockets
! ! ! if total_sent + len > max_send then
! ! ! ! l_o:warn("Send restricted (total: "..total_sent..")")
! ! ! ! return nil, "restricted"
• Very easy thanks to Lua’s metatable
! ! ! end
! ! ! local n, status
! ! ! if math.random(1000) > udp_drop_ratio then
! ! ! ! n, status = sock:send(data)
!
!
!
• Limits chosen by the SPLAYd deployer
!
!
!
!
!
!
else
!
end
n = len
• Both TCP and UDP
! ! ! if n then
! ! ! ! total_sent = total_sent + len
! ! ! end
• Example: UDP’s socket.send
! ! ! return n, status
! ! end
! end
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

32. Sandboxing LuaSocket
!
!
• Same API as plain LuaSocket
if sock.send then
!
!
new_sock.send = function(self, data)
! local len = #data
• On-demand sandboxed sockets
! ! ! if total_sent + len > max_send then
! ! ! ! l_o:warn("Send restricted (total: "..total_sent..")")
! ! ! ! return nil, "restricted"
• Very easy thanks to Lua’s metatable
! ! ! end
! ! ! local n, status
! ! ! if math.random(1000) > udp_drop_ratio then
! ! ! ! n, status = sock:send(data)
!
!
!
• Limits chosen by the SPLAYd deployer
!
!
!
!
!
!
else
!
end
n = len
• Both TCP and UDP
! ! ! if n then
! ! ! ! total_sent = total_sent + len
! ! ! end
• Example: UDP’s socket.send
! ! ! return n, status
! ! end
! end
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

33. for _,module in pairs(splay)
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

34. for _,module in pairs(splay)
luasocket events io
crypto llenc/benc json
misc log rpc
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

35. for _,module in pairs(splay)
luasocket events io
crypto llenc/benc json
misc log rpc
Modules sandboxed to prevent
access to sensible resources
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

36. splay.events
luasocket events io
crypto llenc/benc json
misc log rpc
Modules sandboxed to prevent
access to sensible resources
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

37. splay.events
events
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

38. splay.events
• Distributed protocols can use message-
passing paradigm to communicate
• Nodes react to events events
• Local, incoming, outgoing messages
• The core of the Splay runtime
• splay.socket, splay.io designed to
provide a non-blocking operational mode
• Based on Lua’s coroutines
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

39. splay.rpc
luasocket events io
crypto llenc/benc json
misc log rpc
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

40. splay.rpc
rpc
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

41. splay.rpc
• Default mechanism to communicate
between nodes
• Support for UDP/TCP
• Efficient BitTorrent-like encoding rpc
• Experimental binary encoding
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

42. Life Before !"#$%
• Time spent on developing testbed
specific protocols
• Or fallback on simulations...
• The focus should be on ideas
• Researchers usually have no time to
produce industrial-quality code
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

43. public void declareNeighbourInactive(final DHTNode node) {
// DO NOTHING HERE:
Life Before !"#$%
}
public void addToLeafSet(final DHTNode node) {
// DO NOTHING
}
/**
* Empty UDP message is 64 bytes, to which we must add 1 bytes for the header,
• Time spent on developing testbed
* 1 bytes for the messageId, 28 bytes (IP,port,Bamboo ID) for message src, 28
* bytes for message dest + bytes for the specific data carried in the
specific protocols
• Or fallback on simulations...
* message. NOTE: the size of informations to ack a message is 10 bytes,
*/
@Override
• The focus should be on ideas
public long calculateMessageBytes(final Message msg) {
long result = 122; // 64 + 1 + 1 + 28 + 28;
• Researchers usually have no time to
if (msg.header == MessageType.PING) {
/* nothing to add */
return result;
produce industrial-quality code
}
if (msg.header == MessageType.PONG) {
/* some bytes for the informations about which message is being acked */
result += ACK_INFORMATIONS_SIZE;
return result;
}
if (msg.header == MessageType.BAMBOO_JOIN_LOOKUP_REQUEST
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland
|| msg.header == MessageType.BAMBOO_LOOKUP_REQUEST

44. responseCallback.onResponseReceived(msg);
}/*
* CAN BE COMMENTED AS SEND RETRIAL OF MESSAGES IS OFF else { it is a
Life Before !"#$%
* duplicate lookup response, send a PONG to (hopefully) stop source to
* send us again that message final Message pong = new
* Message(MessageType.PONG, msg.messageId, this, msg.src); pong.ackedMsg
* = msg; simulator.send(pong); }
*/
• Time spent on developing testbed
}
/* ELSE IF IS SOMETHING NOT A LOOKUP RESPONSE */
else {
final Message beingAcked = msg.ackedMsg; specific protocols
• Or fallback on simulations...
final ResponseArrivedCallback handler = this.sentMessagesCallbacks
.remove(beingAcked);
• The focus should be on ideas
/* if there is a mapping */
if (handler != null) {
handler.onResponseReceived(msg);
• Researchers usually have no time to
}/*
* else { this may happen in case of a response arrive later than
* expected... log .info("%%% " + this + " received the response => " +
produce industrial-quality code
* msg +
* " but no handler was expecting it! So, simply send back a PONG to make it stop sending again..n"
* ); send a PONG to (hopefully) stop source to send us again final
* Message pong = new Message(MessageType.PONG, msg.messageId, this,
there is more :-(
* msg.src); pong.ackedMsg = msg; simulator.send(pong); }
*/
}
// /* it replied at last, so "reset" the value */
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland
// this.nodeConsecutiveTimeouts.put((DHTNode) msg.src, 0);

45. such as epidemic diffusion on Erd¨ s-Renyi random
o
graphs [16] and various types of distribution trees [10]
Life With !"#$%
(n-ary trees, parallel trees). As one can note from the fol-
lowing figure, all implementations are extremely concise
7
in terms of lines of code (LOC):
Chord (base) 59 (base) + 17 (FT) + 26 (leafset) = 101
Pastry 265
Scribe Pastry 79
SplitStream Pastry Scribe 58
BitTorrent 420
Cyclon 93
Epidemic 35
Trees 47
Although the number of lines is clearly just a rough
• Lines thepseudocode ~== Lines of executableitcodestill a
indicator of
of
expressiveness of a system, is
5 Splay daemons have been continuously running on these hosts for
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

46. Live Demo
www.splay-project.org
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

47. ay
aw
e- e
ak lid
T S
• Distributed systems raise a number of
issues related to their evaluation
• Their implementation, debug, deployment
and tuning is hard
• !"#$% leverages Lua to produce an
easy to use yet powerful working
environment to solve these issues
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

48. Backup Slides
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

49. The Big Picture
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

50. The Big Picture
Splay
Controller
SQL DB
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

51. The Big Picture
REST API
Splay
Controller
SQL DB
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

52. The Big Picture
Shell Web
REST API
Splay
Controller
SQL DB
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

53. The Big Picture
Shell Web
REST API
Splay
Controller
SQL DB
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

54. The Big Picture
Shell Web
REST API
Splay churn
Controller
jobs
SQL DB blacklist
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

55. The Big Picture
alpha
require”splay.base”
...
Shell Web
REST API
Splay churn
Controller
jobs
SQL DB blacklist
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

56. splayd The Big Picture
splayd
alpha
splayd
require”splay.base”
...
Shell Web
REST API
Splay churn
Controller
jobs
SQL DB blacklist
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

57. splayd The Big Picture
splayd
alpha
splayd
require”splay.base”
...
Shell Web
REST API
Splay churn
Controller
log jobs
log SQL DB blacklist
log
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

58. splayd The Big Picture
splayd
alpha
splayd
require”splay.base”
...
Shell Web
ctrl
REST API
ctrl
ctrl Splay churn
Controller
log jobs
log SQL DB blacklist
log
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

59. splayd The Big Picture
splayd
alpha
splayd
require”splay.base”
...
Shell Web
ctrl
REST API
ctrl
ctrl Splay churn
Controller
log jobs
log SQL DB blacklist
log
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland

60. splayd The Big Picture
splayd
alpha
splayd
require”splay.base”
...
Shell Web
ctrl
REST API
ctrl
ctrl Splay churn
Controller
log jobs
log SQL DB blacklist
log
!"#$% Distributed Systems Made Simple - V. Schiavoni - Lua Workshop 2011, Frick, Switzerland