This was the Kamaelia Tutorial at Europython. It goes from basics - ie building a mini-kamaelia from scratch, through to a file multicaster, through a video recording application all the walk through to a multiuser bulletin board system.
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Kamaelia Europython Tutorial
1. Kamaelia:
Pragmatic Concurrency
A Tutorial
Michael Sparks
Europython '09, Birmingham UK
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
2. About me
• - Been using python for several years
• - Always been interested in concurrency
• - Kamaelia aims to embody safe practices.
• - Work at BBC R&D's Northern Lab, based in
Manchester, moving to Media City:UK in 2011
• - Kamaelia is born out of a variety of R&D
projects, and shaped by needs, not aesthetics
or purity.
• Kamaelia is not an active R&D project. It is used in R&D projects,
and hence actively maintained.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
3. About me
•
Disclaimer: Like you, I'm doing this on my time,
not the BBC's. This doesn't represent BBC
opinion on anything.
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
4. About Kamaelia
• - Also born from a desire to make concurrency
in programs easier to work with
•
- Because it's nice in the shell – I want software
concurrency that easy :-)
• - Expressiveness is favoured over performance,
but not to preclude optimisation
• - Means we're cautious about adding syntactic
sugar.
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
5. About Kamaelia
• - Adapts Unix Philosophy to make a program
concurrent internally, but with the purpose of
simplifying maintenance.
• Unix Philosophy:
•
Write programs that do one thing and do it well.
Write programs to work together.
Write programs to handle text streams, because that is
a universal interface.
• --Doug McIlroy
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
6. About Kamaelia
• - Adapts Unix Philosophy to make a program
concurrent internally, but with the purpose of
simplifying maintenance.
• Kamaelia Philosophy:
•
Write components that do one thing and do it well.
Write components to work together.
Write components to handle python object streams,
because that is a universal interface.
• --With apologies to Doug McIlroy
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
7. What we're covering
• - An overview of Kamaelia
• & it's view on concurrency
• - How to build a mini Kamaelia
• – to get under the hood.
• - Building components & systems
• - Examining larger systems, and debugging.
• - Building a large(ish) system
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
8. Time Estimates
• First part:
• - An overview of Kamaelia (lightning talkstyle : 5-10 mins)
• - How to build a mini Kamaelia (exercise 40-60 mins)
• - Starting building components & systems (remainder)
After break:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
9. Format
• I generally welcome questions at any point
• That said...
• 'Except in the overview
• - the rest of this morning should answer them!!
• Copious notes provided - covers more than
today
• Materials available from URL below
• Is a mixture of “explain then do”.
Perhaps hold questions for during “do” :-) ?
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
10. Caveat
•
• * First time I've given this tutorial !
• * If we run short of time on any section, we'll
skip ahead.
•
• * BUT, have copious notes that cover
everything.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
11. Questions?
• ... before we dive in?
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
12. Part 1
•
• First part:
• - An overview of Kamaelia (lightning talk style : 5-10
mins)
• - How to build a mini Kamaelia (exercise 40-60 mins)
• - Starting building components & systems (remainder)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
13. Kamaelia, a 20:20 overview
20:20 Presentation
A presentation style based on the “Pecha Kucha”
style 20 slides, 20 seconds each
(Similar to a lightning talk)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
14. Why? Hardware finally going
massively concurrent ...
Opportunity! .... PS3, high end servers, trickling down to desktops, laptops)
“many hands make light
work” but Viewed as Hard
... do we just have crap tools?
“And one language to in
Problems
the darkness bind them”
... can just we REALLY abandon 50 years of code for Erlang, Haskell
and occam?
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
15. Missing Something?
Fundamental Control Structures
... in imperative languages number greater than 3! (despite what you get taught...!)
Control Structure Traditional Abstraction Biggest Pain Points
Sequence Function Global Var
Selection Function Global Var
Iteration Function Global Var
Parallel Thread Shared Data
Usually Skipped Lost or duplicate update
are most common bugs
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
16. Regarding using
concurrency, what sort
of applications are we
talking about here?
Desktop gsoc
Media Novice
APPS trainee
Network 3rd Party
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
17. Think backend Speak 'n Write
P2P Whiteboard Programming
needed for Simple
ER DB Modeller (logo)
youtube/flickr Kids Games P2P Radio
Torrent
type systems 3D Systems
Compose Realtime Music
UGC Desktop gsoc Paint App
Backend P2P Web Server
Secure “phone”
Transcoder Social Net Vis
Media Novice ...
Shot Change
Detection APPS trainee
MiniAxon
ScriptReader
Mobile DVB MediaPreview
Reframing on Mobile
Reliable
Macro
Podcasts
Network 3rd Party Multicast
Sedna
“record
everything” XMPP XMLDB
Email & AWS pubsub
Spam Web (Amazon)
SMTP IRC Qt
Serving Gtk
Greylisting
Pop3Proxy ClientSide AIM microblogging
Spam Tools
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
18. Core Approach:
Concurrent things with comms points
Generally send messages
Keep data private, don't share
outbox inbox outbox
inbox
signal control signal
control
... ... ...
...
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
19. But I must share data?
Use Software Transactional Memory
ie version control for variables.
1. Check out the collection
of values you wish to
work on
2. Change them
3. Check them back in
4. If conflict/clash, go
back to 1
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
20. Perspectives in APIs! (1/2)
If you have concurrency it becomes
natural to think in terms of 1st 2nd and 3rd
person. This affects an API's structure, 1st Person - I change my state
and can be vital for understanding it!
This is one we've found that makes sense
2nd Person – YOU 3rd Person –
want to me to do Bla should
something do something
(you send outbox (I send a message)
inbox
me a message)
control signal
... ...
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
21. Perspectives in APIs! (2/2)
If you have concurrency it becomes
natural to think in terms of 1st 2nd and 3rd
person. This affects an API's structure, private real methods
and can be vital for understanding it!
This is one we've found that makes sense
Messages Messages sent
from public to public outboxes
inboxes
inbox outbox
control signal
Also, think ... ... Also, think
about stdin about stdout
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
22. Actor Systems
Distinction can be unclear,
potential source of ambiguity* private real methods
Messages
from public No outbox concept
inboxes Possible issues with
inbox rate limiting*
control
Hardcodes recipient
...
in the sender
*system dependent issue
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
23. Advantages of outboxes
No hardcoding of recipient
allows:
- Late Binding
- Dynamic rewiring
inbox outbox
Concurrency Patterns as
control signal Reusable Code
... ... ... a concurrency DSL
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
24. A Core Concurrency DSL
Pipeline(A,B,C)
Graphline(A=A,B=B, C=C, linkages = {})
Tpipe(cond, C)
Seq(A,B,C), PAR(), ALT()
Backplane(“name”), PublishTo(“name”), SubscribeTo(“name”)
Carousel(...)
PureTransformer(...)
StatefulTransformer(...)
PureServer(...)
MessageDemuxer(...)
Source(*messages)
NullSink
Some of these are work in progress
– they've been identified as useful,
but not implemented as chassis, yet
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
26. Graphline Example
Graphline(
NEXT = Button(...),
PREVIOUS = Button(...), PREVIOUS NEXT
FIRST = Button(...), (button) (button)
LAST = Button(...),
CHOOSER = Chooser(...),
IMAGE = Image(...), FIRST LAST
(button) (button)
...
).run()
Chooser
Image
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
27. Server Example
data
from
user data
Main Socket handler
Server Core to
user
Created at runtime
to handle the
Protocol Handler Factory connection Remote
Protocol handler User
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
28. Server Example
You therefore
need to provide
Main this bit.
Server Core
Protocol Handler Factory
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
29. Server Example
from Kamaelia.Chassis.ConnectedServer import ServerCore
from Kamaelia.Util.PureTransformer import PureTransformer
def greeter(*argv, **argd):
return PureTransformer(lambda x: "hello" +x)
class GreeterServer(ServerCore):
protocol=greeter
port=1601
GreeterServer().run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
30. Backplane Example
# Streaming Server for raw DVB of Radio 1
Backplane(“Radio”).activate()
Pipeline(
DVB_Multiplex(850.16, [6210], feparams), # RADIO ONE
PublishTo("RADIO"),
).activate()
def radio(*argv,**argd):
return SubscribeTo(“RADIO”)
ServerCore(protocol=radio, port=1600).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
31. So that's the 5 minute version
Short Q&A before we move on?
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
32. Part 1
•
• First part:
• - An overview of Kamaelia (lightning talk style : 5-10
mins)
• - How to build a mini Kamaelia (exercise 40-60 mins)
• - Starting building components & systems (remainder)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
33. Mini Axon
• Kamaelia is divided into two halves
• * One part handles all the concurrency
stuff, providing you a component model
• * The other is a large collection of
components and some apps using them.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
34. Mini Axon
• Axon is the part that handles
concurrency and provides the
component model, and is the key to
understanding why & how Kamaelia
works.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
35. Mini Axon
• ... is a collection of exercises where
you build just such a beast.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
36. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
37. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
38. Generators
• * Python's smallest unit of concurrency
• * Single function you call, get a
generator back
• * Can then call it's .next() method to
* Get a new value from it
* Give it CPU time
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
39. Fibonacci Generator
•
def fib(a,b):
while 1:
yield a
a, b = b, b + a
•
•
Demo
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
40. Fibonacci Generator
•
>>> def fib(a,b):
... while 1:
... yield a
... a, b = b, b + a
...
>>> g = fib(1,1)
>>> g
<generator object at 0xb7bf9c6c>
>>> [ g.next() for _ in range(10) ]
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
41. Lots of Generators
•
Using the same fib generator, make 10 of them:
>>> GS = [ fib(x,x) for x in range(10) ]
• And “run” them:
>>> [ G.next() for G in GS ]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [ G.next() for G in GS ]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [ G.next() for G in GS ]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
>>> [ G.next() for G in GS ]
[0, 3, 6, 9, 12, 15, 18, 21, 24, 27]
>>> [ G.next() for G in GS ]
[0, 5, 10, 15, 20, 25, 30, 35, 40, 45]
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
42. Generators as co-routines
•
def fib(a,b):
while 1:
yield 1 # Just to say “keep running me”
print a
a, b = b, b + a
•
•
Demo
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
43. Generators as co-routines
•
def printer(tag):
while 1:
yield 1 # Makes it a generator
print tag
•
•
Demo
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
44. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
45. Microprocesses
• * Generators with context
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
46. microprocess exercise
•
Write a class microprocess with methods:
• __init__(self)
* Takes no arguments
* Uses super to call superclass __init__ method
• main(self)
* No arguments
* Should yield 1
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
47. microprocess answer
•
• class microprocess(object):
def __init__(self):
super(microprocess, self).__init__()
def main(self):
yield 1
•
Generally, we'll skip answers in this presentation,
they're all in the web version of this tutorial here:
•
http://www.kamaelia.org/MiniAxon/
•
(and in the notes :-)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
48. microprocess usage
•
class printer(microprocess):
def __init__(self, tag):
super(printer, self).__init__()
self.tag = tag
def main(self):
while 1:
yield 1
print self.tag
•
• “Printer” isn't particularly interesting, but allows things like
components, but let's see how this can be used.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
50. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
51. Scheduler
•
* Something to run lots of microprocesses
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
52. scheduler exercise
•
Write a class scheduler with 3 methods:
• __init__(self)
* Uses super to call superclass __init__ method
* subclasses microprocess
* Creates 2 queues – active & newqueue
• main(self)
* Is a generator
* Runs the microprocesses activated
•
activateMicroprocess(self, someprocess):
* Calls somprocess.main()
* Adds generator to newqueue
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
53. scheduler exercise
• Scheduler logic
•
main(self)
* loops 100 times, yields 1 at start of loop
* loop through generators in self.active
* call their .next() method
* If result is not -1 and no StopIteration,
append to newqueue
* at end of loop, newqueue becomes active
* newqueue reset to empty list
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
54. scheduler usage
•
Using same printer class..
•
>>> X = printer("Hello World")
>>> Y = printer("Game Over") # :-)
• >>> myscheduler = scheduler()
>>> myscheduler.activateMicroprocess(X)
>>> myscheduler.activateMicroprocess(Y)
• >>> for _ in myscheduler.main():
... pass
• <prints Hello world/Game over repeatedly>
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
55. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
56. Component
•
• * microprocess with a standard interface.
• boxes as inboxes/outboxes
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
57. component exercise
•
Write a class component subclass of
microprocess with 4 methods:
•
__init__(self)
send(self, value, outboxname)
recv(self, inboxname)
dataReady(self, inboxname)
• Behaviour coming up!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
59. component exercise 2/4
•
send(self,value, outboxname) logic:
Finds the list named outboxname in self.boxes,
and appends value to it.
•
Before: X.send(“hello”, “outbox”)
{ “inbox”: [], “outbox”: [] }
• After:
{ “inbox”: [], “outbox”: [“hello”] }
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
60. component exercise 3/4
•
recv(self, inboxname) logic:
Finds the list named inboxname in self.boxes, and
pops the first value
• Given:
{ “inbox”: [“hello”, “world”], “outbox”: [] }
•
self.recv(“inbox”) returns, “hello” leaving...
{ “inbox”: [“world”], “outbox”: [] }
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
61. component exercise 4/4
•
dataReady(self, inboxname) logic:
Finds the list named inboxname in self.boxes:
returns the length of the list.
• Given:
{ “inbox”: [“hello”, “world”], “outbox”: [] }
•
dataReady(“inbox”) --> 2
(allows if self.dataReady(“inbox”) )
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
62. component usage
•
Until we add a means for data to get
from outboxes to inboxes, using
components is no more interesting than
microprocesses
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
63. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
64. Postman
• * Something to do deliveries
•
• Note:
This is more conceptual in real Axon,
but was real in v. old Axon. ie real
Axon is more efficient!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
65. postman exercise
•
Write a class postman subclass of microprocess
with 2 methods:
•
__init__(self, source, sourcebox,
sink, sinkbox)
main(self)
• Behaviour coming up!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
66. postman exercise 1/4
•
__init__(self, source, sourcebox,
sink, sinkbox) logic:
* Copy all the arguments as local attributes
•
* Ensure you call the superclass __init__ method
appropriately!
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
67. postman exercise 2/4
•
main(self) logic:
• In a loop:
• yield a non -1 value (eg 1)
Check if source's sourcebox has
dataReady
• If it has, use recv to collect it from there, and sink's
send method to deliver it.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
68. Mini Axon
•
• Using it!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
70. Producer/Consumer 2/3
•
• class Consumer(component):
def main(self):
count = 0
while 1:
yield 1
count += 1
if self.dataReady("inbox"):
data = self.recv("inbox")
print data, count
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
71. Producer/Consumer 3/3
•
• p = Producer("Hello World")
c = Consumer()
postie = postman(p, "outbox", c, "inbox")
• myscheduler = scheduler()
myscheduler.activateMicroprocess(p)
myscheduler.activateMicroprocess(c)
myscheduler.activateMicroprocess(postie)
•
for _ in myscheduler.main():
pass
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
72. Producer/Consumer Output
•
• Hello World 2
Hello World 3
Hello World 4
• ...
•
Hello World 96
Hello World 97
Hello World 98
•
•
Not 100, because of yields before start of scheduler
loop. (scheduler terminates early)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
73. Mini Axon
•
• Using it for more useful stuff
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
74. File Reader
•
• class FileReader(component):
• def __init__(self, filename):
super(FileReader, self).__init__()
self.file = open(filename, "rb",0)
•
def main(self):
yield 1
for line in self.file.xreadlines():
self.send(line, "outbox")
yield 1
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
78. Mini Axon --> Axon
•
• The rest is:
Syntactic Sugar
Ways of using it
Optimisations
(eg direct delivery – no postman)
+ a couple of other ideas we'll come
to (STM, threadedcomponents, services,
processcomponents)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
79. Questions?
•
• ... before we look how to build some
real components, and how to use them?
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
80. Part 1
•
• First part:
• - An overview of Kamaelia (lightning talk style : 5-10
mins)
• - How to build a mini Kamaelia (exercise 40-60 mins)
• - Starting building components & systems (remainder)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
81. Building a Video Recorder
•
• Before the break, we'll build a simple video recorder.
• Approach:
• 1 Build a webcam
• 2 Clean up the code for “normal” reuse
• 3 Componentise in least effort manner
• 4 Separate input/transform/display parts
• 5 Hook webcam up to a dirac encoder and filewriter
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
82. Step 1: Build a simple Webcam
• Pygame 1.9.1 alpha has basic Linux webcam support
which works nicely, so we're using that.
• First some initialisation
• import pygame
import pygame.camera
• pygame.init()
pygame.camera.init()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
84. Step 1: Build a simple Webcam
• Initialise the display, allocate a camera, and activate it
• display = pygame.display.set_mode(displaysize)
camera = pygame.camera.Camera(device,
capturesize)
camera.start()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
85. Step 1: Build a simple Webcam
• Loop round capturing images and display them
• while 1:
snapshot = camera.get_image()
snapshot = pygame.transform.scale(snapshot,
imagesize)
display.blit(snapshot, imageorigin)
pygame.display.flip()
•
•
Demo!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
86. Step 2: Clean up for reuse
• Given a collection of config options, wrapping this in a
class makes sense.
• As before these parts are unchanged:
• import pygame
import pygame.camera
• pygame.init()
pygame.camera.init()
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
87. Step 2: Clean up for reuse
• We then define a class, and put the config options as
class attributes:
• class VideoCapturePlayer(object):
• displaysize = (800, 600)
capturesize = (640, 480)
imagesize = (352, 288)
imageorigin = (0,0)
device = “/dev/video0”
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
88. Step 2: Clean up for reuse
• Put our initialisation in the __init__. Allow the user to
override our defaults
def __init__(self, **argd):
self.__dict__.update(**argd)
super(VideoCapturePlayer,self).__init__()
self.display = pygame.display.set_mode(self.displaysize)
self.camera = pygame.camera.Camera(self.device,
self.capturesize)
self.camera.start()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
89. Step 2: Clean up for reuse
• Wrap up body of loop in a method
def get_and_flip(self):
snapshot = self.camera.get_image()
snapshot = pygame.transform.scale(snapshot,
self.imagesize)
self.display.blit(snapshot, self.imageorigin)
pygame.display.flip()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
90. Step 2: Clean up for reuse
• Provide a hook to start it going, wrapping the main loop.
def main(self):
while 1:
self.get_and_flip()
•
Then run it!
VideoCapturePlayer().main()
•
•
Demo!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
91. Step 3: Componentise
•
• In this case, least effort approach is to use a threaded
component. Would could make it a generator component
later if needed.
• Changes:
• * imports
* class's baseclass / extraction of display_flip into a
method. (to simplify later integration with existing
components)
* how we run it.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
93. Step 3: Componentise
•
• Baseclass changes & initialiser changes:
class VideoCapturePlayer(threadedcomponent ):
...
def __init__(self, **argd):
# no longer update __dict__ here
super(VideoCapturePlayer,self).__init__(**argd )
...
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
94. Step 3: Componentise
•
• Extract display flipping out to a separate method:
def pygame_display_flip(self):
pygame.display.flip()
def get_and_flip(self):
snapshot = self.camera.get_image()
snapshot = pygame.transform.scale(snapshot,
self.imagesize)
self.display.blit(snapshot, self.imageorigin)
self.pygame_display_flip()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
95. Step 3: Componentise
•
• Change to how we run:
• We had:
•
VideoCapturePlayer().main()
• We now:
• VideoCapturePlayer().run()
•
•
Demo!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
96. Step 4: Split into input & output
•
• Now we can split the component in two:
• * VideoCaptureSource
• * Surface Displayer
• The Video capture source now needs to be self regulating,
so it needs to sleep during the loop, relative to a target
frame rate.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
97. Step 4: Split into input & output
•
• VideoCaptureSource class preamble:
• import time
class VideoCaptureSource(threadedcomponent):
capturesize = (352, 288)
delay = 0
fps = -1
device = “/dev/video0”
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
99. Step 4: Split into input & output
•
• Main loop body now just captures images, capturing a
reference
•
def capture_one(self):
•
self.snapshot = None
self.snapshot = self.camera.get_image()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
100. Step 4: Split into input & output
•
• Main loop still runs it, and sends the image out an
outbox, and then sleeps.
•
def main(self):
•
while 1:
self.capture_one()
self.send(self.snapshot, “outbox”)
time.sleep(self.delay)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
101. Step 4: Split into input & output
•
• The surface displayer takes the other code chunks
• from Axon.Component import component
•
class SurfaceDisplayer(component):
displaysize = (800,600)
imagesize = (352, 288)
imageorigin = (0,0)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
102. Step 4: Split into input & output
•
• Has it's own initialiser...
• def __init__(self, **argd):
super(SurfaceDisplayer, self).__init__(**argd)
self.display = pygame.display.set_mode(self.displaysize)
•
Retains the following method unchanged:
•
def pygame_display_flip(self):
pygame.display.slip()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
103. Step 4: Split into input & output
• Then mainbody waits for surfaces to display, sleeping
when there isn't any:
• def main(self):
•
while 1:
while self.dataReady(“inbox”):
snapshot = self.recv(“inbox”)
snapshot = pygame.transform.scale(snapshot,
self.imagesize)
self.display.blit(snapshot, self.imageorigin)
•
while not self.anyReady():
self.pause()
yield 1
•
yield 1
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
104. Step 4: Split into input & output
•
• We then join these back together in a pipeline:
•
from Kamaelia.Chassis.Pipeline import Pipeline
•
Pipeline(
•
VideoCaptureSource(),
•
SurfaceDisplayer(),
•
).run()
•
Demo!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
105. Step 5: Recording
• Working back...
•
• We want to write to a file... (SimpleWriter)
• We want to write dirac encoded video DiracEncoder
• That expects YUV Frames (ToYUV420_planar)
• That expects RGB data + metadata
PureTransformation of somr RGB data
• Which needs a picture source
•
Which is where we started.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
106. Step 5: Recording
• Our imports therefore need to add:
•
• from Kamaelia.File.Writing import SimpleFileWriter
• from Kamaelia.Codec.Dirac import DiracEncoder
• from Kamaelia.Video.PixFormatConversion import ToYUV420_planar
• from Kamaelia.Util.PureTransformer import PureTransformer
•
• And we delete everything related to display
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
107. Step 5: Recording
• We then need to join it all together:
•
Pipeline(
•
VideoCaptureSource(),
PureTransformer(lambda F :
{"rgb" : pygame.image.tostring(F, "RGB"),
"size" : (352, 288),
"pixformat" : "RGB_interleaved",
}),
ToYUV420_planar(),
DiracEncoder(preset="CIF", encParams={"num_L1":0}),
SimpleFileWriter("X.drc"),
• ).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
108. Step 5: Recording
•
• This then records dirac encoded video, which we can now
playback with a simple dirac player!
•
•
Demo !
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
109. Questions?
• ... before we break?
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
110. End of Part 1
•
• Before the second half, download and install Kamaelia
from the link below, if you haven't already.
•
• http://tinyurl.com/kot49x
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
111. Kamaelia:
Pragmatic Concurrency
A Tutorial
Michael Sparks
Europython '09, Birmingham UK
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
112. Part 2
•
• Second part:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems, embedding and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
113. Questions?
• ... before we carry on?
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
114. Part 2
•
• Second part:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
115. More advanced Stuff
• Have seen how to build a basic component and a basic
system - covers a wide set of problems. Now we
broaden the scope.
•
•
If we have time we'll come back to Kamaelia's STM model &
concepts of services, and (experimental) multiple process
support.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
116. Well behaved Shutdown
• Components should expect to receive these messages on their
“control” inbox, and behave as follows:
• Axon.Ipc.shutdownMicroprocess – if you receive this, you are
expected to shutdown immediately, and to pass this message on.
• Axon.Ipc.producerFinished – if you receive this, someone sending
you data has shutdown. You may want to shutdown depending on your
application's logic. You may process all outstanding messages in this
case. You may wish to pass this message on, or change it to
shutdownMicroprocess if it was unexpected.
• Due to different component needs no syntactic sugar exists for this,
but common cases are being discussed at present.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
122. Pipelines
• Seen how Pipelines can join components together.
• However also can do:
• Pipeline(
Lsystem(),
ConsoleEchoer(tag="n", forwarder=True),
Damage(),
circular = True,
).run()
•
To enable a feedback loop. (Demo)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
125. Graphlines
• Graphlines are like pipelines, but any shape. For
example, a simple presentation tool:
• Graphline(
CHOOSER = Chooser(items = files),
IMAGE = Image(size=(800,600), position=(8,48)),
NEXT = Button(caption="Next", msg="NEXT", position=(72,8)),
PREVIOUS = Button(caption="Previous", msg="PREV", position=(8,8)),
FIRST = Button(caption="First", msg="FIRST",position=(256,8)),
LAST = Button(caption="Last", msg="LAST",position=(320,8)),
linkages = {
("NEXT","outbox") : ("CHOOSER","inbox"),
("PREVIOUS","outbox") : ("CHOOSER","inbox"),
("FIRST","outbox") : ("CHOOSER","inbox"),
("LAST","outbox") : ("CHOOSER","inbox"),
("CHOOSER","outbox") : ("IMAGE","inbox"),
}
).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
126. Graphlines
•
• * Swiss Army Knife of Kamaelia components
• * Allows almost any topology
• - BUT an outbox may only link to one inbox
• - many outbox may link to one inbox
• * Can contain any component – including pipelines and
graphlines
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
132. Backplanes
•
• * For where you want 1 to many or many to many
• * Declare a backplane
• * Components can publish data to it
• * Components may subscribe to it.
• * Subscribers get a copy of all data sent
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
133. Backplanes
• Can be useful for updating a central state.
•
• For example players locations can be posted here:
• Backplane("PLAYERS").activate()
•
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
134. Backplanes
• Players Post their updates like this:
• Pipeline(
• MyGamesEventsComponent(up="p", down="l",
left="a", right="s"),
• BasicSprite("cat.png", name = "cat", border=40),
• PureTransformer(lambda x: ("Cat ", x)),
• PublishTo("PLAYERS"),
• ).activate()
•
•
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
135. Backplanes
• The system can analyse their updates like this:
• Pipeline(
• SubscribeTo("PLAYERS"),
• PlayerAnalyser(),
• Distancer(),
• ConsoleEchoer(),
• ).activate()
•
•
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
136. Backplanes & Servers
• When combined with a server, you instantly get a pub/
sub capable server, or splitter, or merger.
• Backplane(“SPLIT”).activate()
Pipeline(
DiracWebCam(),
PublishTo(“SPLIT”),
).activate()
def VideoProtocol(**argd): return SubscribeTo(“SPLIT”)
• ServerCore(protocol=VideoProtocol, port=1700).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
137. Backplanes & Servers
• When combined with a server, you instantly get a pub/
sub capable server, or splitter, or merger.
• Backplane(“CHAT”).activate()
def ChatProtocol(**argd):
return Pipelines(
SubscribeTo(“CHAT”)
NullComponent(),
PublishTo(“CHAT”)
)
• ServerCore(protocol=ChatProtocol, port=1700).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
138. PAR & Seq
• PAR & Seq are wrapper components with concepts
nabbed from Occam. (hence their names)
•
* Seq runs each component it's given, one after
another, wiring up inboxes/outboxes such that each
component handles it.
Useful in staged protocols (later)
•
* PAR runs all the components concurrently. Their
output is merged. Shutdown messages sent to PAR
are forwarded to all subcomponents – making it
useful for managing system shutdown.
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
139. Using PAR
• PAR can be used to simplify some systems:
• Pipeline(
PAR(
Button(caption="Next", msg="NEXT", position=(72,8)),
Button(caption="Previous", msg="PREV", position=(8,8)),
Button(caption="First", msg="FIRST" ,position=(256,8)),
Button(caption="Last", msg="LAST", position=(320,8)),
),
• Chooser(items = files),
Image(size=(800,600), position=(8,48)),
).run()
• This is the same slideshow as the previous Graphline...
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
140. Using PAR
• ... to run & shutdown multiple subsystems
• Pipeline(
timedShutdown(TTL=15),
PAR(
Pipeline(
ReadFileAdaptor(file, readmode="bitrate",
bitrate = 300000*8/5),
DiracDecoder(),
MessageRateLimit(framerate),
VideoOverlay(position=(260,48), size=(200,300)),
),
Pipeline(
ReadFileAdaptor(file, readmode="bitrate", bitrate = 2280960*8),
DiracDecoder(),
ToRGB_interleaved(),
VideoSurface(size=(200, 300), position=(600,48)),
),
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
142. Using Seq
• An example from the mobile reframer:
• Seq( "Decoding & separating frames...",
Graphline(
MAXF = DetermineMaxFrameNumber(edlfile),
DO = Carousel( ... ),
STOP = TriggeredOneShot(""),
...
),
"Processing edits...",
Graphline(
REFRAMING = ReframeVideo(edlfile...
SOUND = PassThroughAudio(edlfile...
ENCODING = ReEncode(outFileName...
...
),
"Cleaning up...",
StopSelector(),
).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
143. Using Seq
• An example of passing state in a protocol handler:
• def CompositeBulletinBoardProtocol(**argd):
ConnectionInfo = {}
ConnectionInfo.update(argd)
return Seq(
Authenticator(State = ConnectionInfo),
UserRetriever(State = ConnectionInfo),
MessageBoardUI(State = ConnectionInfo),
StateSaverLogout(State = ConnectionInfo),
)
ServerCore(protocol=CompositeBulletinBoardProtocol, ...)
• Worth noting the similarity between this and wsgi.
(Except Seq can contain graphlines, pipelines, etc too)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
144. Part 2
•
• Second part:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
145. Larger Systems & Debugging
•
Large systems which are long running can
develop bugs which are awkward to debug.
•
Kamaelia has a collection of tools you can use
which we'll walk through / demonstrate next.
•
• Practical example.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
148. Part 2
•
• Second part:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
149. Building a Bulletin Board
•
We're going to build an “old school” bulletin board
system.
• * Someone telnets in & authenticates
• * Can get help, exit or read messages
• * Reading messages, they can read, reply,
exit reading, or get help
• * State is restored/saved at session start/end
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
150. Building a Bulletin Board
•
For practicality, we won't implement message posting,
nor message deletion, nor persistent user state. (Though
these would be useful exercises)
• All on-disk objects encoded as json objects:
* users
* Messages – stored in “folders” (directories) with
filenames == messageid
• * message fields: from, to, __body__, date, message,
reply-to, subject
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
151. Building a Bulletin Board
•
Getting started:
• * Start with basic server for request/response
Just echo initially - BB1.py
• * make restarting quicker for debugging - BB2.py
* abstract out “getting a line of data” - BB3.py
* Then use that abstraction - BB4.py
* Simplify “control” box handling - BB5.py
* Abstract out reusable bits from domain specific – BB6.py
•
We'll go through the code for these in the actual code
files rather than on these slides
•
(rather unwieldy as slides)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
152. Building a Bulletin Board
•
Intermediate Plan
• This is the logic of the user protocol
• Seq(
•
Authenticator( <some shared state> ),
• UserRetriever( <some shared state> ),
• MessageBoardUI( <some shared state> ),
•
StateSaverLogout( <some shared state> ),
•
)
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
153. Building a Bulletin Board
•
Intermediate Plan
• * Tidy up control handling a touch more & add netPrint method
to reusable bit - BB7.py
• * Change Authenticator component to support the “passed on
state”, change protocol handler factory to create that Seq
pipeline. - BB8.py
•
* Change Authenticator to authenticate against a password file
and set “remoteuser” in the shared state - BB9.py
•
* Write stubs for UserRetriever & StateSaverLogout - BB10.py
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
154. Building a Bulletin Board
•
Final UI Plan
• * Change usage of waitMsg to yield self.waitMsg() -- BB11.py
• * Copy Authenticator's patten to create initial menu for
MessageBoardUI, including stub for messages menu - BB12.py
• * Use same pattern for messages menu, use waitMsg pattern
for logic. - BB13.py
• * Implement Folders to hold messages.
Numbers as filenames (msg 1, 2, 3, 4)
Messages as json encoded objects - BB14.py
•
* Clean up to add a line oriented buffer to handle partial lines.
(necessary for cross platform/real world) - BB15.py
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
155. Building a Bulletin Board
•
Final UI Plan
• * Change usage of waitMsg to yield self.waitMsg() -- BB11.py
• * Copy Authenticator's patten to create initial menu for
MessageBoardUI, including stub for messages menu - BB12.py
• * Use same pattern for messages menu, use waitMsg pattern
for logic. - BB13.py
• * Implement Folders to hold messages.
Numbers as filenames (msg 1, 2, 3, 4)
Messages as json encoded objects - BB14.py
•
* Clean up to add a line oriented buffer to handle partial lines.
(necessary for cross platform/real world) - BB15.py
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
156. Summary
•
We've covered
• * the 30,000 view of kamaelia
• * Building your own core
•
* Building components, including a video recording application
• * Tools for building systems
• * Tools for debugging systems
•
* Built a large(ish) system (an authenticated staged protocol)
•
* Illustrated the majority of this using real world systems
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
157. Final Questions?
•
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com