While Ruby is known for its flexibility due to high mutability and meta-programming capability, these features make writing thread-safe programs using manual locking very error-prone. For this reason some people are switching to languages with easier to manage concurrency paradigms, such as Erlang/Scala’s message passing, or Clojure/Haskell’s Software Transactional Memory (STM). This talk is about Dataflow, a pure Ruby gem that adds dataflow variables to the Ruby language. Dataflow variables are write-once (or write multiple times with the same value), and suspend execution in the current thread/context if called before being assigned/bound. We will explore how this technique makes writing concurrent but thread-safe code easy, even making it possible to write tests that spawn threads without needing to worry. Declarative concurrency is a relatively unknown programming model that is an alternative to message passing and STM. Ruby’s malleability makes it an ideal host for this model. Besides performance implications, dataflow variables also have an important impact on declarative program modeling. The talk will also go over the differences in performance and memory of the library in various Ruby implementations.

1. Dataflow
The declarative concurrent
programming model

2. Larry Diehl
{:larrytheliquid =>
%w[.com
github
twitter]}

3. Outline
Purpose of presentation
Gradual explanation of concepts
Helpful tips

4. Purpose

5. Lexical Scope
foo = :foo
define_method :foo do
foo
end

6. Dynamic Scope
def foo
@foo
end

7. Mutability
def initialize
@foo = :foo
end
def foo
@foo
end

8. Mutability
def foo
@foo = :foo
@foo
end

9. Mutability+Concurrency
def initialize
Thread.new { loop { @foo = :shazbot } }
end
def foo
@foo = :foo
@foo
end

10. The Declarative Model

11. Declarative Synchronous
my_var = :bound
my_var = :rebind # NOT ALLOWED!

12. Declarative Synchronous
local do |my_var|
my_var.object_id # thread sleeps
end

13. Declarative Synchronous
local do |my_var|
unify my_var, :bound
unify my_var, :rebind # =>
# Dataflow::UnificationError,
# ":bound != :rebind"
end

14. Declarative Synchronous
class MyClass
declare :my_var
def initialize
unify my_var, :bound
end
end

15. Declarative Concurrent
(MAGIC)

16. Declarative Concurrent
local do |my_var|
Thread.new { unify my_var, :bound }
my_var.should == :bound
end

17. Dependency Resolution
local do |sentence, middle, tail|
Thread.new { unify middle, "base are belong #{tail}" }
Thread.new { unify tail, "to us" }
Thread.new { unify sentence, "all your #{middle}" }
sentence.should == "all your base are belong to us"
end

18. Asynchronous Output
def Worker.async(output=nil)
Thread.new do
result = # do hard work
unify output, result if output
end
end
local do |output|
Worker.async(output)
output.should == # hard work result
end

19. Asynchronous Output
local do |output|
flow(output) do
# do hard work
end
output.should == # hard work result
end

20. Anonymous variables
{'google.com' => Dataflow::Variable.new,
'bing.com' => Dataflow::Variable.new
}.map do |domain,var|
Thread.new do
unify var, open("http://#{domain}").read
end
var
end

21. need_later
%w[google.com bing.com].map do |domain|
need_later { open("http://#{domain}").read }
end

22. Chunked Sequential Processing
(1..100).each_slice(10).map do |chunk|
sleep(1)
chunk.inject(&:+)
end.inject(&:+) # => ~10s

23. Chunked Parallel Processing
(1..100).each_slice(10).map do |chunk|
need_later do
sleep(1)
chunk.inject(&:+)
end
end.inject(&:+) # => ~1s

24. Leaving Declarative
via Async

25. Ports & Streams
local do |port, stream|
unify port, Dataflow::Port.new(stream)
port.send 1
port.send 2
stream.take(2).should == [1, 2]
end

26. Ports & Streams (async)
local do |port, stream|
unify port, Dataflow::Port.new(stream)
Thread.new do
stream.each do |message|
puts "received: #{message}"
end
end
%w[x y z].each do |letter|
Thread.new{ port.send letter }
end
stream.take(3).sort.should == %w[x y z]
end

27. FutureQueue
local do |queue, first, second, third|
unify queue, FutureQueue.new
queue.pop first
queue.pop second
queue.push 1
queue.push 2
queue.push 3
queue.pop third
[first, second, third].should == [1, 2, 3]
end

28. Actors
Ping = Actor.new { Pong = Actor.new {
3.times { 3.times {
case receive case receive
when :ping when :pong
puts "Ping" puts "Pong"
Pong.send :pong Ping.send :ping
end end
} }
} }
Ping.send :ping

29. by_need
def baz(num)
might_get_used = by_need { Factory.gen }
might_get_used.value if num%2 == 0
end

30. Tips

31. Modular
local do |my_var|
Thread.new { unify my_var, :bound }
# my_var.wait
my_var.should == :bound
end

32. Debugging
local do |my_var|
my_var.inspect
# => #<Dataflow::Variable:2637860 unbound>
end

33. Class/Module methods
Dataflow.local do |my_var|
Dataflow.async do
Dataflow.unify my_var, :bound
end
my_var.should == :bound
end

34. Use Cases
general purpose
concurrency for elegant program structure with respect
to coordination
concurrency to make use of extra processors/cores
(depending on Ruby implementation)
web development
worker daemons
concurrently munging together data from various rest
api's

35. Ruby Implementations
Pure Ruby library, should work on any implementation
JRuby in particular has a great GC, no GIL, native threads,
and a tunable threadpool option.
Rubinius has more code written in Ruby, so it proxies more
method calls (e.g. Array#flatten).

36. class FutureQueue
include Dataflow
declare :push_port, :pop_port
def initialize
local do |pushed, popped|
unify push_port, Dataflow::Port.new(pushed)
unify pop_port, Dataflow::Port.new(popped)
Thread.new {
loop do
barrier pushed.head, popped.head
unify popped.head, pushed.head
pushed, popped = pushed.tail, popped.tail
end
}
end
end
def push(x) push_port.send x end
def pop(x) pop_port.send x end
end

37. The End
sudo port install dataflow
http://github.com
/larrytheliquid
/dataflow
freenode: #dataflow-gem