This document summarizes concurrency and parallel computing in Julia. It discusses tasks/coroutines, channels for inter-task communication, and parallel for loops and maps for distributing work across processes. Key points: - Tasks allow suspending/resuming execution without context switching and can run in any order. - Channels provide synchronous inter-task communication via put! and take!. - The @spawn and @spawnat macros create remote tasks on other processes. - @parallel distributes loop iterations or map tasks across processes for parallel execution. - Orchestration involves coordinating local and remote asynchronous/parallel tasks.

1. Concurrency in Julia
Julia Taiwan發起人 杜岳華

2. Outline
• Multitasking
• Concurrency
• Easy parallel
• Parallel computing
• Orchestration

3. Multitasking

4. Task
• Aka coroutine
• 類似function，但是可以執行到一半中斷並回傳值，去執行別的Task
julia> f(x) = println(x + 10)
julia> task = Task(f) # no argument
julia> task2 = @task f(2)

5. Task
julia> task2
Task (runnable) @0x00007fdab2199f90
julia> yield(task2)
12
julia> task2
Task (done) @0x00007fdab2199f90
Main() task2
yield(task2)

6. Task
• 與function不同:
• 切換task並不使用其他空間，只會使用存在的process的空間
• CPU不會意識到task的存在，不會做context switch
• Task之間的切換可以是任意順序

7. Single task
julia> function print_to(n)
for i in 1:n
println(i)
end
end
print_to (generic function with 1 method)
julia> foo = @task print_to(100)
Task (runnable)

8. Single task
julia> yield(foo)
1
2
3
…
100
julia> foo
Task (done)

9. Use wait() to pause
julia> function print_to(n)
for i in 1:n
println(i)
wait() # set task state
to :waiting and pause
end
end
print_to (generic function with 1 method)
julia> foo = @task print_to(100)
Task (runnable)

10. Use wait() to pause
julia> yield(foo)
1
julia> yield(foo)
2
julia> yield(foo)
3
julia> foo
Task (runnable)

11. Scheduling
function i_print_to(i, n)
for j in 1:n
println(“Task $(i) print $(j)”)
yield() # switch to scheduler
end
end

12. Scheduling
• @schedule: wrap expression as a Task and add to scheduler
@schedule i_print_to(1, 10)
@schedule i_print_to(2, 5)
@schedule i_print_to(3, 30)

13. Scheduling
Task 1 print 1
Task 2 print 1
Task 3 print 1
Task 1 print 2
Task 2 print 2
Task 3 print 2
Task 1 print 3
…
Task 2 Task 3scheduler Task 1
yield()
yield()
yield()

14. Task state
:runnable
:waiting
:queued :failed
:done

15. Multitasking
Task 2 Task 3scheduler Task 1
id = 1
yield(task1)
yield()
yield()
yield(task2)
yield(task3)

16. Concurrency

17. Concurrency
• A way of flow control
• A way to program
• Concurrency is about dealing with lots of things at once.
• Parallelism is about doing lots of things at once.
• Concurrency is not parallelism -- Rob Pike

18. Concurrency

19. Communicating Sequential Processes
• 善用Task及Channel
• 避免用Shared Array，或是其他share state objects
• 避免race condition
• 避免deadlock
• 不需要lock or synchronization
take!
put!
take!
take!
put!
put!

20. Communicating Sequential Processes
task
task
task
task

21. Channel
• Inter-task communication
• Synchronous (blocking)
ch = Channel{Int64}(50)
put!(ch, 5)
x = take!(ch)
isready(ch) # test if anything in it
close(ch)

22. Channel
open take!put!
closed take!

23. Channel is iterable
ch = Channel{Int64}(100)
for i in 1:100
put!(ch, i)
end
for i in ch
println(i)
end

24. Producer-consumer model
put!
task
take! task
task
task
task
task

25. Channel can be accessed by multi-task
function producer(ch)
for i in 1:100
put!(ch, i)
yield()
end
end
function consumer(ch)
while true
x = take!(ch)
println(x)
yield()
end
end
ch = Channel{Int64}(100)
for _ in 1:10
@schedule producer(ch)
@schedule consumer(ch)
end

26. Concurrency example
function f1(ch1)
for i in 1:100
put!(ch1, i)
end
end
function f2(ch1, ch2)
while true
x = take!(ch1)
y = x^2 + 1
put!(ch2, y)
yield()
end
end
ch1 = Channel{Int64}(100)
ch2 = Channel{Int64}(100)
ch3 = Channel{Tuple}(100)
function f3(ch2, ch3)
while true
x = take!(ch2)
y = log(x)
put!(ch3, (x, y))
yield()
end
end

27. You can bind channel to task
• Binding the lifetime of channel to specific task
• When a channel is bound to multiple tasks, the first task to terminate
will close the channel
t1 = @schedule f1(ch1)
t2 = @schedule f2(ch1, ch2)
t3 = @schedule f3(ch2, ch3)
bind(ch1, t1) # Optional
bind(ch2, t2)
bind(ch3, t3)

28. Concurrency example
• @async
• wrap expression as a Task
• put it to the scheduling queue
• adds it to the nearest enclosing set that @sync
waits for
• @sync
• Wait for all enclosed uses of @async, @spawn,
@spawnat and @parallel are complete
@sync begin
@async f1(ch1)
@async f2(ch1, ch2)
@async f3(ch2, ch3)
end

29. macro tips
• @task
• wrap expression as a Task
• @schedule
• wrap expression as a Task
• put it to the scheduling queue
• @async
• wrap expression as a Task
• put it to the scheduling queue
• adds it to the nearest enclosing set that @sync waits for

30. Events
cond = Condition()
function waiter(cond)
while true
x = wait(cond) # task
change state to :waiting
# do something
yield()
end
end
function notifyer(cond)
while true
x = f()
notify(cond, x) # change
waiter state to queued and pass x
to waiter
yield()
end
end

31. Easy parallel

32. Easy parallel
• Julia提供了非常簡單的parallel介面
• Julia的parallel primitives設計成類似於function call的方式
future = remotecall(rand, 2, 1000, 1000)
result = fetch(future)

33. @spawn
future = @spawnat 2 rand(1000, 1000)
result = fetch(future)
future = @spawn rand(1000, 1000)
result = fetch(future)
future = remotecall(rand, 2, 1000, 1000)
result = fetch(future)

34. Remote API
• remotecall(f, proc_id, args; kwargs) -> Future
• call function asynchronously
• remotecall_fetch(f, proc_id, args; kwargs)
• fetch(remotecall()), context switch
• remotecall_wait(f, proc_id, args; kwargs)
• wait(remotecall())
• remote_do(f, proc_id, args; kwargs) -> Void
• call function asynchronously but return nothing

35. Code accessibility
• Code should be accessible on execution processes.
• `include(“Module.jl”)`: only load file into current process
• `using Module`: load module to every process, but is brought into scope only on
current process
• `@everywhere`: directly define expressions on all processes
@everywhere using Module

36. Data movement
• Explicitly
• fetch
• Implicitly
• @spawn
r = @spawn rand(2,2)
s = @spawn 1 .+ fetch(r)
results = fetch(s)
@everywhere x = 10

37. @spawn API
• @spawnat p expr -> Future
• Create a closure around the expression and run it on specified process
• @spawn expr -> Future
• Create a closure around the expression and run it on automatically-chosen
process

38. Parallel for loop
• Equally distribute tasks into worker processes
• Suitable for lightweight task but huge amount of iterations
@parallel for i in 1:10000
# do something
end
result = @parallel (+) for i in 1:10000
y = f(i)
y
end
results = @parallel (vcat) for i in 1:10000
y = f(i)
[y]
end

39. Parallel map
• Run each heavy_work(args) in a process
args = [i for i in 1:1000]
result = pmap(heavy_work, args)

40. Parallel computing

41. Parallel computing
• Only one process
• myid() == 1 is also a worker
• Multiple processes
• Worker
• myid() == 1 is not a worker process

42. Run tasks on worker processes
func = [f1, f2, f3]
@sync begin
for (i, f) in zip(workers(), func)
@async remotecall(f, i)
end
end

43. RemoteChannel is required
• Channel: use with in local process
• RemoteChannel: pass data to remote process
• Composed of RemoteRef and Channel
ch1 = RemoteChannel(() -> Channel{Int64}(32))
ch2 = RemoteChannel(() -> Channel{Int64}(32))
ch3 = RemoteChannel(() -> Channel{Int64}(32))

44. Future is a specialized RemoteChannel
id =
1
worker
Channel{Any}(1)
Channel{?}(?)
RemoteRef
RemoteRef
Future
RemoteChannel

45. Orchestration

46. Orchestration
Task 2 Task 3scheduler Task 1
id =
1
worker worker
Local Remote
Task 4
worker
ASYNCSYNC
ASYNC I/O

47. Which macro make tasks run remotely?
• Local
• @async
• Remote
• @spawn
• @spawnat
• Hybrid
• @parallel

48. Thank you for attention