1. Operating Systems
CMPSCI 377
Concurrency Patterns
Emery Berger
University of Massachusetts Amherst
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

2. Finishing Up From Last Time
Avoiding deadlock: is this ok?

lock (a); lock (b);
lock (b); lock (a);
unlock (b); unlock (a);
unlock (a); unlock (b);
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 2

3. Finishing Up From Last Time
Not ok – may deadlock.

lock (a); lock (b);
lock (b); lock (a);
unlock (b); unlock (a);
unlock (a); unlock (b);
Solution: impose canonical order (acyclic)

lock (a); lock (a);
lock (b); lock (b);
unlock (b); unlock (b);
unlock (a); unlock (a);
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 3

4. Motivating Example: Web Server
web
server
Client (browser)

Requests HTML, images

not found
Server

Caches requests

http://server/Easter-bunny/
Sends to client
 200x100/75.jpg
client
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

5. Possible Implementation
while (true) {
wait for connection;
read from socket & parse URL;
look up URL contents in cache;
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 5

6. Possible Implementation
while (true) {
wait for connection; // net
read from socket & parse URL; // cpu
look up URL contents in cache; // cpu
if (!in cache) {
fetch from disk / execute CGI;//disk
put in cache; // cpu
}
send data to client; // net
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 6

7. Problem: Concurrency
web
Sequential fine until:
 server
More clients

Bigger server

Multicores, multiprocessors

Goals:

Hide latency of net & disk I/O

Don’t keep clients waiting

Improve throughput

Serve up more pages

clients
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science

8. Building Concurrent Apps
Patterns / Architectures

Thread pools

Producer-consumer

“Bag of tasks”

Worker threads (work stealing)

Goals:

Minimize latency

Maximize parallelism

Keep progs. simple to program & maintain

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 8

9. Thread Pools
Thread invocation & destruction relatively

expensive
Instead: use pool of threads

When new task arrives, get thread from pool

to work on it; block if pool empty
Faster with many tasks

Limits max threads

( ThreadPoolExecutor class in Java)

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 9

10. Producer-Consumer
Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

producer consumer
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 10

11. Producer-Consumer
Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

producer consumer
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 11

12. Producer-Consumer
Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

producer consumer
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 12

13. Producer-Consumer
Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

producer consumer
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 13

14. Producer-Consumer
Can get pipeline parallelism:

One thread (producer) does work

E.g., I/O

and hands it off to other thread (consumer)

producer consumer
LinkedBlockingQueue
Blocks on put() if full, poll() if empty
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 14

15. Producer-Consumer Web Server
Use 2 threads: producer & consumer

queue.put(x) and x = queue.poll();

while (true) { while (true) {
wait for connection; do something…
read from socket & parse URL; queue.put (x);
look up URL contents in cache; }
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
while (true) {
send data to client;
x = queue.poll();
}
do something…
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 15

16. Producer-Consumer Web Server
Pair of threads – one reads, one writes

while (true) { while (true) {
wait for connection; URL = queue.poll();
read from socket & parse URL; look up URL contents in cache;
queue.put (URL); if (!in cache) {
} fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 16

17. Producer-Consumer Web Server
More parallelism –

optimizes common case (cache hit)
while (true) { while (true) {
wait for connection; URL = queue1.poll();
read from socket & parse URL; look up URL contents in cache;
queue1.put (URL); if (!in cache) {
1
} queue2.put (URL); return;
}
send data to client;
}
2
while (true) {
URL = queue2.poll();
fetch from disk / execute CGI;
put in cache;
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 17

18. When to Use Producer-Consumer
Works well for pairs of threads

Best if producer & consumer are symmetric

Proceed roughly at same rate

Order of operations matters

Not as good for

Many threads

Order doesn’t matter

Different rates of progress

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 18

19. Producer-Consumer Web Server
Have to be careful to balance load across

threads
while (true) { while (true) {
wait for connection; URL = queue1.poll();
read from socket & parse URL; look up URL contents in cache;
queue1.put (URL); if (!in cache) {
1
} queue2.put (URL);
}
send data to client;
}
2
while (true) {
URL = queue2.poll();
fetch from disk / execute CGI;
put in cache;
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 19

20. Bag of Tasks
Collection of mostly independent tasks

worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 20

21. Bag of Tasks
Collection of mostly independent tasks

worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 21

22. Bag of Tasks
Collection of mostly independent tasks

worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 22

23. Bag of Tasks
Collection of mostly independent tasks

worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 23

24. Bag of Tasks
Collection of mostly independent tasks

worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 24

25. Bag of Tasks
Collection of mostly independent tasks

worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 25

26. Bag of Tasks
Collection of mostly independent tasks

addWork
worker worker worker worker
Bag could also be LinkedBlockingQueue

(put, poll)
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 26

27. Bag of Tasks Web Server
Re-structure this into bag of tasks:

addWork & worker threads

t = bag.poll() or bag.put(t)

while (true) {
wait for connection;
read from socket & parse URL;
look up URL contents in cache;
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 27

28. Bag of Tasks Web Server
Re-structure this into bag of tasks:

addWork & worker

t = bag.poll() or bag.put(t)

addWork: Worker:
while (true) { while (true) {
wait for connection; URL = bag.poll();
bag.put (URL); look up URL contents in cache;
} if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 28

29. Bag of Tasks Web Server
Re-structure this into bag of tasks:

t = bag.poll() or bag.put(t)

addWork: while (true){
wait for connection;
bag.put (URL);
} worker
worker: while (true) {
addWork URL = bag.poll();
look up URL contents in cache;
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
worker worker }
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 29

30. Bag of Tasks vs. Prod/Consumer
Exploits more parallelism

Even with coarse-grained threads

Don’t have to break up tasks too finely

Easy to change or add new functionality

But: one major performance problem…

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 30

31. What’s the Problem?
addWork
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 31

32. What’s the Problem?
Contention – single lock on structure

Bottleneck to scalability

addWork
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 32

33. Work Queues
Each thread has own work queue (deque)

No single point of contention

executor executor executor executor
Threads now generic “executors”

Tasks (balls): blue = parse, yellow = connect…

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 33

34. Work Queues
Each thread has own work queue (deque)

No single point of contention

executor executor executor executor
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 34

35. Work Queues
Each thread has own work queue (deque)

No single point of contention

executor executor executor executor
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 35

36. Work Queues
Each thread has own work queue (deque)

No single point of contention

executor executor executor executor
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 36

37. Work Queues
Each thread has own work queue

No single point of contention

executor executor executor executor
Now what?

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 37

38. Work Stealing
When thread runs out of work,

steal work from random other thread
worker worker worker worker
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 38

39. Work Stealing
When thread runs out of work,

steal work from top of random deque
worker worker worker worker
Optimal load balancing algorithm

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 39

40. Work Stealing Web Server
Re-structure:

readURL, lookUp, addToCache, output
myQueue.put(new readURL (url))

while (true) {
wait for connection;
read from socket & parse URL;
look up URL contents in cache;
if (!in cache) {
fetch from disk / execute CGI;
put in cache;
}
send data to client;
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 40

41. Work Stealing Web Server
Re-structure:

readURL, lookUp, addToCache, output
myQueue.put(new readURL (url))

readURL(url) {
wait for connection;
read from socket & parse URL;
myQueue.put (new lookUp (URL));
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 41

42. Work Stealing Web Server
Re-structure:

readURL, lookUp, addToCache, output
myQueue.put(new readURL (url))

readURL(url) { lookUp(url) {
wait for connection; look up URL contents in cache;
read from socket & parse URL; if (!in cache) {
myQueue.put (new lookUp (URL)); myQueue.put (new addToCache (URL));
} } else {
myQueue.put (new output(contents));
}
addToCache(URL) {
}
fetch from disk / execute CGI;
put in cache;
myQueue.put (new
output(contents));
}
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 42

43. Work Stealing
Works great for heterogeneous tasks

Convert addWork and worker into units of

work (different colors)
Flexible: can easily re-define tasks

Coarse, fine-grained, anything in-between

Automatic load balancing

Separates thread logic from functionality

Popular model for structuring servers

UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 43

44. The End
UNIVERSITY OF MASSACHUSETTS AMHERST • Department of Computer Science 44