Graphics has been at the forefront of the resurgence in parallel computation. Real-time graphics and games have been the source of many of today’s new programming models and architectures for parallel computation. Modern games are arguably the only successful mainstream application of highly parallel programming in heterogeneous, million-line codebases. But while graphics is thought of as an embarrassingly parallel application, there has been little success in implementing high-performance graphics systems in any single general-purpose parallel programming model, ironically including those which have come from the GPGPU community. I will talk about key patterns of parallelism and locality used in graphics pipelines and games, and how existing tools and monolithic programming models fail to express these patterns with sufficient efficiency. I will try to synthesize some directions for future programming systems based on this experience, including my current thoughts for how a compile-time continuation passing transform could help formalize patterns emerging in how high performance systems are manually overcoming the limitations of existing GPU programming models. This talk will be at least as much informal, educational and speculative as it will be about any currently active research.

1. Why Graphics is Fast,
and what it can teach us about
parallel programming
Jonathan Ragan-Kelley, MIT
7 December 2009, University College London
13 November 2009, Harvard

2. (me)
PhD student at MIT
with Frédo Durand, Saman Amarasinghe
Previously
Stanford
Industrial Light + Magic
The Lightspeed Automatic Interactive Lighting Preview System
(SIGGRAPH 2007)
NVIDIA, ATI
Decoupled Sampling for Real-Time Graphics Pipelines
(ACM Transactions on Graphics 2010)
Intel

3. 1 game frame

4. via Johan Andersson, DICE

5. via Johan Andersson, DICE

6. Game Engine Parallelism
Each task (~200-300/frame) is potentially:
data parallel
physics, sound, AI
image post-processing, streaming/decompression
pipeline parallel
(internally) task parallel
entire invocation of the graphics pipeline
↳ braided parallelism

7. The Graphics Pipeline

8. Vertex Buffer
Input Assembler
Index Buffer
Texture Vertex Shader
memory Primitive Assembler
Texture Geometry Shader
Setup/Rasterizer
Texture
Pixel Shader
Color Buffer
Depth Buffer Output Merger

9. Vertex Buffer
Input Assembler
Index Buffer
Texture Vertex Shader
memory Primitive Assembler
Texture Geometry Shader
Setup/Rasterizer
Texture
Pixel Shader
Color Buffer
Depth Buffer Output Merger

10. Vertex Buffer
Input Assembler
Index Buffer
Texture Vertex Shader
memory Primitive Assembler
Texture Geometry Shader
Setup/Rasterizer
Texture
Pixel Shader
Color Buffer
Depth Buffer Output Merger

11. Vertex Buffer
Input Assembler
Index Buffer
Texture Vertex Shader
memory Primitive Assembler
Texture Geometry Shader
Setup/Rasterizer
Texture
Pixel Shader
Color Buffer
Depth Buffer Output Merger

12. Vertex Buffer
Input Assembler
Index Buffer
Texture Vertex Shader
memory Primitive Assembler
Texture Geometry Shader
Setup/Rasterizer
Texture
Pixel Shader
Color Buffer
Depth Buffer Output Merger

13. Vertex Buffer
Input Assembler
Index Buffer
Texture Vertex Shader
memory Primitive Assembler
Texture Geometry Shader
Setup/Rasterizer
Texture
Pixel Shader
Color Buffer
Depth Buffer Output Merger

14. Vertex Buffer
Input Assembler
Index Buffer
Texture Vertex Shader
memory Primitive Assembler
Texture Geometry Shader
Setup/Rasterizer
Texture
Pixel Shader
Color Buffer
Depth Buffer Output Merger

15. Vertex Buffer
Input Assembler 2%
Index Buffer
Texture Vertex Shader 10%
memory Primitive Assembler 2%
Texture Geometry Shader 8%
Setup/Rasterizer 8%
Texture
Pixel Shader 50%
Color Buffer
Depth Buffer Output Merger 20%

16. Fast Implementation
90% resource utilization
Massive data parallelism
Fine-grained dynamic load balance
Pipeline parallelism
Producer-consumer locality
Global dependence analysis, scheduling
Efficient fixed-function

17. Local Parallelism

18. Shader Execution Model
Highly data-parallel
vertices, primitives, fragments, pixels
Hierarchical parallelism
Instruction bandwidth: SIMD ALUs
Pipeline latency: vector execution/software pipelining
Unpredictable latency: hardware multithreading
Memory latency: dynamic threading/fibering
Task independence: multicore
Regular input, output
Marshaling/unmarshaling from data structures
handled in “fixed-function” for efficiency

19. Shader Implementation
Hardware shader architecture (GPU)
static SIMD warps
many dynamic hardware threads, tens of cores
fine-grained dynamic load balance
many kernels, types simultaneously
heuristic knowledge of specific pipeline
Software shader architecture (Larrabee)
similar static SIMD, inside dynamic latency-hiding,
inside many-core
threading via software fibering (microthreads),
software scheduling

20. Shader Stage-specific Variation
Vertex
less latency hiding needed,
so use less local memory,
[perhaps] no dynamic fibering (simpler schedule).
Geometry
variable output.
more state = less parallelism,
but (hopefully) need less latency hiding.
Fragment
derivatives → neighborhood communication.
texture intensive → more (dynamic) latency hiding.

21. Fixed-function Stages
[Input,Primitive] Assembly
Hairy finite state machine
Indirect indexing
Rasterization
Locally data-parallel (~16xSIMD), No memory access
Globally branchy, incremental, hierarchical
Output Merge
Really data-parallel, but ordered read-modify-write
Implicit data amplification (for antialiasing)

22. Pipeline Implementation

23. Ordering Semantics
Logically sequential
input primitive order defines framebuffer update order
Pixels are independent
spatial parallelism in framebuffer
Buffer to reorder
otherwise
(Could be looser in custom pipelines)

24. Sort-Last Fragment
Shaders fully parallel
buffer to reorder between stages (logically FIFO)
100s-1,000s of triangles in-flight
Output Merger is screen-space parallel
crossbar from Pixel Shade to Output Merge (“sort-last”)
fine-grained scoreboarding
Full producer-consumer locality
all inter-stage communication on-chip
primitive-order processing
framebuffer cache filters some read-modify-write b/w

25. Sort-Middle
Front-end: transform/vertex processing
Scatter all geometry to screen-space tiles
Merge sort (through memory)
Maintain primitive order
Back-end: pixel processing
In-order (per-pixel)
scoreboard per-pixel → screen-space parallelism
One tile per-core
framebuffer data in local store
one scheduler, several worker [hardware] threads collaborating
Pixel Shader + Output Merge together
lower scheduling overhead

26. return to task system
to put back in
context.
reinforce braided
nature.
Game Frame

28. Existing Models

29. Parallel Programming Models
Task parallelism (Cilk, Jade, etc.)
dynamic, heavyweight items
very flexible, high overhead
Data parallelism (NESL, FortranX, C*)
dynamic, lightweight items
flexible, moderate overhead
Streaming (StreamIt, Brook)
static data + pipeline parallelism
very low overhead, but inflexible ➞ load imbalance

30. CUDA, GPGPU
Canonical model Key challenges
thread = work-item
one kernel at a time
} dynamic load balance
purely data parallel
streaming data access
} producer-consumer
locality

31. Future directions

32. Task decomposition continuing to grow
Past: subsystem threads: 2-3 threads utilized
Present: task systems: 6-8 threads, some headroom
Future:
many more tasks, reduce false dependence
orchestration language?
More data, braided parallelism within tasks
Essential to extracting FLOPS in future throughput chips
“Über-kernels”
(NVIDIA ray tracing, Reyes demos; OptiX system)
Dynamic task scheduling inside a single-kernel,
purely data-parallel architecture

33. Über-kernels
stage 1 stage_1():
// do stage 1…
stage_2():
stage 2 // do stage 2…
stage_3():
stage 3 // do stage 3…

34. Über-kernels
while(true):
stage 1
state = scheduler()
switch(state):
case stage_1:
stage 2 // do stage 1…
case stage_2:
// do stage 2…
case stage_3:
stage 3 // do stage 3…

35. Über-kernels
while(true):
stage 1
state = scheduler()
switch(state):
case stage_1:
stage 2 // do stage 1…
case stage_2:
// do stage 2…
case stage_3:
stage 3 // do stage 3…

36. Über-kernels
while(true):
state = scheduler()
stage 1 switch(state):
case stage_1:
// do stage 1…
case stage_2_1:
stage 2 // beginning of 2…
case stage_2_2:
// end of 2…
stage 3 case stage_3_1:
// beginning of 3…
case stage_3_2:
// end of 3…

37. The Ray Tracing Pipeline
Host
Buffers
Texture Samplers
Entry Points Variables
Ray Generation Program
Exception Program
Traversal
Intersection Program
Trace Any Hit Program
Selector Visit Program
Ray Shading
Closest Hit Program
Miss Program

38. Explicit continuation programs
stage_1():
non_blocking_fetch()
stage 1 if(c): recurse(s_1)
else: tail_call(s_2)
non_blocking_fetch():
stage 2
prefetch()
call/cc(myScheduler)
return result
stage 3
recurse(s):
yield(myScheduler)
s()

39. Acknowledgements
CPS ideas
Tim Foley, Mark Lacey, Mark Leone - Intel
General Discussion
Saman Amarasinghe, Frédo Durand - MIT
Solomon Boulos, Kurt Akeley - Stanford/MSR
OptiX details, slides
Austin Robison, Steve Parker - NVIDIA
Current game-engine details, figures
Johan Andersson - DICE

40. extras

41. Need
Locality
across many (potentially competing) axes
application-specific scheduling knowledge
autotuning for complex optimization space?
Braids of parallelism
data-parallelism
static - SIMD, extreme arithmetic intensity
dynamic - latency hiding
task parallelism
static - pipelines)
dynamic - to add dynamism to data-parallel layer

42. Key issues
Hierarchy
Braided decomposition
Managing complexity
separating orchestration from kernel implementation
recursively, at multiple levels
Hybrid static-dynamic
extreme arithmetic intensity + dynamic load balance
Hybrid pipeline and data-parallel
competing axes of locality

43. Task Optimization

44. Draw Batch
Group of primitives bound to common state
JIT compile, optimize on rebinding state
changing any shader stage
potential inter-stage elimination of dead outputs
late-binding constant folding
changing some fixed-function mode state
not on changing inputs (too expensive)
“near-static” compilation

45. Rendering Pass
Common output buffer binding
with no swap/clear
Synthesize renderer from canonical pipeline
render shadow, environment maps
2D image post-processing
Optimize performance
avoid wasted work (Z cull pre-pass, deferred shading)
reorder/coalesce batches (deferred lighting)

46. Inter-pass Optimization
Buffers never consumed
keep in scratchpad
“fast clear”
use optimized/native formats
resolve antialiasing before write-out
Passes overlapped
no startup/wind-down bubbles
Pass folding
e.g. merge 2D post-processing into back-end stage,
on local tile memory

47. Control vs. Data-Dependence
Simple task graph has control dependence
between stages
D3D scheduling based on resource (data)
dependence, not control dependence
‣Finer-grained scheduling
‣Fewer false hazards
Generalize to whole engine core?
AI, sound
Physics: rigid, soft, IK, cloth, particles, fracture

48. Core principles
TODO: update
Hierarchical parallelism
Shader-style data-parallel kernels
Kernel fusion/global optimization
Separate kernel and orchestration
Separate-but-connected languages/runtime models
Resource-based scheduling
Application-specific/user-controlled
scheduling and memory management