April 26, 2017 talk at Austin Waldorf School

1. Computers, Graphics, Engineering,
Math, and Video Games
for High School Students
Mark Kilgard
NVIDIA
April 26, 2017

2. About Me
• Working at NVIDIA
– As Principal System Software Engineer
– Focused on real-time computer graphics
– For 19+ years
– Participated in the specification, standardization, and
implementation of OpenGL
• Popular real-time graphics API for 3D graphics
• It’s in all your cell phones & web browsers
– Before NVIDIA, worked at Silicon Graphics
• Education
– Rice University
– Computer Science & Mathematical Science

3. Along the way, wrote two books

4. Video Games as Technology
Physics & Simulation
Audio
3D Modeling
Level Design & Architecture
Programming

5. Progress in Computer Graphics
1995
2001
2017

6. Progress in Character Portrayal
Laura Croft from 1996
In 1996 the best the industry could do
was an image of 100,000 triangles, 30
times a second, on a 640 x 480 screen.
Laura Croft from 2013
By 2013 it was possible to generate over a 100 million
polygons at 60 to 120 times a second on screens
greater than HD.

7. Laura Croft in 2017

9. Real-time Image Synthesis
• So how do modern video games makes
these animated realistic images?
• Not just realistic
• Not just animated
• But interactive too

10. Geometry
Euclid
Pythagoras

11. Ancient Platonic View
Perfect
Timeless
Unchanging
Essential qualities
Imperfect
Temporary
Dynamic
Accidental qualities

12. Mystical True Reality?

13. Mathematization
Perceived
Reality
Mathematical model
Symbolic manipulation
Transformed
Mathematical model
Understanding or
Simulation of
Reality

14. Computer Graphics
• Nexus of several disciplines
Human
Perception
Artistic
Expression
Physics
of Light
Geometry and
Mathematics of
Surfaces
Computer Science
VLSI Hardware
Design Display & Input
Technology
Animation &
Simulation

15. Roles for Computer Graphics
[Pixar 2010]
Story telling
Product design
[CATIA]

16. Roles for Computer Graphics
Training
[Commercial simulators]
Gaming
[Skyrim]

17. Roles for Computer Graphics
User interfaces
[Android 4.0]
Navigation
[Audi]

18. Roles for Computer Graphics
Printing Digital imaging & video
[HP Deskjet]
[Canon]

19. Example: Modeling Color

20. Biology of Eye’s Color Perception

21. Color as Red-Green-Blue (RGB)

22. Modeling Color as RGB Triples
Scale of 0 to 255 (256 unique intensities per color component, 8 bits)

23. Color Approximations
for Interactive Compute Graphics
• Real color results from continuous electro-
magnetic spectrum visible to the eye
– So-called visible light
• Computer graphics approximates this as
– Discretization: continuous spectrum  three color
component stimuli
• Red, green, and blue
• Based on human visual system’s color sensitivity
– Quantization: limits dynamic range and resolution
with in range
• Real result: three 8-bit RGB channels

24. Color Monitors Imperfectly
Reproduce Human Color Gamut

25. Alternative Color Generation
Approach for Printers

26. Additive versus Subtractive
Color
RGB
Additive color reproduction
for emissive displays
CMYK
Subtractive color reproduction
for printing on paper

27. What makes an apple red?
specular
diffuse
ambient
Child’s view:
“an apple is red”
Image synthesis view:
“light, surface, and material interact to
reflect light perceived as color,
modeled via simplifying assumptions”

28. Computers Today Simulate
Complex Realistic Lighting
ay
OptiX
How?

29. Surface Reflectance Properties
• Smooth surfaces reflect like mirrors
• Rough surfaces scatter light
• Real surfaces act as mixture of both
Smooth reflective
surface
Rough diffuse
surface

30. Emissive Light
• Certain objects emit light
– Various reasons: incandesce, burning, fluorescence,
phosphorescence
– Typically models as a emissive color

31. Types of Reflected Light
• Mirror reflection
– Ideal reflection
– Reflection Law
• Diffuse reflection
– Matte, flat finish
– Lambert’s Law
• Specular
– Highlights and gloss
– Micro-facet model
Phong
lighting
Lambertian
lighting
Blinn
lighting

32. Mirror: Law of Reflection
Angle of incidence = angle of reflection
Planar reflectance
Spherical reflectanc

33. Lambert’s Law, or
Diffuse Illumination
Surface normal points
out from surface
Light source
L
diffuse
θcos
)ˆˆ,0max(
=
•= LN

34. Micro-facet Specular
• Think of the surface
as have a statistical
distribution of facet
orientations
– OpenGL’s fixed-
function specular
model

35. Combining
Lighting Contributions
• Contributions can be summed
– Since RGB color components are additive
– Illumination color modulated by material color
Ambient Diffuse Specular Result

36. Combining
Lighting + Texturing
×× ××) + () + ((( ) =) =
DiffuseDiffuse GlossGlossSpecularSpecularDecalDecal
Final result!Final result!

37. Simulating Shadows
Adds Additional Realism
Shadow
volumesvolumes
Light maps
Projected
planar
shadows
Hybrid
approaches

38. Other Fancy Lighting Effects
Caustic patterns
Light is “focused” as it
refracts through interfaces
Diffraction
Small slits act as a diffraction
grating, separating wavelengths

39. Subsurface Scattering
• Light bounces around within an object
– Translucency

40. Paths of Scattered Photons
• Trace photons statistically as they bounce and eventually
exit the material

41. Accurate Skin Needs to Model
Subsurface Scattering
Skin with subsurface scattering
Looks natural
Without
Looks lifeless
Credit: Stephen Stahlberg

42. Motion Blur

43. Motion Blur

44. Motion Blur

45. Original Image

46. Triangular Mesh Version

47. Velocity Visualization

48. Motion Blur

49. Programming Units within a
Graphics Processing Unit (GPU)
• Multiple programmable domains within the GPU
• Can be programmed in high-level languages
– Examples:
• Cg, HLSL, or OpenGL Shading Language (GLSL)
Geometry
Program
3D Application
or Game
OpenGL API
GPU
Front End
Vertex
Assembly
Vertex
Shader
Clipping, Setup,
and Rasterization
Fragment
Shader
Texture Fetch
Raster
Operations
Framebuffer Access
Memory Interface
CPU – GPU
Boundary
Attribute Fetch
Primitive
Assembly
Parameter Buffer Read
programmable
fixed-function
Legend

50. Mathematization Again
Real World
Problem /
Situation
Real
Model
Mathematical
Model
Conclusions
Insights
Experience
Computer
Model
Distill / Simplify / Approximate
Abstract
Calculate
Program / Train
Interpret /
Appreciate
Simulate

51. Modern Scientific & Engineering Approach
Experiment
Simulation Theory
Check against reality
Often slow & expensive
Dynamically checks theory
Facilitates making predictions
With computers, fast & cheap
Captures essential behavior
Often by approximation
Mathematization