C. Schwartz, R. Klein 3D COFORM Star Day – VAST 2011, October 20 - Prato, Italy

1. Institute of Computer Science II
Computer Graphics
Realistic Object Appearance using
Bidirectional Texture Functions
Christopher Schwartz, Reinhard Klein
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz1 20/10/2011

2. INTRODUCTION
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz2 20/10/2011

3. Object Appearance
• Common presentation: Geometry (+ Texture)
• … but is this sufficient?
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz3 20/10/2011
Geometry only With texture Correct appearance

4. Importance of Surface Appearance
• Same shape, different materials
• Different „look-and-feel“
• Important hints about the object
• Misleading vs. better understanding
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz4 20/10/2011

5. Object Appearance
• Impression of reflection of
incident light
• Influenced by features on
different scales
• Macroscopic
• Mesoscopic
• Microscopic
• Viewpoint and
Illumination dependent
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz5 20/10/2011

6. Form of Representation
Macroscopic scale
•3D shape
•Explicit representation
(e.g. polygon mesh)
Mesoscopic scale
•Individually resolved by
human perception
•Statistical representation
not accurate
•Explicit representation
too costly
Microscopic scale
•Alignment of microscopic
structures
•Statistical representation
(e.g. BRDF)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz6 20/10/2011

7. Bidirectional Reflectance Distribution Function
• Opaque, uniform
Materials
• No texture!
• Ratio of incident
irradiance to outgoing
radiance
• Defined over local
hemisphere
• Depends on
• Solid angle of light ωi
• Solid angle of view ωo
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz7 20/10/2011

8. Bidirectional Reflectance Distribution Function
• Example BRDF
• sampled at discrete angles
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz8 20/10/2011
ωi
ωo
BRDF TabulatedExample: BRDF on Sphere Discrete sample
positions on
Hemisphere
Specular reflection

9. Model-driven vs. Data-driven
• Matusik et al. 2003 [1]:
Measured BRDF ground-truth data
• 100 real world materials
• 1° resolution for view-
and light directions
• > 1,000,000 samples
• Ngan et al. 2005 [2]:
Experimental analysis
of BRDF models
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz9 20/10/2011

10. Model-driven vs. Data-driven
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz10 20/10/2011
Red phenolic
Measured distribution from [1]
Fitted model (Cook-Torrance) from [2]

11. Model-driven vs. Data-driven
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz11 20/10/2011
Red phenolic
Measured distribution from [1]
Fitted model (Cook-Torrance) from [2]

12. Data-driven Reflectance
• Example Surface
with Meso structure
• Results are „ABRDFs“
(apparent BRDFs [3])
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz12 20/10/2011
ωi
ωo
ωi
ωo
Specular reflection
Retro-reflection
Hard to fit with
analytical model
influence from
neighborhood

13. Model-driven Reflectance
• Loss of Mesoscale depth impression…
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz13 20/10/2011
Fitted analytical SVBRDF Photograph
McAllister 2002 [10]

14. Data-driven Reflectance
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz14 20/10/2011
Texture Mesoscale
approximated by
Bump-mapping
Data-Driven
(BTF)
Images taken from
Müller et al. 2005 [7]

15. Form of Representation
Macroscopic scale
•3D shape
•Explicit representation
(e.g. polygon mesh)
Mesoscopic scale
•Individually resolved by
human perception
•Statistical representation
not accurate
•Explicit representation
too costly
Microscopic scale
•Alignment of microscopic
structures
•Statistical representation
(e.g. BRDF)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz15 20/10/2011

16. Form of Representation
Macroscopic scale
•3D shape
•Explicit representation
(e.g. polygon mesh)
Mesoscopic scale
•Individually resolved by
human perception
•Statistical representation
not accurate
•Explicit representation
too costly
Microscopic scale
•Alignment of microscopic
structures
•Statistical representation
(e.g. BRDF)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz16 20/10/2011
Data-driven:
Image based

17. Form of Representation
Macroscopic scale
•3D shape
•Explicit representation
(e.g. polygon mesh)
Mesoscopic scale
•Individually resolved by
human perception
•Statistical representation
not accurate
•Explicit representation
too costly
Microscopic scale
•Alignment of microscopic
structures
•Statistical representation
(e.g. BRDF)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz17 20/10/2011
Bidirectional Texture Function

18. ACQUISITION
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz18 20/10/2011

19. First use of BTF: CUReT Database
• 1996 – 1999 by Dana et al. [4]
• 61 materials
• 205 different view- and light directions
• 24-bit RGB images
 Manual placement of camera
 BTF only partially measured
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz19 20/10/2011

20. University of Bonn BTF Database
• 2001 – 2003 Sarlette et al. [5]
• 6561 view and light directions
• 36-bit RGB images
• Fully automated
• 12 hrs
per acquisition
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz20 20/10/2011

21. University of Bonn Multiview Dome
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz21 20/10/2011
• 2004 – now by Sarlette et al.
• 22,801 view and light directions
• HDR images
• Fully automated
• No moving parts
• 2hrs per acquisition

22. First Capture of Complete Objects with BTF
• Furukawa et al. 2002 [6]
• Laser scanned geometry
• Separate BTF capture
 Very sparse sampling
 Alignment errors
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz22 20/10/2011
Photograph Rendering

23. First Capture of Complete Objects with BTF
• Furukawa et al. 2002 [6]
• Laser scanned geometry
• Separate BTF capture
 Very sparse sampling
 Alignment errors
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz23 20/10/2011
Photograph Rendering

24. Integrated Acquisition of Objects with BTF
• Müller et al. 2005 [7]
• Use University of Bonn Dome
• Dense sampling (22,801), 2hrs/acquisition
• Measurements integrated in one setup
• No registration necessary
• Geometry via Shape-from-Silhouette
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz24 20/10/2011

25. Integrated Acquisition of Objects with BTF
• Müller et al. 2005
Drawbacks:
 No radiometric calibration
• Misleading colors
• Only LDR
 Shape-from-Silhouette
• Not automatable
• Coarse geometry
– Misleading Shape
– Blur due to misalignment
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz25 20/10/2011
Example from Havemann et al. 2008 [12]

26. Integrated HQ Acquisition
• Holroyd et al. 2010 [17]
• Integrated setup
• Geometry with Structured Light
• 42 view and light directions
• 5hrs per acquisition
 Sparse sampling  Model-driven
• Fit Cook-torrance BRDFs
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz26 20/10/2011
Photograph Rendering

27. Integrated HQ Acquisition
• Holroyd et al. 2010 [17]
• Integrated setup
• Geometry with Structured Light
• 42 view and light directions
• 5hrs per acquisition
 Sparse sampling  Model-driven
• Fit Cook-torrance BRDFs
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz27 20/10/2011
Photograph Rendering

28. Integrated HQ Acquisition with BTF
• Schwartz et al. 2011 [8]
• Use University of Bonn Dome
• Extended with projectors for Structured Light
• integrated measurement
• Rapid: 3.7 hrs per acquisition
• Proper calibration and HDR
• Geometry: Weinmann et al. 2011 [11]
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz28 20/10/2011
Visual Hull [7], [12] Laser Scan Proposed Method [11]

29. Quality
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz29 20/10/2011

30. Faithfulness
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz30 20/10/2011
Photographic
picture
(tonemapped HDR)
BTF + Geometry
Schwartz et al. 2011 [8]
(tonemapped HDR)

31. Polynomial Texture Maps
• Malzbender et al. 2001 [9]: PTM
• Image-based, Sampling of Light (ωi)
 Incomplete appearance information
• View-dependent part of reflectance missing
 For 3D Objects: only one fixed viewpoint
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz31 20/10/2011
PTM
Texture

32. Faithfulness
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz32 20/10/2011
Photographic
picture
(tonemapped HDR)
BTF + Geometry
Schwartz et al. 2011 [8]
(tonemapped HDR)
Polynomial Texture Map
Malzbender et al. 2001 [9]
(Single view and LDR!)

33. Multiview PTMs
• Gunawardane et al. 2009 [18]:
• PTMs from multiple viewpoints
 No macro scale
 interpolation of views via optical flow
 Limited amount of views
• Combining multiple objects is hard
• incorrect silhouttes, occlusion,
shadows, etc.
• Full light transport
(i.e. path-tracing)
not possible
33 20/10/2011
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher
Schwartz

34. Full Light Transport with BTFs
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz34 20/10/2011

35. Full Light Transport with BTFs
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz35 20/10/2011

36. COMPRESSION, RENDERING AND
TRANSMISSION
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz36 20/10/2011

37. Datasizes
• Schwartz et al. 2011 [8]:
• Uncompressed BTF:
≈ 500 GB per object
• Not feasible even for
offline rendering…
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz37 20/10/2011

38. BTF Compression
• Fitting analytical models
• Wu et al. 2011 [15]: SPMM
• Model-driven…
• lost meso-structure
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz38 20/10/2011
BTF
SPMM

39. BTF Compression
• High degree of
redundancy in the BTF
• Perform statistical
data analysis
• Find low dimensional basis
•  learn how to best describe the data
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz39 20/10/2011

40. Compression using Statistical Analysis
• Organization of the discrete BTF
• As matrix • As Tensor
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz40 20/10/2011
Angles ωi, ωo
Pixels(x,y,color)
Pixels (x,y)
ωo
Note:
Color (or wavelength) can be
additional dimension

41. Full Matrix Factorization
• E.g. Liu et al. 2004 [14]: FMF
• Representation is compact
and realtime renderable [5]
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz41 20/10/2011
…
...
 
 
  
 
 
M
 
 
 
 
 
…V
Angular components
“Eigen-ABRDFs“
 
 
 
 
 
U …
Spatial components
“Eigen-Textures“SVD
T
M USV
Importance

42. Decorrelated Full Matrix Factorization
• Gero Müller 2009 [13]: DFMF
• Insight from image compression (e.g. JPEG)
• Human perception is more sensitive to variation in
intensity than color
• Also: chromacity in images exhibits less variation
• Decorrelate the BTF data into luminance and
chrominance
– BTFRGB  BTFY BTFU BTFV
• Use fewer components for chrominance channels U, V
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz42 20/10/2011

43. FMF Rendering
• Uncompressed: ≈ 500GB
• Compressed: ≈ 640MB
• 32 components
• Even fits on the GPU
• Random access to BTF
• Angular combination
a = (ωi, ωo)
• Pixel p = (x,y)
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz43 20/10/2011
components
Angular
component #1
Spatial
component #1
Angular
component #2
Spatial
component #2
…
…
pixelangles
BTF(a,p) = < , >

44. Interactive Inspection via GPU Rendering
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz44 20/10/2011

45. Streaming of BTF over the Internet
• Schwartz et al. 2011 [16]:
• Spatial components
≈ natural images
• Angular components
≈ low frequency
• Apply additional wavelet
compression
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz45 20/10/2011
0.4bpp
Wavelet
16bpp
Reference

46. Streaming of BTF over the Internet
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz46 20/10/2011
0.87 MB
First renderable
version
1 MB 7 MB 46.4 MB
Fully
transmitted
534 GB
Reference

47. Questions?
Learn more about
BTFs for Cultural
Heritag on
Wednesday [8]
and Thursday [16]
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz47 20/10/2011

48. References
[1] “A Data-Driven Reflectance Model”, Matusik W., Pfister H., Brand M. and McMillan L., ACM TOG 22, 3(2003),
759-769.
[2] “Experimental Analysis of BRDF models”, Ngan A., Durand F. and Matusik W., Proceedings of EGSR, 2005, 117-
226.
[3] „Image-based Rendering with Controllable Illumination“, Wong T., Heng P., Or S., and Ng W., Proceedings of
EGWR, 1997, 13-22
[4] „Reflectance and Texture of Real-World Surfaces “, Dana K.J., van Ginneken B., Nayar S.K. and Koenderink J.J.,
Proceedings of CVPR, 1997, 151-157
[5] „Efficient and Realistic Visualization of Cloth“, Sattler M., Sarlette R. and Klein R., Proceedings of EGSR, 2003
[6]„Appearance based object modeling using texture database: acquisition, compression and rendering“, Furukawa
R., Kawasaki H. Ikeuchi K. and Sakauchi M., Proceedings of EGRW 2002, 257-266
[7] „Rapid Synchronous Acquisition of Geometry and BTF for Cultural Heritage Artefacts“, Müller G., Bendels G.H.
and Klein R., Proceedings of VAST, 2005
[8] „Integrated High-Quality Acquisition of Geometry and Appearance for Cultural Heritage”, Schwartz C., Weinmann
W., Roland R. and Klein R., Proceedings of VAST, 2011
[9] „Polynomial Texture Maps“, Malzbender T., Gelb D. and Wolters H., Proceedings of SIGGRAPH, 2001
[10] „A Generalized Surface Appearance Representation For Computer Graphics“, McAllister D.K., PhD. Thesis,
University of North Carolina at Chapel Hill, 2002
[11] „A Multi-Camera, Multi-Projector Super-Resolution Framework for Structured Light“, Weinmann M., Schwartz
C., Ruiters R. and Klein R., Proceedings of 3DIMPVT, 2011, 397-404
[12] „The Presentation of Cultural Heritage Models in Epoch“, Havemann S., Settgast V., Fellner D., Willems G., Van
Gool, L., Müller G., Schneider M. and Klein R., EPOCH Conference on Open Digital CH Systems, 2008
[13] „Data-Driven Methods for Compression and Editing of Spatially Varying Appearance“, Müller G., PhD. Thesis,
University of Bonn, 2009
[14] „Synthesis and Rendering of Bidirectional Texture Functions on Arbitrary Surfaces“, Liu X., Hu Y., Zhang J. Tong
X., Guo B. and Shum H.-Y., IEEE Transactions on Visualization and Computer Graphics 10 (3), 2004, 278-289
[15] „A Sparse Parametric Mixture Model for BTF Compression, Editing and Rendering“, Wu H., Dorsey J. and
Rushmeier H., Computer Graphics Forum 30 (2), 2011, 465-473
[16] „WebGL-based Streaming and Presentation Framework for Bidirectional Texture Function“, Schwartz C., Ruiters
R., Weinmann M. and Klein R., Proceedings of VAST, 2011
[17] „A coaxial optica scanner for synchronous acquisition of 3D geometry and surface reflectance“, Holroyd M.,
Lawrence J. and Zickler T., ACM Trans. Graph. 29 (4), 2010
[18] „Optimized Image Sampling for View and Light Interpolation“, Gunawardane P., Wang O., Scher S., Rickards I.,
Davis J. And Malzbender T., Proceedings of VAST, 2009
[19] „Principles and practices of robust, photography-based digital imaging techniques for museums.“, Mudge M.,
Schroer C., Ear G., Martinez K., Pagi H., Toler-Franklin C., Rusinkiewicz S., Palma G., Wochowiak M., Ashley M.,
Matthews N., Noble T. and Dellepiane M., Proceedings of VAST, 2010
3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz48 20/10/2011