Presentation by Andrew Hamilton and Ken Brown from DICE at GDC 2016.
Photogrammetry has started to gain steam within the Games Industry in recent years. At DICE, this technique was first used on Battlefield and they fully embraced the technology and workflow for Star Wars: Battlefront. This talk will cover their research and development, planning and production, techniques, key takeaways and plans for the future. The speakers will cover photogrammetry as a technology, but more than that, show that it's not a magic bullet but instead a tool like any other that can be used to help achieve your artistic vision and craft.
Takeaway
Come and learn how (and why) photogrammetry was used to create the world of Star Wars. This talk will cover Battlefront's use of of the technology from pre-production to launch as well as some of their philosophies around photogrammetry as a tool. Many visuals will be included!
Intended Audience
A content creator friendly talk intended for pretty much any developer, especially those involved in 3D content creation. It is not a technical talk focused on the code or engineering of photogrammetry. The speakers will quickly cover all basics, so absolutely no prerequisite knowledge required.
10. A Retrospective
• The beginning: research and development
• Scanning plans and goals
• Capturing the planets
• Mentality and workflows
• Game assets in a “photogrammetry world”
• Terrain
• Level Construction Kits
• Key learnings and takeaways
11. Big Changes
• New console generation
• New franchise
• Very high expectations to deliver after BF4
• A clean slate; we want to do it right!
• PBR on the way but not ready (in the beginning)
12. • Star Wars!
• Learning from the past
• Photogrammetry pushing the boundaries
• A really tight schedule
• If we’re saying 60, we need to be 60
• Keeping it together
Challenges
13. Dev Structure
• 3 month development cycles
• Content development until the end
• Need for 2 week buffer discovered
• Approval lock
• Stabilization
22. Scan everything!..?
• Bridge the gap between scanned characters and their surroundings.
• Make the best looking Star Wars game ever! ..at 60fps
• We have so little time…
26. • Capture the complete asset libraries of all planets.
• Focus and capture only what is essential, not everything we see.
• Capture meaningful variation and features.
• Visit the real film locations for true inspiration and understanding.
Location capture goals
27. • Simple kit (24mm lens, Canon 6D, tripod, color chart, etc)
• Ideal weather conditions
• On-location breakdown
• Thinking in terms of usable pieces
• Aware of Frostbite strengths/weaknesses
• Focus on quality over quantity of captures
Capturing on location
28. A. Dirt ground
B. Leaves and twigs
C. Clover ground
D. Clover leaves
E. Fern large
F. Fern medium
G. Fern small
H. Log large
I. Log medium
J. Log small
K. Tree large
L. Tree medium
M. Tree small
N. Tree stump
O. Vegetation AC
D
B
E
F
G
H
I J
K
L
M N
O
29. • Overlap!!
• Covering all angles to reduce cleanup.
• Avoid cast shadow - ambient light is easily removed later in the process.
• Most reliable results occurred when full asset was visible in most photos.
• Avoid objects with too intricate details.
• Be sure that all capture methods are consistent and clean.
• Prepare yourself and equipment for various conditions – weather can make
the process sloppy and destroy consistent results.
Best practices
58. Successful location scanning
• A small and manageable group with light equipment is ideal.
• Structured and focused on location.
• Absolutely key to connect and understand how to recreate real locations.
59. Managing the data
• Over 100,000 photos to manually sort
• Over 2000 assets to manually review
• Over 14 hours of video to categorize
• Less than ideal storage solution
• A lot of room for improvement here!
62. High Poly
High Poly
Re-Topo
Re-Topo
UV Layout
UV Layout
Texture
Texture
Setup
Setup
0 2 4 6 8 10 12 14 16
BF4
SW:BF
Days Per Asset
High Poly Re-Topo UV Layout Texture Setup
How long does an asset take?
Is Photogrammetry a savings? (Straw Poll)
63. How long does an asset take?
With Automation
High Poly
High Poly
Re-Topo
Re-Topo
UV Layout
UV Layout
Texture Setup
0 2 4 6 8 10 12 14 16
BF4
SW:BF
Days Per Asset
High Poly Re-Topo UV Layout Texture Setup
64. Photogrammetry workflow
• Remove vignette, keep lens distortion.
• Batch calibration to color chart.
• This entire step should be automated!
PS Camera RAW
65. Photogrammetry workflow
• Most assets need to be masked – very time consuming if not automatic.
• Baking at High gave most useable and manageable results for Objects.
• Baking at Ultra gave best results for Terrain – files are huge though.
Agisoft PhotoScan
66. Photogrammetry workflow
• Mosaic rather than Average blend gives much sharper results.
• Shredded UVs – take to Zbrush and reproject, or bake high res.
Agisoft PhotoScan
67. Photogrammetry workflow
• Some assets completely automated in MeshLab.
• Some needed fine tuning in Topogun – ok if very limited asset library.
Topogun / MeshLab
68. Photogrammetry workflow
• Detail textures applied later means UVs demand very little stretching.
• UV direction important for consistent detail texture alignment, sometimes
may need a second UV just for detail tiling.
Maya / UVLayout
70. Photogrammetry workflow
• Object Space Normals used as a mask to remove ambient lighting.
• AO or cavity map for fine tune lighting removal.
XNormal
71. • 16 bit depth is your friend
• Remove as much as possible from camera raw
• Once removed, move on to ambient
• Object space normal maps ftw
Photogrammetry workflow
Light Removal
72. Light Removal
Object Normal
• In this example, shadows are not
so distinct
• Generate an object normal map
• This gives distinct colors for the
different object normals
75. Texel Density
• Graphical representation of a pixel
• An initial pitfall for us
• The following screenshots are 1080p
• Illustrates texture pixel (texel) to screen
pixel ratio
84. Texel Density
• You’d be surprised what you can get away with
• Detail maps are our friends
• Large textures have a very big streaming overhead
• We have an 800mb texture pool on Gen4
• One 8k texture is almost 90mb
Takeaways
85. Texture Arrays
• The photoscanned asset as a base is generally quite low res.
• All environmental assets use a consistent preset shader.
• Planet-specific Texture Array instance shaders add the detail on top.
Object Preset Shader
86. Texture Arrays
• This texture array instance shader is set up for normal, reflectance, and
color.
Texture Array Instance Shader
87. Texture Arrays
• Brightness is painted in 5 even steps (as there are 5 detail textures).
• The order of these layers must correspond to the order in the Array.
Tree Normals (Texture Array mask in Alpha)
Brightness %
0
25
50
75
100
Detail
Leaves
Bark
Burnt Bark
Rock
Moss
88. Texture Arrays
• Detail textures are fading away towards the end of LOD0 with ‘LodFade’.
• LOD1 has a new simplified shader without detail textures.
Texture Array Instance Shader
89. Texture Arrays
• Like most texture assets in Battlefront, we heavily pack into channels.
• The detail texture uses RG for normals, B for Reflectance, A for Color.
Detail Texture within the Texture Array
90. Texture Arrays
• Unique PhotoScan base is surface blurred,
while keeping important details.
• Drastically reduces texture memory.
91. Texture Arrays
• Texture Array tiling detail on top fill in the
details without overpowering the base.
• Drastically increases texture resolution.
92. How do we ensure all assets fit together consistently?
“Old” Asset Creation
QUALITY
TECHEFFORT
Photogrammetry
TECH+TECHNIQUES
QUALITY
EFFORT
“New” Asset Creation
QUALITY
TECHEFFORT
NEEDED
EFFORTTECH+TECHNIQUES
Photoscan vs. Everything Else
93. • Helps us bridge that gap
• Lessons from Criterion, Guerilla Games, Dontnod etc
• Trying to get everyone to switch together can backfire.
• Some people will adapt instantly. Let them lead!
• For PBR, there isn’t really a middle road.
PBR
Quick Thoughts
94. PBR
• Assets generally fit well but still needed to be balanced against the scene.
• Finding and trusting true PBR values at times was a challenge
• Constant reinforcement helps prevent inconsistencies
104. Vegetation
• Reaching 60fps – lets make a dense forest!
• Integrating hand-built vegetation into a PhotoScanned world was tough
to reach comparable quality!
• Difficult to get good results from scanned vegetation – complex and
noisy, resulting in horrible scans.
• Based on photos of vegetation from location captured on blue board.
105. Vegetation
• All vegetation share the same textures
• There are textures created separately for
close up, mid, and far distances.
• There is a lot of texture packing within
vegetation assets to reduce complexity of
shaders.
• Normals created by broadly modelling
around leaves and branches Maya.
106. Vegetation
• Unique PhotoScanned base blends to tiling
texture that extends upwards.
• Tree base is generally evenly tessellated for
best displacement results.
Tiling trunk texture
Unique base texture
107. Terrain/asset blending
• A lot of work put in to reduce the visible
disconnect between assets and terrain.
• Most assets have terrain radiosity
projected on to them.
• Texture Arrays world projected and details
match.
• Terrain color and normals applied to most
assets.
108. Terrain/asset blending
• A lot of work put in to reduce the visible
disconnect between assets and terrain.
• Most assets have terrain radiosity
projected on to them.
• Texture Arrays world projected and details
match.
• Terrain color and normals applied to most
assets.
109. Mesh Displacement
• Great quality boost for close up detail.
• Object displacement stopped at ~7 meters.
• An ongoing balance between quality and
performance.
112. • Ability to define displacement shape (via mask)
• One of our “Must Have” features
• Critical for ATAT footprints
• Terrain Decal Component
• Combined with detail displacement shader
• (More on this a little later)
Tessellation
Heightfield Displacement
113. • Quality vs. Speed ratio is extremely high
• Feel free to get creative
Tessellation and Photogrammetry
Heightfield Displacement
121. Terrain
• 8x8km with a 1x1km playable space.
• Huge background terrain meshes.
• Terrain textures captured from real world.
• Layers limited to control draw calls.
127. Best practices - Terrain
• We aimed for 3x3 meter patches
• 5x5 photo grid ensures enough overlap
• Capture only 1 defined element at a time
• Low ISO and high F-Stop for sharp image
• Avoid noisy surfaces such as grass
128.
129.
130.
131. Terrain Tessellation
• Great quality boost for close up detail on terrain
• Major upgrade for us. Can never go back!
• Terrain displacement stopped at ~15 meters.
134. Level Construction Kits
• Content and documentation packages.
• Provides fast level iteration.
• Shifts focus to building levels rather than
content during Production.
• Allows us to more quickly understand
performance and keep track of it.
135. Level Construction Kits
• 8x8km highly detailed terrain.
• Ready to paint terrain materials.
• Easily accessible limited library of content.
• Lighting presets based on the films.
161. Key learnings and takeaways
• Solid processes and workflows in place before Production was a big win.
• Well planned and structured Photogrammetry trips results in clean and
manageable assets.
• Photogrammetry is not a silver bullet – it was only one part of achieving
the final results in game.
• It was crucial to have artists own and be responsible for their
locations/planets - Consistency is key when working with scanned assets.
• Working towards PBR was an ongoing process to success.
• Much of our creative process was incredibly repetitive, much of which
can and should be automated.
• There was way too much manual labor, with levels being built by brute
force without the help of workflow-improving tools.
162.
163. Special Thanks
John Troive -
Pontus Ryman -
Petter Sköld -
Tim McLeod -
Simon Barsky -
Darko Pracic -
Daniel Cambrand -
Oscar Carlen -
Madina Chionidi -
Björn Arvidsson -
Robert Kihl -
… and of course, the entire Battlefront team at DICE!
Senior Environment Artist
Senior Environment Artist
Senior Environment Artist
Environment Artist
Environment Artist
Environment Artist
Environment Artist
Senior Lighting Artist
Character Artist
Character Artist
Rendering Engineer
164. Thanks!
Andrew Hamilton – Lead Environment Artist
Andrew.Hamilton@dice.se
Kenneth Brown – Technical Art Director
Kenneth.Brown@dice.se
Notas del editor
Photogrammetry - the science of making measurements from photographs, especially for recovering the exact positions of surface points. Moreover, it may be used to recover the motion pathways of designated reference points located on any moving object, on its components and in the immediately adjacent environment. Photogrammetry may employ high-speed imaging and remote sensing in order to detect, measure and record complex 2-D and 3-D motion fields (see also sonar, radar, lidar, etc.). Photogrammetry feeds the measurements from remote sensing and the results of imagery analysis into computational models in an attempt to successively estimate, with increasing accuracy, the actual, 3-D relative motions within the researched field. (source: Wikipedia)
Photogrammetry - the science of making measurements from photographs, especially for recovering the exact positions of surface points. Moreover, it may be used to recover the motion pathways of designated reference points located on any moving object, on its components and in the immediately adjacent environment. Photogrammetry may employ high-speed imaging and remote sensing in order to detect, measure and record complex 2-D and 3-D motion fields (see also sonar, radar, lidar, etc.). Photogrammetry feeds the measurements from remote sensing and the results of imagery analysis into computational models in an attempt to successively estimate, with increasing accuracy, the actual, 3-D relative motions within the researched field. (source: Wikipedia)
A detailed breakdown of our step-by-step workflow can be found here.
Align photos.
Generate point cloud.
Generate highpoly mesh.
Generate/project textures.
Opportunity to change the way we think and create content
Available manpower (artists transitioning from BF4)
Photogrammetry can make great high res, largely organic assets
We still need man (alien?) made asset
We need to significantly increase our efficiency
Let’s look at were we are and re-evaluate everything we know. Nothing is off limits.
Today we’ll discuss some of the software we settled on
The beginnings of photogrammetry with Battlefield 4
An example of the range of captured assets from the Lucas archives.
An example of the range of captured assets from the Lucas archives.
Supporting scanned assets with other tools such as Marvellous Designer
Hard-Surface materials created with traditional methods
An example of the level of detail we’re able to get with combining photoscan with new (to us) software.
An example of the level of detail we’re able to get with combining photoscan with new (to us) software.
An example of the ingame assets using the combination of techniques.
A screenshot from Battlefield: Armored Kill that was considered ‘realistic’ during its time…
…to a screenshot from Star Wars: Battlefront, using photogrammetry on a very limited number of selective content to build up a realistic environment
A slideshow of locations we have visited on scanning trips, for a quick glimpse into the range and variety of settings at our disposal.Iceland
Iceland
Death Valley, USA
Utah, USA
Death Valley, USA
National Redwood Forests , USA
National Redwood Forests , USA
Finse, Norway
Finse, Norway
Iceland
Iceland
Iceland
A small selection from our large library of scanned materials that would be used for Sullust terrain surfaces
A small selection from our large library of scanned materials that would be used for Tatooine terrain surfaces
A small selection from our large library of scanned materials that would be used for Endor terrain surfaces
A small selection from our large library of scanned materials that would be used for Hoth terrain surfaces
A small selection from our large library of scanned materials that would be used for Hoth terrain surfaces
Vegetation and ground scattering assets captured on a blue board for easy background extraction
An example of a scanned asset, showing how intentionally generic and tilable they are.
An example of a scanned asset, showing how intentionally generic and tilable they are.
An example of a scanned asset, showing how intentionally generic and tilable they are.
An example of a scanned asset, showing how intentionally generic and tilable they are.
An example of a scanned asset, showing how intentionally generic and tilable they are.
An example of a scanned asset, showing how intentionally generic and tilable they are.
An example of a scanned asset, showing how intentionally generic and tilable they are.
An example of a scanned asset, showing how intentionally generic and tilable they are.
An example of a scanned asset, showing how intentionally generic and tilable they are.
An example of a scanned asset, showing how intentionally generic and tilable they are.
Opportunity to change the way we think and create content
Available manpower (artists transitioning from BF4)
Opportunity to change the way we think and create content
Available manpower (artists transitioning from BF4)
Data taken from a cross sampling of content creator disciplines. Straw poll – not necessarily accurate 1:1, but representative of reality.
A basic step by step guide through the Photogrammetry workflow
A basic step by step guide through the Photogrammetry workflow
A basic step by step guide through the Photogrammetry workflow
A basic step by step guide through the Photogrammetry workflow
A basic step by step guide through the Photogrammetry workflow
A basic step by step guide through the Photogrammetry workflow
A basic step by step guide through the Photogrammetry workflow
Shadows
Need to mask out shadows
Super important to have 16 bit depth
Before that, remove as much as possible from the camera raw (shadow slider photoshop camera raw plugin)
Create mask from what’s dark.
Flatten luminance in the colors first
Shadows tend to be colder, so you can use color to grab them.
Once you’ve gotten the shadows removed, you can treat it like something that’s been captured overcast
Get rid of ambient light
Two ways
Use the normal map (baked out in object space) to get the angle.
Use the occlusion, baked from the geometry.
Also need to use the mask to remove the color (bounced light)
Overcast is best
Let’s look at other aspects of the asset creation
Texel refers to the graphical representation of a pixel on a 2d object (eg. a texture). Basically, when you look at a pixel of a texture rendered on a 3d model, you are looking at a“texel”.
In this shot ~25x’s more pixels available than the screen can render.
Mips always try to keep a 1:1 ratio. So detail will get mipped away anyways!
Not only is this extremely close for 1p, it’s an even more extreme distance for 3p. At this resolution, there is certainly an absolute difference in quality, but a trivial relative difference in quality.
To give you a sense of the memory impact (and subsequent streaming impact) that this has.
The quality of these assets are great. This just reinforces what we’ve been saying – focus on the craftsmanship. More resolution isn’t what’s going to make these assets look better – the overall quality of the asset is what’s going to sell it.
In this example, the multiply node is set to a factor of 4, give a range from 0-4 (or 5 distinct integers). Color information (only greyscale though) is stored in the alpha.
- Anything we can do to increase efficiency
PBR is a talk in and of itself, but there a few things worth mentioning.
Render passes that compose the final image
Render passes that compose the final image
Render passes that compose the final image
Render passes that compose the final image
Render passes that compose the final image
Render passes that compose the final image
Render passes that compose the final image
Render passes that compose the final image
Render passes that compose the final image
Battlefield titles extensively used WorldMachine for the terrain, but we needed to support our up-close Photoscanned assets with realistic landscape in the distance
We instead complemented Photoscanned assets with real-world landscape data
The square in the middle represents the 8x8km Frostbite terrain, where mesh extends the backdrop a further 50km in all directions
Example of a scanned surface
Example of a scanned surface
Example of a scanned surface
A Level Construction Kit was created for each planet or biome
LCK video
This scene appears to be comprised of many elements with much variation…
…but it is only comprised of a very limited amount of generic assets that work well to populate a large area without appearing repetitive.
This scene appears to be comprised of many elements with much variation…
…but it is only comprised of a very limited amount of generic assets that work well to populate a large area without appearing repetitive.
This scene appears to be comprised of many elements with much variation…
…but it is only comprised of a very limited amount of generic assets that work well to populate a large area without appearing repetitive.
This scene appears to be comprised of many elements with much variation…
…but it is only comprised of a very limited amount of generic assets that work well to populate a large area without appearing repetitive.
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits
A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits