Bickerstaff benson making3d games on the playstation3
1. Making 3D Games on the PlayStation 3 ®
by Ian Bickerstaff & Simon Benson,
Sony Computer Entertainment Stereoscopic 3D Team
2. Making 3D Games on the PlayStation 3 ®
•
What is stereoscopic 3D and why have 3D games?
•
Stereoscopic theory and how to make high quality 3D images
•
Rendering 3D images on the PlayStation 3
®
•
Case studies
•
Future work
3. What is stereoscopic 3D?
Images from slightly different
viewpoints are presented to each eye
The difference between the two
images is called parallax
The brain combines the images to
reconstruct the 3D world
4. No th ing ne w …
Le petit journal pour rire 1859
5. No th ing ne w …
•
1 8 3 2 firs t 3 D v ie w ing de v ic e
Sir Charles Wheatstone reflecting stereoscope
6. No th ing ne w …
Brewster
•
1 8 3 2 firs t 3 D v ie w ing de v ic e stereoscope &
stereocards
•
1 8 4 4 3 D ph o to v ie w e r
Col Ian Bickerstaff
Robert Howlett’s 3D image
of “Great Eastern” Captain
William Harrison, 1857
7. No th ing ne w … Jules Duboscq
Bioscope disc
•
1 8 3 2 firs t 3 D v ie w ing de v ic e
•
1 8 4 4 3 D ph o to v ie w e r
•
1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks
Col Joseph Palteau, University of Ghent
For Cross-eyed viewing
8. No th ing ne w …
•
1 8 3 2 firs t 3 D v ie w ing de v ic e
•
1 8 4 4 3 D ph o to v ie w e r
•
1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks
•
1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)
Look through here!
D’Almeida Eclipse
stereoscope
9. No th ing ne w …
•
1 8 3 2 firs t 3 D v ie w ing de v ic e
•
1 8 4 4 3 D ph o to v ie w e r
•
1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks
•
1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)
•
1 8 9 3 3 D po la ris e d gla s s e s
John Anderson’s
“Stereoscope for
polarised light”
1930s polarised 3D glasses
10. No th ing ne w …
Paul Mortier
•
1 8 3 2 firs t 3 D v ie w ing de v ic e
electrically
•
1 8 4 4 3 D ph o to v ie w e r synchronised shutter
glasses
•
1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks
•
1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)
•
1 8 9 3 3 D po la ris e d gla s s e s
•
1 8 9 6 e le c tric a lly s ync h ro nis e d s h utte r gla s s e s
1930s 3D shutter glasses
11. No th ing ne w …
Lumiere brothers’
•
1 8 3 2 firs t 3 D v ie w ing de v ic e Autochrome
process
•
1 8 4 4 3 D ph o to v ie w e r
•
1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks
•
1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)
•
1 8 9 3 3 D po la ris e d gla s s e s
•
1 8 9 6 e le c tric a lly s ync h ro nis e d s h utte r gla s s e s
•
1 9 0 3 c o lo ur 3 D ph o to gra ph s
Col Ian Bickerstaff
12. No th ing ne w …
•
1 8 3 2 firs t 3 D v ie w ing de v ic e
•
1 8 4 4 3 D ph o to v ie w e r
•
1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks
•
1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)
•
1 8 9 3 3 D po la ris e d gla s s e s
•
1 8 9 6 e le c tric a lly s ync h ro nis e d s h utte r gla s s e s Radio News, November 1928
•
1 9 0 3 c o lo ur 3 D ph o to gra ph s Baird
Stereoscopic television
•
1 9 2 8 3 D TV de m o ns tra te d
Modification by Erik Kurland
13. No th ing ne w … Ivan Sutherland’s
Head Mounted
•
1 8 3 2 firs t 3 D v ie w ing de v ic e Display
•
1 8 4 4 3 D ph o to v ie w e r
•
1 8 5 2 3 D m o v ie s s upplie d o n inte rc h a nge a b le dis ks
•
1 8 5 8 3 D s h utte r gla s s e s (m e c h a nic a l)
•
1 8 9 3 3 D po la ris e d gla s s e s
•
1 8 9 6 e le c tric a lly s ync h ro nis e d s h utte r gla s s e s
•
1 9 0 3 c o lo ur 3 D ph o to gra ph s
•
1 9 2 8 3 D TV de m o ns tra te d
•
1 9 6 5 inte ra c tiv e 3 D CGI
14. Why has 3D returned?
New LCD TVs make high quality 3D affordable
H I s ta nda rd no w inc lude s 3 D
DM
The PlayStation 3 is an ideal 3D image
®
generator
3D gaming will be widely available soon
15. Why have 3D games?
Increases the immersion
and realism
16. Why have 3D games?
Improves the clarity of visual
information
17. Why have 3D games?
Improves the clarity of visual
information
19. How stereoscopic 3D works
The difference in the position of our eyes
generates parallax
The closer the object, the more parallax is
observed
Distant objects produce virtually no
parallax
Replace our eyes with cameras to capture
the 3D effect
The distance between the cameras is
called the “Interaxial”
Increasing the interaxial increases the
parallax
26. How stereoscopic 3D works –what is required
“convergence”: controls the depth placement of the image
27. How stereoscopic 3D works – back in the room it now looks correct
“ortho-stereoscopic” viewing
28. How to implement interaxial and convergence shifts
y
z
Interaxial: a local world space translation in the x axis
x
y
Convergence: a screen space translation in the x axis
x
29. How to implement interaxial and convergence shifts
y
z
Interaxial: a local world space translation in the x axis
x
y
Convergence: a screen space translation in the x axis
x
asymmetrical viewing frusta
“point of convergence”
...determines on which side of the screen objects lie
30. Interaxial and convergence in real cameras
Stereoscopic camera with built in
convergence
Stereo Realist 35mm camera 1947
Non-stereoscopic cameras toed in to
achieve convergence
31. The problem with toeing in…
Keystone distortion!
Always avoid toeing in your cameras!
32. Story so far…
Adjusted the interaxial
Adjusted the convergence
We have replicated reality!
37. The limitations of parallax: cinemas –the screen is far away
Close objects require too much negative parallax
38. The limitations of parallax: televisions –the screen is close to the viewer
Distant objects require too much positive parallax
39. Exactly how much positive parallax?
We need to know...
The size of the screen – maybe but not guaranteed
The viewing distance – need head tracking
The amount of light entering the
viewer’s eyes – not practical to measure
The viewer’s stereoscopic ability – not practical to measure
40. Parallax Management – use an arbitrary value
•
A de fa ult m a xim um +v e pa ra lla x o f
1 / 3 0 th s c re e n w idth
•
A o ids div e rge nc e fo r s c re e ns up to 9 0
v
inc h e s
1/30th screen width
41. Parallax Management – use an arbitrary value
•
A de fa ult m a xim um +v e pa ra lla x o f
1 / 3 0 th s c re e n w idth
•
A o ids div e rge nc e fo r s c re e ns up to 9 0
v
inc h e s
1/30th screen width
On a typic a l s c re e n th e m a xim um de pth
is a ppro xim a te ly th e s a m e a s th e
v ie w ing dis ta nc e
A o inc lude a 3 D s tre ngth s lide r
ls
42. Limiting the positive parallax: solution 1
as it originally was...
Restrict the depth in your world to be within the required parallax range
43. Limiting the positive parallax: solution 1
with a wall added...
Restrict the depth in your world to be within the required parallax range
44. Limiting the positive parallax: solution 1 – how the depth is perceived
max parallax
real world viewed world
Accurate depth but large restrictions on the game design
45. Limiting the positive parallax: solution 2
as it originally was...
Reduce the amount of convergence
46. Limiting the positive parallax: solution 2
convergence reduced
...but the objects are now in front of the screen
Reduce the amount of convergence
47. An aside: placement and scaling in stereoscopic 3D
Decrease the convergence Everything moves closer
y
x Everything decreases in size
48. An aside: placement and scaling in stereoscopic 3D
Decrease the convergence Everything moves closer
y
x Everything decreases in size
Decrease the interaxial Objects in the foreground move further
y
away
z
Objects in the foreground increase in
x
size
Objects in the distance are unchanged
49. An aside: placement and scaling in stereoscopic 3D
First adjust your convergence to position
y
your distant objects
x
y
z Then adjust the interaxial until your
foreground objects are correct
x
...see how this affects the room scene
50. Limiting the positive parallax: solution 2 with reduced interaxial
...foreground objects now the correct side of the screen
51. Limiting the positive parallax: solution 2
max parallax
display infinity
compressed
expanded
real world viewed world
...acceptable parallax, but the world appears distorted
52. What happens if you alter your viewing distance?
real world viewed world
The amount of perceived depth increases with viewing distance
53. Choice of lenses
real world viewed world
What happens if the viewer is stationary but the camera and field of view are altered?
54. Choice of lenses – wide angle lenses
real world viewed world
A wide angle lens exaggerates depth
55. Choice of lenses – telephoto style lenses
real world viewed world
A telephoto-style lens will compress depth – not good for 3D
56. Lens choice to improve the solution 2 distortion: before
max parallax
display infinity
real world viewed world
Can the distortion from before be improved?....
57. Lens choice to improve the solution 2 distortion: after
max parallax
display infinity
real world viewed world
The foreground objects are now closer to their correct positions
58. Summary – a compromise
Televisions limit the amount of positive parallax we can use
Either limit the depth of our world – game design issues
Or
Alter the 3D settings - distortion
Adjusting the 3D settings can control the distortion
This is the challenge of stereography!
63. Summary about negative parallax
• Keep most objects behind the screen
• Best for fast moving objects and special FX
• Occasional negative parallax is entertaining!
64. Other issues: Fusion – regions of similar parallax
The brain can fuse together regions of
similar parallax to form a single image
Large variations in parallax can’t be
fused
Handy for extracting objects from
backgrounds
Repeatedly switching between fusion
zones is tiring
65. Scaling the world to maximise the parallax
Our stereoscopic vision
works best for objects
close to us
What if our game world
operates outside of that
range?
66. Scaling the world to maximise the parallax
Here the cameras are
metres apart
It is as if we are a giant
looking down on the world
The apparent size of our
3D world is a creative
decision
67. Summary of other issues
• Avoid the requirement to switch from fusion zone to zone
• The world can be scaled to bring it into our 3D perception range
68. Implementation on the PlayStation 3 – SDK facilities available
®
• supports 3D output
• tells you if the TV supports 3D
• gives you the screen size
• The choice to switch to 3D is up to the user from within your game
69. Implementation on the PlayStation 3 – a 3 step process
®
1280 pixels
Step 1: Generate two images
Left 720 pixels
Render into a 1280x1470 buffer
Images automatically converted into
HDMI 3D output at 59.94Hz 1470 pixels 30 pixel gap filled with any uniform colour 30 pixels
Any frame rate up to 59.94 can be used
Avoid frame tearing Right
720 pixels
70. Implementation on the PlayStation 3 – a 3 step process ®
1280 pixels
Step 1: Generate two images
•
H rdw a re ups c a ling is a v a ila b le
a
Left 720 pixels
•
U c a le d 3 D im a ge s te nd to lo o k m uc h
ps
b e tte r th a n 2 D im a ge s
•
U e go o d a nti- a lia s ing
s 1470 pixels 30 pixel gap filled with any uniform colour 30 pixels
•
Lo w - re s a nti- a lia s e d im a ge s lo o k m uc h
b e tte r th a n h igh re s im a ge s w ith no a nti-
a lia s ing Right
720 pixels
71. Implementation on the PlayStation 3 – a 3 step process
®
Step 2: Apply convergence to
define the maximum positive
parallax
•
A 2D x axis translation in screen space
•
Shift the left image to the left and the right
image to the right
•
Normally use 1/30th screen width as the
default
alternating left/right frames
72. Implementation on the PlayStation 3 – a 3 step process
®
Step 2: Apply convergence to
define the maximum positive
parallax
•
Take care the convergence is applied
to all your rendering pipeline
alternating left/right frames
73. Implementation on the PlayStation 3 – a 3 step process
®
Step 3: Apply the interaxial by horizontally
offsetting the cameras
•
Keep the cameras parallel
•
Maximise the depth by increasing the interaxial
until the closest objects are just behind the screen
alternating left/right frames
•
Surprisingly, dynamically altering the interaxial is
not that noticeable to the viewer
interaxial = 2 X tan (horizontal field of view/2) x closest distance
30
74. Implementation on the PlayStation 3 – a 3 step process
®
Step 3: Apply the interaxial by horizontally
offsetting the cameras
•
Be careful with large interaxials – miniaturisation?
•
The field of view and camera distance can be
adjusted too
alternating left/right frames
•
Try to avoid window violations and depth conflicts
interaxial = 2 X tan (horizontal field of view/2) x closest distance
30
75. Implementation on the PlayStation 3 – a 3 step process
®
Step 3: Apply the interaxial by horizontally
offsetting the cameras
Take care with any 2D rendering tricks-
are they OK in 3D?
alternating left/right frames
Reflections will need to be calculated for each eye
92. Future work- a rendering technique used by Sony Pictures Entertainment
•
Pa ra lla x fro m de pth m a p
•
R nde r a
e s ingle im a ge a nd c re a te th e s e c o nd im a ge
us ing th e de pth m ap
93. Future work- a rendering technique used by Sony Pictures Entertainment
•
Pa ra lla x fro m de pth m a p
•
La rge pa ra lla x c re a te s ga ps w h ic h ne e d to b e b a c kfille d
•
Tra ns pa re nc y a nd re fle c tio ns w o n’ t w o rk w ith th is
a ppro a c h
•
Ne e d to c o m b ine it w ith full re nde ring fo r diffic ult
o b je c ts
•
Ta ke s a b o ut 3 % o f th e to ta l SPUtim e pe r 6 0 H fra m e
z
94. Acknowledgements
F u rt h e r re a din g :
Introduction to stereoscopic 3D:
Mendiburu, 3D Movie Making; Stereoscopic Digital Cinema from Script to Screen,2009
History References:
Reynaud, Tambrun & Timby Paris in 3D; from Stereoscope to Virtual Reality 1850-2000, 2000
F Drouin, The Stereoscope and Stereoscopic Photography 1894
