Presented at FITC Toronto 2019 More info at www.fitc.ca/toronto Amelie Rosser Jam3 Overview For the past two years Jam3 worked alongside Joy Kogawa and the NFB to create East of the Rockies, an augmented reality storytelling experience. East of the Rockies is the first interactive AR game of its kind. The story takes users through a piece of Canadian history where Japanese Canadians were forced to leave their homes and live at internment camps during WWII. This talk will cover the creation of the game: from concept and storyboarding, to the development process in Unity and various challenges and questions to consider from a creator’s perspective. Objective To let the audience in on the behind the scenes of developing an AR experience like East of the Rockies. Target Audience For those interested in Augmented Reality storytelling and game development. Five Things Audience Members Will Learn AR techniques using Unity Storytelling in AR Prototyping interactions in AR Game state management using Unidux Game optimization techniques in Unity

1. FITCToronto2019
1 TORONTO / LOS ANGELES / MONTEVIDEO
East of the Rockies:
Developing an AR
Game

2. FITCToronto2019
2
Intro - Historical context - Internment
1942
- Imperial Japanese Army
bombed Pearl Harbour
- The Canadian government
declared Japanese Canadians
as the enemy
- In British Columbia they were
forced to leave their homes
and businesses
- They were pushed into
internment camps

3. FITCToronto2019
3
Joy Kogawa
Author, poet, activist, order of Canada
- Joy Kogawa was one of
those interned Japanese
Canadians.
- She was able to turn these
negative experiences into
creative outputs
- She wrote Obasan and was
rewarded with Order of
British Columbia and Japan's
Order of the Rising Sun.
FITCToronto2019

4. FITCToronto2019
4
Intro - Jason and Dirk’s initiative
Jason Legge and Dirk van Ginkel
wanted to do something for the 150th
anniversary of Canada.
- Wanted to highlight Canada's
injustices as well as celebrate
their achievements
- Jason was always a fan of Joy
- They met and played video
games together
- Joy was on board!

6. FITCToronto2019
6
East of the Rockies Partners

7. FITCToronto2019
7
Our goal was to shed light and
acknowledge Canada’s past
injustices
We wanted to use immersive storytelling to educate and
empower our audience.

8. FITCToronto2019
8
Our Audience
Gen Z’ers are digital natives ready to embrace brands and
services that understand their needs and interests. “When we
match a digital relationship with the influence of storytelling,
we have the ability to empower young people to make a
difference.” – Huffington Post
Gen Z’ers
- 55% are willing to pay for products and services that
push forward social impact issues –Neilson, 2015
- The most diverse and inclusive generation – Business
Insider
- Want to make a difference and change the world –
Forbes
- Crave innovative ways to consume content and
embrace new forms of storytelling – Forbes

9. FITCToronto2019
9
Augmented Reality
What I find amazing about this technology is the fact it’s integrated into your phone or tablet, so
there’s no extra hardware barrier for people to start using the content.
Augmented Reality
- Augmented Reality has become a
popular way for people to
experience content
- There’s no hardware barriers

10. FITCToronto2019
10
Intro - EOTR Script
We started with
a script.
- Joy Kogawa teamed up with
Walt Williams
- Jason and Dirk adapted the
script for gameplay

11. FITCToronto2019
11
- The narrative is told from the
perspective of Yuki, a 17-year-old girl
- The story follows Yuki as she and her
family experience good times as well
as the bad
- Joy’s granddaughter Anne voices Yuki

12. FITCToronto2019
12
Our next step as a team
was to breakdown the
script into a storyboard.
Storyboarding

13. FITCToronto2019
13
Storyboarding - Legend
- We created a legend to
breakdown the narrative and
gameplay
- A pattern started to develop for
the main gameplay objectives
and interactions

14. FITCToronto2019
14

15. FITCToronto2019
15
Storyboarding - Act 1, Scene 2

16. FITCToronto2019
16
UX
- With the initial storyboarding in place we
would then begin to map out a clear user
journey path
- UX then starts to define the gameplay
journey
- We’d define key interactions
- Optional interactions to enrich the
storytelling
UX MappingFITCToronto2019

17. FITCToronto2019
17
Gameplay structure
We started with another legend, this time
breaking down the gameplay
- Critical path for the users
- Side and optional interactions

18. FITCToronto2019
18
Gameplay structure

19. FITCToronto2019
19

20. FITCToronto2019
20
Gameplay breakdown by location
- Gameplay was then broken down
by location
- This helped us understand the
order interactions would go in

21. FITCToronto2019
21
Draw Skippy With Tomio 🎯
Align device UI on the dots to connect them and
reveal the drawing (x5 dots)
5
5
Open Luggages 👆
Tap on locks to open the luggages (x2 locks)
1
4
Light On Stove 👆
Tap on stove to light it on
4
1
Open Curtain ⚡
Swipe on curtain to open & close it
2
2
3
3
Chop Logs ⚡
Swipe on logs to chop them (x5 logs)
Act 1 - Scene 2 - Yuki’s House

22. FITCToronto2019
22
6
1
2
5
3
4
Interact With Mrs Tucker 👆
Tap on Mrs Tucker’s to trigger news & radio
6
Raise Union Jack Flag 🎯
Align device UI on the dots to raise the flag (x3
dots)
1
Swipe Steam ⚡
Swipe to remove steam from the screen
5
Close Bag With Japanese Doll ⚡
Swipe to the bottom to close the bag
2
3
Look Through Walls 🔍
Get closer to look through the walls of the
Bathhouse & the Odd Fellows Hall.
4
Interact With NPC Characters 👆
Tap on NPC characters to trigger their actions (x3
characters)
Act 1 - Scene 5 - Camp Scene

23. FITCToronto2019

24. FITCToronto2019
24
FITCToronto2019
- Harshness of winter
- Only sources of heat was the stove

25. FITCToronto2019

26. FITCToronto2019

27. FITCToronto2019
27
UI Discovery
- Initial exploration into how
we would overlay
graphical elements within
the game

28. FITCToronto2019
28

29. FITCToronto2019
29

30. FITCToronto2019
30

31. FITCToronto2019
31
Technical Discovery
When we reached technical kickoff we had a lot of areas to explore.
Our key areas to focus on were:
- Game engines
- Augmented Reality
- Interactions
- Lighting and rendering
- Visual fx
Our team already had knowledge of these areas but not within Unity.
We needed to figure these things out to give us insight on the right
way forward

32. FITCToronto2019
32
Development - Technical Discovery - Deciding on a Game Engine
- Our team had the most experience
with it
- Team workflow was important - two
options Unity Collaborate and Git
- Big active and open community
Choosing Unity

33. FITCToronto2019
33
We used Vuforia and
it was hell.
Development - Technical Discovery - Vuforia
Pros Cons
Cross Platform
In Editor Simulator
Updating Unity would
often break Vuforia
API and examples
were not
straightforward
Performance was
bad at times, drifting
occured
In-Editor simulation
eventually broke from
an update

34. FITCToronto2019
34
- We prototyped with it to test the api and performance
- We noticed less drifting and a performance increase
- The API is completely open source
- Unity are also working on a remote AR tool
https://forum.unity.com/threads/ar-remote-tool-alpha-
preview.573229/
- When removing the Vuforia Framework from our project,
XCode failed to build . . .
Development - Technical Discovery - AR Foundation
Meanwhile, Unity
released AR Foundation

35. FITCToronto2019
35
We needed a way to focus
the players attention within
the world
Development - Technical Discovery - Content scaling
- We couldn’t scale the models as it would break the
lighting
- However, we found an article by Unity that
documented two approaches we could use
https://blogs.unity3d.com/2017/11/16/dealing-with-scale-in-ar/

36. FITCToronto2019
36
Development - Technical Discovery - Content scaling

37. FITCToronto2019
37
Interactions needed a manager
- We used a web inspired approach to
handle touches / mouse interactions
- Attach a script to a gameobject to
make it interactable
- Raycasting is expensive
Development - Technical Discovery - Interactions

38. FITCToronto2019
38
Inspect interaction
- Object would animate
towards the camera
- User could turn the item
around and find details
- Seperate camera attached to
the main camera

39. FITCToronto2019
39
Development - Technical Discovery -
Lighting explorations
Unity Concept art

40. FITCToronto2019
40
Rig lighting explorations continued

41. FITCToronto2019
41
World Masking
- Mask to focus the region of
the world
- We called it the
“circle of occlusion”
- Mask happened in the
fragment shader
- Radius of the mask needed to
scale with the camera zoom
- Unity Post processing caused
us problems

42. FITCToronto2019
42
Assets
- We worked with two Technical Artists
for the project that created the 3D
game assets
- Models were created in Meters to align
with Unity’s measurement space
- Avoid textures where possible for
performance
- Houdini, Zbrush and Maya
Game Assets

43. FITCToronto2019
43
Development - Assets - Import settings
- Import assets with the right settings
- Track memory usage with the profiler
- Use streaming sounds to reduce
memory overload
- Set max texture size
- Use the memory profiler & look for
unusual memory sizes
Unity Asset Import Tips

44. FITCToronto2019
44
Colour Palette
Creating a cohesive colour tone
https://docs.unity3d.com/Manual/DrawCallBatching.html
- We created a reusable colour palette of
multiple materials to use throughout the game

45. FITCToronto2019
45
Assets - Characters
High poly sculpt of Yuki (that turned out we couldn’t use)

46. FITCToronto2019
46
Assets - Characters
The Tanaka Family

47. FITCToronto2019
47
Development - Assets - Character animations
- A decision early on in the project was to use
motion capture suits for the character
animation
- In theory it sounded like a great idea since we
required a lot of animations for the game

48. FITCToronto2019
48
Development - Assets - Character animations
- We later discovered though a lot of the
mocapped animations would require cleaning
up.
- We came to the conclusion we should have
opted for an animator instead
Oh dear.

49. FITCToronto2019
49
Assets - Buildings
- Models separated into
prefabs
- Combed prefabs together to
build the world
- Static objects together,
dynamic game objects into
gameplay prefab

50. FITCToronto2019
50
Assets - Buildings
- Gameplay prefab
- Objects require realtime
lights
- Interactive objects would
appear here

51. FITCToronto2019
51
Scene Structure
This is the typical structure for a
scene in the game
- Gameplay prefab
progresses user through the
story
- Scene prefab attached to
the AR anchor point

52. FITCToronto2019
52
Assets - Photographs

53. FITCToronto2019
53
Assets - Photographs

54. FITCToronto2019
54
We then needed a solution
for the team to collaborate
on the project
- We had up to 5 developers on the project
- It needed to be based on Git
Collaboration Strategy

55. FITCToronto2019
55
Development - Workflow - Unity Collaborate vs Git
Unity Collaborate
- Team licensing
- Basic interface
- Limited branch features
(no git flow)
Github
- No limit on contributors
- Use the tools we know and love
- Git flow workflow

56. FITCToronto2019
56
Development - Setting up Git
- Github for Unity was still in it’s early stages
- Use .gitattributes and .gitignore
- Used Gitflow
Branch Structure:
- Master
- Develop
- Feature
- [Jira-ticket-id]-feature-name
Setting up Git = lots of trial and error!

57. FITCToronto2019
57
Development - Game Development
Every Game Developers
nightmare.
- We wanted our game architecture
to be clean!

58. FITCToronto2019
58
- Our team has a strong web development
background
- We needed a robust approach to managing the
core logic of our game
- We came across Unidux, a Redux inspired
application architecture
- Features:
- Includes a useful UI inspector for your
store’s state.
- State is serialised to json
- We built on top of it to add in the missing
features we need
Development - Game Development - Unidux

59. FITCToronto2019
59
Development - Game Architecture - UniduxUnidux Inspector
Unidux Inspector
- Redux dev tools equivalent

60. FITCToronto2019
60
Development - Game Architecture - Unidux
Game reducers used

61. FITCToronto2019
61
Action Creators
Action Creators for a
gameplay scene

62. FITCToronto2019
62
Development - Game Architecture - Unidux
Reducer for a gameplay scene

63. FITCToronto2019
63
Subscribe
- Subscribing to the store was
simple
- Unity OnDestroy would
automatically unsubscribe

64. FITCToronto2019
64
Handle Props Changes
- This was the manual part
- Handle prop changes ourselves to
see what changes

65. FITCToronto2019
65
Props Changed
If props have changed, do something.

66. FITCToronto2019
66
The Game Prefab is the brain
- Contains all core logic
- Every scene in the game needs it
- A singleton and never gets destroyed
Development - Game Development - Game Prefab

67. FITCToronto2019
67
Development - Closed Captions
The game required
both English and French
- We used https://poeditor.com as our CMS
- Exported JSON files to use in-game

68. FITCToronto2019
68
Development - Localised Text Script
We have a localised text script which
fetches the text and renders it

69. FITCToronto2019
69
Development - Voice Over
Voice over required a
different approach
- Youtube for captions
- Node scripts to fetch VTT and export to json

70. FITCToronto2019
70
We stored game assets in Asset Bundles
- Included in the app and loaded
from Streaming Assets
- Unity depreciated their ABM so we
created our own to suit our needs
Development - Game Development - Asset Bundles

71. FITCToronto2019
71
Scene Flow Structure
- Each scene would start with a voice over
and a scripted timeline
- Once the voice over ends interactions
would activate
- We created our own solution to
subscribe to timeline events
- Sometimes things would go wrong...
Development - Scene Structure

72. FITCToronto2019
72
Development - Game Development - Building for testing
Building for devices would require a few steps:
- If content changed in the asset bundles, asset
bundles would need to be re-generated which
would take ~4 mins to build
- If the changes we’re only scripts we could build
the Xcode project straight away
- Then building the Game’s Xcode project would
take 1 minute
- It could take up to 5 mins to test changes on a
device

73. FITCToronto2019
73
Unity & UI
- NO HTML CSS :(
- Unity’s UI system takes time to learn
- How we approached it
Unity’s UI

74. FITCToronto2019
74
Our main game UI lived in our Game Prefab
- We used Canvas scale with resolution
setting
- We should of made the UI more
component based
Development - UI - Canvas

75. FITCToronto2019
75
Key Learnings & Insights
- Start with a solid game architecture
- Define a good naming convention for Assets
- Always keep an eye on memory
- Make sure the Unity project is kept clean, avoid
messy folders
- Ensure code is reviewed by a lead developer
- Keep the workflow optimised and fast for device
testing

76. FITCToronto2019
76
IDFA
- We showed the game off at IDFA -
International Film and Documentary festival
in Amsterdam
- It gave us a good insight to how people
played the game
- It allowed us time for final UX amends for
release

77. FITCToronto2019
77

78. FITCToronto2019
78
We reached game release!
2+
Years in
development
2000+
Hours worked on
the project
~87
Pizzas eaten
(we can’t be too sure)

79. FITCToronto2019
79
Results and Reception
Game Launch
- We finally launched the game
- Apple promoted us for App of the day
- Maintaining a rating of 4.5 or higher
100,000+
Downloads in the first
two weeks
Rank in Education
category
#3

80. FITCToronto2019
80
FITCToronto2019

81. FITCToronto2019
81
FITCToronto2019

82. FITCToronto2019

83. FITCToronto2019 The Bathhouse

84. FITCToronto2019
84
Launch - JCC
We showed the app of at the Japanese
Canadian Centre
- It was the first time showcasing the full
experience to members of the Japanese
Canadian community
- Some had been in internment camps
themselves as young children