Games have been leveraging AI since the 1950s, when people built a rules-based AI engine that played tic-tac-toe. With technological advances over the years, AI has become increasingly popular and widely used in the gaming industry. The typical characteristics of games and game development makes them an ideal playground for practicing and implementing AI techniques, especially deep learning and reinforcement learning. Most games are well scoped; it is relatively easy to generate and use the data; and states/actions/rewards are relatively clear. In this talk, I will show a couple of use cases where ML/AI helps in-game development and enhances player experience. Examples include AI agents playing game and services that provide personalized experience to players.

1. AI in Gaming
Long Lin
Director of Engineering
Data & AI @ EA
lolin@ea.com

2. 2

4. “At EA, we envision a future in which games go
even further beyond the immersive experiences
players enjoy today. I’m talking about games
that offer living, breathing worlds that
constantly evolve.”
- Ken Moss

5. AI Everywhere
Creating games:
• AI Agent & Simulation
• Animation
• AI-driven game balancing
• AI development tools
Operating games:
• Bad actor detection
• Content creation
• Player acquisition
Playing games:
• Personalization
• Mentors & Assistants
• Matchmakers
• Conversational interfaces
• Dynamic experiences
Not an exhaustive list

6. Games Have Been Instrumental to AI
● Crushing humans is not the only use-case of AI in games
● Can we create AI agents that play the games similar to humans do
● Can we leverage AI to benefit games

7. Gaming - Playground for AI
• Human Interaction
• All virtual - faster iteration
• Can be well scoped
• Relatively easy to generate and use the data
• States/Actions/Rewards are relatively clear
• Result is measurable, and can be visualized

8. AI Agents
Agents that learn to play the game with different goals
● NPC
● Simulation
● Exploration of game space
● Game design
● Game balancing
● Optimal solution
● Fast test and feedback loops
● Find defects

9. ● Dev Build
● Fast Simulation
● Near-Realtime
Metrics
● Multiple agents

10. Designer Questions
● Is there a significant imbalance in
relationship categories?
● How many actions are needed to
progress in the careers?
● How do objects impact career
progress?
● How much impact did the build
changes actually make, and does
these change align with the
designer’s plan?

11. Markov Decision Process
Tuple (S, A, P, R)
● States: S
● Action: A
● State Dynamics/Simulator: P(S,A) → S’
● Reward: R
Learn a Policy: π(S) → A
*Randomness of Simulator and Policy

12. AI Agents Can be Difficult to Train
• Large state space
• Large action space
• Large number of steps in game episodes
• Reward sparse environments
• Simultaneous actions
• Multi-agent interaction
• Complex reward function and long term strategy

13. Luckily
• Better/cheaper computing power
• Distributed model - Computation, Data
• A lot of Data
• More mature AI libraries & frameworks
• Framebuffer vs Game state parameters, Joystick vs abstract actions
• Scope the problem - start with something simple
• Does not have to start from scratch - Demonstration & IL

14. Game Interface
Platform Game

15. Reinforcement Learning
Images by Stephane Ross
Calculate Reward
Initialize Policy Trials
Update Policy

16. (Deep) Reinforcement Learning
Learning from Rewards, Trials and Errors
Goal: Optimal solution (i.e., maximize cumulative reward)
Pros:
➔ Explore the world beyond
the skill of experts
➔ Superhuman game play
➔ Fast simulation/iteration
Cons:
➔ Complexity around Knowledge
representation for non-framebuffer
approach
➔ Algorithmic efficiency (time to converge,
might not converge) in transfer data to
policy depends on high efficient
representation of knowledge

17. Imitation Learning
Images by Stephane Ross
Expert Demonstration State/Action Pair Policy

18. (Deep) Imitation Learning
Learning from experts/players’s demonstration & feedback
Goal: Human-like behavior (i.e., minimize the difference between policy and the
demonstration)
Pros:
➔ Simple, efficient
➔ Works well when at state
space that has enough
demonstrations coverage
➔ Great at picking up styles
Cons:
➔ Limited state-space coverage of
the expert data, tend to over-fit
➔ Limited by the speed/scale
human player can generate data
➔ No long term planning

19. IL & RL
Images by Stephane Ross
Calculate Reward
Initialize Policy Trials
Update Policy
Expert
Demonstration
State/Action
Pair

20. Agent Training Workflow
Platform Game
● Training Environment
● Policy Storage
● Agent Management
● Agent Execution
● Data Pipeline

21. Looking Ahead
• State of Art Methodologies
• More Complex Environment
• Multi-Agent Interaction
• Distributed Training

22. Q&A