*** Presentation created by Federico Liberatore *** Flocking strategies are sets of behavior rules for the interaction of agents that allow to devise controllers with reduced complexity that generate emerging behavior. In this paper, we present an application of genetic algorithms and flocking strategies to control the Ghost Team in the game Ms. Pac-Man. In particular, we define flocking strategies for the Ghost Team and optimize them for robustness with respect to the stochastic elements of the game and effectivity against different possible opponents by means of genetic algorithm. The performance of the methodology proposed is tested and compared with that of other standard controllers. The results show that fl ocking strategies are capable of modeling complex behaviors and produce effective and challenging agents.

1.
Evolving Evil: Optimizing Flocking
Strategies through Genetic
Algorithms for the Ghost Team in
the Game of Ms. Pac­Man
F. Liberatore
A. Mora
P. Castillo
J.J. Merelo

2.
Outline
­ The Game of Ms. Pac­Man.
­ State of the Art.
­ Flocking.
­ Flocking Strategies for the Ghost
Team.
­ A Genetic Algorithm for Flocking
Strategies.
­ Experiments.
­ Conclusions.

3.
What is this about?
In this work we propose:
­ A Controller for the Ghost
Team in the game of Ms. Pac­Man
…
­ … based on Flocking
Strategies (more about this in
the following) …
­ … optimized by a Genetic
Algorithm.

4.
The game of Ms. Pac­Man
[1/2]
The game of Ms. Pac­Man:
­ Released on 1981.
­ Chosen for the Ms. Pac­Man vs
Ghosts competition.
­ Participants can submit
controllers for Ms.Pac­Man and
the Ghosts.
­ The objective of Ms. Pac­Man
is to maximize the final score.
­ The objective of the Ghosts is
to minimize it.

5.
The game of Ms. Pac­Man
[2/2]
Movement restrictions for the
Ghosts:
­ A ghost can never stop.
­ A ghost can choose its
direction only at a junction.
­ A ghost cannot turn back on
itself.
­ At every tick of the game all
the ghosts obligatorily reverse
their direction according to a
small random probability.

6.
State of the Art [1/2]
Most contributions proposed controllers for Ms.
Pac­Man:
­ Genetic Programming (Alhejali and Lucas 2010, 2011,
Brandstetter and Ahmadi 2012),
­ Genetic Algorithms for parameters optimization
(Thawonmas 2010),
­ Evolved Neural Networks (Burrow and Lucas 2009,
Keunhyun and Sung­Bae 2010),
­ Ant Colonies (Martin et al. 2010),
­ Monte Carlo Tree Search (Samothrakis et al. 2011,
Ikehata and Ito 2011, Alhejali and Lucas 2013),
­ Reinforcement Learning (Bom et al. 2013).

7.
State of the Art [2/2]
Controllers for the Ghosts:
­ Monte Carlo Tree Search (Nguyen and
Thawonmas 2011, 2013),
­ Influence Maps (Svensson and Johansson
2012),
­ Neural Networks (Jia­Yue et al. 2011),
­ Genetic Algorithm + Rules (Gagne and
Congdon 2012),
­ Competitive Co­Evolution of Ms. Pac­Man and
Ghosts controllers (Cardona et al. 2013).

8.
Flocking [1/2]
Swarm Intelligence technique:
­ Coordinated intelligence that arises from the
collective behavior of decentralized,
self­organized systems.
Flocking:
­ Developed to mimic lifelike behaviors of groups
of beings.
­ Flocking systems consist of a population of
simple agents (or boids) interacting locally with
one another depending on the distance between
them.

9.
Flocking [2/2]
­ The agents follow very simple steering
behaviors:
∙ Separation: makes the agent steer away from close
flock mates.
∙ Alignment: makes the agent steer toward the
average heading of the flock.
∙ Cohesion: makes the agent steer toward the
average position of distant flock mates.
­ The interactions between such agents lead to
the emergence of "intelligent" global behavior

10.
Flocking Strategies [1/4]
­ Generalization of the concept of Flocking.
­ The direction taken by a ghost at a junction
depends on:
∙ State:
­ Normal,
­ Vulnerable,
­ Flashing.
∙ Position WRT that of the other actors:
­ Ms. Pac­Man,
­ Powerpills,
­ Other Ghosts (depending on their state).

11.
Flocking Strategies [2/4]
­ Each Ghost is surrounded by
concentric rings called
Neighborhoods.
­ The neighborhoods can have
different radius and size.
­ Each neighborhood defines
the strength of the
interaction between the ghost
and the other actors:
∙ Negative value Separation→
∙ Positive value Cohesion→

12.
Flocking Strategies [3/4]
Formally, a Flocking Strategy is represented
by a tuple (N, δ, Α):
­ N, number of neighborhoods, indexed by n=1,…,N.
­ δ∈ℝN

, vector of N real positive numbers. Each
element δn
is associated to one neighborhood and
represents its maximum radius.
­ Α∈Μ, three­dimensional matrix with real
entries. Each element αs,a,n
is the magnitude of
the steering force on the current ghost in state
s, resulting from the interaction with the actor
a, falling into neighborhood n.

13.
Flocking Strategies [4/4]
­ First, all the interactions
with the other actors are
calculated as vectors.
­ Then, all the vectors are
summed to find the total
interaction vector.
­ Finally, the direction
taken by the ghost (UP, DOWN,
LEFT, RIGHT) depends on the
component of the total
interaction vector having
maximum absolute value.

14.
A Genetic Algorithm for
Flocking Strategies [1/2]
Flocking Strategies could be designed manually
by an expert.
We propose a standard Genetic Algorithm for
the automatic definition of effective Ghosts
agents:
­ Each individual is a Flocking Strategy,
represented as a tuple (N, δ, Α).
­ Random initial population.
­ Roulette wheel selection.
­ Cross­over: two parents generate one child by
random recombination of their chromosomes.

15.
A Genetic Algorithm for
Flocking Strategies [2/2]
­ Fitness function:
∙ The objective is to create effective Ghosts that
perform well against different Ms. Pac­Man strategies.
∙ Each individual is tested against two Ms. Pac­Man
agents: StarterPacMan (SPM) and NearestPillPacMan
(NPPM).
∙ To ensure robustness in the face of random events,
the game is simulated 30 times for each Ms. Pac­Man
agent and 95% confidence intervals are calculated.
∙ Fitness is given by:
FITNESS = 1/CI+
SPM
+ 1/CI+
NPPM
Where CI+
represents the upper limit of the 95%
confidence interval.

16.
Experiments [1/2]
Comparison of the performance of the Ghosts
controllers obtained with different values of
the parameter N (number of Neighborhoods)
N=1 N=2 N=3 N=4 N=5
Best
FITNESS­1
783.38 726.84 815.66 766.96 720.20
Avg.
FITNESS­1
871.30 ±
55.45
861.57 ±
70.13
876.31 ±
65.06
905.86 ±
62.66
863.17 ±
72.15
Avg. CPU
time (s)
1373 ±
150.66
1484.3 ±
122.01
1561 ±
193.94
1562.60
± 109.90
1473.00
± 74.02

17.
Experiments [2/2]
Performances of the controllers included in
the competition framework.
Controller Aggressive
Ghosts
Legacy Legacy 2
The
Reckoning
Random
Ghosts
Starter
Ghosts
FITNESS­1
1893.13 2210.9 1429.20 4200.70 1603.49

18.
Conclusions [1/2]
­ Sample controller (link)
­ Flocking Strategies: sets of behavior rules that
determine the next move of an agent as a force
resulting from the interaction of the agents in
the game.
­ Genetic Algorithm presented to design optimized
strategies.
­ The fitness function evaluates each individual
by pitting it against two Ms. Pac­Man controllers
30 times, so as to avoid noise in the function.

19.
Conclusions [2/2]
­ Future lines of research:
∙ Include in the fitness function the best Ms.
Pac­Man controllers that took part to the
competition.
∙ Compare the Flocking Strategy­based
controller with the best Ghosts controllers that
took part to the competition.
∙ Explore optimization methods other than
Genetic Algorithms.
∙ Change the fitness function structure to
reduce the computation requirements.

20.
The End!
Questions?