Seminar in AI, with chosen topic Hierarchical Task Networks. Here is provided an overview of HTNs, based on Erol(94) paper. An idea of HTNs application is given by presenting some example based on video-game domain and, in general, real-time application.

1. Hierarchical Planning in
Video Games
by Andrea Tucci

2. Outline

3. Outline
Introduction
Hierarchical Task Networks
Overview
Definition
Expressivity
Complexity
Applications
Video Games AI : A Hierarchical Task Networks approach
Planning in games
HTNs in games
Example of Study : Killzone
Conclusions
References
Outline
Introduction
HTNs
Video Games Ai : HTNs
Conclusions
References

4. Introduction

5. Introduction
Hierarchical Planning in a real-time environment
application : video games
Overview of Hierarchical Task Networks
Video Games AI, focusing on planning
Killzone : a commercial game with HTNs
Show issues and advantages of Hierarchical Planning in
dynamic worlds
Index
Introduction
HTNs
Video Games Ai : HTNs
Conclusions
References

6. Hierarchical Task
Networks

7. Overview - Classical planning vs HTNs
Classical AI planning : how reach a goal? → actions
an action is represented by an operator < c, e>, c is a precondition, e is the effect of
the action
states : configurations of the world generated by applying actions
planner aim is to find the (minimal) path from initial state to the goal state, the path is
composed of actions
Hierarchical Task Network (HTN) planning : actions to reach a (composite) task
task : composite or primitive
composite tasks have to be decomposed in subtasks
primitive tasks can be executed directly by applying an operator
planner decomposes compound tasks of a given task network until only primitive tasks
are found
Outline
Introduction
HTNs
>>Overview
>Definition
>Expressivity
>Complexity
>Applications
Video Games Ai : HTNs
Conclusions
References

8. Definition
Given a first order HTN language L , we have a vocabulary < V, C, P, F, T, N > where
V = { v1, v2, ... } is an infinite set of variables
C is a finite set of constant symbols
P is a finite set of predicate symbols
F is a finite set of primitive tasks
T is a finite set of compound tasks
N = {n1, n2 ... } is an infinite set of task labels
Goals replaced by tasks. A task can be:
goal task, tasks to achieve
primitive task, obtained by executing the corresponding action
compound task, require decomposition
e.g. “travelByPlane” may require subtasks like buyTicket, goToAirport, fly, goToDestination
Outline
Introduction
HTNs
>Overview
>>Definition
>Expressivity
>Complexity
>Applications
Video Games Ai : HTNs
Conclusions
References

9. Definition
A Task Network is a collection of tasks and constraints : (( n1 : α1), ... , (nm : αm), Φ)
Each αi is a task labeled ni
Φ is a boolean formula; can contain :
variable constraints (e.g x = y, x,y ∈ V or x ∈ V, y ∈ C )
temporal ordering (e.g n ≺ n’, n,n’ ∈ N)
truth constraints (e.g (n,l) i.e.“n must be true immediately after n, n ∈ N and l is a
literal)
Operator is of the form (f(v1, ..., vk),l1, ..., lk) where f ∈ F, v ∈ V and li is a literal
in this representation there are no preconditions in operators : they are expressed as goal tasks in
task networks
A method is a pair (α, d) where α is not a primitive task and d is a task network
This means : in order to achieve α, we use d (i.e. all its subtasks)
Example : (on(v1,v2),d) where d is a task network.
Outline
Introduction
HTNs
>Overview
>>Definition
>Expressivity
>Complexity
>Applications
Video Games Ai : HTNs
Conclusions
References

10. Definition
A planning problem is composed of:
an initial state
a task network, that encode the problem to be solved
a planning domain, that contains methods and operators
Outline
Introduction
HTNs
>Overview
>>Definition
>Expressivity
>Complexity
>Applications
Video Games Ai : HTNs
Conclusions
References

11. Definition
Given a planning problem with task network d, a rough Algorithm is :
1.Perform a compound task by decomposing it in subtask (use a method);
replace the compound task with the subtasks provided by the used method
2.Are there other compound tasks? If yes goto (1.);
3.Find a totally-ordered ground instantiation of d that satisfies all the
constraints.
Example (from blocks world domain) : put v1 on v2
Outline
Introduction
HTNs
>Overview
>>Definition
>Expressivity
>Complexity
>Applications
Video Games Ai : HTNs
Conclusions
References
Syntax used :
do precedes a primitive task
perform precedes a compound task
achieve precedes a goal task

12. Expressivity
HTNs provides all the STRIPS concepts (states, actions, goals)
translation of a STRIPS encoded domain to HTNs domain done in polynomial
time
Is the opposite true?
NO!
HTNs can express situations impossible to express using STRIPS operators
HTNs provide a constraint language to express many type of
interactions
STRIPS does not have the same level of abstraction
There exist HTN planning domains that can not be represented by
any finite number of STRIPS-style operators1
Outline
Introduction
HTNs
>Overview
>Definition
>>Expressivity
>Complexity
>Applications
Video Games Ai : HTNs
Conclusions
References

13. Complexity
A planning problem P can be
propositional : no variables are allowed
totally ordered : all tasks in any task network are totally ordered
regular : all the task networks contain max one non-primitive task and is the first or the last
task
primitive : the task network we need to plan for contains only primitive tasks
Plan existence is
strictly semi-decidable2;
decidable if methods are restricted to be acyclic3
is EXPSPACE-hard and DOUBLE-EXPTIME if the planning probem(P) is restricted to be totally ordered;
PSPACE-hard and EXP-TIME if it is propositional4
EXPSPACE-complete if P is regular5
PSPACE-complete if P is regular and propositional6
is NP-complete if P is primitive7
Outline
Introduction
HTNs
>Overview
>Definition
>Expressivity
>>Complexity
>Applications
Video Games Ai : HTNs
Conclusions
References

14. Complexity
Outline
Introduction
HTNs
>Overview
>Definition
>Expressivity
>>Complexity
>Applications
Video Games Ai : HTNs
Conclusions
References

15. Applications
Scheduling
Video Games
Automated Composition of Web Services
Evaluating Terrorist Threats
Evacuation
Military Tasks
Others
Outline
Introduction
HTNs
>Overview
>Definition
>Expressivity
>Complexity
>>Applications
Video Games Ai : HTNs
Conclusions
References

16. Applications - SHOP
Simple Hierarchical Ordered Planner 2 (SHOP 2) is a
hierarchical planner developed in 2001, written in LISP.
Outline
Introduction
HTNs
>Overview
>Definition
>Expressivity
>Complexity
>>Applications
Video Games Ai : HTNs
Conclusions
References
(defdomain basic-example (
(:operator (!pickup ?a) () () ((have ?
a)))
(:operator (!drop ?a) ((have ?a))
((have ?a)) ())
(:method (swap ?x ?y)
((have ?x))
((!drop ?x) (!pickup ?y))
((have ?y))
((!drop ?y) (!pickup ?x)))))
(defproblem problem1 basic-example
((have banjo)) ((swap banjo kiwi)))
Plans:
(((!DROP BANJO) (!PICKUP KIWI)))

17. Video Games AI :
A Hierarchical Task Networks
Approach

18. Planning in Games
Planning in games arises in the last years
Planning approach makes video games agents (bots,NPCs) more dynamic w.r.t. old approaches
Planning approach can split action planning from action execution
Planning increases autonomy of NPCs : they have alternate solutions to fulfill a goal
Planning can handle new situations
First Encounter Assault Recon (F.E.A.R.) developed by Monolith was the first commercial game
that used planning in its AI
in 2005 won “best AI award” from GameSpot
still considered as one of the best games for AI
GOAP : Goal Oriented Action Planning,decide what to do AND how to do it
Hierarchical planning : more natural in some situations (reasoning on tasks rather than
actions)
Outline
Introduction
HTNs
Video Games Ai : HTNs
>>Planning in Games
>HTNs in Games
>Example of Study
>Killzone
Conclusions
References

19. Planning in Games
Outline
Introduction
HTNs
Video Games Ai : HTNs
>>Planning in Games
>HTNs in Games
>Example of Study
>Killzone
Conclusions
References

20. Planning in Games
Outline
Introduction
HTNs
Video Games Ai : HTNs
>>Planning in Games
>HTNs in Games
>Example of Study
>Killzone
Conclusions
References

21. Hierarchical Task Networks in Games
Used in both individual and coordination decisions
In video games AI, actions can be dictated also by
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>>HTNs in Games
>Example of Study
>Killzone
Conclusions
References- error prone +little CPU overhead
- human effort
- case specific
Script
HTNs can be used as script generators
offline planning, then the output actions are converted to scripts
Other approach : real-time planning

22. Examples of Study
Understand how HTNs work in practice
Real-Time planning
How planners are integrated with the entire system
Design of HTN based agents
Differences with other implementations
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>>Example of Study
>Killzone
Conclusions
References

23. Killzone 2

24. Killzone 2
First Person Shooter, developed by Guerrilla Games published in 2009
Need for a new multiplayer bot system...why?
In its predecessor, Killzone, coordination between bots of the same team were
missing
Goals :
Extensible bot architecture
Improve Killzone AI
Include cooperative behavior
Outperform old AI
Entertain and challenge players
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References

25. Killzone 2
Quick overview of AI
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References
Strategy
Bot
Squad
•Goal-based
•Defines what to achieve in the
mission by using heuristics
•Creates squads•Execution of mission tasks
•Coordinate bots
(e.g. targets)
•Action synchronization
•Execution of
Squad task
• Inform their squad

26. Killzone 2 vs Killzone
Killzone 2
Killzone
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References

27. Killzone 2 vs Killzone (2)
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References
Killzone Killzone 2
Approach
Goal selection
Goal satisfaction
Atom part
Front end
Goal-based HTNs
Heuristics
High-level HTN
choices
Behavior Plan
Action Primitive task
Body goals Body goals

28. Killzone 2 - AI
Goal behavior system of Killzone, replaced by task planning mechanism
Improve flexibility
Dynamic : decomposition of a compound task may lead to different plans (i.e.
different behavior)
HTN planner based on SHOP syntax
Specialized primitive tasks : is possible to add facts in the HTN database for a certain
amount of time and delete them in any moment
(!remember ?period ?fact) (!forget ?fact)
HTN planner is used for evaluate the relevance of goals and find how to reach them
Knowledge domains provides how tasks are performed and also tasks priority
Agents use a combination of generalized and specialized knowledge rules
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References

29. Killzone 2 - AI (2)
What should AI do?
Decided by high-level HTNs
Compound task(s) as output
How agent should perform decisions?
Low-level HTNs decisions
Each agent has a set of daemons that provide info about the world
The planner requires those facts before it starts
HTNs used in squad
There is an invisible leader that uses planner for coordination purposes
Informations shared
Squad plans are series of orders for team members
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References

30. Killzone 2 - Squad AI
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References
Strategy
Team member
messages
Facts
Daemons
Planner Plan
Task
execution
Order to
members

31. Killzone 2 - AI (2)
(:method (order_member_defend ?inp_member ?inp_id ?inp_level ?inp_marker ?inp_context_hint)
…
(branch_advance
()
(
(!forget member_status ?inp_member **)
(!remember - member_status ?inp_member ordered_to_defend ?inp_id)
(!start_command_sequence ?inp_member ?inp_level 1)
(do_announce_destination_waypoint_to_member ?inp_member)
(!order ?inp_member clear_area_filter)
(!order ?inp_member
set_area_restrictions (call find_areas_to_wp ?inp_member (call get_entity_wp ?
inp_marker)))
(!order_custom ?inp_member move_to_defend ?inp_marker ?inp_context_hint)
(!order_custom ?inp_member send_member_message_custom completed_defend ?inp_id)
(order_set_defend_area_restriction ?inp_member (call get_entity_area ?inp_marker))
(!order_custom ?inp_member defend_marker ?inp_marker ?inp_context_hint)
(!end_command_sequence ?inp_member)
)
)
)
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References

32. Killzone 2 - AI (2)
HTN used in individual bot’s decisions
combat behavior
mission specific tasks
other (heal, repair ... )
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References(:method (select_weapon_and_attack_as_turret ?inp_threat)
(branch_use_bullets // Only use bullets against humanoids and turrets.
(and
(or (threat ?inp_threat humanoid) (threat ?inp_threat turret) )
(distance_to_threat ?inp_threat ?threat_distance)
(call lt ?threat_distance @weapon_bullet_max_range) )
((attack_as_turret_using_weapon_pref ?inp_threat wp_bullets))
)
( branch_use_rockets // Don't use rockets against humanoids and turrets.
(and
(not (threat ?inp_threat humanoid)) (not (threat ?inp_threat turret))
(distance_to_threat ?inp_threat ?threat_distance)
(call lt ?threat_distance @weapon_rocket_max_range) )
((attack_as_turret_using_weapon_pref ?inp_threat wp_rockets))
))

33. Killzone 2 - agent design
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References

34. Killzone 2 - real time environment
Planning ahead is not so good, we are in a real-time environment
Long plans tend to invalidate
☛ Local plans are made
Planning done 5 times per second, 2 times per second for squad
Invalidation of a decided plan of actions : better plan or changing environment
Problem : re-planning so often is
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References
HEAVY
How to deal with this?

35. Killzone 2 - real time environment (2)
While re-planning, if we come up with a current task, just update it.
Only decompose the part of the plan which we are interested at the moment, delaying
the other.
Re-plan only if a certain percentage of preconditions is not valid anymore (i.e. the world
changed too much)
Strategies for optimize re-planning
tasks updating
subtasks priority order changes
partial decomposition of tasks
delayed decomposition of tasks
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References

36. Killzone 2 - conclusions
Old AI (bots from Domination game mode) vs new AI (Group Strategy) : 174 tests
battles in a multiplayer modality
Outline
Introduction
HTNs
Video Games Ai : HTNs
>Planning in Games
>HTNs in Games
>Example of Study
>>Killzone
Conclusions
References

37. Conclusions

38. Conclusions
Overview of HTNs
Use of HTNs in video games
offline planning (e.g. Script generation)
online planning
Real-Time (HTN) planning
much more interesting
deal with re-planning
Outline
Introduction
HTNs
Video Games Ai : HTNs
>>Conclusions
References

39. Conclusions - HTN in real-time environment
Make planners for real-time environments
build connection to the problem to be solved ... but is like “hard-code” the planner
make full logic representation, gaining expressive power and more generality ... but loosing
efficiency
How can we maintain a “generic” planner with efficiency?
Some optimizations:
Use critics to prevent backtracking in an unfeasible plan
critics are used to remove conflicts within a plan as soon as possible
Partial re-planning
why re-plan an entire plan? We can adapt it :(example) agent A needs an object
O, so a plan is made. A can’t find O, but he can replace it with another object that
has the same function, no need to re-plan for the whole task (go to the shop etc..)
Outline
Introduction
HTNs
Video Games Ai : HTNs
>>Conclusions
References

40. Conclusions - HTN in real-time environment
Domain compilation
Introduced for JSHOP planner
unique instance of the planner, generated compiling both the domain and the problem
☛ Expert planner for that problem
In real-time environment we only can compile the domain
Time slicing
Allows to pause and restart the planner
Spread time of planning in more frame
☛ Characters are able to perceive events
Other optimizations based on the problem
Make precondition checking light : heavy computations (e.g. pathfinding) are done by sensors of the agent and
stored in a working memory
☛ Sensors provide up-to-date data to the planner
Caching perceptions as working memory facts, containing important informations
Garbage collector : a sensor should delete a fact added that is no relevant anymore
Outline
Introduction
HTNs
Video Games Ai : HTNs
>>Conclusions
References

41. Conclusions - HTN in real-time environment
So...are HTNs good or not? Can be used well in a real time
environment?
In general, planning in more than one layer is better
at any given level of the HTN, the number of applicable
operators is smaller than the total number (reduction of a
search space)
invalid plan can be detected at a top level and never be
considered for expansion
more suitable for dynamic world : support of re-planning is
good
Outline
Introduction
HTNs
Video Games Ai : HTNs
>>Conclusions
References

42. References
Presentation based on
HTN Planning : Complexity and Expressivity (K. Erol, J. Hendler, D. Nau)
Ms thesis : A hierarchically-layered multiplayer bot system for a first-person shooter (T. Verweij)
Papers consulted
SquadSmart Hierarchical Planning and Coordinated Plan Execution for Squads of Characters (P.
Gorniak, I. Davis)
Agent Architecture Considerations for Real-Time Planning in Games (J. Orkin)
Hierarchical Plan Representations for Encoding Strategic Game AI (H. Hoang, S. Lee-Urban, H.
Muñoz-Avila
Planning with Hierarchical Task Networks in Video Games (J.Kelly, A. Botea, S. Koening)
Other material
Hierarchical Planning in Dynamic Worlds (N.Wallace, Ai Programming Wisdom vol. 2)
Killzone 2 Multiplayer Bots slides (Slides by A.Champandard, T.Verweij, R.Straatman)
Images of STRIPS-based and HTNs-based games, are taken from AiGameDev.com
Outline
Introduction
HTNs
Video Games Ai : HTNs
Conclusions
>>References

43. References
Notes (from HTN Planning : Complexity and Expressivity)
1. Theorem 7
2. Theorem 1
3. Theorem 2
4. Theorem 3
5. Theorem 4
6. Theorem 5
7. Theorem 6
Outline
Introduction
HTNs
Video Games Ai : HTNs
Conclusions
>>References

44. Thanks
Thank you for your attention!
@andreatux
Outline
Introduction
HTNs
Video Games Ai : HTNs
Conclusions
References