Disclaimer: the slide deck does not contain all the content, this was a mostly spoken/hands-on talk. Code used in the talk: http://github.com/inequation/ggd Video games are highly complex and indeterministic systems working closely to hardware. This means that sometimes the regular troubleshooting measure of setting up breakpoints just doesn't cut it. How to filter out noise, when our breakpoint is hit dozens of thousands of times per second? Or if our bug, despite perfect reproduction rate, manifests itself in a different way each time? How to find the culprit overwriting our precious data? This talk will present some interesting cases from the life of a game programmer along with tips and tricks to have up your sleeve when fighting them.

1. inequation.org
inequation.org
Gamedev-grade debugging
Leszek Godlewski
Freelance Programmer
lg@inequation.org
SpreadIT 2013 · October 19th, 2013
SpreadIT 2013 · October 19th, 2013

2. Code snippets
Code snippets
All code used in the talk available online:
github.com/inequation/ggd
2
2
inequation.org
inequation.org

3. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
3
3
inequation.org
inequation.org

4. Who is this guy?
Who is this guy?
Freelance Programmer
●
●
●
(Sep 2013 – onwards)
inequation.org
Painkiller Hell & Damnation
Linux port finalization (2013)
Unannounced project
Generalist Programmer,
The Farm 51
●
●
●
●
4
4
(Mar 2010 – Aug 2013)
thefarm51.com
Painkiller Hell & Damnation
(2012-2013; Win/Linux/X360/PS3)
Deadfall Adventures
(2011-2013; Win/X360)
A few unannounced projects
inequation.org
inequation.org

5. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
5
5
inequation.org
inequation.org

6. Why the talk?
Why the talk?
Because THIS has happened to me:
http://imgur.com/yBa1OGm
6
6
inequation.org
inequation.org

7. Why the talk?
Why the talk?
Intern: I've read* at all the code and
still don't see the bug.
Me:
So just debug it!
*read, as in „stare without actually running it”
7
7
inequation.org
inequation.org

8. Why the talk?
Why the talk?
Intern:
© DreamWorks
8
8
http://imgur.com/yBa1OGm
inequation.org
inequation.org

9. Why the talk?
Why the talk?
The three uses of debugging
●
Bug hunting (doh)
●
●
9
9
inequation.org
inequation.org

10. Why the talk?
Why the talk?
The three uses of debugging
Bug hunting (doh)
● Reverse engineering
●
●
10
10
inequation.org
inequation.org

11. Why the talk?
Why the talk?
The three uses of debugging
Bug hunting (doh)
● Reverse engineering
● Testing a new feature
●
11
11
inequation.org
inequation.org

12. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
12
12
inequation.org
inequation.org

13. A taste of gamedev debugging
A taste of gamedev debugging
Code: 01-taste
13
13
inequation.org
inequation.org

14. A taste of gamedev debugging
A taste of gamedev debugging
Can you spot the culprit?
// class declaration
class Crasher extends ActorComponent;
var int DummyArray[1024];
// in ammo consumption code
Crash = new class'Crasher';
Comp = new class'ActorComponent' (Crash);
14
14
inequation.org
inequation.org

15. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
15
15
inequation.org
inequation.org

16. Right tool for the job
Right tool for the job
„Fix a bug for an intern, they will get stuck on
the next one.
Teach them debugging, they will fix most bugs
they encounter on their own.”
– Paulo Coelho
16
16
inequation.org
inequation.org

17. Right tool for the job
Right tool for the job
Debugging tools are essential to this
profession!
When joining a new team or starting development for a
new platform, demand debugging tools
● Ask senior teammates
● If they don't know, there must be documentation
● Be proactive!
● Don't give up until the debugger is fully working
● No tools? Roll your own!
● You are a coder after all, right?
●
17
17
inequation.org
inequation.org

18. Right tool for the job
Right tool for the job
Not all bugs are in the code
Animation graphs
● Flow graphs/visual scripts
● Post-process effects
●
Still need a way to debug them
18
18
inequation.org
inequation.org

19. Right tool for the job
Right tool for the job
Code: 02-tools
19
19
inequation.org
inequation.org

20. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
20
20
inequation.org
inequation.org

21. Noise filtering
Noise filtering
There are parts of code executed thousands
of times each frame
Object transformation
● Collision detection
●
Also rarer, but still impractical to track
Setting materials on objects
● Attaching and detaching of components
●
21
21
inequation.org
inequation.org

22. Noise filtering
Noise filtering
Code: 03-material
04-ragdoll
05-assert
22
22
inequation.org
inequation.org

23. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
23
23
inequation.org
inequation.org

24. Memory corruption
Memory corruption
Look closely:
// class declaration
class Crasher extends ActorComponent;
var int DummyArray[1024];
// in ammo consumption code
Crash = new class'Crasher';
Comp = new class'ActorComponent' (Crash);
24
24
inequation.org
inequation.org

25. Memory corruption
Memory corruption
Look closely:
// class declaration
class Crasher extends ActorComponent;
var int DummyArray[1024];
// in ammo consumption code
Crash = new class'Crasher';
Comp = new class'ActorComponent' (Crash);
25
25
inequation.org
inequation.org

26. Memory corruption
Memory corruption
●
UnrealScript object construction syntax
new <class> [(<template object>)];
●
But:
sizeof(Crasher) > sizeof(ActorComponent)
●
Verdict:
26
26
inequation.org
inequation.org

27. Memory corruption
Memory corruption
●
UnrealScript object construction syntax
new <class> [(<template object>)];
●
But:
sizeof(Crasher) > sizeof(ActorComponent)
●
Verdict:
27
27
BUFFER OVERFLOW!
inequation.org
inequation.org

28. Memory corruption
Memory corruption
But this can happen anywhere! How to find it?
Use a memory fence
● Many related techniques
● Allocate additional space in front and behind actual
allocations
● Then protect them from writing...
● Or write a byte pattern and periodically assert its
consistency
● Also it's useful to log stack traces
● Memory and CPU overhead!
● Use a debug memory allocator (dmalloc)
● Use a memory debugger (Valgrind)
● Use a memory analysis tool (HeapInspector)
●
28
28
inequation.org
inequation.org

29. Memory corruption
Memory corruption
Memory fences
malloc
Regular allocation
29
29
malloc
Fenced allocation
inequation.org
inequation.org

30. Takeaway
Takeaway
You can't be an effective programmer without
debugging tools
● If there are no tools, make some
● Noise filtering techniques save your time
● Time is not only money – nerves are just as important!
● Know your machine (physical or virtual) down to the
metal
● Instruction opcodes, registers etc. come in handy
● Tons of resources available
● Random crashes and/or content glitches may indicate
memory corruption
● Memory corruption can be defeated!
●
30
30
inequation.org
inequation.org

31. inequation.org
inequation.org
Questions?
lg@inequation.org
SpreadIT 2013 · October 19th, 2013
SpreadIT 2013 · October 19th, 2013

32. inequation.org
inequation.org
Thank you!
inequation.org
lg@inequation.org
@TheIneQuation
SpreadIT 2013 · October 19th, 2013
SpreadIT 2013 · October 19th, 2013