White Paper for Guerrilla Games' presentation by Frank Compagner

1. Note: ““Attached is a very rough first draft of my whitepaper. The first half of it is more or less
complete (though it still needs quite a lot of editing); the second half is still very sketchy. Not
sure if this is enough detail for now, I will try flesh it out in the coming days and can probably
send you an updated version early next week if that is any help.
Frank.”
Frank Compagner, Guerrilla Games
Abstract
Over the past years we have developed a strategy for managing our release branches that goes
against accepted wisdom in the Perforce and wider SCM community, but which we have found to
work very well nevertheless. Instead of having our developers work on the release branch, and
integrating their changes back to the main branch, we keep everybody working on main and
integrate selected changes to the release branch. We will explain why this works well in our case, and
what processes and tools we needed to adopt to make this a key factor in a very smooth running
release pipeline.
What did we do?
For every notable deliverable we produced over the last few years, we have created a release
branch. Sometimes a further (say) 1.1 release branch was made from the 1.0 release branch, as
usual. Initially we had most people working on the release branch, fixing bugs which would then be
integrated back to the main branch. Over time, however, we often found it much easier to let people
continue working on the main branch, and only integrate the changes to the release branch under
complete control of the release managers. At first this process was managed by manually keeping
track of the status of all changes, but this did not scale well. In fact, we found it hard to maintain for
more than a week or so, at which point it became very hard to figure out the status of any particular
change.
Still, this allowed us to set up a release cycle as follows: at the beginning of the week, we would do a
complete re synchronization of the main branch to the release branch. The result of this then gets
tested by our internal QA department, any problems found would be fixed on the main branch and
integrated onto the release branch, where the fix would be verified. This normally would enable us to
deliver a fairly high quality build at the end of every week, which would then go on to more stringent
external testing. Several weeks before the final delivery of the release we would stop the full re

2. synchronizations from the main branch, and spent the remainder of the time ironing out the last
remaining problems.
Release branch
Main branch
1 week
Full (re )synchronization Bug fix Release
This worked well as long as we restricted ourselves to just fixing the highest priority bugs, but it did
not scale well to more complex releases where we could tolerate less problems in the final product.
It was simply not possible to keep track of the status of all needed change s, and the inevitable
divergence between main and release branch meant that after a week or two we stated to see lots of
conflicts in the integration results, often making it next to impossible to get certain changes on the
release branch.
It became clear that if we wanted to continue in this way we would need tools to keep track of all
relevant changes. As an important part of the problem was the lack of visibility of change status to
everybody in the company, we quickly settled on a web app that allowed all developers to easily see
the status of all changes. It also allowed everybody to request promotion of a change to the release
branch, while the members of the release team were additionally allowed to approve or deny
changes. To facilitate the actual integrations we created a simple tool that obtains a list of eligible
changes from the web app, and allows the release team to simply integrate those changes to the
release branch. Together, these tools improved the process considerably, making it possible to stay
in control of the inevitable divergence for much longer, especially after we extended the web app to
show dependency information of all changes.
These tools are shown in more detail below, but let’’s first step back and ask ourselves:
Why did we do it?
The original reason for letting people continue working on the main branch during a release cycle
was a very prosaic one. Even though Perforce branches are shallow on the server, on the client they
are still pretty expensive, especially if you want to branch an entire project with a head revision of
about 300GB. We would often find that we needed somebody to fix a bug who did not yet have the
release branch locally; just getting the branch from the depot would then take the best part of the
day, to say nothing of the subsequent build effort. Allowing people to fix the bug on the main branch
and then integrating them to the Release branch was often by far the most practical solution.
We have investigated the feasibility of creating tools that through a combination of clever workspace
mappings, judiciously applied ‘‘p4 sync –k’’ commands and possibly some client side md5 hash
tables would make branching a shallow process on the client as well as one the server. But while this
does appear to be possible, it is definitely non trivial, and making the process solid and fool proof is
likely to be very hard indeed.

3. Another reason for not wanting people to work directly on the release branch is that all of our tools
needed to be branch aware. As a high end games production company, we use a large number of
specialized in house developed tools, and we kept finding circumstances under which these would
refuse to work correctly on the release branch. Supporting this became a considerable amount of
work. Finally, we have a considerable amount of users without much technical background, and we
often found people being confused as to what branch they were working on, and submitting changes
to the wrong (or even both) branches.
So far, the reasons for keeping people working on the main branch are essentially negative. But as
more and more people were working this way, we discovered an unexpected benefit: we gained
control over the order and timing of changes on the release branch. This allowed us, for instance, to
test certain more complex features in isolation on the relatively stable release branch, without much
interference from other unrelated changes. It also allowed us to sometimes postpone risky changes
until just after an intermediate deliverable, thereby maximizing testing time. In the end the amount
of control over the entire release cycle we regained in this way is the main reason we have really
grown to like this process.
Of course, there are a number of obvious downsides to this idea. First and foremost is the fact that
nobody in the entire company has the actual release branch on their machine; it exists solely on our
build servers and as a finished build at the QA department. This means users cannot easily see the
actual state of the release branch, so that reproduction of bugs is potentially a problem. It also
becomes vitally important that everybody has an easy way of finding out whether their change has
made it onto the release build, and, if so, in which build it ended up. As the order in which changes
are integrated to the release branch is sometimes completely different from the order in which they
were originally submitted on main, there is much potential for very complex resolve conflicts.
However, through the use of a small amount of process and some well thought out tools we have
found it very possible to keep these problems under control, resulting in a pretty smooth release
process.
How did we do it?
Created web app that shows all changes on the main branch
Using Python, Cherrypy and p4python.
Easy overview of own changes for each user
All users can request any change to be integrated
Supports any number of simultaneous release branches
For each release branch, we have a page with pending changes and a page with submitted
changes
Pending changes can be categorized by the release team.
Pending page also lists dependencies (color coded for easy viewing), showing potential
merge conflicts.
Comments can be added by everyone
Changes can be viewed in detail, with through links to the p4web diff pages
Released changes list also shows builds done from release branch
All released changes (also on user tab) will show first build that that change ended up in.

4. Simple integration with bug tracking system (devtrack)
Not (yet) using jobs
‘‘Manual’’ changes directly to the Release branch also supported.
Link between changes on release and main branches made through changelist comments;
sounds brittle, but not a problem in practice

5. User changes:
s

6. Pending changes:
n

7. Released changes:
e n

8. Inte
egration tool:
For build/rel
lease team o
only
Also
A python, with wx GU lib
, UI
Simple and foolproof
S f
Commandlin variant ex
C ne xists

9. Conclusion
Not for everyone (list necessary conditions)
But can be very good when you get it to work
Visibility and transparency to all users is key (and beneficial in any branching strategy)
Pretty easy to implement with a couple of Perforce api’’s