Category Archives Technical Fu

A build system for games

The fact that we make games put a series of differences into our production and deployment process that doesn’t generally affect non game software development cycles.

For a start art and animation are key in a game, so are visual effects. In build server terms, that generally means two things: It requires space(for content building) and quite probably a graphics card (for the visual effects). Our ideal solution would be to have a virtual machine per supported platform , but we found that none of the virtualization software (VirtualBox, VMWare, etc) virtualizes the graphics card reliably (if you found a way, please let me know)  so we are running our server on a non virtual machine.

The objective is to build the whole game, tools, etc. at each commit. There are a series of tools available in the market at the moment that allows you to do just that. For example: TeamCity, Jenkins, CruiseControl, Hudson.. etc. There is a good comparison matrix here.

We chose TeamCity, mostly because we know it, we find it easy to use and it is free.

Our setup

Our code is based around XNA and monoGame (recently on version 3).

We have many projects: a game and engine project, a tool, a library and the content builder. This means we have a diverse project output. For example, in the case of the content, we are not interested in the actual build results (yet) however we are interested in knowing that it did build, and to report errors if it didn’t.

The game and engine project reside in one solution and have many unit tests. We need the build server to not only build the project and after that run the unit and integration tests. Initially I set this up using psake (a tool that does the build with PowerShell) this is a great solution if you want to have a one click build, so I gave it a try, I found that setting up the tests assemblies was a manual process and that to get psake to talk to TeamCity nicely it would take more time that I had. So I tried running the build directly from TeamCity, and it was easier and I didn’t loose to much flexibility so, I had to go for that solution (however is a decision I will revisit shortly).

Capture

Setting up each of the steps took me approximately 10~20 minutes if done this way. There are a few things that I want to improve on, however the end result is that in a very short time I can have a new project building and reporting.

Some of the disadvantages of the process as it is are:

  1. If I want to change the build type I need to do it manually, in this case, the target test assemblies will need to be updated
  2. At the moment we are using dropbox as our way to distribute the builds, it is a process some members of our team really like because they just get the new binaries and they can choose when to update (so that there is no friction). I would prefer if the tools would just auto-update, but this is working for the moment.

Summary

As you can see the server setup is not perfect, there is loads to improve on, but setting this up from early on has proven very helpful. It allows us to have a one click build and a functional way to deliver our tools and product to the people in our team.

Content hot loading in XNA

There’s a really interesting post here about content hot loading in XNA. Unfortunately there are no implementation details, so I thought I’d share my own.

There are three parts to our solution. First, a tool that monitors the content directory for new content. When new content is detected, the content project file is updated with that content, and a bat file is kicked off that uses msbuild to rebuild the content project. Second, a piece of code that runs in the engine, that monitors the content project output directory for file changes. When a change is detected there, the offending file is reloaded through a custom ContentManager that bypasses the default content managers’ caching. Third and lastly, a double reference class. Resources are wrapped in this class and returned to the game, so when a content change is detected, the engine only replaces the wrapped reference, and all code using that reference updates automatically.

So details. First the tool. .Net has a nice class called FileSystemWatcher. Guess what it does! Unfortunately it’s a bit awkward to use because the OS can raise multiple events for a single change to the same file. We have a class called FileWatcher which wraps a FileSystemWatcher instance and uses Reactive extensions to buffer changes over 1 second, which so far seems to work pretty well. Here’s the guts of it.

C#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
public class Resource
    {
        public Resource(T reference)
        {
            Reference = reference;
        }
        public T Reference { get; set; }
        public static implicit operator T(Resource x)
        {
            return x.Reference;
        }
    }

We also use this class in the engine runtime to watch the content project output directory. In the runtime though, the class implementation is wrapped in conditional debug directives, so that at runtime, an implementation of the class does nothing. There are various ways to achieve the same thing. Perhaps using interfaces and passing a null implementation would be better. I dislike conditional compilation. I find it makes the code harder to read. Something to look into.

Anyway, when a change is detected, we modify the content project file (which is just XML) to include the new file. We use a convention based approach so the tool knows that when a file is dropped into the textures folder, for example, it’s a texture and should have TextureImporter and TextureProcessor set. This is great because Visual Studio is just a terrible, terrible tool for managing content. Default values for any content file can be set in a separate XML file in the content folder, just in case you need something different i.e. a Dxt compressed texture. When the project file is modified, the tool runs a bat file containing this:

C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild /P:XNAContentPipelineTargetPlatform=Windows /P:XNAContentPipelineTargetProfile=HiDef

and that’s it for the tool side of things. This has really made my workflow a lot easier when dealing with content. I don’t even need a content project in Visual Studio anymore, which is great.

So on to the runtime. As I mentioned, we use our FileWatcher class to monitor the content build directory, and a custom content manager that we can ask to ignore the content cache. Here’s the important bits:
[crayon-58da012ea0175 lang='csharp']
public override T Load(string assetName)
{
return Load(assetName, false);
}

public T Load(string assetName, bool forceReload)
{
return forceReload == false ? base.Load(assetName) : ReadAsset(assetName, null);
}
[/crayon]

We don’t expose an instance of this class to the game code. Instead, we have a ResourceManager class that takes our custom content manager.

[crayon-58da012ea0175 lang='csharp']
public class ResourceManager
{
private readonly ForceableContentManager _content;
private Dictionary> _references = new Dictionary>();
private FileWatcher _fileWatcher;

public ResourceManager(ForceableContentManager content)
{
_content = content;
_fileWatcher = new FileWatcher(_content.RootDirectory);
_fileWatcher.ContentChangedEvent += ContentChanged;
}

public Resource Load(string assetName)
{
var resource = _content.Load(assetName);
KeyValuePair reference;

if(_references.TryGetValue(assetName, out reference) == false)
{
reference = new KeyValuePair(typeof(T), new Resource(resource));
_references.Add(assetName, reference);
}

return (Resource)reference.Value;
}

private void ContentChanged(object sender, ContentChangedEventArgs e)
{
var assetName = e.FilePath.Replace(_content.RootDirectory, “”).TrimStart(‘\\’);
assetName = assetName.Substring(0, assetName.LastIndexOf(‘.’));

if (_references.ContainsKey(assetName) == false)
{
assetName = assetName.Replace(‘\\’, ‘/’);

if (_references.ContainsKey(assetName) == false)
return;
}

var reference = _references[assetName];

var loadMethod = _content.GetType().GetMethod(“Load”, new[] {typeof (string), typeof (bool)});
var genericLoadMethod = loadMethod.MakeGenericMethod(reference.Key);

reference.Value.GetType().GetProperty(“Reference”).SetValue(reference.Value, genericLoadMethod.Invoke(_content, new object[]{assetName, true}), null);
}
} [/crayon]

Game code that wants to load a resource calls ResourceManager.Load<T>() much like you would call ContentManager.Load<T>(). The difference is that our Load method returns a Resource<T> which is an instance of the double reference class I mentioned earlier. When a resource is loaded, the ResourceManager maintains a reference to it. When a resource is subsequently modified outside the game, we look up the reference and change it’s value. Here is the Resouce class. It’s quite simple.

[crayon-58da012ea0175 lang='csharp']
public class Resource
{
public Resource(T reference)
{
Reference = reference;
}

public T Reference { get; set; }

public static implicit operator T(Resource x)
{
return x.Reference;
}
}
[/crayon]

Note that this is definitely work in progress and has some bugs, but I wanted to give everyone something to fall asleep to to enjoy on Christmas day after you’ve all gorged yourselves on Santa’s sack.