↓Skip to main content

Continuous Delivery in LOOT

·6 mins

The LOOT project makes heavy use of Continuous Integration (CI) products that we get for free by virtue of being an open source project, so I thought I’d write a post about how much our soon-to-be robot overlords do for us.

The Masterlists #

Each of the games LOOT supports has a masterlist of metadata that is used to help sort users’ load orders and display helpful messages. These masterlists are YAML files that have a particular structure, and some values (like conditions for conditional metadata) have specific syntax. If the structure or syntax are wrong, LOOT can’t parse the masterlist and an error occurs.

This used to be a significant problem, every so often someone would push a change that would accidentally break a masterlist for everyone until it could be fixed. That’s why LOOT updates its masterlists using local Git repositories, so that if it fails to parse a masterlist, it can efficiently roll back through the masterlist’s history until it finds a version that works, and let the user know which versions are broken so they can report it. This is good, but obviously it would be better to catch these issues earlier, or at least get automatically notified if something breaks, rather than relying on our users to tell us.

That’s why each masterlist repository uses CI to test each Git push using a validator that’s built on top of the LOOT API, so it spots any errors that our users would. If the validator finds any issues, the test fails, and we get email notifications saying so. By preferring (though not enforcing) a pull-request-based workflow, we can see if commits would cause problems before they hit our release branches.

We use Travis CI to run our masterlist ‘builds’ because it integrates well with GitHub and offers parallel builds for free, so multiple masterlists can be tested at the same time, unlike other free offerings.

The LOOT, the LOOT API, the Python API wrapper and the metadata validator #

Each of our libraries and applications lives in a separate repository, and they all use AppVeyor and Travis for Windows and Linux CI environments respectively. We use them to:

  • Lint our JavaScript with ESLint and enforcing a consistent code style using Prettier.
  • Run our hundreds of C++, JavaScript and Python tests.
  • Build and package our applications and libraries, and deploy every build to Bintray.
  • Build and deploy our documentation to Read the Docs.
  • Publish builds of tagged commits as GitHub releases.

Although we deploy every build as a snapshot of the code at that particular commit, Bintray’s free offering isn’t really suited to that kind of usage. To avoid filling up the space we have on Bintray, each build runs a script that deletes build artifacts that are:

  • more than 30 builds old, or
  • are for commits that are no longer part of the repository history (to clean up after deleted branches and rebases), or
  • are for builds that were done on commits in a non-default branch that have been merged into the repository’s default branch (to clean up after merged branches)

The snapshot builds that get uploaded to Bintray are identical to those that get published as releases on GitHub. The only difference is that releases come from tagged commits, and this means that making a release involves:

  1. Updating version numbers in docs and code
  2. Updating the version history with the new release’s changes
  3. Committing the version number and history changes
  4. Tagging the commit.
  5. Pushing the tag to GitHub.

Beyond that, our whole release process is fully automated, and is handled by a “robot” account of mine to make it easier to audit what our automated pipelines are doing, and to reduce the impact if the account’s authorisation tokens were somehow obtained by anyone else.

With this setup, every time changes are pushed, they are tested and built in a relatively consistent environment, and we get documentation and and release-like artifacts from them. This means:

  • I don’t need to worry about how my local dev environment could affect releases, because they don’t come from my PC.
  • It’s easy to debug issues users raise, as builds with potential fixes are automatically made available to them to verify, and having a history of builds available makes tracking down when something broke much easier.
  • Having very little overhead to releases means there’s less of a reason to sit on changes, so users get them sooner.

I used to do alpha, beta and release candidate builds, but having snapshot builds available has made that redundant, with the only technological barriers to continuous deployment being:

  • Setting version numbers, which could instead be incremented just after release instead of just before.
  • Updating the version history, which could be done as features land instead of when deciding to release.
  • Faith in our automated tests’ ability to catch issues. This is non-trivial, as it’s a combination of code coverage and being able to catch non-functional issues like performance or GUI styling bugs.

Even if we removed those barriers, I’d still hold back from continuous deployment, because experience has shown that people get update fatigue, and in reality there’s no benefit for LOOT to be releasing more often than I can already.

Rough Edges #

We get an awful lot for free, but our setup does have a few annoyances and limitations:

  • We don’t currently have code coverage measurements for our C++ tests, so while there are a lot of them, their coverage might not be great. I know for a fact that our LOOT C++ test coverage is pretty poor, because so much of the code is tied up in CEF integration (perhaps unnecessarily).
  • Our LOOT API builds are slow, largely due to having to build a bunch of dependencies as part of the build. Some of them are also my work, so I could publish binaries for them and use those binaries instead, but I haven’t gotten around to trying that yet.
  • If we merge a branch, the HEAD commit gets rebuilt, because in general it’s possible to have branch-specific behaviour (and we do for some repositories). This slows everything down, and a more intelligent CI pipeline would encapsulate branch-specific parts and track commit builds so that rebuilding a commit on a different branch or in a tag would cause only the branch-specific parts to run.
  • AppVeyor’s organisation/team handling is very weird, you need a separate user account for your team, and then you grant access to that account to team members’s personal accounts, and have to log in and out to switch between the repositories built on that account and the repositories built on your personal account.
  • AppVeyor only offers one build at a time in its free tier, so it’s often the bottleneck. It’s hard to complain there being a limit to their generosity though.
↑