Lost In Permutation

Cheezburger DVCS Workflow: A History

I wrote an article on my company's technology blog!

 

In the beginning was the Code, and the Code was with Subversion, and the Code was Subversion. Cheezburger was with Subversion in the beginning.

 

Subversion was good, but, as a globally distributed team, Cheezburger really needed a way for developers to work offline, without having to coordinate with each other. Somebody ran an import script, and just like that Cheezburger was on Mercurial. There was one repository, “Current”, on the Kiln server, and we released by tagging and shipping from that same repository.

 

 

Eventually we realized that sometimes features take longer than a single release cycle to create, but we still want to push them to a staging server or show them to someone else. Using Kiln, branching was supremely easy, so we created branches for long-lived features and teams. The central integration repository was called Current, and we would tag and release from that repository.

A new problem arose with this system; namely, that it wasn’t clear which changes had been tested by our automatic continuous integration system, so in May 2011, we added a new branch called “Approved” that the CI system would push approved changesets to, and we shipped from there after tagging Current.

 

This system worked pretty well for a while, but every once in a while, a team would want to push a gigantic merge with old changes from their branch into Current in order to release. Then, the CI system would detect a failure, or the QA team would identify a fatal flaw, or the team would get cold feet for some other reason. Unfortunately, all the other teams would do the sensible thing and merge with Current, so their branches were also infected with the unreleasable code. Changesets got backed up behind the misfired release, tempers flared, developers threatened violence on lolcats. The first time this happened, promises were made that it wouldn’t happen again.

 

Then it happened again, with predictable results. Lolcats flared, developers got backed up behind the misfired release, and changesets threatened violence on tempers. A company crisis was declared, and a Serious Meeting was held. There was talk of separating the app into siloed services, which is certainly a viable solution, but would take a very significant amount of code and work. Eventually, the team decided on another change to our repository layout that would avoid this infuriating problem.

The rules of the system are pretty simple:

1. Don’t pull from PendingTests, pull from Released instead
2. Don’t push to PendingRelease or Released — let the tools automate this

Current had two functions: it was a place to put code that was about to be released, and a place to synchronize with other teams’ code. This new workflow has an identical number of entry points, but it separates the release functionality from the team integration functionality.

PendingTests and PendingRelease are both disposable: if unreleasable code makes it into either of these repositories, they can be deleted and recreated as clones of Released.

This workflow also imposes a natural lock on release: if there are changes waiting in PendingTests, Mercurial won’t let you push until you pull those changes down. But you can’t pull any new changes until the release is complete, because of rule #1. As a nice bonus, Kiln has a page that shows you related repositories. If you visit PendingTests and check out the outgoing changes to Released, you can see who is releasing, and exactly what their release will include.

Our release process has grown more complicated over time, but these complications actually allow us to develop code independently from each other without stepping on one anothers’ toes. In addition, we have been able to incrementally grow an automated testing system, which lets us move ever closer to our goal of continuous deployment. I’m totally jealous of IMVU’s nine minute release pipeline — ours takes over an hour: 38 minutes in continuous integration and 30-45 minutes to release. There’s clearly more room for us to improve! Let us know what your team’s workflow looks like in the comments.