Scaling

Recently (the past few years actually) we’ve been experiencing that Mercurial has problems scaling to it’s activity. Here are some statistics for example:

• 24550 Mercurial heads (this is reset every few months)
• Head count correlated with the degraded performance
• 4.3 GB in size, 203509 files without a working copy

One of the methods we’re attempting is to modify try so that each push is not a head, but is instead a bundle that can be applied cleanly to any [mozilla-central](https://hg.mozilla.org/mozilla-central" target="_blank) tree.

We have quite a bit of infrastructure around this including Tinderbox Pushlog (TBPL) and  more. This post deals with the infrastructure and problem we face while trying to scale the ’try’ repository.

A few statistics:

• The try repository currently has 17943 heads. These heads are never removed.
• The try repository is about 3.6 GB in size.
• Due to Mercurial’s on-wire HTTP protocol, this number of heads causes HTTP cloning to fail
• There are roughly 81000 HTTP requests to try per day
• To fix problems (mentioned below), the try repository is deleted and re-cloned from mozilla-central every few months

There are a number of problems associated with such a repository. One particularly nasty one has been present through several years of Mercurial development, and has been tricky in that it is seemingly unreproducible. The scenario is something like: