Debugging

We have quite a bit of infrastructure around this including Tinderbox Pushlog (TBPL) and  more. This post deals with the infrastructure and problem we face while trying to scale the ’try’ repository.

A few statistics:

• The try repository currently has 17943 heads. These heads are never removed.
• The try repository is about 3.6 GB in size.
• Due to Mercurial’s on-wire HTTP protocol, this number of heads causes HTTP cloning to fail
• There are roughly 81000 HTTP requests to try per day
• To fix problems (mentioned below), the try repository is deleted and re-cloned from mozilla-central every few months

There are a number of problems associated with such a repository. One particularly nasty one has been present through several years of Mercurial development, and has been tricky in that it is seemingly unreproducible. The scenario is something like: