I am leaving my responsibilities in the capable hands of my teammates. Although I will no longer be here, the work will still get done.
I’d like to thank all of you who helped me along the way. In particular, the release engineering team for introducing me to the reality of operations at an impressive scale. I’d also like to thank IT for teaching me how large of a scope an org can have, and for civilizing this operations cowboy.
When I posted a graph comparing the size of the mozilla-central repository by Firefox version my colleague gszorc was quick to point out that the 4k blocksize of the filesystem meant that the on-disk size of a working copy might not accurately reflect the true size of the repository. I considered this and compared the working copy size (with blocksize =1) to the typical 4k blocksize. This is the result.
As part of my recent duties I’ve been looking at trends in Mozilla’s monolithic source code repository mozilla-central. As we’re investigating growth patterns and scalability I thought it would be useful to get metrics about the size of the repositories over time, and in what ways it changes.
It should be noted that the sizes are for all of mozilla-central, which is Firefox and several other Mozilla products. I chose Firefox versions as they are useful historical points.
Joining me at the sprint were two of my colleagues Gregory Szorc (gps) and Mike Hommey (glandium). They took part in some of the serious discussions about core bugfixes and features that will help Mozilla scale its use of Mercurial. Impressively, glandium had only been working on the project for mere weeks, but was able to make serious contributions to the bundle2 format (an upcoming feature of Mercurial). Specifically, we talked to Mercurial developers about some of the difficulties and bugs we’ve encountered with Mozilla’s “try” repository due to the “tens of thousands of heads” and the events that cause a serving request to spin forever.
We have quite a bit of infrastructure around this including Tinderbox Pushlog (TBPL) and more. This post deals with the infrastructure and problem we face while trying to scale the ’try’ repository.
A few statistics:
The try repository currently has 17943 heads. These heads are never removed. The try repository is about 3.6 GB in size. Due to Mercurial’s on-wire HTTP protocol, this number of heads causes HTTP cloning to fail There are roughly 81000 HTTP requests to try per day To fix problems (mentioned below), the try repository is deleted and re-cloned from mozilla-central every few months There are a number of problems associated with such a repository.
The Mercurial developers were concerned about race conditions and concurrent write/reads causing service inconsistency between hosts. This became evident when stale file handles started appearing in our apache logs. An extension we wrote (pushlog) was also being served off of NFS. This is a problem not because we have multiple hosts writing at once, but because the file is kept in memory for the lifetime of the hgweb-serving WSGI process, and we’ve experienced that sometimes requests to the pushlog can be served old information.