I am leaving my responsibilities in the capable hands of my teammates. Although I will no longer be here, the work will still get done.
I’d like to thank all of you who helped me along the way. In particular, the release engineering team for introducing me to the reality of operations at an impressive scale. I’d also like to thank IT for teaching me how large of a scope an org can have, and for civilizing this operations cowboy.
When I posted a graph comparing the size of the mozilla-central repository by Firefox version my colleague gszorc was quick to point out that the 4k blocksize of the filesystem meant that the on-disk size of a working copy might not accurately reflect the true size of the repository. I considered this and compared the working copy size (with blocksize =1) to the typical 4k blocksize. This is the result.
As part of my recent duties I’ve been looking at trends in Mozilla’s monolithic source code repository mozilla-central. As we’re investigating growth patterns and scalability I thought it would be useful to get metrics about the size of the repositories over time, and in what ways it changes.
It should be noted that the sizes are for all of mozilla-central, which is Firefox and several other Mozilla products. I chose Firefox versions as they are useful historical points.
Joining me at the sprint were two of my colleagues Gregory Szorc (gps) and Mike Hommey (glandium). They took part in some of the serious discussions about core bugfixes and features that will help Mozilla scale its use of Mercurial. Impressively, glandium had only been working on the project for mere weeks, but was able to make serious contributions to the bundle2 format (an upcoming feature of Mercurial). Specifically, we talked to Mercurial developers about some of the difficulties and bugs we’ve encountered with Mozilla’s “try” repository due to the “tens of thousands of heads” and the events that cause a serving request to spin forever.
Recently (the past few years actually) we’ve been experiencing that Mercurial has problems scaling to it’s activity. Here are some statistics for example:
24550 Mercurial heads (this is reset every few months) Head count correlated with the degraded performance 4.3 GB in size, 203509 files without a working copy One of the methods we’re attempting is to modify try so that each push is not a head, but is instead a bundle that can be applied cleanly to any [mozilla-central](https://hg.
We have quite a bit of infrastructure around this including Tinderbox Pushlog (TBPL) and more. This post deals with the infrastructure and problem we face while trying to scale the ’try’ repository.
A few statistics:
The try repository currently has 17943 heads. These heads are never removed. The try repository is about 3.6 GB in size. Due to Mercurial’s on-wire HTTP protocol, this number of heads causes HTTP cloning to fail There are roughly 81000 HTTP requests to try per day To fix problems (mentioned below), the try repository is deleted and re-cloned from mozilla-central every few months There are a number of problems associated with such a repository.
For the uninitiated, Hackerbeach involves a group of hackers (historically 15-20) gathering in a tropical location for a month to hack on various open source projects. It can be thought of as a month-long hackathon or code sprint for nomadic open source developers. All of the code so far has been focused on the open web ecosystem.
Last year it took place on a small island of Vietnam named Phú Quốc.