The ‘Try’ repository and its evolution

As the primary maintainers of the back end of the Try repository (in addition to the rest of Mercurial infrastructure) we are responsible for its care and feeding, making sure it is available, and a safe place to put your code before integration into trees.

Recently (the past few years actually) we’ve been experiencing that Mercurial has problems scaling to it’s activity. Here are some statistics for example:

  • 24550 Mercurial heads (this is reset every few months)
  • Head count correlated with the degraded performance
  • 4.3 GB in size, 203509 files without a working copy

One of the methods we’re attempting is to modify try so that each push is not a head, but is instead a bundle that can be applied cleanly to any mozilla-central tree.

By default when issuing a ‘hg clone’ from and to a local disk, it will create a hardlinked clone:


$ hg clone --time --debug --noupdate mozilla-central/ mozilla-central2/
linked 164701 files
listing keys for "bookmarks"
time: real 3.030 secs (user 1.080+0.000 sys 1.470+0.000)

$ du --apparent-size -hsxc mozilla-central*
1.3G   mozilla-central
49M    mozilla-central2
1.3G   total

These lightweight clones are the perfect environment to apply try heads to, because they will all be based off existing revs on mozilla-central anyway. To do that we can apply a try head bundle on top:


$ cd mozilla-central2/
$ hg unbundle $HOME/fffe1fc3a4eea40b47b45480b5c683fea737b00f.bundle
adding changesets
adding manifests
adding file changes
added 14 changesets with 55 changes to 52 files (+1 heads)
(run 'hg heads .' to see heads, 'hg merge' to merge)
$ hg heads|grep fffe1fc
changeset:   213681:fffe1fc3a4ee

This is great. Imagine if instead of ‘mozilla-central2′ this were named fffe1*, and could be stored in portable bundle format, and used to create repositories whenever they were needed. There is one problem though:


$ du -hsx --apparent-size mozilla-central*
1.3G	mozilla-central
536M	mozilla-central2

Our lightweight 49MB copy has turned into 536MB. This would be fine for just a few repositories, but we have tens of thousands. That means we’ll need to keep them in bundle format and turn them into repositories on demand. Thankfully this operation only takes about three seconds.

I’ve written a little bash script to go through the backlog of 24,553 try heads and generate bundles for each of them. Here is the script and some stats for the bundles:


$ ls *bundle|wc -l
24553

$ du -hsx
39G    .

$ cat makebundles.sh
#!/bin/bash

/usr/bin/parallel --gnu --jobs 16 \
    "test ! -f {}.bundle && \
    /usr/bin/hg -R /repo/hg/mozilla/try bundle --rev {} {}.bundle ::: \
    $(/usr/bin/hg -R /repo/hg/mozilla/try heads --template "{node} ")

If this tooling works well I’d like to start using this as the future method of submitting requests to try. Additionally if developers wanted, I could create a Mercurial extension to automate the bundling process and create a bundle submission engine for try.

Tags: , , , , , , , ,
| August 5th, 2014 | Posted in Uncategorized |

Leave a Reply