Recently (the past few years actually) we’ve been experiencing that Mercurial has problems scaling to it’s activity. Here are some statistics for example:

• 24550 Mercurial heads (this is reset every few months)
• Head count correlated with the degraded performance
• 4.3 GB in size, 203509 files without a working copy

One of the methods we’re attempting is to modify try so that each push is not a head, but is instead a bundle that can be applied cleanly to any [mozilla-central](https://hg.mozilla.org/mozilla-central" target="_blank) tree.

We have quite a bit of infrastructure around this including Tinderbox Pushlog (TBPL) and  more. This post deals with the infrastructure and problem we face while trying to scale the ’try’ repository.

A few statistics:

• The try repository currently has 17943 heads. These heads are never removed.
• The try repository is about 3.6 GB in size.
• Due to Mercurial’s on-wire HTTP protocol, this number of heads causes HTTP cloning to fail
• There are roughly 81000 HTTP requests to try per day
• To fix problems (mentioned below), the try repository is deleted and re-cloned from mozilla-central every few months

There are a number of problems associated with such a repository. One particularly nasty one has been present through several years of Mercurial development, and has been tricky in that it is seemingly unreproducible. The scenario is something like:

1. The Mercurial developers were concerned about race conditions and concurrent write/reads causing service inconsistency between hosts. This became evident when stale file handles started appearing in our apache logs.
2. An extension we wrote (pushlog) was also being served off of NFS. This is a problem not because we have multiple hosts writing at once, but because the file is kept in memory for the lifetime of the hgweb-serving WSGI process, and we’ve experienced that sometimes requests to the pushlog can be served old information.
3. During times of peak activity there was non-trivial IOWait which caused clone times to increase.
4. Netapp licenses aren’t cheap. 😉

This took a lot of effort and coordination with the release engineering team to ensure that downtime was kept minimal and there were no feature or performance regressions along the way.

This year the prestigious Hacker Beach event is taking place on the island of Lamu off the eastern coast of Kenya. The island is serviced by a single UMTS tower located above the hospital in the main town of Lamu City. However, our accommodation is on the other side of the island.

Our accommodation had a previously installed directional antenna on the roof to provide internet access. Unfortunately the access was very slow, with only 14% signal strength. This was complicated by strong winds blowing against antenna, causing it to be pointed in a wrong direction. This further reduced the cellular reception, sometimes making it disconnect completely.

For the next week (and the previous week) I’m spending time in Paris. This turned out to be largely a convenient set of circumstances, since I had an excellent experience when I was here two weeks ago, and I wished I could spend more time here.

I was fortunate enough to be here for two concerts I wanted to attend, namely Morcheeba and Parov Stelar. Both were at the excellent venue L’Olympia. The concert halls were different than I’m used to in the US, being much smaller, serving actual beer in pint sizes, and being a much more stream-lined affair in terms of coat checks, getting in, and getting out. The Morcheeba concert had excellent music (Ross Godfrey can play some mean guitar solos), however the overall experience had a certain quality missing that I cannot put into words.

For the uninitiated, Hackerbeach involves a group of hackers (historically 15-20) gathering in a tropical location for a month to hack on various open source projects. It can be thought of as a month-long hackathon or code sprint for nomadic open source developers. All of the code so far has been focused on the open web ecosystem.

Last year it took place on a small island of Vietnam named Phú Quốc. It was a loosely organized event with no set dates (besides the month of January). Hackers from all around the world showed up, found places to stay, rented motorbikes, and met up at coffee shops for coworking in paradise.

Woke up with a sore throat. A bad omen for a long travel stint. I hadn’t packed the night before, so it was all done day-of, which surprisingly didn’t result in me not packing important items (that I can think of yet). Maybe I’m getting used to this, or maybe I’m just being more reserved in what I consider necessary.

Managed to get to the airport 2 hours early and thanks to my GlobalEntry card, was able to bypass the security line affording me enough time to sit at Rogue and enjoy some nourishing lunch before catching my 10 hour flight from Portland to Amsterdam. Sadly, I was unable to sleep on the flight, but did catch up on the last 10 episodes of Adventure Time, re-watched Cloud Atlas, and played Shadowrun Returns on my N7. SR had just enough personality to make me realize why I fell in love with the dark, gritty cyberpunk world all those years ago. I wish I had more consciousness to appreciate it. Sadly, I only got an hour or two of sleep which I feel made my oncoming illness worse.

In order to enter the airport, you must show proof of identity and flight documentation (such as an itinerary) to uniformed, armed guards stationed outside the entrance of the airport. After reading a bit, this heightened security must be due to some of the recent bombings in Mumbai. Metal detecters are everywhere in the country, including the entrance to shopping malls and other high profile public places. Ironically, none of the places that employ these devices actually CARE about the result. At each location I visited I continually set off the alarms, and none of the attending parties ever batted an eye.

That said, they’re an excellent way to access the city. They’re affordable, readily available, and their open-air nature really allows one to take in the spectacles, cacophony, and (sometimes) smells of the city.

One of the perils of being a westerner in India (or really anybody that one can identify as not being born and raised locally) is that often others will try to take advantage of you. This certainly isn’t restricted to auto drivers either, as this will also happen on the street, in shopping malls, at the airport, and generally everywhere.

1. Sign up for an account on www.notifymyandroid.com.  Log in, click on ‘My Account’, then ‘Generate New Key’
2. Install the NMA Android app (around $3 USD) onto your android device.  Launch the app and login
3. Install pynma.py and nma.py to $HOME/.weechat/python.
4. (Optional) Symlink nma.py to $HOME/.weechat/python/autoload
5. In weechat:

/python load python/nma.py
/set plugins.var.python.nma.apikey "$myapikey"

6. (Optional) To inly send notifications when detached from the screen: install and load the screen_away.py, then

/python load python/screen_away.py
/set plugins.var.python.nma.only_away on

Now you too can be harassed by inane IRC highlights no matter where in the world you are!  Thankfully, the NMA Android app has a ‘quiet hours’ in case you’d rather not be woken up in the middle of the night.