Size of mozilla-central compared

As part of my ongoing work I’ve been measuring the size and depth of mozilla-central to extrapolate future repository size for scaling purposes. Part of this was figuring out some details such as average file size, distribution of types of files, and on-disk working copy size versus repository size.

When I posted a graph comparing the size of the mozilla-central repository by Firefox version my colleague gszorc was quick to point out that the 4k blocksize of the filesystem meant that the on-disk size of a working copy might not accurately reflect the true size of the repository. I considered this and compared the working copy size (with blocksize =1) to the typical 4k blocksize. This is the result.

Continue reading Size of mozilla-central compared

Trends in Mozilla’s central codebase

UPDATE: By popular demand I’ve added numbers for beta, aurora, and m-c tip

As part of my recent duties I’ve been looking at trends in Mozilla’s monolithic source code repository mozilla-central. As we’re investigating growth patterns and scalability I thought it would be useful to get metrics about the size of the repositories over time, and in what ways it changes.

It should be noted that the sizes are for all of mozilla-central, which is Firefox and several other Mozilla products. I chose Firefox versions as they are useful historical points. As of this posting (2015-02-06) version 36 is Beta, 37 is Aurora, and 38 is tip of mozilla-central.

Source lines of code and repo size

UPDATE: This newly generated graph shows that there was a sharp increase in the amount of code around v15 without a similarly sharp rise of working copy size. As this size was calculated with ‘du’, it will not count hardlinked files twice. Perhaps the size of the source code files is insignificant compared to other binaries in the repository. The recent (v34 to v35) increase in working copy size could be due to added assets for the developer edition (thanks hwine!)

My teammate Gregory Szorc has reminded me that since this size is based off a working copy, it is not necessarily accurate as stored in Mercurial. Since most of our files are under 4k bytes they will use up more space (4k) when in a working copy.

From this we can see a few things. The line count scales linearly with the size of a working copy. Except at the beginning, where it was about half the ratio until about Firefox version 18. I haven’t investigated why this is, although my initial suspicion is that it might be caused by there being more image glyphs or other binary data compared to the amount of source code.

Also interesting is that Firefox 5 is about 3.4 million lines of code while Firefox 35 is almost exactly 6.6 million lines. That’s almost a doubling in the amount of source code comprising mozilla-central. For reference, Firefox 5 was released around 2011/06/21 and Firefox 35 was released on 1/13/2015. That’s about two and a half years of development to double the codebase.

If I had graphed back to Firefox 1.5 I am confident that we would see an increasing rate at which code is being committed. You can almost begin to see it by comparing the difference between v5 and v15 to v20 and v30.

I’d like to continue my research into how the code is evolving, where exactly the large size growth came from between v34 and v35, and some other interesting statistics about how individual files evolve in terms of size, additions/removals per version, and which areas show the greatest change between versions.

If you’re interested in the raw data collected to make this graph, feel free to take a look at this spreadsheet.

The source lines of code count was generated using David A. Wheeler’s SLOCCount.

My Gear Post

bird

Whenever I encounter people as I travel, they are often curious about my luggage. It seems to be invisible. They’ll often ask where my bag is, assuming that it must have gotten lost in transit. Their eyes go wide and confusion sets in when I tell them that the bag on my back is the only one.

It is my estimation that at least some people would be curious about what gear I travel with. They ask how I’m able to pack all the necessities into such a small space. There is no great secret to traveling light. All it takes is a little research and compromise in creature comforts. If you have browsed the postings of other nomadic hackers, there might be little to be gleaned from this post. Here’s a basic rundown, with almost each article deserving its own article.

It should go without saying that nobody paid for me to write this post, and likewise nobody as sent me any products to test.

Continue reading My Gear Post

Working around flaky internet connections

In many parts of the world a reliable internet connection is hard to come by. Ethernet is essentially non-existent and WiFi is king. As any digital nomad can testify, this is ‘good enough’ for doing productive work.

Unfortunately not all WiFi connections work perfectly all the time. They’re fraught with unexpected problems including dropping out entirely, abruptly killing connections, and running into connection limits.

Thankfully with a little knowledge it is possible to regain productivity that would otherwise be lost to a flaky internet connection. These techniques are applicable to coffee shops, hotels, and other places with semi-public WiFi.

Continue reading Working around flaky internet connections

Attempts source large E-Ink screens for a laptop-like device

One idea that’s been bouncing around in my head for the last few years has been a laptop with an E-Ink display. I would have thought this would be a niche that had been carved out already, but it doesn’t seem that any companies are interested in exploring it.

I use my laptop in some non-traditional environments, such as outdoors in direct sunlight. Almost all laptops are abysmal in a scenario like this. E-Ink screens are a natural response to this requirement. Unlike traditional TFT-LCD screens, E-Ink panels are meant to be viewed with an abundance of natural light. As a human, I too enjoy natural light.

Continue reading Attempts source large E-Ink screens for a laptop-like device

The cost of ‘free’ relocation cars in New Zealand

Raspberries on the lawn of the Christchurch Riccarton Bush farmers market
Raspberries on the lawn of the Christchurch Riccarton Bush farmers market

As many of my friends know, I’ve been spending the last few weeks in New Zealand. Primarily it’s been for a small holiday, but also in preparation inimitable Linux.conf.au 2015 conference. Leading up to that, one of the goals of my visit was to experience the variety of places across both islands.

Continue reading The cost of ‘free’ relocation cars in New Zealand

Finding the perfect ancillary travel device

Hackerbeach attendees at the upper dining table
Hackerbeach attendees at the upper dining table

As would be familiar to anybody who knows me, I’m always interested in new tech, especially when it’s running free software and portable enough to be in my every-day carry arsenal.

For the past month or so I’ve been looking at a few devices as a secondary to my laptop to carry with me. In a few weeks I’ll be joining those already there at third installment of Hackerbeach, on the Caribbean island of Dominica.

I wanted an embedded Linux system that could do everything. The scope of this device just kept getting bigger the more I thought about it.

It should: Continue reading Finding the perfect ancillary travel device

September ’14 Mercurial Code Sprint

A week ago I was fortunate enough to attend the latest code sprint of the Mercurial project. This was my second sprint with this project, and took away quite a bit from the meeting. The attendance of the sprint was around 20 people and took the form of a large group, with smaller groups splitting out intermittently to discuss particular topics. I had seen a few of the attendees before at a previous sprint I attended.

Joining me at the sprint were two of my colleagues Gregory Szorc (gps) and Mike Hommey (glandium). They took part in some of the serious discussions about core bugfixes and features that will help Mozilla scale its use of Mercurial. Impressively, glandium had only been working on the project for mere weeks, but was able to make serious contributions to the bundle2 format (an upcoming feature of Mercurial). Specifically, we talked to Mercurial developers about some of the difficulties and bugs we’ve encountered with Mozilla’s “try” repository due to the “tens of thousands of heads” and the events that cause a serving request to spin forever.

By trade I’m a sysadmin/DevOps person, but I also do have a coder hat that I don from time to time. Still though, the sprint was full of serious coders who seemingly worked on Mercurial full-time. There were attendees who had big named employers, some of whom would probably prefer that I didn’t reveal their identities here.

Unfortunately due to my lack of familiarity with a lot of the deep-down internals I was unable to contribute to some of the discussions. It was primarily a learning experience for me for both the process which direction-driving decisions are made for the project (mpm’s BDFL status) and all of the considerations that go into choosing a particular method to implement an idea.

That’s not to say I was entirely useless. My knowledge of systems and package management meant I was able to collaborate with another developer (kiilerix) to improve the Docker package building support, including preliminary work for building (un)official Debian packages for the first time.

I also learned about some infrequently used features or tips about Mercurial. For example, folks who come from a background of using git often complain about Mercurial’s lack of interactive rebase functionality. The “histedit” extension provides this feature. Much like many other features of Mercurial, this is technically “in core”, but not enabled by default. Adding a line in the “[extensions]” section your “hgrc” file such as “histedit =” enables it. It allows all the expected picking, folding, dropping, editing, or modifying commit messages.

Changeset evolution is another feature that’s been coming for a long time. It enables developers to safely modify history and be able to propagate those changes to any down/upstream clones. It’s still disabled by default, but is available as an extension. Gregory Szorc, a colleague of mine, has written about it before. If you’re curious you can read more about it here.

One of the features I’m most looking forward to is sparse checkouts. Imagine a la Perforce being able to only check out a subtree or subtrees of a repository using ‘–include subdir1/’ and –exclude subdir2/’ arguments during cloning/updating. This is what sparse checkouts will allow. Additionally, functionality is being planned to enable saved ‘profiles’ of subdirs for different uses. For instance, specifying the ‘–enable-profile mobile’ argument will allow a saved list of included and excluded items. This seems like a really powerful way of building lightweight build profiles for each different type of build we do. Unfortunately to be properly implemented it is waiting on some other code to be developed such as sharded manifests.

One last thing I’d like to tell you about is an upcoming free software project for Mercurial hosting named Kallithea. It was borne from the liberated code from the RhodeCode project. It is still in its infancy (version 0.1 as of the writing of this post), but has some attractive features for viewing repositories, such visualizations of changelog graphs, diffs, code reviews, a built-in editor, LDAP support, and even a JSON-RPC API for issue tracker integration.

In all I feel it was a valuable experience for me to attend that benefited both the Mercurial project and myself. I was able to lend some of my knowledge about building packages and familiarity with operations of large-scale hgweb serving, and was able to learn a lot about the internals of Mercurial and understand that even the deep core code of the project isn’t very scary.

I’m very thankful for my ability to attend and look forward to attending the next Sprint in the following year.

The ‘Try’ repository and its evolution

As the primary maintainers of the back end of the Try repository (in addition to the rest of Mercurial infrastructure) we are responsible for its care and feeding, making sure it is available, and a safe place to put your code before integration into trees.

Recently (the past few years actually) we’ve been experiencing that Mercurial has problems scaling to it’s activity. Here are some statistics for example:

  • 24550 Mercurial heads (this is reset every few months)
  • Head count correlated with the degraded performance
  • 4.3 GB in size, 203509 files without a working copy

One of the methods we’re attempting is to modify try so that each push is not a head, but is instead a bundle that can be applied cleanly to any mozilla-central tree.

By default when issuing a ‘hg clone’ from and to a local disk, it will create a hardlinked clone:


$ hg clone --time --debug --noupdate mozilla-central/ mozilla-central2/
linked 164701 files
listing keys for "bookmarks"
time: real 3.030 secs (user 1.080+0.000 sys 1.470+0.000)

$ du --apparent-size -hsxc mozilla-central*
1.3G   mozilla-central
49M    mozilla-central2
1.3G   total

These lightweight clones are the perfect environment to apply try heads to, because they will all be based off existing revs on mozilla-central anyway. To do that we can apply a try head bundle on top:


$ cd mozilla-central2/
$ hg unbundle $HOME/fffe1fc3a4eea40b47b45480b5c683fea737b00f.bundle
adding changesets
adding manifests
adding file changes
added 14 changesets with 55 changes to 52 files (+1 heads)
(run 'hg heads .' to see heads, 'hg merge' to merge)
$ hg heads|grep fffe1fc
changeset:   213681:fffe1fc3a4ee

This is great. Imagine if instead of ‘mozilla-central2′ this were named fffe1*, and could be stored in portable bundle format, and used to create repositories whenever they were needed. There is one problem though:


$ du -hsx --apparent-size mozilla-central*
1.3G	mozilla-central
536M	mozilla-central2

Our lightweight 49MB copy has turned into 536MB. This would be fine for just a few repositories, but we have tens of thousands. That means we’ll need to keep them in bundle format and turn them into repositories on demand. Thankfully this operation only takes about three seconds.

I’ve written a little bash script to go through the backlog of 24,553 try heads and generate bundles for each of them. Here is the script and some stats for the bundles:


$ ls *bundle|wc -l
24553

$ du -hsx
39G    .

$ cat makebundles.sh
#!/bin/bash

/usr/bin/parallel --gnu --jobs 16 \
    "test ! -f {}.bundle && \
    /usr/bin/hg -R /repo/hg/mozilla/try bundle --rev {} {}.bundle ::: \
    $(/usr/bin/hg -R /repo/hg/mozilla/try heads --template "{node} ")

If this tooling works well I’d like to start using this as the future method of submitting requests to try. Additionally if developers wanted, I could create a Mercurial extension to automate the bundling process and create a bundle submission engine for try.

Mozilla’s “try” repository

At Mozilla we use Mercurial for Firefox development. We have several repositories/trees that are used depending on where the code should be. If a developer wishes to test the code they have been developing, they can submit it to a Mercurial repository called ‘try‘, since running our entire test suite is not feasible on developer machines.

We have quite a bit of infrastructure around this including Tinderbox Pushlog (TBPL) and  more. This post deals with the infrastructure and problem we face while trying to scale the ‘try’ repository.

A few statistics:

  • The try repository currently has 17943 heads. These heads are never removed.
  • The try repository is about 3.6 GB in size.
  • Due to Mercurial’s on-wire HTTP protocol, this number of heads causes HTTP cloning to fail
  • There are roughly 81000 HTTP requests to try per day
  • To fix problems (mentioned below), the try repository is deleted and re-cloned from mozilla-central every few months

There are a number of problems associated with such a repository. One particularly nasty one has been present through several years of Mercurial development, and has been tricky in that it is seemingly unreproducible. The scenario is something like:

  • User ‘hg push’es some changes to a new head onto try
  • The push process takes a long time (sometimes between 10 minutes and hours)
  • A developer could issue an interrupt signal (ctrl+C) which causes the client to gracefully hang up and exit (his typically has no effect on the server
  • Subsequent pushes will hang with something similar to ‘remote: waiting for lock on repository /repo/hg/mozilla/try/ held by ‘hgssh1.dmz.scl3.mozilla.com:23974′
  • When this happens a hg process is running on the server has the following characteristics:
    • A ‘hg serve’ process runs single-threaded using 100% CPU
    • strace-ing and ltrace-ing reveal that the process is not making any system calls or external library calls
    • perf reveals that the process is spending all of its time inside some ambiguous python function
    • pdb yields that the process is spending all of its time in a function that (along some point in the stack trace) is going through ancestor calculations
    • The process will eventually exit cleanly
  • As operators there is nothing we can do that to alleviate the situation once the repository gets in this state. We simply inform developers and monitor the situation.

There have been several ideas on ways to alleviate the problem:

  • Periodically reset ‘try’. This is considered bad because 1) it loses history, and 2) it is disruptive to developers, who might have to re-submit try jobs again
  • Reset try on the SSH servers, but keep old try repositories on the HTTP servers. This has the potential to create unforeseen problems of growing these repositories even further on the HTTP servers. If reset (staggered from SSH server resets) this will remove unforeseen problem potential, but still lose history.
  • Creating bundle files out of pushes to ‘try’, then hosting these in an accessible location (S3, http webroot, etc). I will detail this method in a future blog post.

As of now though, try will periodically need to be reset as a countermeasure to the hangs mentioned in this post. Getting a reproducible test case might allow us to track down a bug or inefficiency in Mercurial to fix this problem after all. If you’d like to help us with this, please ping fubar or me (bkero) on irc.mozilla.org.