Running 512 containers on a laptop

LXC is a management system for lightweight virtual machines called containers.  LXC relies on the Linux kernel for several key features including process isolation and resource limitation.  Compared to virtual machines, LXC containers have a few limitations, however none of them affect the vast majority of use cases.

My laptop (Thinkpad X220) is by no measure a speed demon, although it still has the required resources to host a seemingly arbitrarily large amount of containers thanks to a few key deduplication technologies.  Kernel Samepage Merging or KSM allows fragments of memory with identical contents to be merged together, thus freeing up memory for other uses.

The other technology required to make this work is BTRFS.  In particular the required part is BTRFS’s snapshotting ability, which is the ability to take one directory and use it as a template so that any derivative works will only write down the modifications to the template, and not alter the original.  This allows massive space savings which is required to get a high number of containers without linear disk usage.

For this demonstration I’ll be using debootstrap to create a minimal system image to use as a template, and then clone that to many containers using a simple shell script.  I’ve accomplished this previously on Ubuntu Linux 12.04, however for this demonstration I’ll be using Gentoo Linux because that’s what I happen to be booted in at the moment.

To start with, be sure that your root filesystem uses btrfs and that you have the app-emulation/lxc and sys-fs/btrfs-progs packages installed.  On Debian or Ubuntu you’ll need the ’lxc’ package installed.

First we’ll create our template image and call it ’template'.

# lxc-create -n template -t ubuntu -B btrfs -- precise
...
# The default user is 'ubuntu' with password 'ubuntu'!
# Use the 'sudo' command to run tasks as root in the container.

'ubuntu' template installed
'template' created

Next let’s make a big loop in bash and use it to create a bunch of snapshots called ‘ubuntu1’ to ‘ubuntu512’.

# for i in $(seq 1 512); do lxc-clone -s -o template -n ubuntu${i}; done
...
Tweaking configuration
Copying rootfs...
Updating rootfs...
'ubuntu512' created
#

Now we have 512 stopped Linux containers. Let’s start them all up now. Be warned: When I did this my load average grew to around 300 as the containers went through their init sequence. My system remained responsive enough to type this blog post in Firefox.

# for i in $(seq 1 512); do lxc-start -d -n ubuntu${i}; done

512 Ubuntu containers should be stampeding through your system, consuming quite a few resources as their startup scripts run. On Debian and Ubuntu what will happen is that the first container starting will create an Ethernet bridge called ‘br0’ and a virtual ethernet device ‘veth0’, and add the device to the bridge. Subsequent containers starting will create ‘veth’ devices and add them to the bridge. A DHCP server (dnsmasq) will also start and is responsible for serving DHCP requests from the containers.

Now it’s up to you to get creative about how to use these. How dense can active containers be packed before hardware falls over? What are the symptoms of resource over-utilization? How can one provision these in a scalable way? These are questions that I’d like to tackle in subsequent blog posts. Please leave a comment if you’ve tried this and have interesting results.