This morning at Mozilla, we had out first ganeti node failure, which meant we got to learn how to deal with a node failure! So far all of our ganeti clusters have been 2 node clusters, so while relatively simple, it meant an extra quirk in bringing everything up. Thankfully we used the iallocator so as to not overallocate nodes on the cluster. Here’s how the process went.
I recently bought a Sharp Netwalker netbook for daily use in class and on the go. One of the parts that I didn’t like was the crusty old Ubuntu 9.04 install that came with it. Thankfully that’s not the only option available to me. I read some reports online that booting from the MicroSD card is possible. I also found a really cool tool in Ubuntu called rootstock. There’s even an android port for it.
I wanted a recent Ubuntu distribution though, which I couldn’t find provided for me online. So I rolled my own, and am reasonably pleased with the results. All of the hardware works, although there are some annoyances, such as no battery meter in the GNOME notification area, and the wireless card isn’t supported by NetworkManager, although still works with iwconfig and wpasupplicant. Additionally, the hotkeys on the top are not bound to any programs.
So here it is.
To install this guy, format an SD card with one big ext3 partition and untar the tarball to it.
$ wget http://files.blueheaven.ws/netwalker/netwalker-lucid-0.1.tar.bz2
$ sudo fdisk /dev/mmcblk0 # Make a big partition
$ sudo mkfs.ext3 /dev/mmcblk0p1
$ sudo mount /dev/mmcblk0p1 /mnt
$ cd /mnt
$ sudo tar xvjpf ~/netwalker-lucid.0.1.tar.bz2
$ cd ~
$ sudo umount /mnt
Afterwards, eject it and insert it into the Netwalker(unless you’re doing all this FROM the netwalker). Then shut the system completely off. To boot off of the MicroSD card, hold both mouse buttons down, then press and hold the power button for 2 seconds. Continue holding the mouse buttons down until black text bleeds through the white ‘SHARP’ splash screen. The first boot should take noticeably longer than other boots. The default username and password are both ‘ubuntu’.
I’m currently updating the image to use Ubuntu Maverick, and will publish a new post once I have a working image.
Hopefully you’ve read the first part(Munin-Node), and have one or more nodes already configured. It’s alright if the munin-node is running on the same machine as the munin-master. The master is more of a pain to set up than the client, and could require significantly more debugging.
After becoming increasingly frustrated with cacti’s lack of sane repeatable configuration and extensibility I began to explore other options.
Munin showed the most promise and compatibility with many of the services we run at the OSL, such as memcached and varnish. I liked how the plugin system is set up independently on each host, and that each plugin can be managed, configured, and consolidated through symlinks.