Ganeti, dealing with node failure

This morning at Mozilla, we had out first ganeti node failure, which meant we got to learn how to deal with a node failure! So far all of our ganeti clusters have been 2 node clusters, so while relatively simple, it meant an extra quirk in bringing everything up. Thankfully we used the iallocator so as to not overallocate nodes on the cluster. Here’s how the process went.

Continue reading “Ganeti, dealing with node failure”