Categories
cron fix gentoo load monitoring munin osuosl

Munin-cron overlap fix

Recently I’ve migrated my munin instance onto a Linode virtual machine. Having fewer resources was a potential source of problems.
One of the problems that I ran into early on was that sometimes a munin-cron run would still be executing when another munin-cron was scheduled to start. This caused many contention issues, ridiculous load(35+), and once caused a reboot of the VM.

One solution I tried was to turn up the time between tests from 5 minutes to 10 minutes. This still had generated problems, as munin-cron runs still weren’t done after 10 minutes, and any greater time between polls meant subpar graph granularity.

My final solution was to modify munin’s cron entry to the following:


*/7 * * * * munin test ( ! "$(pidof /usr/bin/munin-cron)" ) && test -x /usr/bin/munin-cron && nice -19 /usr/bin/munin-cron

I ran a test to see if there’s still a PID active from munin-cron, and if not check to see if the munin-cron executable exists. If it does exist, run a munin-cron job with the nicest setting possible.

This has solved the problem while still keeping granular graphs, and has significantly reduced the load on the virtual machine.
load average: 0.09, 0.42, 0.38

Categories
centos cfengine fedora gentoo munin munin-node osuosl

Part III: Munin[node] in cfEngine

At work we use munin and cfEngine extensively, therefore it is accurate to assume that we have a tight-knit setup that allows us to scale munin-node to all hosts in cfEngine, and have the munin-master configured through it as well. This post describes our configuration, and how it takes place.

Categories
centos debian fedora gentoo Linux monitoring munin osuosl ubuntu

Part II: The Munin-master

Hopefully you’ve read the first part(Munin-Node), and have one or more nodes already configured. It’s alright if the munin-node is running on the same machine as the munin-master. The master is more of a pain to set up than the client, and could require significantly more debugging.

Categories
centos debian fedora gentoo Linux monitoring munin munin-node osuosl ubuntu

Part I: Setting up Munin-Node

After becoming increasingly frustrated with cacti’s lack of sane repeatable configuration and extensibility I began to explore other options.

Munin showed the most promise and compatibility with many of the services we run at the OSL, such as memcached and varnish. I liked how the plugin system is set up independently on each host, and that each plugin can be managed, configured, and consolidated through symlinks.