Monitoring

One solution I tried was to turn up the time between tests from 5 minutes to 10 minutes. This still had generated problems, as munin-cron runs still weren’t done after 10 minutes, and any greater time between polls meant subpar graph granularity.

My final solution was to modify munin’s cron entry to the following:

*/7 * * * *     munin test ( ! "$(pidof /usr/bin/munin-cron)" ) && test -x /usr/bin/munin-cron && nice -19 /usr/bin/munin-cron

I ran a test to see if there’s still a PID active from munin-cron, and if not check to see if the munin-cron executable exists. If it does exist, run a munin-cron job with the nicest setting possible.

First off, you’re going to have to install the munin package. You’ll also need to have a web server installed, I prefer apache:

# For Gentoo,
emerge munin apache
# For CentOS/Fedora
yum install munin apache
# For Debian
aptitude install munin apache

Gentoo and CentOS both install the HTTP root in /var/www/localhost/htdocs/munin. After setup, the graphs will be available at http://localhost/munin

Debian requires a bit more attention. Debian munin installs the HTTP root in /var/www/munin. so after the setup, a vhost will have to be created to access it. To do this(on Gentoo) I created an apache vhost file, /etc/apache2/vhosts.d/08_munin.conf. On Debian, the file is the same but the path is different, /etc/apache2/sites-available/08munin

Munin showed the most promise and compatibility with many of the services we run at the OSL, such as memcached and varnish. I liked how the plugin system is set up independently on each host, and that each plugin can be managed, configured, and consolidated through symlinks.

For the benefit of the uninitiated the setup on each node goes something like this:

#For gentoo
emerge munin
#For Fedora/CentOS
yum install munin-node
#For Debian/Ubuntu
apt-get install munin-node

Each client is a “node”, and runs the daemon ‘munin-node’. It’s configuration file exists at /etc/munin/munin-node.conf. This file merely tells the daemon which user to run as, which interface/port to listen to, where to log, and which munin masters are allowed to poll statistics.