Munin showed the most promise and compatibility with many of the services we run at the OSL, such as memcached and varnish. I liked how the plugin system is set up independently on each host, and that each plugin can be managed, configured, and consolidated through symlinks.
For the benefit of the uninitiated the setup on each node goes something like this:
yum install munin-node
apt-get install munin-node
Each client is a “node”, and runs the daemon ‘munin-node’. It’s configuration file exists at /etc/munin/munin-node.conf. This file merely tells the daemon which user to run as, which interface/port to listen to, where to log, and which munin masters are allowed to poll statistics.
bkero@ponderosa /etc/munin $ cat munin-node.conf
Inside the /etc/munin/plugins directory are symlinks to the actual plugins that are polled. Note that both varnish checks point to the file varnish_. Plugins ending in ‘_’ have multiple uses, and their behavior changes depending on the file name.
bkero@ponderosa /etc/munin/plugins $ find . -type l -printf "%P -> %ln"
varnish_objects -> /usr/local/libexec/munin/plugins/varnish_
varnish_request_rate -> /usr/local/libexec/munin/plugins/varnish_
cpu -> /usr/libexec/munin/plugins/cpu
if_eth0 -> /usr/libexec/munin/plugins/if_
if_tun0 -> /usr/libexec/munin/plugins/if_
uptime -> /usr/libexec/munin/plugins/uptime
apache_accesses -> /usr/libexec/munin/plugins/apache_accesses
apache_volume -> /usr/libexec/munin/plugins/apache_volume
The only other configuration is in the /etc/munin/plugin-conf.d/ directory. This contains arbitrarily named configuration files sourced in to configure plugins.
bkero@ponderosa /etc/munin/plugin-conf.d $ cat mysql-settings
env.mysqlopts -u maintenance
For less organized and stringent configurations, the /usr/sbin/munin-node-configure script can be run to automatically detect which plugins a system should have and enable(symlink) them accordingly.
That’s the entire structure, behavior, and configuration of a munin-node. Note that there is no additional security on the munin-node. To test that a node is working correctly simply ensure the daemon is started, then netcat(or telnet) to localhost on port 4949.
bkero@ponderosa ~ $ nc localhost 4949
# munin node at ponderosa.osuosl.org
apache_processes mysql_slowqueries if_eth0 processes apache_accesses users apache_volume mysql_threads varnish_hit_rate swap varnish_backend_traffic uptime mysql_bytes load varnish_request_rate if_tun0 cpu varnish_ hddtemp mysql_queries threads iostat memory varnish_objects varnish_memory_usage
You’re done! If you feel like continuing and making Munin-Node useful at all, read Part II: Munin Master.