Part I: Setting up Munin-Node
Munin showed the most promise and compatibility with many of the services we run at the OSL, such as memcached and varnish. I liked how the plugin system is set up independently on each host, and that each plugin can be managed, configured, and consolidated through symlinks.
For the benefit of the uninitiated the setup on each node goes something like this:
#For gentoo emerge munin #For Fedora/CentOS yum install munin-node #For Debian/Ubuntu apt-get install munin-node
Each client is a “node”, and runs the daemon ‘munin-node’. It’s configuration file exists at /etc/munin/munin-node.conf. This file merely tells the daemon which user to run as, which interface/port to listen to, where to log, and which munin masters are allowed to poll statistics.
bkero@ponderosa /etc/munin $ cat munin-node.conf log_level 4 log_file /var/log/munin/munin-node.log pid_file /var/run/munin/munin-node.pid background 1 setseid 1 user root group root setsid yes allow ^127.0.0.1$ host * port 4949
Inside the /etc/munin/plugins directory are symlinks to the actual plugins that are polled. Note that both varnish checks point to the file varnish_. Plugins ending in ‘_’ have multiple uses, and their behavior changes depending on the file name.
bkero@ponderosa /etc/munin/plugins $ find . -type l -printf "%P -> %ln" varnish_objects -> /usr/local/libexec/munin/plugins/varnish_ varnish_request_rate -> /usr/local/libexec/munin/plugins/varnish_ cpu -> /usr/libexec/munin/plugins/cpu if_eth0 -> /usr/libexec/munin/plugins/if_ if_tun0 -> /usr/libexec/munin/plugins/if_ uptime -> /usr/libexec/munin/plugins/uptime apache_accesses -> /usr/libexec/munin/plugins/apache_accesses apache_volume -> /usr/libexec/munin/plugins/apache_volume
The only other configuration is in the /etc/munin/plugin-conf.d/ directory. This contains arbitrarily named configuration files sourced in to configure plugins.
bkero@ponderosa /etc/munin/plugin-conf.d $ cat mysql-settings [mysql*] env.mysqlopts -u maintenance
For less organized and stringent configurations, the /usr/sbin/munin-node-configure script can be run to automatically detect which plugins a system should have and enable(symlink) them accordingly.
That’s the entire structure, behavior, and configuration of a munin-node. Note that there is no additional security on the munin-node. To test that a node is working correctly simply ensure the daemon is started, then netcat(or telnet) to localhost on port 4949.
bkero@ponderosa ~ $ nc localhost 4949 # munin node at ponderosa.osuosl.org list apache_processes mysql_slowqueries if_eth0 processes apache_accesses users apache_volume mysql_threads varnish_hit_rate swap varnish_backend_traffic uptime mysql_bytes load varnish_request_rate if_tun0 cpu varnish_ hddtemp mysql_queries threads iostat memory varnish_objects varnish_memory_usage . fetch uptime uptime.value 25.96 . quit