My Profile Photo

chris.heald.me


My personal blog. Software, devops, security, music, and general nerditry.


Monitoring Rails: Getting instant monitoring alerts

Monitoring is big. Having an automated daemon watch your stuff and make sure it's running properly can let you sleep at night, knowing that if something blows up, there's an ever-watchful guardian ready to wake you up so you can fix it. There are a number of monitoring solutions that are popular these days, such as monit, god, and Nagios. They're all fantastic, but sometimes you just want something simple and to-the-point, right? With jabberish in your project, this becomes a no-brainer. I'm already using Jabberish in my project, so I whipped up a little script that checks system load, available memory, and any changes in swap usage and shoots me an IM under certain conditions. My monitoring still handles automated maintenance in the case of a runaway process or whatnot, but this keeps me instantly informed of any problems that my system might be running in to. ~~~ruby require 'rubygems' require 'drb' require 'daemons' MAX_MEMORY = 95 MAX_LOAD = 4.0 DELIVER_TO = "cheald@gmail.com" JABBERISH_SERVER = "druby://localhost:35505" $warned = {} $hostname = `hostname`.strip def im $im_service ||= DRbObject.new(nil, JABBERISH_SERVER) end def deliver(msg) im.deliver DELIVER_TO, "[#{$hostname}] #{msg}" end def check_stats meminfo = open("/proc/meminfo").read mtotal = meminfo.match(/MemTotal:\s+(\d+)/)[1].to_i mfree = meminfo.match(/MemFree:\s+(\d+)/)[1].to_i mused = mtotal - mfree stotal = meminfo.match(/SwapTotal:\s+(\d+)/)[1].to_i sfree = meminfo.match(/SwapFree:\s+(\d+)/)[1].to_i sused = stotal - sfree begin if $warned[:swap] and sused > $warned[:swap] then deliver "WARNING: Swap has increased from #{$warned[:swap]} to #{sused}" end $warned[:swap] = sused pct = mused / mtotal.to_f * 100.0 if pct > MAX_MEMORY then unless $warned[:memory] deliver sprintf("ALERT: Memory free: %2.2fmb (%2.2f%% used)", mfree / 1024.0, pct) $warned[:memory] = true end else $warned[:memory] = false end load = open("/proc/loadavg").read.split(" ").first if load > MAX_LOAD then unless $warned[:load] deliver sprintf("WARNING: Load average is %s", load) $warned[:load] = true end else $warned[:load] = false end rescue puts "Error: #{$!}" end end Daemons.daemonize(:backtrace => true) loop { check_stats sleep(10) } ~~~ Not too bad, huh? This is written for a CentOS installation, so you may need to change things like the meminfo regexes depending on your system. It could probably use a YAML config file to be truly correct - configuration options in constants works, but is a little ugly. Now I get alerts like these via instant message: ~~~ruby [iceman.tagteam] WARNING: Load average is 4.44 [iceman.tagteam] ALERT: Memory free: 99.82mb (93.38% used) [polaris.tagteam] ALERT: Memory free: 72.20mb (95.14% used) ~~~ This lets me respond to changing system conditions extremely rapidly, and serves as a high-level alert log when when I'm not at the keyboard - when I get back, I check my messages from blippr, and can see when and how often certain marginal conditions are being met. Hope it's useful!
comments powered by Disqus