Monitor memcached with Monit and alert with Gmail
I setup monit on a production slice running Ubuntu to monitor memcached using the directions on Luc Castera's blog post. And for lack of a reliable smtp server setup, I'm just using GMail's SMTP server to send us the alert emails. GMail SMTP requires SSL, and using monit version 4.8.x won't work.
Write a monit configuration file:
sudo touch /etc/monitrc
chmod 700 /etc/monitrc
Create a directory to hold individual monit monitoring configurations (for memcached:)
sudo mkdir /etc/monit.d
sudo touch /etc/monit.d/memcached.monit
The above creates an empty monit file for you to add some configurations. Here is the monitrc file configuration I'm using (minus a few password related stuff):
set daemon 120 # Poll at 2-minute intervals
set logfile /home/arthur/logs/monit.log # saves the log to a log file
set httpd port 2812 and use address MY_IP_ADDRESS # this is monit's httpd server, shows you graphically what's happening
allow MY_IP_ADDRESS # this is what IP's are allowed to access the monit page
allow localhost # localhost is good for testing locally
allow username:password # Allow Basic Auth
set mailserver smtp.gmail.com port 587 username "emailname@gmail.com" password "gmailpass" using tlsv1 with timeout 30 seconds # this basically allows you to use gmail's smtp server. tlsv1 gets things going in SSL.
set alert globalemail@email.com # set this as a default email to send alerts to if you don't specify later on
include /etc/monit.d/* # includes your memcached.monit config file, and any others you write
So before you can test it out, since we're asking to include config files for stuff, we have to write one or else monit will complain it can't find any to load, so here's what I'm doing for memcached.monit:
check process memcached with pidfile "/home/arthur/logs/memcached.pid" # this is where the pid file will be
start = "/usr/bin/memcached -u root -d -P /home/arthur/logs/memcached.pid" # this starts it with the root user, daemonized, and stores the pid in the file mentioned, the pid is important to be used all over.
stop = "/bin/kill -9 cat `/home/arthur/logs/memcached.pid`; rm /home/arthur/logs/memcached.pid" # kills it and also removes the pid file to stop
if failed host 127.0.0.1 port 11211 then restart # restart it if it dies on localhost on the default port of 11211
if cpu usage is greater than 60 percent for 2 cycles then alert # if it uses too much cpu we'll send out an alert
if cpu usage > 98% for 5 cycles then restart # if it uses tons of cpu, restart. this should probably not be so high
if 2 restarts within 3 cycles then timeout # timeout if you try too many times
alert email@email.com # alert email
sudo monit -t
sudo monit
