Monitor memcached with Monit and alert with Gmail

I setup monit on a production slice running Ubuntu to monitor memcached using the directions on Luc Castera's blog post.  And for lack of a reliable smtp server setup, I'm just using GMail's SMTP server to send us the alert emails.  GMail SMTP requires SSL, and using monit version 4.8.x won't work.

Write a monit configuration file:

sudo touch /etc/monitrc

chmod 700 /etc/monitrc

Create a directory to hold individual monit monitoring configurations (for memcached:)

sudo mkdir /etc/monit.d

sudo touch /etc/monit.d/memcached.monit

The above creates an empty monit file for you to add some configurations.  Here is the monitrc file configuration I'm using (minus a few password related stuff):

set daemon 120 # Poll at 2-minute intervals

        set logfile /home/arthur/logs/monit.log # saves the log to a log file

        set httpd port 2812 and use address MY_IP_ADDRESS # this is monit's httpd server, shows you graphically what's happening

                allow MY_IP_ADDRESS # this is what IP's are allowed to access the monit page

                allow localhost # localhost is good for testing locally

            allow username:password # Allow Basic Auth

        set mailserver smtp.gmail.com port 587 username "emailname@gmail.com" password "gmailpass" using tlsv1 with timeout 30 seconds # this basically allows you to use gmail's smtp server.  tlsv1 gets things going in SSL.

        set alert globalemail@email.com # set this as a default email to send alerts to if you don't specify later on

include /etc/monit.d/* # includes your memcached.monit config file, and any others you write

So before you can test it out, since we're asking to include config files for stuff, we have to write one or else monit will complain it can't find any to load, so here's what I'm doing for memcached.monit:

check process memcached with pidfile "/home/arthur/logs/memcached.pid" # this is where the pid file will be

                start = "/usr/bin/memcached -u root -d -P /home/arthur/logs/memcached.pid" # this starts it with the root user, daemonized, and stores the pid in the file mentioned, the pid is important to be used all over.

                stop = "/bin/kill -9 cat `/home/arthur/logs/memcached.pid`; rm /home/arthur/logs/memcached.pid" # kills it and also removes the pid file to stop

                if failed host 127.0.0.1 port 11211 then restart # restart it if it dies on localhost on the default port of 11211

                if cpu usage is greater than 60 percent for 2 cycles then alert # if it uses too much cpu we'll send out an alert

                if cpu usage > 98% for 5 cycles then restart # if it uses tons of cpu, restart.  this should probably not be so high

                if 2 restarts within 3 cycles then timeout # timeout if you try too many times

        alert email@email.com # alert email

The cpu usage stuff is just some random stuff, so make sure you tune those to what you need.

Check your syntax by:

sudo monit -t

It should all check out OK, if not it will give you a bunch of errors.  Also if it can't find your monitrc config file, specify it with the -c flag.  If you did everything posted in the blog post the -c flag shouldn't be needed, as it'll find the config in the path.

Now big drumroll, start up monit

sudo monit

That's it, check the status with sudo monit status, and you can manually start memcached if not already with sudo monit start memcached.  Let me know if any of the info above is inaccurate or you have questions!

Posted