Memcached is pretty easy to setup, but there are a few items that are super strange. This is how I set it up locally to test and on a production slice:
Production: Passenger 2.2.1, REE, rails 2.3.2, memcached, memcache-client gem, systemtimer gem, slicehost slice
Development: mongrels as usual, rails 2.3.2, ruby 1.8.6, memcached, memcache-client gem, systemtimer gem, Mac OS X Leapard
Development:
- Install macports if you haven't already
- sudo port install memcached
- sudo gem install memcache-client
- sudo gem install systemtimer
- memcache -vv # this is the verbose for testing
- I have yet to figure out how to get development working without memcached running when you don't care for it.
Production:
- sudo apt-get memcached
- sudo gem install memcache-client
- sudo gem install systemtimer
- memcache -d # this daemonizes it with the default IP to 127.0.0.1 and port 11211
Important Notes:
- There is no configuration needed for apache/passenger.
- memcache-client 1.5.0 is actually bundled with rails 2.1, but I would highly suggest upgrading to the memcache-client 1.7.2. Read why here. In a nutshell, it's WAY faster.
- Check out his commit recently about system timer, this is extremely important for those running with REE. He has not released this yet, but I've tried it on my slice and it works fine, so go ahead and edit your memcache.rb file with these changes.
- Make sure you install all our gems with ruby enterprise. Symlink your /usr/bin/ruby to the enterprise one, or make the enterprise ruby the first and only one to show up in your path.
- Marshal serializes objects into memcached, and de-serializes them (is that the way to say it?) when you want to pull it out. This way you can store more than just strings. By passing in true as the third parameter of a fetch, you can do a raw add to memcache, but when you pull it back out, it will only come out as a string. See more stuff later in this post about this.
- Objects added or get'd from memcached need to be serializable! Passenger gives you a horrible error message pointing to a line in memcache.rb where it does a Marshal.load or Marshal.dump, but tells you nothing else. That's a good indication to check the objects you're returning in a fetch/add/get block to see if they are serializable. If not write your own, see more further along in this post about it.
- Passenger does smart spawning, which is great, but also freaks out memcached. This is where we use memcache-client gem to do a reset whenever Passenger forks. What smart spawning does, in short, is that whenever passenger needs a new worker process it loads it up with the an already loaded Rails application/framework, rather than loading the entire app and framework for each worker process. It only does this once. Really fast with REE and Passenger lined up. REE improves this because it is copy-on-write friendly which means the worker processes will share as much memory as possible. If you want to know more, read the two links in this bullet point that i mentioned.
- How to re-establish connection with memcached in your rails app then specifically? See below:
Setting up your production.rb to use memcached and to solve the smart spawning issue that Passenger has
# set cache classes to true
config.cache_classes = true
config.action_controller.consider_all_requests_local = false
# of course you want to perform caching
config.action_controller.perform_caching = true
config.cache_store = :mem_cache_store
memcache_options = {
:c_threshold => 10000,
:compression => true,
:debug => false,
:namespace => 'a',
:readonly => false,
:urlencode => false}
# require the new gem, this will load up 1.7.2 instead of using the built in 1.5.0
require 'memcache'# make a CACHE global to use in your controllers instead of Rails.cache, this will use the new memcache-client 1.7.2
CACHE = MemCache.new memcache_options# connect to your server that you started earlier
CACHE.servers = '127.0.0.1:11211'# this is where you deal with passenger's forking
begin
PhusionPassenger.on_event(:starting_worker_process) do |forked|
if forked
# We're in smart spawning mode, so...
# Close duplicated memcached connections - they will open themselves
CACHE.reset
end
end
# In case you're not running under Passenger (i.e. devmode with mongrel)
rescue NameError => error
end
And finally, the magical part of caching in your controllers:
CACHE.fetch('cachekey', 1.hour) { # block }
The above is using our CACHE global, that uses memcache-client 1.7.2. memcache-client gem basically helps rails talk to memcached server. No apache settings needed here at all.
The cache key should be unique enough so that the items in the block will be valid.
The second parameter is the timed expiration. Be careful, Rails.cache.fetch accepts expiration as a hash parameter, :expires_in => 1.hour. This is not the case for memcache-client, you must pass it in as a regular parameter.
The block can hold any ruby code you want. It has access to any variables etc. normally accessible at this point. The very last thing in that block is returned. Great! But something very important:
The returned object (and if a hash or array or an enum'd object, all objects there within) MUST be serializable. If not, you're going to get some crazy ass error messages that says nothing about being serializable, and will give you headaches. If something is not serializable, Marshalling will error out without telling you exactly what's happening. You can write your own serialization for special objects to get around this, or save things in a custom hash if you don't really care about the entire object. That custom hash would just hold all the values you need outside of the fetch block, and that hash can be returned. here's an example of the error I got from Passenger when trying to hit an action that had an unserializable object returning:
[Sun May 03 23:14:57 2009] [error] [client 76.239.166.13] Premature end of script headers: amazon, referer: [ADDRESS_REMOVED_FOR_THIS_BLOG_POST]
[ pid=5227 file=ext/apache2/Hooks.cpp:546 time=2009-05-03 23:14:57.657 ]:
Backend process 5255 did not return a valid HTTP response. It returned no data.
/opt/ruby-enterprise-1.8.6-20090201/lib/ruby/gems/1.8/gems/memcache-client-1.7.2/lib/memcache.rb:335: [BUG] non-initialized struct
If you see the above, immediately check the objects returning from the fetch block.
That's all for now, hope that was helpful for those with passenger, ree, and the memcache-client gem!
Big thanks to all the help from Mike Perham, who maintains the memcache-client gem (amongst other amazing feats), Michael Simons who wrote about the Passenger issues with more emphasis with solutions when using memcache-client gem, and all the folks at memcached google group and phusion passenger google group. And big help from joe who helped troubleshoot all the code the whole time and dealt with all my crap.
