Arthur Chang

Internet Explorer doesn't play nice with Rails 2+ respond_to blocks

One out of the millions of things wrong with Internet Explorer, is that it has strange Accept headers when sending a request to a server.  What happens in the end of it all is that IE request hits the first format you define in a respond_to block.  I found this funny transcript from http://garrickvanburen.com/archive/workaround-for-ie-overly-accepting-in-rail...

Here’s a conversation between Rails & Firefox

Firefox: “Hey Rails, I want this url”
Rails: “No problem, which format would you like it in?”
Firefox: “HTML, please.”
Rails: “Here you go.”

Here’s the same conversation with Internet Explorer

IE: “Hey Rails, I want this url”
Rails: “No problem, which format would you like it in?”
IE: “Whatcha got?”
Rails: “I’ve got Atom, and…”
IE: (interputting) “OK THANKS!”
Rails: “…um, what? I wasn’t finished, really? ok, here you go.”

The workarounds are pretty gross.  Every html request just pass along a :format => :html.  You can also make sure your AJAX calls goto .js if you put format.js blocks before anything else, or specifically set the request header for all AJAX calls as well like so:

xhr.setRequestHeader("Accept", "text/javascript" )

This google group thread is also a good quick read: http://groups.google.com/group/wellrailed/browse_thread/thread/619ad1b3c5a487cd

(photo is unrelated to topic! picture I took of my friend Anna)

XML Builder Partials in Rails

The past few days I've been creating an API with Rails and found that writing xml Builder files is slightly different then writing the usual erb in views.  The biggest problem was keeping things DRY and using partials, it just doesn't work the same way.  The following attempt to render does nothing in the xml.builder file:

render :partial => 'somepartial'

The reason is because the partial will receive a new xml Builder object rather than using the one already setup.  You can do a few tricks such as passing in the xml builder object in as a local variable, but I found out that the cleanest way to do it is:

xml << render :partial => 'somepartial'

Hope that helps people looking to make partials with the XML Builder in Rails

Create a simple API with Ruby on Rails

Here are the few easy steps to creating a simple API in a Ruby on Rails project.

• Start a new rails app with the restful_authentication plugin by technoweenie: http://github.com/technoweenie/restful-authentication/
• The restful_authentication plugin allows for user accounts, but doesn't have any API authentication built in.  If you want to make publicly available API keys for your users, you'll need to put this in so you can track API usage and deter any unauthorized use.  So assuming you have restful_authentication all setup with defaults, follow this tutorial for setting up API authentication: http://www.compulsivoco.com/2009/05/rails-api-authentication-using-restful-authentication/
• Once you have the above api authentication applied, make sure all the actions that you want protected by the API authentication by adding a before filter:

before_filter :login_required, :only => [...array of actions to be protected...]

• To render out xml for a certain object, you can simply use a respond_to when you're ready to render xml in the controller.

• The above assumes you have an object that you want to return, and will dump the columns as needed.  If you want a prettier or custom return xml, I would recommend using the built in Builder that allows you to specify exactly what xml you want by creating a new view file called action_name.xml.builder and changing the respond_to line to the following:

• In your action_name.xml.builder, use the xml builder syntax to create your own xml file.  Here's a quick example:

• You should test all of this using curl

http://localhost:3000/controller/action/param.xml?api_key=SOME_API_KEY

Snow Leopards and Rails

My turn to put in a few notes on what I did to get Rails to work nice with my newly updated Mac OS: Snow Leopard.  I got most of my information from Riding Rails blog post here: http://weblog.rubyonrails.org/2009/8/30/upgrading-to-snow-leopard.  I would recommend reading that through, but I found a few new things along the way:

• Save your .bash_profile and your paths before doing the upgrade to Snow Leopard, you'll need them later.
• Backup your databases, and be ready to reload them in after this is all done since we need to reinstall mysql with the 64 bit version
• DO NOT use the latest mysql gem 2.8.x, it doesn't work.  Use the 2.7 version instead like this:
  • sudo env ARCHFLAGS="-arch x86_64" gem install mysql --version 2.7 -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
• You must reinstall MacPorts before running the port commands listed in the blog post, you can now grab the latest Snow Leopard binary package here
• You will have to upgrade --force install with your MacPorts to get the libraries updated correctly.  Takes a long time!  If any fail, just try deactivating the failed one first, then uninstalling it with -f, then reinstalling it.  If it's good, you will need to restart the upgrade --force install once more.  This will help resolve the issues there.
• Use the script they have at the bottom of the post to see which gems you want to reinstall

After a lengthy MacPorts update, reinstalling gems, figuring out that I need the mysql 2.7 gem, getting git and some other paths like mysql back into my environment, everything fired right back up.  Now I have to reload my databases and I should be good to go.

Hope you guys get your stuff working!  If anything, read the comment thread on the Riding Rails blog, that'll help a ton!

memcached with passenger, ree, and the memcache-client gem

Memcached is pretty easy to setup, but there are a few items that are super strange.  This is how I set it up locally to test and on a production slice:

Production: Passenger 2.2.1, REE, rails 2.3.2, memcached, memcache-client gem, systemtimer gem, slicehost slice

Development: mongrels as usual, rails 2.3.2, ruby 1.8.6, memcached, memcache-client gem, systemtimer gem, Mac OS X Leapard

Development:

• Install macports if you haven't already
• sudo port install memcached
• sudo gem install memcache-client
• sudo gem install systemtimer
• memcache -vv # this is the verbose for testing
• I have yet to figure out how to get development working without memcached running when you don't care for it.
  

Production:

• sudo apt-get memcached
• sudo gem install memcache-client
• sudo gem install systemtimer
• memcache -d # this daemonizes it with the default IP to 127.0.0.1 and port 11211

Important Notes:

• There is no configuration needed for apache/passenger.
  
• memcache-client 1.5.0 is actually bundled with rails 2.1, but I would highly suggest upgrading to the memcache-client 1.7.2.  Read why here.  In a nutshell, it's WAY faster.
  
• Check out his commit recently about system timer, this is extremely important for those running with REE.  He has not released this yet, but I've tried it on my slice and it works fine, so go ahead and edit your memcache.rb file with these changes.
• Make sure you install all our gems with ruby enterprise.  Symlink your /usr/bin/ruby to the enterprise one, or make the enterprise ruby the first and only one to show up in your path.
  
• Marshal serializes objects into memcached, and de-serializes them (is that the way to say it?) when you want to pull it out.  This way you can store more than just strings.  By passing in true as the third parameter of a fetch, you can do a raw add to memcache, but when you pull it back out, it will only come out as a string.  See more stuff later in this post about this.
• Objects added or get'd from memcached need to be serializable!  Passenger gives you a horrible error message pointing to a line in memcache.rb where it does a Marshal.load or Marshal.dump, but tells you nothing else.  That's a good indication to check the objects you're returning in a fetch/add/get block to see if they are serializable.  If not write your own, see more further along in this post about it.
  
• Passenger does smart spawning, which is great, but also freaks out memcached.  This is where we use memcache-client gem to do a reset whenever Passenger forks.  What smart spawning does, in short, is that whenever passenger needs a new worker process it loads it up with the an already loaded Rails application/framework, rather than loading the entire app and framework for each worker process.  It only does this once.  Really fast with REE and Passenger lined up.  REE improves this because it is copy-on-write friendly which means the worker processes will share as much memory as possible.  If you want to know more, read the two links in this bullet point that i mentioned.
• How to re-establish connection with memcached in your rails app then specifically?  See below:

Setting up your production.rb to use memcached and to solve the smart spawning issue that Passenger has

# set cache classes to true
config.cache_classes = true
config.action_controller.consider_all_requests_local = false

# of course you want to perform caching
config.action_controller.perform_caching             = true

config.cache_store = :mem_cache_store
memcache_options = {
  :c_threshold => 10000,
  :compression => true,
  :debug => false,
  :namespace => 'a',
  :readonly => false,
  :urlencode => false

}

# require the new gem, this will load up 1.7.2 instead of using the built in 1.5.0
require 'memcache'

# make a CACHE global to use in your controllers instead of Rails.cache, this will use the new memcache-client 1.7.2
CACHE = MemCache.new memcache_options

# connect to your server that you started earlier
CACHE.servers = '127.0.0.1:11211'

# this is where you deal with passenger's forking
begin
   PhusionPassenger.on_event(:starting_worker_process) do |forked|
     if forked
       # We're in smart spawning mode, so...
       # Close duplicated memcached connections - they will open themselves
       CACHE.reset
     end
   end
# In case you're not running under Passenger (i.e. devmode with mongrel)
rescue NameError => error
end

And finally, the magical part of caching in your controllers:

CACHE.fetch('cachekey', 1.hour) { # block }

The above is using our CACHE global, that uses memcache-client 1.7.2.  memcache-client gem basically helps rails talk to memcached server.  No apache settings needed here at all.

The cache key should be unique enough so that the items in the block will be valid.

The second parameter is the timed expiration.  Be careful, Rails.cache.fetch accepts expiration as a hash parameter, :expires_in => 1.hour.  This is not the case for memcache-client, you must pass it in as a regular parameter.

The block can hold any ruby code you want.  It has access to any variables etc. normally accessible at this point.  The very last thing in that block is returned.  Great!  But something very important:

The returned object (and if a hash or array or an enum'd object, all objects there within) MUST be serializable.  If not, you're going to get some crazy ass error messages that says nothing about being serializable, and will give you headaches.  If something is not serializable, Marshalling will error out without telling you exactly what's happening.  You can write your own serialization for special objects to get around this, or save things in a custom hash if you don't really care about the entire object.  That custom hash would just hold all the values you need outside of the fetch block, and that hash can be returned.  here's an example of the error I got from Passenger when trying to hit an action that had an unserializable object returning:

[Sun May 03 23:14:57 2009] [error] [client 76.239.166.13] Premature end of script headers: amazon, referer: [ADDRESS_REMOVED_FOR_THIS_BLOG_POST]
[ pid=5227 file=ext/apache2/Hooks.cpp:546 time=2009-05-03 23:14:57.657 ]:
  Backend process 5255 did not return a valid HTTP response. It returned no data.
/opt/ruby-enterprise-1.8.6-20090201/lib/ruby/gems/1.8/gems/memcache-client-1.7.2/lib/memcache.rb:335: [BUG] non-initialized struct

If you see the above, immediately check the objects returning from the fetch block.

That's all for now, hope that was helpful for those with passenger, ree, and the memcache-client gem!

Big thanks to all the help from Mike Perham, who maintains the memcache-client gem (amongst other amazing feats), Michael Simons who wrote about the Passenger issues with more emphasis with solutions when using memcache-client gem, and all the folks at memcached google group and phusion passenger google group.  And big help from joe who helped troubleshoot all the code the whole time and dealt with all my crap.

Rails cache store class and time-based expiry support with :expires_in option

Cache Store Class

Rails 2.x has an abstract cache store class, which is great to use for caching queries in the controller, but there are a few big gotchas that you'll need to figure out.  The fine print of the docs say little with a lot of links.  The basics of it are that you need to worry about your cache store implementation.  The docs recommends MemCacheStore.  I'm not sure what else you could use out of the box.

MemCacheStore uses memcached as cache storage, and is required to use :expires_in

:expires_in

So :expires_in won't work unless you specify in your settings that you're using the MemCacheStore implementation (or something similar) because MemCacheStore supports the :expires_in option with the write commands.  Otherwise your cache will not expire over time.  It's probably a better idea to use memcached and MemCacheStore on production as it's probably the best solution currently, than to write something to the database that saves off cache times and such.

If anyone has other suggestions to better solutiosn other than MemCacheStore, please post!  This is the only solution I've found looking through the docs.

Rails and Twitter Signin

For awhile I've been looking for a nice Twitter solution for Rails.  Sure I could've built something on my own, but I have been mostly looking out of curiosity.  The usual suspects were not the greatest, and I couldn't find a lightweight and elegant solution.  Then came TwitterAuth, a plugin written by Michael Bleigh.  It's made for Rails 2.3, mostly because of the Rails engine use, which is pretty slick and a whole other discussion altogether.

The fun part of TwitterAuth is that it uses oauth but is heavily influenced by the restful_authentication that the rails community has adopted as a very standard / solid way to do user authenticated accounts.  What does that mean?  that means it uses controller extensions like "logged_in?" and "current_user" so if you already use restful_authentication, this makes total sense.

Install TwitterAuth as a gem, or as a plugin.  Remember: you need oauth gem installed as well which is taken care of automatically with the gem install, but will be needed with the plugin instal method.

To quickly get into authenticating your users, goto the gem or plugin directory, and checkout his app directory that comes with it.  In there you'll see a user.rb model, a sessions_controller.rb, and some view partials!  This is exactly what you'll need, if not the only things you'll need to get your users immediately working with Twitter OAuth.  No need to write these yourself, grab these from his examples, and modify as needed.  Out of the box they worked perfectly for me.

Don't forget to get your consumer key and secret from Twitter.  Remember that if you send direct messages and stuff to twitter, it will come from the user you apply for the twitter key / secret with.  Meaning, if you use your FooBar twitter account to signup for the Twitter API key / secret, all direct messages will come from FooBar.  I have yet to figure out if we can send them from one specific person who we've authenticated in the past.  Should be easy.

To get the key and secret, you'll need to goto: http://twitter.com/apps.  This link is so buried, it took me forever to find.  That and I hadn't had coffee all day and I was on my 15th hour of working for the day.

Lastly, the OAuth callback is a bit tricky, because if you're working on localhost as a developer, it won't be able to... well, callback, unless you can give it a visible IP.  Without getting into tricks and sorcery, I just gave it a fake callback, and copy and pasted the parameters in the GET callback request and appended it to http://localhost:3000/oauth_callback. ; UPDATE: In the API Changeset of April 23rd, 2009, the oauth_callback is deprecated due to security issues, so no more localhost callback.  UPDATE: cleaverness of sorcery is actually attributed to joe.

Anyway, hope that was fun, go and authenticate yourself like crazy with Twitter =)

Rails and processing inbound emails

The TMail library is now included in the latest Rails, and handles inbound emails amazingly well.  No need to go into parsing MIME/SMIME for different email clients anymore, this does it all for you.  I found a lot of resources through searches on the subject, but they were slightly outdated and a bit confusing.  The quick and dirty of it is:

1. Create a mailer

ruby script/generate mailer SomeNameMail


2. def a receive method that accepts a TMail object - the email parameter is the TMail object in the example below:
   

def receive(email)
... # handle mail here (see next step) ...
end


3. Retrieve to, from, subject, attachments at more with very simple commands.  Note that email.to and email.from returns arrays in the case that there's more than one person being sent or recieved from, so make sure you grab all of them or just the first one:
   

@to = email.to.first
@from = email.from.first


4. And to get the body of the email with the 'text/html' content-type (meaning it comes with all the nice html tags) you need to do a little extra below:

@body = body_html(email)
...
def body_html(email)
result = nil
        if email.multipart?
            email.parts.each do |part|
                if part.multipart?
                    part.parts.each do |part2|
                        result = part2.unquoted_body if part2.content_type =~ /html/i
                    end
                elsif !email.attachment?(part)
                    result = part.unquoted_body if part.content_type =~ /html/i
                end
            end
        else
            result = email.unquoted_body if email.content_type =~ /html/i
        end
        result = email.body if result.nil?
        return result.strip
    end

TMail RDOC: http://tmail.rubyforge.org/rdoc/index.html

To get mail into the receive action is another story.  Read #46 in the Advanced Rails Recipes book for more information on how to do this.  The basics of it is to run a daemon that fetches mail from an inbox and feeds it as TMail to your mailer.  The mail_fetcher script mentioned in the Advanced Rails Recipes does a good job of it.

Attachments are also very simple, just email.attachments returns the array of attachments, which you can then save off to something like Paperclip or filecolumn.

Cheers!

Connection presence with Orbited

Orbited is a comet technology, allowing you to hookup a message broker and protocol to it to manage messages and other such features you may have.  Orbited currently ships with MorbidQ and STOMP as the default broker and protocol, but if you're looking for more control, as in a chatroom and knowing how many people are in a certain "room", a different protocol would be needed.  I'm going to be looking into plugging in a XMPP protocol into Orbited to replace the current one we're using, which is ActiveMQ with STOMP.  I'll update with more information on that soon.

Here are some resources to learn more about comet, orbited, and such:

Orbited main page (sometimes goes down) and google group:

http://orbited.org/

http://groups.google.com/group/orbited-users

About the Comet neogism:

http://en.wikipedia.org/wiki/Comet_(programming)

Protocol information for Orbited:

http://orbited.org/wiki/Protocols