Arthur Chang

Almost a year ago, I wrote about how to override caching when developing here, and only turning caching on when testing.  As it turns out that might not be a great idea.  Too many times I have had strange bugs on production that I could never figure out locally due to caching issues.  Only after a few hours of debugging did I realize it could have to do with caching.

There's also a really big reason why nobody's written an easy way to turn off caching in development, mainly because it's bad for you to see different behavior in development vs. production at any time, especially with queries and fragment caching.

The kind of caching you do want to turn off is class/controller caching for the sake of avoiding restarting your server just so it will pickup your new code.  

config.action_controller.perform_caching = false
config.cache_classes = false

So run memcached, do your fragment caching in development, and you should be good to go.  No reason not to cache queries or view renders.

The delayed_job gem has been around for a little while.  It's a great way to run cron jobs as well as any other background running process.  Some people think it can only be spawned once in awhile, triggered by some other action, but it's a lot more flexible.

At Fanvibe, I've built a huge array of delayed_job workers that are helping us with jobs such as fetching new sport stats every half second, then crunching all that data and figuring out who needs to see that data, as well as grabbing cool heuristics to help us create awesome polls with a mashup of user activity, player and team stats, time, and more.  This is all getting crunched in the background with delayed_job.

The trick is quick re-queueing of the job in an ensure block.  Unlike a message broker system (like rabbitMQ), this stores jobs in the database as one row with a marshalled string to hold things like parameters.  The read/writes are fast enough for things other than chat messaging and real-time collaborative stuff (i'm talking about actual real time within 20ms).

It also has total access to the rails environment, so you can do things like access ActiveRecord etc.

Below is a quick gist of how you would run the delayed_job worker:

Also know, you'll have to kick it off manually the first time.  Just go into your console and do the Delayed::Job.enqueue

(photo taken in San Diego a little while ago)

Official Update from Olark: "hey folks, just posted up some new information about how we load Olark after all other parts of the page, hope this helps explain how we improved times. if you grab the latest code from www.olark.com/install you should be all set :)"

Olark has stalled my document ready event from firing one too many times!  Yes, they're pretty fast most of the time, but still, it's time I could shave off.

The problem
Olark gives you one code snippet to add to the bottom of your dom, right before the end body tag. It creates a div with some info in it, then loads an external js library and initializes olark.  The whole time it does that, your document ready bound events are still waiting.  If olark takes some time, anything you setup in the ready event handler will not actually work, including more event binds.  Most use cases for the document ready event handler are to attach more event handlers to the dom that's now rendered out.  Without the dom finishing, there's no way to attach these js event handlers.  Since Olark could potentially block these from binding, you're left with a limbo stage where your users aren't firing off the correct js events.

The Goal
Render the divs Olark requires at the end of the dom, but loading the external olark js and initializing should come absolutely last, even after all other js in other document ready event handlers.

The Tools
I'm using jQuery to do some nice javascripting here, you can probably find equivalent ways to do it with your own javascript library or you can write them by hand.  I'm also using the Rails framework, so you'll see me say partials and the gists also shows them, etc.

First step
Dissect the snippet of code Olark gives you into parts (in this case I'm using partials in Rails found in app/shared/_olark.html.erb and app/shared/_olark_script.html.erb).  Then use jQuery's awesome getScript function to actually fetch the script, and on success call the initialization function.

Second step
Put the script part right before the end head tag, basically after any other javascript that runs in the head.  It's just a bit of text, so it won't slow anything down on initial page render.  For Rails users, I have it in my application.html.erb layout file, after all the javascript_include_tags renders, right before the end head tag.  The purpose here is to make sure this runs after all other document ready event handlers.  Olark is probably the lowest priority for rendering on your page.  See gist after the third step for the code example.

Third step
Put the div part right before the end of the body tag as usual.  It's not going to be slow to just put in one div and one a tag here.  Example below 

Done
That should basically be it.  Olark js and initialization will now happen after all your other document ready event handlers.

(photo taken a few weeks ago when I went on a spontaneous shoot in San Francisco.  It was really windy so the cloud movement was perfect)

My friend Calvin (IntoMobile Developer) and I were just talking about keeping your app from caching with Rails.cache in development mode.  One nice way to do it is to wrap all your Rails.cache calls in your application_controller.rb.  In this wrapping call, you can determine whether or not to execute the caching or to just let it through in development.  Another benefit is being able to change your caching behavior in the future without needing to mess around with every single line of code that you've written for caching.

In my development.rb file, I define the following:

I am using the memcache gem, rather than the more popular memcache-client gem, and running memcached on my development box as well (I like testing with it running, rather than using the built in rails way of storing it in memory).  The reason for using memcache gem is because I'm currently hosting our app on Heroku, which requires this gem to work in tandem with their cloud solution for memcached.

The CACHE constant is setup to use this memcache gem's caching mechanism, which would be equivalent to the built in Rails.cache.  Setting this constant up allows me to change this in my environment files once, without ever needing to worry about all the places I set it up.

Then there's the cache_store, which I set correctly to use Memcached gem.  This will help me store the cache correctly.

CACHE_DEVELOPMENT constant is for me to turn caching on / off for development mode.  You'll see how it works in the next piece of code below.

The last two lines that are commented out basically control action and class caching, which doesn't really effect if Rails.cache even goes through.  So just setting these two things to false, Rails.cache will still try to cache.  In order to keep this from happening, we have to do some logic in our application controller wrapper, here it is:

The cache_fetch method will be called everytime I want a cache call from a controller.  I pass in the cache_key as the first parameter to basically name this cache result, and I also accept the period of time the cache is kept before it expires.  0 means it never expires, so that is the default value.

The conditional statement basically allows me to turn caching on or off.  If I'm in the development environment and the constant CACHE_DEVELOPMENT is set to false, I just let the block through without hitting any caching.  Otherwise it goes into the caching logic.

Check out that # CACHE.fetch(key, time_expire){yield} block.  This is how it used to work, but recently a few things changed so that I needed to do a get, and if that didn't exist I'd have to manually set it.  I'm not sure why this happened, or if it's still needed, but you can see how I was able to change this detail in one place, rather than everywhere I tried to cache.

The rest shows you how I'm caching.  This would be ugly if I had to do all of that everytime I wanted to cache a result, and even worse if something changed.  This way I can control the development environment for testing caching, as well as keeping code as DRY as possible.

Here's a simple example of how I call the cache_fetch action from my controller:

Hope that helps!

Paperclip is really good at saving images from local repositories or through forms, but there's little documentation on how to save an image from a remote location, say from a public URL somewhere.

The above uses a quick example using the Flickraw plugin to grab the most interesting photo from Flickr and saving it as a users' favorite photo.

Bloggers love being promoted inside apps like FanPulse!  We also love it when we can depend on some awesome people to write great sport posts to help people keep up with their favorite teams and games.  The problem is, lots of infrastructure needs to be put together on the backend to support the synchronization of our algorithms with the premium content from blogs and news sources.  Superfeedr fortunately has some tricks for me to automate the process of adding and removing sources on the fly.

When a new source is added to our database, I fire off an after filter that subscribes Superfeedr to that feed.  When the source is destroyed, I also unsubscribe with Superfeedr.  The API makes it super easy.  Here are some examples from FanPulse's Rails framework:

Adding a feed to Superfeedr to subscribe to:

I have a few constants in there that are specific to our app.  

• SUPERFEEDR_URL = 'http://superfeedr.com/hubbub'
• SUPERFEEDR_LOGIN = my_superfeedr_login_name
• SUPERFEEDR_PASSWORD = my_superfeedr_password
• SUPERFEEDR_CALLBACK = my_servers_callback_method

I'm using the RestClient gem by adamwiggins to make the calls, as you can see it's pretty darn easy.  "hub.topic" is the url for the feed,  "hub.callback" is the url I want future feeds to be pushed to using webhooks as well as subscribing internally, "hub.verify" should be set to sync since we're not doing an asynchronous call, and lastly "hub.mode" is set to subscribe since we want to add this new feed.

There are a few slight subtleties I did not mention, including the fact that your callback has to implement the basic PubSubHubbub subscribe / unsubscribe spec as usual.  That will have to be in another discussion though.

Lastly, if you want to destroy the source from your database, just make sure you hit the Superfeedr API to unsubscribe the feed as well.

Hope that helps!  Big thanks to Julien for all the support and help getting FanPulse running smoothly with the awesome Superfeedr.

Quick post about something strange I came across recently.  When you concat two ruby strings, it does an extra escape on each string.

The \r\n were double escaped so when you print the string, you don't actually get carriage return or newline characters, you literally get the slash r slash n.  That's all =)

Finding all rows in one table that has a certain amount or no associations at all is a little tricky with ActiveRecord.  In fact, you just sort of have to hack it up or use find_by_sql instead.  Here's a smooth solution I used to find all entries without tags and notes (as an example):

Entry.find_in_batches(
  :select => "entries.*",   :joins => ["LEFT OUTER JOIN tags ON entries.id = tags.entry_id", "LEFT OUTER JOIN notes ON entries.id = notes.entry_id"],
  :conditions => "entries.entry_type = 0 and tags.id is NULL and notes.id is NULL",
  :batch_size => 500
) do |batch|
  # do something
end

Above, I'm using find_in_batches, which is an awesome new Rails 2.3 feature that batches your finds for you.  No more limiting and offsetting manually needed!  It works great.  I'm using this in a daily cron to clean up weird stuff, so there is sometimes a lot of entries to play with.  Read more about the Rails 2.3 release.

Took me awhile to finally figure out exactly what the Property Lists were all about, once it came together, putting together a nice property list was pretty easy.  The problem I'm solving is giving a fairly static list of teams to an iPhone app.  The iPhone parses out binary plist files the most efficiently, so here it is if you want to generate these plists from a rails app:

First get an array of all the objects you want to put into the property list and create an plist array

teams = Team.find(:all)

This creates a nice array with a few attributes we want to add.  If you want all attributes added to the plist_array, instead of adding in a hash just add in team.attributes, which returns a hash of all attributes.  What comes next is the use of the plist gem

$ sudo gem install plist

The above will install the plist gem onto your machine, make sure it's unpacked and/or avai lable on your production servers if you want this to be performed there as well.  Once you have the gem installed, require the gem in your controller and you can use the .to_plist to basically dump the plist_array into the accepted XML Property List format.

plist_array.to_plist

This of course does not give you a binary file yet.  For that, use the plutil that is usually installed on mac osx machines.  Mine just came with it.  If your machine doesn't have the util, it's part of the developer tools on your OSX install disks.

$ plutil -convert binary1 filename.plist

The option -convert basically takes one conversion parameter, either being binary1 or xml1.  binary1 converts the file, designated in the next parameter (in our case filename.plist), that contains the output of the to_plist.  The to_plist outputs the XML format of the Property List, and -convert binary1 will convert it to a nice binary representation.  You can go backwards with -convert xml1

$ plutil convert xml1 filename.plist

When it converts, it resaves the file, filename.plist for example.  Once you get that binary file, you can use it easily on the iPhone for retrieving the data.

For some more examples of how people are using the plist gem see this post to get you started: http://stackoverflow.com/questions/1264329/getting-active-records-to-display-...

A good administration dashboard is key.  The whole reason behind an admin panel is to allow for human heuristic fine tuning of an app.  We need these either constantly or often, in order to maintain some aspect that just cannot be solved with computer magic (though many will argue that there is always a way, but the tradeoffs in some cases are not worth it).  Because of this, an aesthetically beautiful and equally easy to use admin interface is as important, if not more, than a user facing interface.  For computers, you want them to crank through really gross algorithms that would otherwise be impossible to maintain in your own brain.  For administrators, you want to make the experience of administering a site as pleasant and easy as humanly possible.

The big reason why people neglect and cut corners on admin facing pages is because "nobody ever sees it except us."  But isn't that the most important thing?  If you as employees only see the shittiest parts of your app all the time, or subject poor interns and new hires to this awful task, it just shows how little attention to detail you give to making your app hum like a well oiled engine.  When you develop any part of your code, always ask yourself if this is quality you are proud of, internal and external to your own company.  At some point horrible admin panels will just pile up, and there's no turning back.

If you start with a slick looking admin panel that works great and is easy to maintain when things change, then you will always have an awesomely tuned app with consistently great admin controls.  If you start with quite a grotesque arrangement of links and badly styled form fields, anything slicker than that just makes what you've built already look even uglier.  It's a bad snowball to be in.

It's easy to choose admin pages to cut corners on, because there's never an imminent need for them.  The time it saves is clearly good initially, but leads to plenty of cursing and swearing later on.  So what's the solution?  In some frameworks, such as Ruby on Rails, there are nice little things like Active Scaffold.  Unfortunately DHH and the rails maintainers aren't the world's best designers.  The usability, looks, and consistency are all pretty bad using the built in utilities.  At FanPulse, our current admin looks good, works great, is easy to maintain, and is even easier to expand on.  What we did is we used the somewhat popular web app theme utility for our rails based app called, "web-app-theme" by Pilu.  I guess it's not that clever of a name, but it works like it reads.

The tool is actually a ruby script that can be used with the script/generate command.  It basically sweetens up an existing active scaffold, or any model based controller.  It even creates all the files from scratch to you.  It comes prepackaged with a dozen nice themes that are super clean and easy to use.  It took me less than a day to create an expansive admin panel that can help us keep track of, modify, and add new teams, players, leagues, divisions, conferences, sports, beta users... you name it!  This will save us hundreds and hundreds of hours of work, and make administering the app just a tad less tedious.  Of course, if your app requires tons of administrative work constantly, there's some other decision that has gone wrong.  In our case, we need to do some fine tuning once to make sure all our data is totally perfect.  It's also great for adding new beta users and all that jazz.

The web-app-theme generator is geared towards being a good launch point for an entire web app, not just an admin panel.  I'm not that excited to take these designs and showing them to the world, but they look and feel great for administrative purposes.  So I've tapped into using it exclusively in an admin only side of our app.

They key to making it an admin only app, is to use elegant Rails Namespaces, which came about in the rails Changeset 6594.  This allows you to group controllers for specific models inside a namespace.  I chose the namespace called "admin".  This effectively gave me the ability to create a subdirectory in the app/controllers directory called "admin".  Then under there were all my individual controllers for individual models.  My routes.rb looked as so:

 

  map.namespace :admin do |admin|

    # The below resource mapping directs /admin/teams/* to Admin::TeamsController (app/controllers/admin/teams_controller.rb)

    admin.resources :teams

    admin.resources :players

    admin.resources :users

  end

 

The above snippet shows how I have a model controller for each model.  By doing this, you can maintain a super restful way to setup your controllers, using the usual index, show, edit, update, new, create, destroy actions with their respective get, put, and delete request methods.  Also maintains a great way to use named routes with: admin_teams_path to goto the index action of the teams controller under the admin namespace.  Another example is new_admin_team_path for going to the new controller.  One thing to note is for other mentions of controllers outside of this admin namespace, use a prepended slash to specify no namespace.  For example:

redirect_to :controller => '/session', :action => 'new'

The above takes you out of the admin namespace because of the prepended slash in front of session.  Otherwise it will try to hit you with admin/session controller, which probably doesn't exist.

Once you've installed the web-app-theme generator, and setup your routes, you can then create initial admin layout and styles:

$ ruby script/generate themed admin --theme="drastic-dark"

The generator is called "themed" and the parameter "admin" is what we want the layout to be named, in this case it will automatically create an "admin.html.erb" layout file for you.  The option "--theme" is where you designate what theme style you want to go with.  Pick from here: http://pilu.github.com/web-app-theme

Once that's done, create a view for your first model controller (controller can be created after):

$ ruby script/generate themed admin/teams --layout=admin --with_will_paginate

The above will generate all the views for you under app/views/admin/teams using the layout we generated before as well as scaffolding the views up with will_paginate automatically.  Good if you have more than like 50 entries.  Then create your controller using the usual ActiveScaffold, but give it a namespace as well as a "global" layout:

class Admin::TeamsController < ApplicationController

layout 'admin'

...

The controller should be created inside the app/controllers/admin directory.  Given your routes were setup with admin.resources :teams, it should all be setup.  Oh and one last thing, if you used the will_paginate option, make sure you change the controller finds to paginate.  And that's it!  Do the same for all the other models you want to administer.  If you want to do one offs from models, it's easy to just use the usual template to put whatever you want in there.  Takes minutes, looks consistent, and is much nicer to use and expand on.

Make your employees happy in your startup, make the interns feel like they are not doing really sad work in, but actually being part of a really polished startup company.  Having a solid admin in your app is a huge indication of how well you plan everything you work on, including planning, researching options, and finding the most elegant and robust solution available.  If you can learn to use these tools quickly, you will go far in the rest of your development, user facing or not.

NOTE: I didn't do any editing of this blog post, just wrote what was in my head (as I usually do), so please excuse my crap grammatical errors and horrible organization of it all.  I'm not here to publish a book, just write my thoughts down.