Starting the Beanstalk Worker From Capistrano

I recently changed SmallPayroll to use Beanstalkd instead of delayed_job for background processing. delayed_job is an awesome tool and makes asynchronous processing so simple. However I wanted to have multiple queues so that I could have different workers processing different queues, and have some upcoming needs to process the jobs quicker than the 5 second polling time.

After watching railscasts on beanstalkd and stalker I decided to use that. Beanstalkd is a lightweight job processor, and stalker makes it very simple to use from the client end.

I used to have an observer that said

<pre><code>class UserObserver < ActiveRecord::Observer<br />
  def after_create(user)<br />
    UserMailer.send_later(:deliver_welcome_email, user)<br />
    UserMailer.send_later(:deliver_notify_admin, user)<br />
  end<br />
end</code></pre></p>

This became:

<pre><code>class UserObserver < ActiveRecord::Observer<br />
  def after_create(user)<br />
    Stalker.enqueue("email.new_user", :user_id => user.id)<br />
  end<br />
end</code></pre></p>

delayed_job was nice in that the job would just run against the model, but now I have to process the job in config/worker.rb:

require 'stalker'
include Stalker
require File.expand_path("../environment", __FILE__)
job 'default' do |args|
  puts "I don't support the default queue"
end
job 'email.new_user' do |args|
  user = User.find(args["user_id"])
  UserMailer.deliver_welcome_email(user)
  UserMailer.deliver_notify_admin(user)
end

One thing about stalker is that it wants you to pass simple objects instead of ActiveRecord objects, so I queue the user_id instead of the user model.

The script above monitors the default tube, which I don’t use, because the nagios plugin for beanstalk expects someone to monitor it (after setting it all up, I guess I could have set it up to ignore that tube). I’m also using the munin plugin for beanstalkd to graph the activity in the daemon.

Then, script/worker uses the daemons gem to start the job and restart it if it crashes:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
pwd  = File.dirname(File.expand_path(__FILE__))
file = pwd + '/../config/worker.rb'
Daemons.run_proc(
  'payroll-generic-worker', # name of daemon
  :dir_mode => :normal,
  :dir => File.join(pwd, '../tmp/pids'),
  :backtrace => true,
  :monitor => true,
  :log_output => true
) do
  exec "stalk #{file}"
end

Finally, some capistrano magic to start the worker with the deploy inside config/deploy.rb

namespace :beanstalk do
  desc "Start beanstalk process"
  task :start, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker start"
  end
  desc "Stop beanstalk process"
  task :stop, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker stop"
  end
  desc "Restart beanstalk process"
  task :restart, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker restart"
  end
end
after "deploy:start", "beanstalk:start"
after "deploy:stop", "beanstalk:stop"
after "deploy:restart", "beanstalk:restart"

The only problem I’ve run into so far is that my HTML email seems to go out without the text/html content-type. Fixing that was a simple matter of putting content_type 'text/html' inside my mailer, which wasn’t needed when I was using delayed_job.

Sean’s Obsessions

Sean Walberg’s blog

Starting the Beanstalk Worker From Capistrano

Comments