Sean's Obsessions

  • I'm a happy Linode customer. This is a Linode 540 VPS. Linode periodically increases RAM and disk while keeping costs steady, which makes them the natural choice.
  • Archives

16 Jan

Starting the beanstalk worker from capistrano

I recently changed SmallPayroll to use Beanstalkd instead of delayed_job for background processing. delayed_job is an awesome tool and makes asynchronous processing so simple. However I wanted to have multiple queues so that I could have different workers processing different queues, and have some upcoming needs to process the jobs quicker than the 5 second polling time.

After watching railscasts on beanstalkd and stalker I decided to use that. Beanstalkd is a lightweight job processor, and stalker makes it very simple to use from the client end.

I used to have an observer that said

class UserObserver < ActiveRecord::Observer
  def after_create(user)
    UserMailer.send_later(:deliver_welcome_email, user)
    UserMailer.send_later(:deliver_notify_admin, user)

This became:

class UserObserver < ActiveRecord::Observer
  def after_create(user)
    Stalker.enqueue("email.new_user", :user_id =>

delayed_job was nice in that the job would just run against the model, but now I have to process the job in config/worker.rb:

require 'stalker'
include Stalker
require File.expand_path("../environment", __FILE__)

job 'default' do |args|
  puts "I don't support the default queue"

job 'email.new_user' do |args|
  user = User.find(args["user_id"])

One thing about stalker is that it wants you to pass simple objects instead of ActiveRecord objects, so I queue the user_id instead of the user model.

The script above monitors the default tube, which I don't use, because the nagios plugin for beanstalk expects someone to monitor it (after setting it all up, I guess I could have set it up to ignore that tube). I'm also using the munin plugin for beanstalkd to graph the activity in the daemon.

Then, script/worker uses the daemons gem to start the job and restart it if it crashes:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'

pwd  = File.dirname(File.expand_path(__FILE__))
file = pwd + '/../config/worker.rb'

  'payroll-generic-worker', # name of daemon
  :dir_mode => :normal,
  :dir => File.join(pwd, '../tmp/pids'),
  :backtrace => true,
  :monitor => true,
  :log_output => true
) do
  exec "stalk #{file}"

Finally, some capistrano magic to start the worker with the deploy inside config/deploy.rb

namespace :beanstalk do
  desc "Start beanstalk process"
  task :start, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker start"

  desc "Stop beanstalk process"
  task :stop, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker stop"

  desc "Restart beanstalk process"
  task :restart, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker restart"

after "deploy:start", "beanstalk:start"
after "deploy:stop", "beanstalk:stop"
after "deploy:restart", "beanstalk:restart"

The only problem I've run into so far is that my HTML email seems to go out without the text/html content-type. Fixing that was a simple matter of putting content_type 'text/html' inside my mailer, which wasn't needed when I was using delayed_job.

Leave a Reply

© 2014 Sean's Obsessions | Entries (RSS) and Comments (RSS)

Powered by Wordpress, design by Web4Sudoku, based on Pinkline byGPS Gazette