Like many Rails people, I use New Relic to monitor my Rails app. At Wave Accounting we even pay for it. It’s well worth the money, as you get a lot of extra visibility into your app.

At the standard level, New Relic is pretty good, but sometimes it seems like I’m missing out on something. RPM will show me that a controller is running slowly but most of the time is spent in the controller itself, not really saying what’s happening. I’ve recently discovered a few tweaks that have made a huge difference.

Turn on garbage collection

It’s pretty embarrassing, but this isn’t on by default. It’s so simple to figure out how much of your time is spent in GC, the only caveat is that you have to be running REE or 1.9.x. This doc explains how, but all you have to do is turn on the stats and New Relic does the rest.

# For REE
GC.enable_stats
# For 1.9.2
# GC::Profiler.enable

Put that in an initializer, and you get GC activity in almost all your graphs:

Local development mode

If you go to http://localhost:3000/newrelic you will get some in depth information about what’s going on in your app when in dev mode. If you’re using pow then add the following to ~/.powconfig:

export NEWRELIC_DISPATCHER=pow
export POW_WORKERS=1

There’s a wealth of information here.

Trace specific sections of your code

Your controller does a bunch of stuff but New Relic reports it as one big block. block tracing to the rescue!

respond_to do |format|
  format.html do
    self.class.trace_execution_scoped(['Custom/MyController#show/render_html']) do
      do_it_in_html
    end
  end
  format.pdf do
    self.class.trace_execution_scoped(['Custom/MyController#show/render_pdf']) do
      do_it_in_pdf
    end
  end
end

Then, these blocks will be counted separately.

Trace methods in your code, or other people’s code

Want to do the same thing as before on someone else’s code, or at a method level? Add a method tracer in your initializer:

require 'new_relic/agent/method_tracer'
CalculationEngine.class_eval do
    include NewRelic::Agent::MethodTracer
    add_method_tracer :calculate_taxes
    add_method_tracer :calculate_totals
    add_method_tracer :send_invoice
end

Poof, all those are now traced and broken out separately:

Other things

You can also trace custom metrics, such as user signups or report views. I’m still working on that, along with monitoring background jobs.

a

I’ve wanted to play with the Twilio client for a while. They have this great quick start but it’s written in PHP. Now I don’t mind PHP, but I prefer Ruby. If I’m going to write anything using the client, it’s going to be in Ruby, so I don’t see the point in learning it on PHP.

So, here is the meat of the quickstart done up as a Rails 3.1 application.

First, generate the application.

$ rails new twilio
      create
      create  README
      create  Rakefile
      create  config.ru
      create  .gitignore
      create  Gemfile
      create  app
      ...

This creates a new Rails 3.1 app in the current directory called twilio. Change to this directory, and add a line to your Gemfile:

gem 'twilio-ruby'

Run bundle install to add the official Twilio gem to your bundle.

Next, head on over to your Twilio account and get your SID and auth token. Those can go in an initializer, config/initializers/twilio.rb:

TwilioAccountSID="AC........"
TwilioAuthToken="......."

Those are the magic tokens that let you authenticate yourself to the Twilio API, and more importantly for them, let them bill you.

Next, head on over to app/helpers/application_helper.rb:

module ApplicationHelper
  def twilio_javascript_include_tag
    javascript_include_tag "http://static.twilio.com/libs/twiliojs/1.0/twilio.min.js"
  end
end

Then in app/views/layouts/application.html.erb add that helper in the head:

  <%= stylesheet_link_tag    "application" %>
  <%= javascript_include_tag "application" %>
  <%= twilio_javascript_include_tag  %>
  <%= csrf_meta_tags %>

Yea, you could have put the code right in the layout, but I like sparse layout files.

Next up, create a controller:

$ rails generate controller client index
      create  app/controllers/client_controller.rb
       route  get "client/index"
      invoke  erb
      create    app/views/client
      create    app/views/client/index.html.erb
      invoke  test_unit
      create    test/functional/client_controller_test.rb
      invoke  helper
      create    app/helpers/client_helper.rb
      invoke    test_unit
      create      test/unit/helpers/client_helper_test.rb
      invoke  assets
      invoke    coffee
      create      app/assets/javascripts/client.js.coffee
      invoke    scss
      create      app/assets/stylesheets/client.css.scss

Then add root :to => ‘client#index’ to config/routes.rb so that your new view is displayed in the root url.

Run rails server or whatever you do to start your dev instance and browse to it. You should get the usual “find me in app/views/client/index.html.erb” message. Check the headers to make sure the library is being installed. The rest of the examples then deal with app/views/client/index.html.erb and app/helpers/client_helper.rb.

For the first example you want:

Helper:

module ClientHelper

  def twilio_token
    capability = Twilio::Util::Capability.new TwilioAccountSID, TwilioAuthToken
    capability.allow_client_outgoing "APabe7650f654fc34655fc81ae71caa3ff"
    capability.generate
  end
end

View:

<%= javascript_tag do %>
function call() {
  Twilio.Device.connect();
}

function hangup() {
  Twilio.Device.disconnectAll();
}

$(function() {
  Twilio.Device.setup("<%= twilio_token %>");

  Twilio.Device.ready(function (device) {
    $("#log").text("Ready");
  });

  Twilio.Device.error(function (error) {
    $("#log").text("Error: " + error.message);
  });

  Twilio.Device.connect(function (conn) {
    $("#log").text("Successfully established call");
  });
});
<% end %>
<button class="call" onclick="call();">
  Call
</button>

For the second example, you just change the view.

<%= javascript_tag do %>
function call() {
  Twilio.Device.connect();
}

function hangup() {
  Twilio.Device.disconnectAll();
}

$(function() {
  Twilio.Device.setup("<%= twilio_token %>");

  Twilio.Device.ready(function (device) {
    $("#log").text("Ready");
  });

  Twilio.Device.error(function (error) {
    $("#log").text("Error: " + error.message);
  });

  Twilio.Device.disconnect(function (conn) {
    $("#log").text("Call ended");
  });

  Twilio.Device.connect(function (conn) {
    $("#log").text("Successfully established call");
  });
});
<% end %>
<button class="call" onclick="call();">
  Call
</button>

<button class="hangup" onclick="hangup();">
  Hangup
</button>

<div id="log">Loading pigeons...</div>

For the third example we’ll have to change the helper and the view accordingly:

app/helpers/client_helper.rb (note I’m using my own sandbox id. You get your own inside the Twilio account page!)

module ClientHelper
  def twilio_token
    capability = Twilio::Util::Capability.new TwilioAccountSID, TwilioAuthToken
    capability.allow_client_outgoing "AP..."
    capability.allow_client_incoming 'jenny'
    capability.generate
  end
end

app/views/client/index.html.erb

%= javascript_tag do %>
function call() {
  Twilio.Device.connect();
}

function hangup() {
  Twilio.Device.disconnectAll();
}

$(function() {
  Twilio.Device.setup("<%= twilio_token %>");

  Twilio.Device.ready(function (device) {
    $("#log").text("Ready");
  });

  Twilio.Device.error(function (error) {
    $("#log").text("Error: " + error.message);
  });

  Twilio.Device.disconnect(function (conn) {
    $("#log").text("Call ended");
  });

  Twilio.Device.connect(function (conn) {
    $("#log").text("Successfully established call");
  });

  Twilio.Device.incoming(function (conn) {
    $("#log").text("Incoming connection from " + conn.parameters.From);
    // accept the incoming connection and start two-way audio
    conn.accept();
  });

});
<% end %>
<button class="call" onclick="call();">
  Call
</button>

<button class="hangup" onclick="hangup();">
  Hangup
</button>

<div id="log">Loading pigeons...</div>

Now, hook up a new action in the client controller to redirect the call from Twilio to the app inside app/controllers/client_controller.rb

def incoming
  response = Twilio::TwiML::Response.new do |r|
    r.Dial  do |d|
      d.Client 'jenny'
    end
  end

  render :text => response.text
end

Don’t forget to add post “client/incoming” to config/routes.rb. Then point your sandbox URL to your dev box, such as http://yourhome.com:3000/client/incoming.xml.

As a bonus, here’s a rake task to log in to a remote host and set up an ssh tunnel on remote port 3000 to local port 3000:

namespace :tunnel do
  desc "Start a ssh tunnel"
  task :start => :environment do

    public_host_username = "sean"
    public_host = "home.ertw.com"
    public_port = 3000

    local_port = 3000

    puts "Starting tunnel #{public_host}:#{public_port} \
          to 127.0.0.1:#{local_port}"

    exec "ssh -nNT -g -R *:#{public_port}:127.0.0.1:#{local_port} \
                           #{public_host_username}@#{public_host}"
  end
end

There are two more examples in the quickstart, but as they are more of the same, I’ll leave them for another post. I’d also like to try rewriting the Javascript in Coffeescript.

Update – Code is at https://github.com/swalberg/twilio-client-ruby

a

But I was OK with that. If I was to get anywhere with SmallPayroll, people had to submit a timesheet. They didn’t care if the server side code was efficient as long as it worked. And, as I was to find out, they didn’t seem to care if it was slow. In order to build the rest of the app I had to have a timesheet. So I left my ugly inefficient code in, along with the tests that exercised it, and got to building the rest of the application.

Between Rails Analyzer and New Relic I kept an eye on things. The timesheet did get worse as time passed. Now that SmallPayroll has become more successful and I can spend more time on it, I’ve come back to look at fixing this. But before I know if I’m doing a better job, I have to know how I’m doing now.

I found Advanced Performance Optimization of Rails Applications that had a neat trick to capture the queries generated by a test.

Put this in test_helper.rb:

module ActiveSupport
  class BufferedLogger
    attr_reader :tracked_queries
    def tracking=(val)
      @tracked_queries = []
      @tracking = val
    end

    def debug_with_tracking(message)
      @tracked_queries << $1 if @tracking && message =~ /3[56]\;1m(.* (Load|Create|Update|Destroy)) \(/
      debug_without_tracking(message)
    end
    alias_method_chain :debug, :tracking
  end
end
class ActiveSupport::TestCase
  def track_queries(&block)
    RAILS_DEFAULT_LOGGER.tracking = true
    # Had to add this to get queries to be logged
    Rails.logger.level = ActiveSupport::BufferedLogger::DEBUG
    yield
    result = RAILS_DEFAULT_LOGGER.tracked_queries
    RAILS_DEFAULT_LOGGER.tracking = false
    Rails.logger.level = ActiveSupport::BufferedLogger::INFO
    result
  end
end

Then wrap your test inside a test_queries block:

def test_submit_timesheet
    visit "/timesheet?date=2011-07-10"
    10.upto(16) do |day|
      fill_in /.*_shift_REG_2011-07-#{day}_hours/, :with => 8
    end

    queries = track_queries do
      click_button "Update"
    end
    puts queries.inspect
  end

After that it was a matter of making a performance test, copying over some of my functional tests that represented a case I was trying to optimize, then doing some before/after.

Before
------

TimesheetTest#test_submit_timesheet (283 ms warmup)
        process_time: 707 ms
        memory: 18915.89 KB
        objects: 232124
Queries: ["User Load", "User Update", "Employer Load", "Employee Load", "Shift Load", "Shift Create", "Shift Load", "Shift Create",
 "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load",
 "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create",
 "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load",
 "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create",
 "Shift Load", "Shift Create", "Employee Load", "Shift Load", "Property Load"]

After
-----

TimesheetTest#test_submit_timesheet (243 ms warmup)
        process_time: 578 ms
              memory: 14788.10 KB
             objects: 204551

Queries: ["User Load", "User Update", "Employer Load", "Employee Load", "Shift Load", "Shift Create", "Shift Create", "Shift Create",
"Shift Create", "Shift Create", "Shift Create", "Shift Create", "Employee Load", "Shift Load", "Property Load"]

So it would seem I've been able to knock off some time and memory consumption, along with lots of queries, by optimizing my code. Since I had already written test cases I was able to show that it worked the same as before.

But I think the better point to make here is that I could have spent a lot of time trying to build these optimizations in from day 1 and detracted from building a good product. Instead, I deferred the hard work until the time that it mattered. And now that I have more Ruby and Rails experience, doing the optimization was much easier. Something that might have taken several evenings over the course of a couple of weeks was done in less than a day. And while I don't follow TDD, having existing tests to start from made a huge difference.

a

I ran into a few problems.

The freshbooks gem doesn’t work under 1.9.2, which I found out after trying to deploy to Heroku and then trying locally under RVM. The error was:

NoMethodError in OutstandingController#index

undefined method `gsub!' for :invoice_id:Symbol

Someone made a compatible gem on Github that you can use by using the following in your Gemfile instead of gem “freshbooks”:

gem 'bcurren-freshbooks.rb',
 :require => 'freshbooks',
  :git => 'git://github.com/bcurren/freshbooks.rb'

There were a couple of differences I found:

• Instead of using FreshBooks.setup to enter your credentials, use FreshBooks::Base.establish_connection
• The original gem let you do FreshBooks::Invoice.send_by_email(id), the bcurren one makes it an instance method… FreshBooks::Invoice.get(id).send_by_email

Those were the only two changes I had to make.

The second problem was with the Twilio gem. I got SSL errors if I had to make calls to Twilio (as opposed to incoming requests from the Twilio API).

OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed):
  app/controllers/phone_controller.rb:8:in `call'

The reason for this problem is that the OpenSSL library is making a call to an HTTPS resource, but it has no way to verify the certificate. There are two ways to fix this problem:

1. Tell OpenSSL not to verify the certificate
2. Give OpenSSL the proper Certification Authority (CA) certificates to verify the certificate

I’ll confess that my normal approach here is #1, but this time I felt like doing it properly… Since the Twilio module includes HTTPParty you can call methods right on the Twilio module. So add an intializer inside config/initializers, such as twilio.rb:

Twilio::ssl_ca_file File.join(Rails.root, "config/cacert.pem")

Then, all you have to do is grab cacert.pem from somewhere else. Many other gems include it so if you look for the file on your hard drive you should find it. FWIW, the NewRelic one doesn’t work, it only includes the certificates they need. I ended up finding mine in ActiveMerchant.

*edit* I forked the twilio gem to try and bundle the cacerts.pem file, and as I was going through the code I saw that they look for the certs in /etc/ssl/certs unless you use the method above.

After that, things just worked!

a

Controller:

    @commits = Array.new
    git = `git log -15 --abbrev-commit --pretty=format:"%H - %cr - %s - %d"`
    git.split(/\n/).each do |commit|
      elements = commit.split(/ - /)
      @commits << {
        :hash => elements[0],
        :time => elements[1],
        :subject => elements[2],
        :refs => (elements[3] || "").match(/deploy_[^,]*/)
      }
    end

The :refs never got used, I had thought that I’d tag all my deploys and be able to highlight them later, but it never ended up working out.

View: (It’s in HAML)

 - @commits.each do |commit|
      %li
        = link_to truncate(commit[:subject], :length => 80), "http://git.example.com/git/myapp.git/commit/#{commit[:hash]}"
        = "(#{commit[:time]})"
        %b= commit[:refs]

It’s pretty simple, it just parses the output of git log and spits it out as a list, showing the description and how long ago it was checked in. If you have gitweb or something similar installed, you get a link to the repo.

It’s helped me to find production bugs, and also when I deploy without pushing my code to the git repo, and end up forgetting some changes!

a

After watching railscasts on beanstalkd and stalker I decided to use that. Beanstalkd is a lightweight job processor, and stalker makes it very simple to use from the client end.

I used to have an observer that said

class UserObserver < ActiveRecord::Observer
  def after_create(user)
    UserMailer.send_later(:deliver_welcome_email, user)
    UserMailer.send_later(:deliver_notify_admin, user)
  end
end

This became:

class UserObserver < ActiveRecord::Observer
  def after_create(user)
    Stalker.enqueue("email.new_user", :user_id => user.id)
  end
end

delayed_job was nice in that the job would just run against the model, but now I have to process the job in config/worker.rb:

require 'stalker'
include Stalker
require File.expand_path("../environment", __FILE__)

job 'default' do |args|
  puts "I don't support the default queue"
end

job 'email.new_user' do |args|
  user = User.find(args["user_id"])
  UserMailer.deliver_welcome_email(user)
  UserMailer.deliver_notify_admin(user)
end

One thing about stalker is that it wants you to pass simple objects instead of ActiveRecord objects, so I queue the user_id instead of the user model.

The script above monitors the default tube, which I don't use, because the nagios plugin for beanstalk expects someone to monitor it (after setting it all up, I guess I could have set it up to ignore that tube). I'm also using the munin plugin for beanstalkd to graph the activity in the daemon.

Then, script/worker uses the daemons gem to start the job and restart it if it crashes:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'

pwd  = File.dirname(File.expand_path(__FILE__))
file = pwd + '/../config/worker.rb'

Daemons.run_proc(
  'payroll-generic-worker', # name of daemon
  :dir_mode => :normal,
  :dir => File.join(pwd, '../tmp/pids'),
  :backtrace => true,
  :monitor => true,
  :log_output => true
) do
  exec "stalk #{file}"
end

Finally, some capistrano magic to start the worker with the deploy inside config/deploy.rb

namespace :beanstalk do
  desc "Start beanstalk process"
  task :start, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker start"
  end

  desc "Stop beanstalk process"
  task :stop, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker stop"
  end

  desc "Restart beanstalk process"
  task :restart, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker restart"
  end
end

after "deploy:start", "beanstalk:start"
after "deploy:stop", "beanstalk:stop"
after "deploy:restart", "beanstalk:restart"

The only problem I've run into so far is that my HTML email seems to go out without the text/html content-type. Fixing that was a simple matter of putting content_type 'text/html' inside my mailer, which wasn't needed when I was using delayed_job.

a

Step 1 – Install the plugin

 rails plugin install https://github.com/rails/exception_notification.git

Step 2 – Edit config/application.rb:

module MyApp
  class Application < Rails::Application
    # .....
    #  somewhere in this block put the following:

    config.middleware.use "::ExceptionNotifier",
      :email_prefix => "[MyApp Error] ",
      :sender_address => %{"notifier" },
      :exception_recipients => %w{youraddress@example.com}

3. Verify:

$ rake middleware | grep ExceptionNotifier
use ExceptionNotifier

Now you’ll get any application errors emailed to the addresses in the exception_recipients array.

a

One great feature is that git can work with an SVN repository. You use git to check out some SVN code into a local git repository, you work with the repo as if it were git, then you commit your changes back to svn.

The lowdown on this is:

# Check out the svn repo
# It may take a while because it's copying all the changes, not just a checkout
git svn init http://myrepo/myproject/trunk myproject
cd myproject
# do some work
# commit to git
git commit -a -m "did some work"
# send updates to svn
git svn dcommit

But wait, someone else has done some work and you want to pull the latest changes from svn to your git repo

git svn rebase

Nice eh?

So let’s say you’re working in your git repo, and want to branch

git branch somefeature
git checkout somefeature

You’re working along, and you want to pull changes from svn into master and your branch.

git checkout master
git svn rebase
# master is up to date
# so merge master into the branch
git checkout somefeature
git merge master

This is so easy to use! Previously, with SVN, I’d branch, work on the feature, and then merge back to trunk. Given how painful merges were, there was no hope that I’d be merging trunk changes into my feature branch.

I’m sure my tune will change the first time I blow up my repository though

a

Each form has a hidden form element:

<div style="margin:0;padding:0;display:inline">
<input name="authenticity_token" type="hidden"
value="16iUP1J2tdSKyvHKgYR/I/og6K7NgPPmTHCZ+idQP4k=" />
</div>

The token is also encrypted in the session so that the response to the form has to match. This value changes every time the form is loaded and can’t be set to a known value, otherwise it would be easy to defeat the CSRF protection. This means that JMeter has to figure out the token and put it into the POST.

Googling around, I found some people that said they did it, and a lot of people who couldn’t get it to work, but no solid walkthroughs.

Here’s my solution:


There are three elements. The first is the HTTP request sampler that gets the login page. Under that is a regular expression extractor post processor that gets the authenticity token. The extractor uses a simple regexp to pull out the value parameter and saves it to the LOGIN_AUTH_TOKEN variable. The login is then done by making reference to the variable – ${LOGIN_AUTH_TOKEN}. Make sure you have the Encode? button checked, as the authenticity_token is not always base-64 friendly!

The final step, not pictured, is that you have an HTTP Cookie Manager in your thread group to take care of cookies. You probably already have one, though.

Sean

a

So I ran into a case where I do an action and use should_change to make sure that a certain number of rows were added to a table, then I do it again and make sure they don’t change. So I used a nested set of contexts to handle this:

context "do it once" do
  setup do
    post :method, :foo => {...}
  end
  should_change("something", :by => 6) {Something.count}
  #... other tests ...
  context "do it a second time" do
     setup do
       post :method, :foo => {...}
     end
     # should it change by 6 or 0?
  end
end

I wasn’t sure if the should_change applied to both the setup blocks or only the second. Turns out it’s only the second, so it should not change in the second test. Here’s proof:

require 'test_helper'

class SomethingControllerTest < ActionController::TestCase

  context "outer" do
    setup do
      @user = Factory.create(:valid_user)
    end
    should_change("number of users", :by => 1) { User.count }
    context "inner" do
      setup do
      end
      should_change("number of users", :by => 0) { User.count }
    end
  end
end

Since I didn’t write my tests before I wrote the code I wasn’t sure if my code was bad or my test was. Now I know how the test could be structured and I could test properly.

a