Sean’s Obsessions

икона за подаръкAt my awesome job we use HipChat for team collaboration. I also use the Pomodoro app to try and manage my time. One problem is that I often get interrupted while working.

Long story short, the Pomodoro app lets you run AppleScript when various events happen, so I wrote some stuff to change my HipChat status to DND when I’m in the middle of a work cycle. Here’s the code:

tell application "System Events" to tell UI element "HipChat" of list 1 of process "Dock"
	perform action "AXShowMenu"
	delay 0.5
	click menu item "Status" of menu 1
	click menu item "DND" of menu 1 of menu item "Status" of menu 1
 end tell

All that remains is to insert that into the Pomodoro app through Preferences -> Scripts:

Just note that you have to change “DND” to “Available” for some of the events.

This was my first foray into AppleScript, so it’s possible I’m sending my banking details off to Nigeria, but it seems to work so far.

Edit you need to enable access for assistive devices from System Preferences -> Universal Access:

ХудожникБогородицаUpdate – if you sign up through this link, New Relic will send you an RC helicopter for trying out New Relic.

Like many Rails people, I use New Relic to monitor my Rails app. At Wave Accounting we even pay for it. It’s well worth the money, as you get a lot of extra visibility into your app.

At the standard level, New Relic is pretty good, but sometimes it seems like I’m missing out on something. RPM will show me that a controller is running slowly but most of the time is spent in the controller itself, not really saying what’s happening. I’ve recently discovered a few tweaks that have made a huge difference.

Turn on garbage collection

It’s pretty embarrassing, but this isn’t on by default. It’s so simple to figure out how much of your time is spent in GC, the only caveat is that you have to be running REE or 1.9.x. This doc explains how, but all you have to do is turn on the stats and New Relic does the rest.

# For REE
GC.enable_stats
# For 1.9.2
# GC::Profiler.enable

Put that in an initializer, and you get GC activity in almost all your graphs:

Local development mode

If you go to http://localhost:3000/newrelic you will get some in depth information about what’s going on in your app when in dev mode. If you’re using pow then add the following to ~/.powconfig:

export NEWRELIC_DISPATCHER=pow
export POW_WORKERS=1

There’s a wealth of information here.

Trace specific sections of your code

Your controller does a bunch of stuff but New Relic reports it as one big block. block tracing to the rescue!

respond_to do |format|
  format.html do
    self.class.trace_execution_scoped(['Custom/MyController#show/render_html']) do
      do_it_in_html
    end
  end
  format.pdf do
    self.class.trace_execution_scoped(['Custom/MyController#show/render_pdf']) do
      do_it_in_pdf
    end
  end
end

Then, these blocks will be counted separately.

Trace methods in your code, or other people’s code

Want to do the same thing as before on someone else’s code, or at a method level? Add a method tracer in your initializer:

require 'new_relic/agent/method_tracer'
CalculationEngine.class_eval do
    include NewRelic::Agent::MethodTracer
    add_method_tracer :calculate_taxes
    add_method_tracer :calculate_totals
    add_method_tracer :send_invoice
end

Poof, all those are now traced and broken out separately:

Other things

You can also trace custom metrics, such as user signups or report views. I’m still working on that, along with monitoring background jobs.

икони цени

I’ve wanted to play with the Twilio client for a while. They have this great quick start but it’s written in PHP. Now I don’t mind PHP, but I prefer Ruby. If I’m going to write anything using the client, it’s going to be in Ruby, so I don’t see the point in learning it on PHP.

So, here is the meat of the quickstart done up as a Rails 3.1 application.

First, generate the application.

$ rails new twilio
      create
      create  README
      create  Rakefile
      create  config.ru
      create  .gitignore
      create  Gemfile
      create  app
      ...

This creates a new Rails 3.1 app in the current directory called twilio. Change to this directory, and add a line to your Gemfile:

gem 'twilio-ruby'

Run bundle install to add the official Twilio gem to your bundle.

Next, head on over to your Twilio account and get your SID and auth token. Those can go in an initializer, config/initializers/twilio.rb:

TwilioAccountSID="AC........"
TwilioAuthToken="......."

Those are the magic tokens that let you authenticate yourself to the Twilio API, and more importantly for them, let them bill you.

Next, head on over to app/helpers/application_helper.rb:

module ApplicationHelper
  def twilio_javascript_include_tag
    javascript_include_tag "http://static.twilio.com/libs/twiliojs/1.0/twilio.min.js"
  end
end

Then in app/views/layouts/application.html.erb add that helper in the head:

  <%= stylesheet_link_tag    "application" %>
  <%= javascript_include_tag "application" %>
  <%= twilio_javascript_include_tag  %>
  <%= csrf_meta_tags %>

Yea, you could have put the code right in the layout, but I like sparse layout files.

Next up, create a controller:

$ rails generate controller client index
      create  app/controllers/client_controller.rb
       route  get "client/index"
      invoke  erb
      create    app/views/client
      create    app/views/client/index.html.erb
      invoke  test_unit
      create    test/functional/client_controller_test.rb
      invoke  helper
      create    app/helpers/client_helper.rb
      invoke    test_unit
      create      test/unit/helpers/client_helper_test.rb
      invoke  assets
      invoke    coffee
      create      app/assets/javascripts/client.js.coffee
      invoke    scss
      create      app/assets/stylesheets/client.css.scss

Then add root :to => ‘client#index’ to config/routes.rb so that your new view is displayed in the root url.

Run rails server or whatever you do to start your dev instance and browse to it. You should get the usual “find me in app/views/client/index.html.erb” message. Check the headers to make sure the library is being installed. The rest of the examples then deal with app/views/client/index.html.erb and app/helpers/client_helper.rb.

For the first example you want:

Helper:

module ClientHelper

  def twilio_token
    capability = Twilio::Util::Capability.new TwilioAccountSID, TwilioAuthToken
    capability.allow_client_outgoing "APabe7650f654fc34655fc81ae71caa3ff"
    capability.generate
  end
end

View:

<%= javascript_tag do %>
function call() {
  Twilio.Device.connect();
}

function hangup() {
  Twilio.Device.disconnectAll();
}

$(function() {
  Twilio.Device.setup("<%= twilio_token %>");

  Twilio.Device.ready(function (device) {
    $("#log").text("Ready");
  });

  Twilio.Device.error(function (error) {
    $("#log").text("Error: " + error.message);
  });

  Twilio.Device.connect(function (conn) {
    $("#log").text("Successfully established call");
  });
});
<% end %>
<button class="call" onclick="call();">
  Call
</button>

For the second example, you just change the view.

<%= javascript_tag do %>
function call() {
  Twilio.Device.connect();
}

function hangup() {
  Twilio.Device.disconnectAll();
}

$(function() {
  Twilio.Device.setup("<%= twilio_token %>");

  Twilio.Device.ready(function (device) {
    $("#log").text("Ready");
  });

  Twilio.Device.error(function (error) {
    $("#log").text("Error: " + error.message);
  });

  Twilio.Device.disconnect(function (conn) {
    $("#log").text("Call ended");
  });

  Twilio.Device.connect(function (conn) {
    $("#log").text("Successfully established call");
  });
});
<% end %>
<button class="call" onclick="call();">
  Call
</button>

<button class="hangup" onclick="hangup();">
  Hangup
</button>

<div id="log">Loading pigeons...</div>

For the third example we’ll have to change the helper and the view accordingly:

app/helpers/client_helper.rb (note I’m using my own sandbox id. You get your own inside the Twilio account page!)

module ClientHelper
  def twilio_token
    capability = Twilio::Util::Capability.new TwilioAccountSID, TwilioAuthToken
    capability.allow_client_outgoing "AP..."
    capability.allow_client_incoming 'jenny'
    capability.generate
  end
end

app/views/client/index.html.erb

%= javascript_tag do %>
function call() {
  Twilio.Device.connect();
}

function hangup() {
  Twilio.Device.disconnectAll();
}

$(function() {
  Twilio.Device.setup("<%= twilio_token %>");

  Twilio.Device.ready(function (device) {
    $("#log").text("Ready");
  });

  Twilio.Device.error(function (error) {
    $("#log").text("Error: " + error.message);
  });

  Twilio.Device.disconnect(function (conn) {
    $("#log").text("Call ended");
  });

  Twilio.Device.connect(function (conn) {
    $("#log").text("Successfully established call");
  });

  Twilio.Device.incoming(function (conn) {
    $("#log").text("Incoming connection from " + conn.parameters.From);
    // accept the incoming connection and start two-way audio
    conn.accept();
  });

});
<% end %>
<button class="call" onclick="call();">
  Call
</button>

<button class="hangup" onclick="hangup();">
  Hangup
</button>

<div id="log">Loading pigeons...</div>

Now, hook up a new action in the client controller to redirect the call from Twilio to the app inside app/controllers/client_controller.rb

def incoming
  response = Twilio::TwiML::Response.new do |r|
    r.Dial  do |d|
      d.Client 'jenny'
    end
  end

  render :text => response.text
end

Don’t forget to add post “client/incoming” to config/routes.rb. Then point your sandbox URL to your dev box, such as http://yourhome.com:3000/client/incoming.xml.

As a bonus, here’s a rake task to log in to a remote host and set up an ssh tunnel on remote port 3000 to local port 3000:

namespace :tunnel do
  desc "Start a ssh tunnel"
  task :start => :environment do

    public_host_username = "sean"
    public_host = "home.ertw.com"
    public_port = 3000

    local_port = 3000

    puts "Starting tunnel #{public_host}:#{public_port} \
          to 127.0.0.1:#{local_port}"

    exec "ssh -nNT -g -R *:#{public_port}:127.0.0.1:#{local_port} \
                           #{public_host_username}@#{public_host}"
  end
end

There are two more examples in the quickstart, but as they are more of the same, I’ll leave them for another post. I’d also like to try rewriting the Javascript in Coffeescript.

Update – Code is at https://github.com/swalberg/twilio-client-ruby

No Starch Press sent me Practical Packet Analysis, 2ed a little while back. At about 250 pages it’s a lot smaller than Chappell’s “Wireshark Network Analysis”, and more appropriate for someone who wants to get up and running quickly rather than going for a certification.

The book assumes no knowledge of Wireshark, and a basic understanding of networking. More than half the book is devoted to teaching the Wireshark interface and how the popular protocols work. So, if you don’t know anything about DNS recursion, you’ll get a taste of it here along with what it looks like in Wireshark. The first half covers everything from filtering inside Wireshark to how different protocols work.

The second half of the book follows fairly typical examples, such as decoding HTTP streams and troubleshooting the causes of network congestion. Of special interest is Chapter 10, which is about using wireshark for security analysis. This chapter is merely an introduction to a huge topic, but the author has chosen some interesting examples such as an ARP poisoning attack and analysis of a trojan horse.

One theme the author continually comes back to is appropriate placement of the analysis tool. The early chapters discuss the matter in theory, and every example in the second half has some text that analyzes the options for where to use Wireshark and where the best spot is.

Some of the highlights of the book:

• A great discussion of TCP congestion and analysis of a congestion scenario
• A good tradeoff between depth and breadth. This is a “getting started” guide/
• Uses many of the features of Wireshark in a practical context
• A good, though basic, chapter about wireless sniffing

Some of the downsides:

• No IPv6 (other than a brief mention of a host filter)
• Would have liked to see more use about IO graphs and TCP stream graphs especially when talking about congestion.

On the whole, a great book for the IT administrator who wants to quickly get started using Wireshark. Cover price is $49.95 US, Amazon.com is showing it for $30 which is a bargain.

иконописI use Nagios to monitor the health of a few servers, and would like to be paged if something goes wrong.

When I set it up a couple of years ago, I used SMS Gateway which was $10 for 100 SMSes. I was able to page with a simple curl command. However I’d get the odd page that wouldn’t go through, and despite being very responsive, the support wasn’t very reassuring.

Now that I’ve depleted my 100 pages, I figured I’d move over to Twilio because they’re pretty slick, and the reliability has to be better.

Some Nagios code, first:

define contactgroup{
        contactgroup_name       important
        alias                   Sean Buzz
        members                 sean, page
}
define contact{
        contact_name                    page
        alias                           page
        service_notification_commands   notify-by-page
        host_notification_commands      host-notify-by-page
        service_notification_period 24x7
        host_notification_period 24x7
        service_notification_options w,u,c,r
        host_notification_options d,u,r
        pager                           nobody@localhost
}
define service{
        use                             local-service         ; Name of service template to use
        host_name                       localhost
        service_description             smallpayroll.ca
        contact_groups                  important
        check_command                   check_http_string!smallpayroll.ca!Easy to use
        }

The first stanza creates a contact group called “important” that emails sean, and pages. The second stanza implements the “host-notify-by-page” and “notify-by-page” which do the actual paging. The final stanza is an example of a service that would get paged on. If the check_http_string check fails, the “important” contact group is notified.

The code to page is as follows:

define command {
        command_name    notify-by-page
        command_line    curl --data-urlencode "From=YOURTWILIONUMBER" --data-urlencode "To=YOURCELL" --data-urlencode "Body=[Nagios] $NOTIFICATIONTYPE$ $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$" https://SID:TOKEN@api.twilio.com/2010-04-01/Accounts/SID/SMS/Messages >> /tmp/sms
}
define command {
        command_name    host-notify-by-page
        command_line    curl --data-urlencode "From=YOURTWILIONUMBER" --data-urlencode "To=YOURCELL" --data-urlencode "Body=[Nagios] $HOSTSTATE$ alert for $HOSTNAME$" https://SID:TOKEN@api.twilio.com/2010-04-01/Accounts/SID/SMS/Messages >> /tmp/sms
}

To get the SID and TOKEN (note there are two instances of the SID, the second is in the URL right after accounts) go to your dashboard and look at the top:

To get a number click on Numbers then Buy a Number:

Then search for a number. It should be in the USA, as it looks like Canadian numbers don’t support SMS. You can verify this by clicking on “Buy”:

Buy the number for $1/month. You don’t have to set up any URLs with it if you’re doing outbound only.

YOURCELL should be obvious It could also be templated within Nagios.

КартиниSmallPayroll.ca was my first big Rails project, and looking back at some of the code, it shows. One of the first things I did was the timesheet. The form has 21 input fields per employee, then it has to go through the database and figure out if days have changed or deleted. So it’s doing a lot, and at the time I was trying to figure out how both Ruby and Rails worked, and the code ended up being a mess.

But I was OK with that. If I was to get anywhere with SmallPayroll, people had to submit a timesheet. They didn’t care if the server side code was efficient as long as it worked. And, as I was to find out, they didn’t seem to care if it was slow. In order to build the rest of the app I had to have a timesheet. So I left my ugly inefficient code in, along with the tests that exercised it, and got to building the rest of the application.

Between Rails Analyzer and New Relic I kept an eye on things. The timesheet did get worse as time passed. Now that SmallPayroll has become more successful and I can spend more time on it, I’ve come back to look at fixing this. But before I know if I’m doing a better job, I have to know how I’m doing now.

I found Advanced Performance Optimization of Rails Applications that had a neat trick to capture the queries generated by a test.

Put this in test_helper.rb:

module ActiveSupport
  class BufferedLogger
    attr_reader :tracked_queries
    def tracking=(val)
      @tracked_queries = []
      @tracking = val
    end

    def debug_with_tracking(message)
      @tracked_queries << $1 if @tracking && message =~ /3[56]\;1m(.* (Load|Create|Update|Destroy)) \(/
      debug_without_tracking(message)
    end
    alias_method_chain :debug, :tracking
  end
end
class ActiveSupport::TestCase
  def track_queries(&block)
    RAILS_DEFAULT_LOGGER.tracking = true
    # Had to add this to get queries to be logged
    Rails.logger.level = ActiveSupport::BufferedLogger::DEBUG
    yield
    result = RAILS_DEFAULT_LOGGER.tracked_queries
    RAILS_DEFAULT_LOGGER.tracking = false
    Rails.logger.level = ActiveSupport::BufferedLogger::INFO
    result
  end
end

Then wrap your test inside a test_queries block:

def test_submit_timesheet
    visit "/timesheet?date=2011-07-10"
    10.upto(16) do |day|
      fill_in /.*_shift_REG_2011-07-#{day}_hours/, :with => 8
    end

    queries = track_queries do
      click_button "Update"
    end
    puts queries.inspect
  end

After that it was a matter of making a performance test, copying over some of my functional tests that represented a case I was trying to optimize, then doing some before/after.

Before
------

TimesheetTest#test_submit_timesheet (283 ms warmup)
        process_time: 707 ms
        memory: 18915.89 KB
        objects: 232124
Queries: ["User Load", "User Update", "Employer Load", "Employee Load", "Shift Load", "Shift Create", "Shift Load", "Shift Create",
 "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load",
 "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create",
 "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load",
 "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create", "Shift Load", "Shift Create",
 "Shift Load", "Shift Create", "Employee Load", "Shift Load", "Property Load"]

After
-----

TimesheetTest#test_submit_timesheet (243 ms warmup)
        process_time: 578 ms
              memory: 14788.10 KB
             objects: 204551

Queries: ["User Load", "User Update", "Employer Load", "Employee Load", "Shift Load", "Shift Create", "Shift Create", "Shift Create",
"Shift Create", "Shift Create", "Shift Create", "Shift Create", "Employee Load", "Shift Load", "Property Load"]

So it would seem I've been able to knock off some time and memory consumption, along with lots of queries, by optimizing my code. Since I had already written test cases I was able to show that it worked the same as before.

But I think the better point to make here is that I could have spent a lot of time trying to build these optimizations in from day 1 and detracted from building a good product. Instead, I deferred the hard work until the time that it mattered. And now that I have more Ruby and Rails experience, doing the optimization was much easier. Something that might have taken several evenings over the course of a couple of weeks was done in less than a day. And while I don't follow TDD, having existing tests to start from made a huge difference.

This site is hosted on a Linode 768 VPS, and has been for a couple of years now, along with some other domains. I have hosted it at home, and also on a GoDaddy VPS which didn’t end up being all that good, but am now very happy with Linode. I host a combination of PHP (mostly WordPress) and Ruby on Rails applications.

Over the years Linode has kept the price the same ($30/month for my plan) but have increased the disk and memory of their plans every year. When I started out my plan had 18G of disk space and around 512MB of RAM, now it has 30G of disk and 768MB of RAM. So the value for money keeps on getting better.

I’ve also set up Linode VPSes for a few people, including TwiTip.com and TopMMANews.com and continue to assist in their management. Both of them are fairly heavy sites and also run on a Linode 768. TwiTip hit 11mbps of traffic when it was tweeted by Ashton Kutcher, and TopMMANews has a fairly active site.

I’ve found the service to be very reliable. At one point one of their data centres was having problems but they were fixed reasonably quickly and the company kept the customers updated.

You get a control panel that lets you see your cpu/disk/io status, along with how much disk and network you’ve used. The screenshot below shows my system (you can see that I haven’t yet taken advantage of the 6GB of disk space they added to my account)

One feature I really like about the service is that you get free DNS hosting, and the interface is very simple (I mean “simple” as “does not get in your way”, not as in “stripped of features”). You can do AAAA, TXT, and SRV records, or control the whole thing through an API.

I can’t speak highly enough about Linode VPSes. If you’re looking for a VPS service they offer great value for money and a high service level. If you’re wondering about which size to buy, I’ve found the 768 to be a real workhorse. You can also upgrade/downgrade your plan with minimal downtime and no loss of data, so there’s little risk in picking the wrong plan.

I have been playing with the Freshbooks API and the Twilio API as part of a contest that Twilio is running. It’s a great excuse to try something I’ve been meaning to do for a while.

I ran into a few problems.

The freshbooks gem doesn’t work under 1.9.2, which I found out after trying to deploy to Heroku and then trying locally under RVM. The error was:

NoMethodError in OutstandingController#index

undefined method `gsub!' for :invoice_id:Symbol

Someone made a compatible gem on Github that you can use by using the following in your Gemfile instead of gem “freshbooks”:

gem 'bcurren-freshbooks.rb',
 :require => 'freshbooks',
  :git => 'git://github.com/bcurren/freshbooks.rb'

There were a couple of differences I found:

• Instead of using FreshBooks.setup to enter your credentials, use FreshBooks::Base.establish_connection
• The original gem let you do FreshBooks::Invoice.send_by_email(id), the bcurren one makes it an instance method… FreshBooks::Invoice.get(id).send_by_email

Those were the only two changes I had to make.

The second problem was with the Twilio gem. I got SSL errors if I had to make calls to Twilio (as opposed to incoming requests from the Twilio API).

OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed):
  app/controllers/phone_controller.rb:8:in `call'

The reason for this problem is that the OpenSSL library is making a call to an HTTPS resource, but it has no way to verify the certificate. There are two ways to fix this problem:

1. Tell OpenSSL not to verify the certificate
2. Give OpenSSL the proper Certification Authority (CA) certificates to verify the certificate

I’ll confess that my normal approach here is #1, but this time I felt like doing it properly… Since the Twilio module includes HTTPParty you can call methods right on the Twilio module. So add an intializer inside config/initializers, such as twilio.rb:

Twilio::ssl_ca_file File.join(Rails.root, "config/cacert.pem")

Then, all you have to do is grab cacert.pem from somewhere else. Many other gems include it so if you look for the file on your hard drive you should find it. FWIW, the NewRelic one doesn’t work, it only includes the certificates they need. I ended up finding mine in ActiveMerchant.

*edit* I forked the twilio gem to try and bundle the cacerts.pem file, and as I was going through the code I saw that they look for the certs in /etc/ssl/certs unless you use the method above.

After that, things just worked!

I have an administration panel for my Rails application that shows various information. I’ve found it helpful to show the last few commits along with a link to the repository. Here’s the code:

Controller:

    @commits = Array.new
    git = `git log -15 --abbrev-commit --pretty=format:"%H - %cr - %s - %d"`
    git.split(/\n/).each do |commit|
      elements = commit.split(/ - /)
      @commits << {
        :hash => elements[0],
        :time => elements[1],
        :subject => elements[2],
        :refs => (elements[3] || "").match(/deploy_[^,]*/)
      }
    end

The :refs never got used, I had thought that I’d tag all my deploys and be able to highlight them later, but it never ended up working out.

View: (It’s in HAML)

 - @commits.each do |commit|
      %li
        = link_to truncate(commit[:subject], :length => 80), "http://git.example.com/git/myapp.git/commit/#{commit[:hash]}"
        = "(#{commit[:time]})"
        %b= commit[:refs]

It’s pretty simple, it just parses the output of git log and spits it out as a list, showing the description and how long ago it was checked in. If you have gitweb or something similar installed, you get a link to the repo.

It’s helped me to find production bugs, and also when I deploy without pushing my code to the git repo, and end up forgetting some changes!

ПодаръцивикI was doing some work that involved moving between several directories. Remembering about pushd and popd, I googled around to try and find out how to use them properly. I found this article which was helpful, but what was even better was one of the comments talking about “cd -”.

[root@host tmp]# pwd
/tmp
[root@host tmp]# cd -
/auto/tmp/sean
[root@host sean]# pwd
/auto/tmp/sean
[root@host sean]# cd -
/tmp
[root@host tmp]# pwd
/tmp

What it does is cycle you between your last two directories. It also operates outside of the pushd/popd stack:

[root@host tmp]# dirs
/tmp

So you should still be able to use those!