After coming across a few sites that were serving pharma spam to Google Bot but not regular browsers, I thought it would be fun to give Sinatra a try and come up with a quick web app that checks for cloaked links. That lead to a few more checks, then some improvements, and Is it hacked? was launched. I’ve got some ideas for where I want to go with the project, but in the meantime it’s catching stuff that other sites in the space are missing.

There’s also a bookmarklet at the bottom of the page. You can drag it to your button bar and check the site you’re on for any infection.

Update: I’ve sold the site to someone else and am no longer involved.

Capistrano is a deployment framework for Rails, though it can be extended to do pretty much anything. The system comes with a bunch of built in tasks, and each task can be made up of other tasks and code. Plugins can add their own tasks and hook into default tasks, and the user can do the same through the configuration files.

Like any popular open source project capistrano has gone through changes. Documentation on the Internet is plentiful but often refers to old ways of doing things, so copy/pasting recipes can often result in stuff that doesn’t work quite right.

One common thing people need to do is to run some command after the deploy or at some time during the deploy. Capistrano has a very easy way of doing this.

1

after 'deploy:update_code', 'deploy:symlink_shared'

This says “after running deploy:update_code, run deploy:symlink_shared”. The latter is custom task that’s defined elsewhere.

The problem comes in when you look at the way the “deploy” and “deploy:migrations” tasks differ. I’ve seen a common problem where something works when deploying without migrations but not when migrations are used. Usually this is because the hook used is not the right one, either because of bad information or the user figured out where to hook into by looking at the output of a deploy.

If you look at Capistrano’s default deploy.rb you can piece together the tasks that are run in both cases.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

deploy:migrations
  update_code
   strategy.deploy!
   finalize_update
  migrate
  create_symlink
  restart
deploy
  update
    update_code
      strategy.deploy!
      finalize_update
    create_symlink
  restart

From this, you can see that the sequence is somewhat different. The “update” task isn’t used in the migrations case. Instead, the components are replicated.

If you want to hook in, use

• update_code to run before/after the code is put into the release directory, such as if you want to make more symlinks or do something before migrations are potentially run.
• create_symlink to run before/after the symbolic link pointing to the current release is made. Note that symlink is deprecated. You can run it from the command line, but if you try and hook in to it, you won’t be happy.
• restart to run before/after the application server is restarted, e.g. to restart background workers or do deployment notifications

The Unix shell has a whole bunch of features to make your life easier. One has to do with the history. Some I have managed to ingrain into muscle memory, others I have to remember which often means I do it the long way. I hope these examples help you out.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

# Start off with some files
$ touch one two three four
# !$ expands to all the arguments from the last command
$ ls !*
ls one two three four
four  one three   two
# !foo runs the last command that starts with foo
$ !touch
touch one two three four
# Carat (^) does a search/replace in the previous command
$ ^touch^ls
ls one two three four
four  one three   two
# !$ is the last item in the previous command, !^ is the first _argument_
$ head !$
head four
$ ls four three two one
four  one three   two
$ cat !^
cat four
# !:n is the n'th item on the previous command line, 0 indexed
$ ls four three two one
four  one three   two
$ cat !:3
cat two
$ ls four three two one
four  one three   two
$ which !:0
which ls
/bin/ls

There are a lot more, run a man bash and look for the section called HISTORY EXPANSION.

I’ve long been a fan of MixPanel and was happy that I got to get to use them again over at Wave Payroll. Last time I used them you could only track a page view, but now they have added track_links and track_forms which fire the event asynchronously after the link is clicked or the form is submitted.

I started off by using the event_tracker gem which only handles the page view events, but it does make it easier to fire events in the view based on controller actions so it is well worth using. I talked to the author about adding support for track_links and track_forms, but after a good discussion he convinced me that the gem was not the right place for this and that I should pursue something more elegant such as tagging the links.

Ideally, what we wanted to arrive at was something like

1

<a href="blah" class="track-with-mixpanel" data-event="clicked on login">

or with Rails helpers:

1

=link_to login_path, :class => "track-with-mixpanel", :data => { :event => "clicked on login" }

which would magically call

1

mixpanel.track_links("#id_of_element", "clicked on login")

One problem is that not all the elements had IDs and I didn’t want the added friction of having to add that in.

What I came up with was:

1
2
3
4
5
6
7
8
9
10
11
12

$(".track-with-mixpanel").each( function(i) {
  var obj = $(this);
  // Make up an ID if there is none
  if (!obj.attr('id')) {
    obj.attr('id', "mixpanel" + Math.floor(Math.random() * 4294967296));
  }
  if (obj.is("form")) {
    mixpanel.track_forms("#" + obj.attr("id"), obj.data('event'), obj.data('properties'));
  } else {
    mixpanel.track_links("#" + obj.attr("id"), obj.data('event'), obj.data('properties'));
  }
});

So this simply finds all the links with the track-with-mixpanel class, assigns them a random ID if they don’t have one, and then calls the appropriate mixpanel function on them.

A couple of notes, though…

The first is that the data-properties doesn’t quite work right. Pulling the data into a hash requires some finesse that I haven’t been able to figure out.

The second is that track_links and track_forms are horribly broken and will mess up any onClick handlers you have. So if you have links that you use :method => :post for, some third party javascript like validations or Stripe, you’re better off with the old track method because Mixpanel won’t play nicely with it. But for regular links and forms, this is great.

One of my favourite applications on my Mac is Quicksilver. It’s an application launcher that can be extended to do almost anything. I’ve been able to set up hot keys to rate songs in iTunes, and also set shortcuts to URLs.

These screenshots explain how to make a “jira” shortcut that will take you to a particular issue in JIRA. The instructions will work equally well for anything where the URL is well known. For example, ticket PR-139 is at https://mycompany.atlassian.net/browse/pr-139. To go there, you’ll hit your Quicksilver hot key (e.g. Control-space), then type “jira”, enter, then the ticket number, and enter.

First, make sure the Web Search Module plugin is loaded by checking the box in the plugins menu.

Then, add the shortcut to your catalog by adding it as a custom source under Web Search List.

Looking more closely at what you add:

The name is whatever you want to type when first opening Quicksilver. The url is the URL you want to go to, with the dynamic part replaced by ***.

You may have to rescan the catalog from the main Quicksilver menu.

I just wanted to mention that I’ve started a podcast over at LinuxAdminShow.com. It’s me talking to a guest every week about issues that matter to the Linux administrator. The first episode was about systems management and admins-who-code, the second was about SSL.

Always looking for feedback and guest/topic suggestions. Have a listen and let me know what you think!

Deploying Rails
Automate, Deploy, Scale, Maintain, and Sleep at Night
Anthony Burns and Tom Copeland

“Deploying Rails” is more than just about deploying a Rails application, it’s about that and everything that goes into managing servers, from provisioning to monitoring. This book explains how to do these tasks with the help of some popular Open Source tools and a focus on automation.

The flow of the book is fairly logical. Start by building virtual machines with Vagrant. Learn how to automate configuration with Puppet. Nail down deployment and remote tasks with Capistrano. Monitor the server and application with Nagios and Ganglia. Delve into some side topics like systems administration. Even though there is ample free documentation on all of these topics, this book sets itself apart in two ways.

First is that the tools from previous chapters are used to augment later chapters. You’ll learn how to use Vagrant to set up a virtual machine in the first chapter, and from them on when you need a server you’ll configure a Vagrant box. You’ll learn how to automate configuration management with Puppet in the second chapter, and all successive chapters build on that. You don’t simply install Nagios, you write Puppet scripts that install Nagios for you. By the end of the book you have a collection of tools that you can start using in your own real world environment.

Secondly, you’re doing everything on an environment you can build yourself without needing to know how to install Linux, owning spare servers, or knowing how to manage servers. You just install Vagrant and follow the book. The book is heavy on code samples and screen captures – it is the exception to open up to a random page and not see some code or example. You can have a replicated MySQL setup and work on your database recovery practices, destroy it, and know you can rebuild it with a few keystrokes.

Puppet is a large part of this book. Almost every task is done in a Puppet manifest, from installing the web server to setting up monitoring. The authors walk you through creating couple of simple manifests and then refactoring the code to be more reusable. The basics of Puppet are covered such as installing packages, starting services, and copying files. Later on Puppet is used to interact with the existing system by managing cron jobs and using templates to edit existing configuration files.

The popular deployment suite, Capistrano, is the topic of two chapters. The first looks at a simple deployment, then goes on to examine roles and adding hooks that automate tasks at points during the deployment. The advanced chapter delves into remote command invocation and parsing, multistage deployments (such as a separate staging and production deployment) and further automation of the deployment. People who have used Capistrano before will not be surprised by much in the basic chapter, but are almost certain to find something helpful in the advanced chapter. It opened my eyes to what Capistrano can do outside of the deployment – it can automate maintenance and support tasks, too.

The last three chapters discuss various topics, from managing multiple Ruby interpreters with RVM to backing up your database and how to manage a master-slave setup. Some of these topics can be books in themselves, though Deploying Rails does a good job at getting you started. Even though the examples throughout the book use Apache and Phusion Passenger, the appendixes have a chapter on using Nginx and Unicorn.

Despite all the remarkable content, I did feel there were some areas that could have been covered. Given the extensive use of Vagrant throughout the book I found it surprising there was no discussion about using it for its intended purpose of managing developer’s environments. There’s a brief mention that Vagrant can run the Puppet scripts and you can save the step of running it manually, but other than that I found little that would tell the reader that they could reuse the work they had been doing so that all the developers would have a production like environment in which to work. Similarily, since the environment is well defined the authors were able to make several assumptions in their coniguration that would not necessarily work in a typical production environment. Some of these are simple, such as IP addresses and SSH keys being hard coded, but some are more involved, such as how to distribute the Puppet manifests when that’s not taken care of by Vagrant. Books, like software, have to draw the line somewhere though.

As a systems administrator turned developer I was encouraged to see this book being released. It shows the ideal marriage of the systems administration mindset, with its relentless focus on automation and monitoring, with the tools available to the modern programmer. In some circles this practice is called DevOps, but even in shops that keep these two separate, this book will benefit both teams.

At my awesome job we use HipChat for team collaboration. I also use the Pomodoro app to try and manage my time. One problem is that I often get interrupted while working.

Long story short, the Pomodoro app lets you run AppleScript when various events happen, so I wrote some stuff to change my HipChat status to DND when I’m in the middle of a work cycle. Here’s the code:

tell application "System Events" to tell UI element "HipChat" of list 1 of process "Dock"
    perform action "AXShowMenu"
    delay 0.5
    click menu item "Status" of menu 1
    click menu item "DND" of menu 1 of menu item "Status" of menu 1
 end tell

All that remains is to insert that into the Pomodoro app through Preferences -> Scripts:

Just note that you have to change “DND” to “Available” for some of the events.

This was my first foray into AppleScript, so it’s possible I’m sending my banking details off to Nigeria, but it seems to work so far.

Edit you need to enable access for assistive devices from System Preferences -> Universal Access:

Update - if you sign up through this link, New Relic will send you $25 gift card for trying it out.

Like many Rails people, I use New Relic to monitor my Rails app. At Wave Accounting we even pay for it. It’s well worth the money, as you get a lot of extra visibility into your app.

At the standard level, New Relic is pretty good, but sometimes it seems like I’m missing out on something. RPM will show me that a controller is running slowly but most of the time is spent in the controller itself, not really saying what’s happening. I’ve recently discovered a few tweaks that have made a huge difference.

Turn on garbage collection

It’s pretty embarrassing, but this isn’t on by default. It’s so simple to figure out how much of your time is spent in GC, the only caveat is that you have to be running REE or 1.9.x. This doc explains how, but all you have to do is turn on the stats and New Relic does the rest.

1
2
3
4

# For REE
GC.enable_stats
# For 1.9.2
# GC::Profiler.enable

Put that in an initializer, and you get GC activity in almost all your graphs:

Local development mode

If you go to http://localhost:3000/newrelic you will get some in depth information about what’s going on in your app when in dev mode. If you’re using pow then add the following to ~/.powconfig:

export NEWRELIC_DISPATCHER=pow

export POW_WORKERS=1

There’s a wealth of information here.

Trace specific sections of your code

Your controller does a bunch of stuff but New Relic reports it as one big block. block tracing to the rescue!

1
2
3
4
5
6
7
8
9
10
11
12

respond_to do |format|
  format.html do
    self.class.trace_execution_scoped(['Custom/MyController#show/render_html']) do
      do_it_in_html
    end
  end
  format.pdf do
    self.class.trace_execution_scoped(['Custom/MyController#show/render_pdf']) do
      do_it_in_pdf
    end
  end
end

Then, these blocks will be counted separately.

Trace methods in your code, or other people’s code

Want to do the same thing as before on someone else’s code, or at a method level? Add a method tracer in your initializer:

1
2
3
4
5
6
7

require 'new_relic/agent/method_tracer'
CalculationEngine.class_eval do
    include NewRelic::Agent::MethodTracer
    add_method_tracer :calculate_taxes
    add_method_tracer :calculate_totals
    add_method_tracer :send_invoice
end

Poof, all those are now traced and broken out separately:

Other things

You can also trace custom metrics, such as user signups or report views. I’m still working on that, along with monitoring background jobs.

I’ve wanted to play with the Twilio client for a while. They have this great quick start but it’s written in PHP. Now I don’t mind PHP, but I prefer Ruby. If I’m going to write anything using the client, it’s going to be in Ruby, so I don’t see the point in learning it on PHP.

So, here is the meat of the quickstart done up as a Rails 3.1 application.

First, generate the application.

1
2
3
4
5
6
7
8
9

$ rails new twilio
      create
      create  README
      create  Rakefile
      create  config.ru
      create  .gitignore
      create  Gemfile
      create  app
      ...

This creates a new Rails 3.1 app in the current directory called twilio. Change to this directory, and add a line to your Gemfile:

1

gem 'twilio-ruby'

Run bundle install to add the official Twilio gem to your bundle.

Next, head on over to your Twilio account and get your SID and auth token. Those can go in an initializer, config/initializers/twilio.rb:

1
2

TwilioAccountSID="AC........"
TwilioAuthToken="......."

Those are the magic tokens that let you authenticate yourself to the Twilio API, and more importantly for them, let them bill you.

Next, head on over to app/helpers/application_helper.rb:

1
2
3
4
5

module ApplicationHelper
  def twilio_javascript_include_tag
    javascript_include_tag "http://static.twilio.com/libs/twiliojs/1.0/twilio.min.js"
  end
end

Then in app/views/layouts/application.html.erb add that helper in the head:

1
2
3
4
5

<pre><code>
  <%= stylesheet_link_tag    "application" %>
  <%= javascript_include_tag "application" %>
  <%= twilio_javascript_include_tag  %>
  <%= csrf_meta_tags %><br />

Yea, you could have put the code right in the layout, but I like sparse layout files.

Next up, create a controller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

$ rails generate controller client index
      create  app/controllers/client_controller.rb
       route  get "client/index"
      invoke  erb
      create    app/views/client
      create    app/views/client/index.html.erb
      invoke  test_unit
      create    test/functional/client_controller_test.rb
      invoke  helper
      create    app/helpers/client_helper.rb
      invoke    test_unit
      create      test/unit/helpers/client_helper_test.rb
      invoke  assets
      invoke    coffee
      create      app/assets/javascripts/client.js.coffee
      invoke    scss
      create      app/assets/stylesheets/client.css.scss

Then add root :to => ‘client#index’ to config/routes.rb so that your new view is displayed in the root url.

Run rails server or whatever you do to start your dev instance and browse to it. You should get the usual “find me in app/views/client/index.html.erb” message. Check the headers to make sure the library is being installed. The rest of the examples then deal with app/views/client/index.html.erb and app/helpers/client_helper.rb.

For the first example you want:

1
2
3
4
5
6
7

module ClientHelper
  def twilio_token
    capability = Twilio::Util::Capability.new TwilioAccountSID, TwilioAuthToken
    capability.allow_client_outgoing "APabe7650f654fc34655fc81ae71caa3ff"
    capability.generate
  end
end

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<%= javascript_tag do %>
function call() {
  Twilio.Device.connect();
}
function hangup() {
  Twilio.Device.disconnectAll();
}
$(function() {
  Twilio.Device.setup("<%= twilio_token %>");
Twilio.Device.ready(function (device) {
    $("#log").text("Ready");
  });
Twilio.Device.error(function (error) {
    $("#log").text("Error: " + error.message);
  });
Twilio.Device.connect(function (conn) {
    $("#log").text("Successfully established call");
  });
});
<% end %>
<button class="call" onclick="call();">
  Call
</button>

For the second example, you just change the view.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

<%= javascript_tag do %>
function call() {
  Twilio.Device.connect();
}
function hangup() {
  Twilio.Device.disconnectAll();
}
$(function() {
  Twilio.Device.setup("<%= twilio_token %>");
Twilio.Device.ready(function (device) {
    $("#log").text("Ready");
  });
Twilio.Device.error(function (error) {
    $("#log").text("Error: " + error.message);
  });
Twilio.Device.disconnect(function (conn) {
    $("#log").text("Call ended");
  });
Twilio.Device.connect(function (conn) {
    $("#log").text("Successfully established call");
  });
});
<% end %>
<button class="call" onclick="call();">
  Call
</button>
<button class="hangup" onclick="hangup();">
  Hangup
</button>
<div id="log">Loading pigeons...</div>

For the third example we’ll have to change the helper and the view accordingly:

app/helpers/client_helper.rb (note I’m using my own sandbox id. You get your own inside the Twilio account page!)

1
2
3
4
5
6
7
8

module ClientHelper
  def twilio_token
    capability = Twilio::Util::Capability.new TwilioAccountSID, TwilioAuthToken
    capability.allow_client_outgoing "AP..."
    capability.allow_client_incoming 'jenny'
    capability.generate
  end
end

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

<%= javascript_tag do %>
function call() {
  Twilio.Device.connect();
}
function hangup() {
  Twilio.Device.disconnectAll();
}
$(function() {
  Twilio.Device.setup("<%= twilio_token %>");
Twilio.Device.ready(function (device) {
    $("#log").text("Ready");
  });
Twilio.Device.error(function (error) {
    $("#log").text("Error: " + error.message);
  });
Twilio.Device.disconnect(function (conn) {
    $("#log").text("Call ended");
  });
Twilio.Device.connect(function (conn) {
    $("#log").text("Successfully established call");
  });
Twilio.Device.incoming(function (conn) {
    $("#log").text("Incoming connection from " + conn.parameters.From);
    // accept the incoming connection and start two-way audio
    conn.accept();
  });
});
<% end %>
<button class="call" onclick="call();">
  Call
</button>
<button class="hangup" onclick="hangup();">
  Hangup
</button>
<div id="log">Loading pigeons...</div>

Now, hook up a new action in the client controller to redirect the call from Twilio to the app inside app/controllers/client_controller.rb

1
2
3
4
5
6
7
8

def incoming
  response = Twilio::TwiML::Response.new do |r|
    r.Dial  do |d|
      d.Client 'jenny'
    end
  end
  render :text => response.text
end

Don’t forget to add post “client/incoming” to config/routes.rb. Then point your sandbox URL to your dev box, such as http://yourhome.com:3000/client/incoming.xml.

As a bonus, here’s a rake task to log in to a remote host and set up an ssh tunnel on remote port 3000 to local port 3000:

1
2
3
4
5
6
7
8
9
10
11
12
13

namespace :tunnel do
  desc "Start a ssh tunnel"
  task :start => :environment do
    public_host_username = "sean"
    public_host = "home.ertw.com"
    public_port = 3000
    local_port = 3000
    puts "Starting tunnel #{public_host}:#{public_port} \
          to 127.0.0.1:#{local_port}"
    exec "ssh -nNT -g -R *:#{public_port}:127.0.0.1:#{local_port} \
                           #{public_host_username}@#{public_host}"
  end
end

There are two more examples in the quickstart, but as they are more of the same, I’ll leave them for another post. I’d also like to try rewriting the Javascript in Coffeescript.

Update - Code is at https://github.com/swalberg/twilio-client-ruby