Sean's Obsessions


  • I'm a happy Linode customer. This is a Linode 540 VPS. Linode periodically increases RAM and disk while keeping costs steady, which makes them the natural choice.
  • Archives

25 Oct

Using Google Universal Analytics with NationBuilder

We spent a lot of time trying to understand our visitors at Bowman for Winnipeg and part of that was using Google Analytics. The site was built with NationBuilder but they only support the async version of Analytics and it’s difficult to customize. In particular, we used the demographic and remarketing extensions and there was no easy way to alter the generated javascript to get it to work.

Normally you’d just turn off your platform’s analytics plugins and do it yourself, but NationBuilder has a great feature that fires virtual page views when people fill out forms, and we wanted to use that for goal tracking.

The solution was to turn off NationBuilder’s analytics and do it ourselves but write some hooks to translate any async calls into universal calls. Even with analytics turned off in our NationBuilder account, they’d fire the conversion events so this worked out well.

In the beginning of our template:


<script type="text/javascript">
  var _gaq = _gaq || []; // leave for legacy compatibility
   var engagement = {% if request.logged_in? %}"member"{% else %}"public"{% endif %};

  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-XXXXXXXX-1', 'www.bowmanforwinnipeg.ca');
  ga('require', 'displayfeatures');
  ga('require', 'linkid', 'linkid.js');
  ga('send', 'pageview', { "dimension1": engagement});
}
</script>

It’s pretty vanilla Google Analytics code with a couple of features — display features for layering on demographic features and retargeting integration and the enhanced link attribution for better tracking of clicks within the site. We also added a custom dimension so we could separate out people who took the time to create an account in our nation vs those that were public.

Then, at the bottom:


 $(function() {
    // Send anything NB tried to inject
    for(i in _gaq) { var c = _gaq[i]; if(c[0] == "_trackPageview"){ ga('send', 'pageview', c[1]) } }
  });

That’s a simple loop that iterates through the _gaq array (that the async version of the tracker uses as an event queue) and pushes out any page views using the universal API. We didn’t have to worry about the initial page view because turning off NationBuilder’s analytics feature removes all of that.

21 Nov

Review: Agile Web Development with Rails 4

сондажиI had the good fortune to receive Agile Web Development with Rails 4 from the publisher to review.

The book is really two books in one. The first is a walkthrough of building a simple Rails 4 application for an online bookstore. The second is a component by component look at Rails 4. If you want to be generous there’s a quick introduction to Ruby and Rails concepts at the beginning of the book.

The online bookstore walkthrough is interesting especially if you are new to Rails and the ideas behind Agile development. You take the role of a developer who is building an online bookstore for a client. You start off with a general idea of what needs to be done, but you build it incrementally, showing it to your customer at each step. Based on feedback you decide what to tackle next, such as showing pictures, getting details, or adding a shopping cart. Along the way there are some discussions of deployment, internationalization, authentication, and sending email.

Through the examples you learn the basics of creating Rails models, views, and controllers. Though the examples lean heavily on scaffolding to automatically generate the CRUD actions, you do extend it somewhat to handle XML and JSON responses. You also do some AJAX and automated testing. The book does stick pretty closely to the default Rails toolset including the test framework, though at the very end of the book there are some quick examples on using HAML for views.

At the end of each chapter are some exercises. Unlike many other books I found them to be of appropriate difficulty, with answers available online.

The second half of the book is a detailed look at the components of Rails. This is the more technical part of the book as it gets into specifics of how a request is processed and how a response is generated. There’s no bookstore application anymore, it’s just discussion of what’s available, code samples, and diagrams.

Along the way there are some interesting sidebars that explain some of the Rails philosophies or some real world scenarios. For example, one of the sidebars talks about when you want to use a database lookup that raises an exception on failure versus one that returns a nil or an empty set.

I didn’t read any of the previous editions so I can’t say with authority how much has changed. The book is up to date on the many changes that came in Rails 4 so it is current in that respect. However there are times when you read it and some older terminology, like fragment or page caching, creep in. This is more a matter of editing than it is about it being out of date, as the associated examples are correct. The index is fairly weak — many of the terms I tried to look up, including those featured on the back cover, were not found.

If you’re an experienced Rails developer this book will not help you much. But if you’re looking to get into Ruby on Rails, either from another language or even with a weak programming background, this book is an excellent way to get started. At a little over 400 pages it’ll take you a few weekends to get through depending on your skill level.

05 Nov

Using an IP phone with Twilio

Twilio has supported SIP termination for a while but recently announced SIP origination. This means that previously you could receive calls with SIP but now you can also make calls from a hard phone using SIP instead of using the browser client or going through the PSTN.

It was this second announcement that got my interest. I have an IP phone that I use in my office, currently it’s through Les.net but I like the pricing and interface of Twilio and would rather use them.

For some reason everything I read about SIP and Twilio uses a separate SIP proxy even if they have a compliant SIP device. Even their own blog takes a working SIP ATA and puts it behind Asterisk. I knew I could do better.

What you’ll need

  • An IP phone. I used a Cisco 7960 converted to SIP
  • A publicly available web server running PHP (feel free to use another language, we have to do some basic processing of the request so static won’t work)
  • A Twilio account

When thinking about VoIP, always think in two directions. Inbound and outbound. Sending and receiving. Talking and listening.

Receiving calls

Get a number and point the Voice Request URL to your web server. Please don’t use mine.

Phone Number | Dashboard | Twilio 2013-10-21 08-04-09

Your outbound.php script will send some TwiML to dial your phone:


<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Dial>
    <Sip>
        sip:line_number@address_of_phone
    </Sip>
</Dial>
</Response>

Note: this part was a lot of trouble. After some packet tracing and some brilliant detective work by Brian from Twilio support, it turns out that the address of the phone in the SIP invite had to be an IP address, not a hostname. With a hostname the phone received the INVITE but always returned 404.

Your phone will need to be on the Internet, either with a public address or with UDP port 5060 port forwarded to it. The “line_number” has to match the name of the line on your phone. In my case, I named my line after my phone number:


proxy2_address: "ertw.sip.twilio.com"
line2_name: "204xxxxxxx"
line2_shortname: "204xxxxxxx"
line2_displayname: "204xxxxxxx"
line2_authname: "204xxxxxxx"

One thing to note is that you don’t register your phone to Twilio. I left the proxy address there so that the requests will keep the NAT translation alive. After detailed packet tracing it looks like the Twilio SIP messages originate from different IP addresses so this might not be helping as much as I thought.

At this point you should be able to dial your Twilio number from a regular phone. Twilio will request inbound.php and then do a SIP INVITE to the phone. The phone will accept it and then you have voice!

Making calls

The first step is to set up a SIP domain in Twilio:

Twilio User - Account Sip Domains 2013-10-21 08-15-14

Call it whatever you want, but you’ll need to set the Voice URL.

Twilio User - Account Sip Domains 2013-10-21 08-15-57

The script you point it at has to parse the data coming in from Twilio to find the phone number and then issue a Dial instruction to get Twilio to dial the phone and connect the two ends.


<?php

  $called = preg_replace('/sip:1?(.*?)@.*/', '{$1}', $_POST['Called']);

  header("content-type: text/xml");
  echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n";
  ob_start();
  var_dump($called);
  var_dump($_POST);
  $contents = ob_get_contents();
  ob_end_clean();
  error_log($contents, 3, "/tmp/twilio.txt");

?>
<Response>
  <Dial timeout="10" record="false" callerId="YOURTWILIONUMBER"><?= $called ?></Dial>
</Response>

All we’re doing here is extracting the phone number from the Called header that Twilio sends us, stripping any leading 1’s, and then sending a TwiML response to dial that number. The ob_start through to the error_log is just logging the parameters if you’re interested.

Don’t forget to change the caller ID to your phone number, otherwise you get errors in the Twilio console.

So now when you place a call on your phone, Twilio will send the digits to the application which will return a Dial verb and the proper 10 digit number. Twilio links the two calls.

Conclusions

It took a bit of playing around to get this going but now I’ve shown that you don’t need an Asterisk server to integrate a SIP phone with Twilio. If you are setting up a phone bank or something with hard phones you can just configure them to hit Twilio, and for Twilio to hit them.

Of course, if you are expecting significant inbound traffic the benefit of a SIP proxy is that it can direct calls to multiple numbers without needing Twilio to be able to reach the phone directly. I’m hoping that Twilio can improve on that in the future!

24 Jun

Find method definitions in vim with ctags

Ever asked yourself one of these questions?

  • Where is the foo method defined in my code?
  • Where is the foo method defined in the Rails source?

Then you spend a couple of minutes either grepping your source tree or looking on GitHub and then going back to your editor.

This weekend I went through VIM for Rails developers. There’s a lot that’s out of date, but there’s also some timeless stuff in there too. One thing I saw in there was the use of ctags which is a way of indexing code to help you find out where methods are defined.

Install the ctags package with brew/yum/apt/whatever. Then generate the tags with

ctags -R –exclude=.git –exclude=log *

You may want to add tags to your ~/.gitignore because you don’t want to check this file in.

Also add set tags=./tags; to your .vimrc which tells vim to look for the tags in the current directory. If you have it in a parent directory, use set tags=./tags;/ which tells vim to work backward until it’s found.

Then, put your cursor on a method and type control-] and you’ll jump to the definition. control-T or control-O will take you back to your code. control-W control-] opens it up in a horizontal split. Stack Overflow has some mappings you can use to reduce the number of keystrokes or use vertical splits.

If you use bundler and have it put the gems in your vendor directory, ctags will index those too. So you can look up Rails (or any gem) methods.

13 May

Is it hacked?

After coming across a few sites that were serving pharma spam to Google Bot but not regular browsers, I thought it would be fun to give Sinatra a try and come up with a quick web app that checks for cloaked links. That lead to a few more checks, then some improvements, and Is it hacked? was launched. I’ve got some ideas for where I want to go with the project, but in the meantime it’s catching stuff that other sites in the space are missing.

There’s also a bookmarklet at the bottom of the page. You can drag it to your button bar and check the site you’re on for any infection.

13 Mar

Understanding Capistrano hooks and deploy vs deploy:migrations

Capistrano is a deployment framework for Rails, though it can be extended to do pretty much anything. The system comes with a bunch of built in tasks, and each task can be made up of other tasks and code. Plugins can add their own tasks and hook into default tasks, and the user can do the same through the configuration files.

Like any popular open source project capistrano has gone through changes. Documentation on the Internet is plentiful but often refers to old ways of doing things, so copy/pasting recipes can often result in stuff that doesn’t work quite right.

One common thing people need to do is to run some command after the deploy or at some time during the deploy. Capistrano has a very easy way of doing this.

after 'deploy:update_code', 'deploy:symlink_shared'

This says “after running deploy:update_code, run deploy:symlink_shared”. The latter is custom task that’s defined elsewhere.

The problem comes in when you look at the way the “deploy” and “deploy:migrations” tasks differ. I’ve seen a common problem where something works when deploying without migrations but not when migrations are used. Usually this is because the hook used is not the right one, either because of bad information or the user figured out where to hook into by looking at the output of a deploy.

If you look at Capistrano’s default deploy.rb you can piece together the tasks that are run in both cases.

deploy:migrations
  update_code
   strategy.deploy!
   finalize_update
  migrate
  create_symlink
  restart
deploy
  update
    update_code
      strategy.deploy!
      finalize_update
    create_symlink
  restart

From this, you can see that the sequence is somewhat different. The “update” task isn’t used in the migrations case. Instead, the components are replicated.

If you want to hook in, use

  • update_code to run before/after the code is put into the release directory, such as if you want to make more symlinks or do something before migrations are potentially run.
  • create_symlink to run before/after the symbolic link pointing to the current release is made. Note that symlink is deprecated. You can run it from the command line, but if you try and hook in to it, you won’t be happy.
  • restart to run before/after the application server is restarted, e.g. to restart background workers or do deployment notifications
24 Jan

BASH history expansion

The Unix shell has a whole bunch of features to make your life easier. One has to do with the history. Some I have managed to ingrain into muscle memory, others I have to remember which often means I do it the long way. I hope these examples help you out.

# Start off with some files
$ touch one two three four
# !$ expands to all the arguments from the last command
$ ls !*
ls one two three four
four	one	three	two
# !foo runs the last command that starts with foo
$ !touch
touch one two three four
# Carat (^) does a search/replace in the previous command
$ ^touch^ls
ls one two three four
four	one	three	two
# !$ is the last item in the previous command, !^ is the first _argument_
$ head !$
head four
$ ls four three two one
four	one	three	two
$ cat !^
cat four
# !:n is the n'th item on the previous command line, 0 indexed
$ ls four three two one
four	one	three	two
$ cat !:3
cat two
$ ls four three two one
four	one	three	two
$ which !:0
which ls
/bin/ls

There are a lot more, run a man bash and look for the section called HISTORY EXPANSION.

05 Jan

Mixpanel track_links and track_forms

I’ve long been a fan of MixPanel and was happy that I got to get to use them again over at Wave Payroll. Last time I used them you could only track a page view, but now they have added track_links and track_forms which fire the event asynchronously after the link is clicked or the form is submitted.

I started off by using the event_tracker gem which only handles the page view events, but it does make it easier to fire events in the view based on controller actions so it is well worth using. I talked to the author about adding support for track_links and track_forms, but after a good discussion he convinced me that the gem was not the right place for this and that I should pursue something more elegant such as tagging the links.

Ideally, what we wanted to arrive at was something like

<a href="blah" class="track-with-mixpanel" data-event="clicked on login">

or with Rails helpers:

=link_to login_path, :class => "track-with-mixpanel", :data => { :event => "clicked on login" }

which would magically call

mixpanel.track_links("#id_of_element", "clicked on login")

One problem is that not all the elements had IDs and I didn’t want the added friction of having to add that in.

What I came up with was:

$(".track-with-mixpanel").each( function(i) {
  var obj = $(this);
  // Make up an ID if there is none
  if (!obj.attr('id')) {
    obj.attr('id', "mixpanel" + Math.floor(Math.random() * 4294967296));
  }
  if (obj.is("form")) {
    mixpanel.track_forms("#" + obj.attr("id"), obj.data('event'), obj.data('properties'));
  } else {
    mixpanel.track_links("#" + obj.attr("id"), obj.data('event'), obj.data('properties'));
  }
});

So this simply finds all the links with the track-with-mixpanel class, assigns them a random ID if they don’t have one, and then calls the appropriate mixpanel function on them.

A couple of notes, though…

The first is that the data-properties doesn’t quite work right. Pulling the data into a hash requires some finesse that I haven’t been able to figure out.

The second is that track_links and track_forms are horribly broken and will mess up any onClick handlers you have. So if you have links that you use :method => :post for, some third party javascript like validations or Stripe, you’re better off with the old track method because Mixpanel won’t play nicely with it. But for regular links and forms, this is great.

16 Nov

JIRA shortcuts with Quicksilver

One of my favourite applications on my Mac is Quicksilver. It’s an application launcher that can be extended to do almost anything. I’ve been able to set up hot keys to rate songs in iTunes, and also set shortcuts to URLs.

These screenshots explain how to make a “jira” shortcut that will take you to a particular issue in JIRA. The instructions will work equally well for anything where the URL is well known. For example, ticket PR-139 is at https://mycompany.atlassian.net/browse/pr-139. To go there, you’ll hit your Quicksilver hot key (e.g. Control-space), then type “jira”, enter, then the ticket number, and enter.

First, make sure the Web Search Module plugin is loaded by checking the box in the plugins menu.

Then, add the shortcut to your catalog by adding it as a custom source under Web Search List.

Looking more closely at what you add:

The name is whatever you want to type when first opening Quicksilver. The url is the URL you want to go to, with the dynamic part replaced by ***.

You may have to rescan the catalog from the main Quicksilver menu.

23 Sep

Podcast – Linux Admin Show

I just wanted to mention that I’ve started a podcast over at LinuxAdminShow.com. It’s me talking to a guest every week about issues that matter to the Linux administrator. The first episode was about systems management and admins-who-code, the second was about SSL.

Always looking for feedback and guest/topic suggestions. Have a listen and let me know what you think!

© 2014 Sean's Obsessions | Entries (RSS) and Comments (RSS)

Powered by Wordpress, design by Web4Sudoku, based on Pinkline byGPS Gazette