Sean’s Obsessions

I am doing some work with WordPress, where we have a development server and a production server. The development side is set up as a git repo, and the production side pulls from the dev repo when we want to pull in changes:

git pull origin master

I move between the two environments using the host file, which sometimes means I’m not sure which environment I’m in. I put the following in functions.php to help me out:

function mysite_i_am_in_dev() {
  // Red border if we're in dev mode
  echo '<!-- dev mode --><style type="text/css"> body { border: 2px solid #FF0000; } </style>';
}

if ($_SERVER["SERVER_ADDR"] == "x.x.x.x") { // x.x.x.x is the IP address of your dev server
  add_action('wp_head', 'mysite_i_am_in_dev');
  add_action('admin_head', 'mysite_i_am_in_dev');
}

So now, anyone viewing the development server will have a small red border around the screen.

I recently changed SmallPayroll to use Beanstalkd instead of delayed_job for background processing. delayed_job is an awesome tool and makes asynchronous processing so simple. However I wanted to have multiple queues so that I could have different workers processing different queues, and have some upcoming needs to process the jobs quicker than the 5 second polling time.

After watching railscasts on beanstalkd and stalker I decided to use that. Beanstalkd is a lightweight job processor, and stalker makes it very simple to use from the client end.

I used to have an observer that said

class UserObserver < ActiveRecord::Observer
  def after_create(user)
    UserMailer.send_later(:deliver_welcome_email, user)
    UserMailer.send_later(:deliver_notify_admin, user)
  end
end

This became:

class UserObserver < ActiveRecord::Observer
  def after_create(user)
    Stalker.enqueue("email.new_user", :user_id => user.id)
  end
end

delayed_job was nice in that the job would just run against the model, but now I have to process the job in config/worker.rb:

require 'stalker'
include Stalker
require File.expand_path("../environment", __FILE__)

job 'default' do |args|
  puts "I don't support the default queue"
end

job 'email.new_user' do |args|
  user = User.find(args["user_id"])
  UserMailer.deliver_welcome_email(user)
  UserMailer.deliver_notify_admin(user)
end

One thing about stalker is that it wants you to pass simple objects instead of ActiveRecord objects, so I queue the user_id instead of the user model.

The script above monitors the default tube, which I don't use, because the nagios plugin for beanstalk expects someone to monitor it (after setting it all up, I guess I could have set it up to ignore that tube). I'm also using the munin plugin for beanstalkd to graph the activity in the daemon.

Then, script/worker uses the daemons gem to start the job and restart it if it crashes:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'

pwd  = File.dirname(File.expand_path(__FILE__))
file = pwd + '/../config/worker.rb'

Daemons.run_proc(
  'payroll-generic-worker', # name of daemon
  :dir_mode => :normal,
  :dir => File.join(pwd, '../tmp/pids'),
  :backtrace => true,
  :monitor => true,
  :log_output => true
) do
  exec "stalk #{file}"
end

Finally, some capistrano magic to start the worker with the deploy inside config/deploy.rb

namespace :beanstalk do
  desc "Start beanstalk process"
  task :start, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker start"
  end

  desc "Stop beanstalk process"
  task :stop, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker stop"
  end

  desc "Restart beanstalk process"
  task :restart, :roles => :app do
    run "cd #{current_path}; RAILS_ENV=production script/worker restart"
  end
end

after "deploy:start", "beanstalk:start"
after "deploy:stop", "beanstalk:stop"
after "deploy:restart", "beanstalk:restart"

The only problem I've run into so far is that my HTML email seems to go out without the text/html content-type. Fixing that was a simple matter of putting content_type 'text/html' inside my mailer, which wasn't needed when I was using delayed_job.

I was just fighting with this, and being new to the Rails 3 way of things, the docs didn’t quite make sense.

Step 1 – Install the plugin

 rails plugin install https://github.com/rails/exception_notification.git

Step 2 – Edit config/application.rb:

module MyApp
  class Application < Rails::Application
    # .....
    #  somewhere in this block put the following:

    config.middleware.use "::ExceptionNotifier",
      :email_prefix => "[MyApp Error] ",
      :sender_address => %{"notifier" },
      :exception_recipients => %w{youraddress@example.com}

3. Verify:

$ rake middleware | grep ExceptionNotifier
use ExceptionNotifier

Now you’ll get any application errors emailed to the addresses in the exception_recipients array.

I just upgraded my nginx rpm and my site refused to start. /var/log/nginx/error.log reported the following:

2010/11/19 09:30:08 [emerg] 27885#0: eventfd() failed (38: Function not implemented)
2010/11/19 09:30:08 [alert] 27884#0: worker process 27885 exited with fatal code 2 and can not be respawn

It turns out the new nginx rpm was compiled with –with-file-aio which uses the eventfd() call:

[root@smallpayroll ~]# nginx -V
nginx version: nginx/0.8.53
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)
TLS SNI support disabled
configure arguments: --user=nginx --group=nginx --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-mail --with-file-aio --with-mail_ssl_module --with-ipv6 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables' --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables'

eventfd() wasn’t available until Kernel 2.6.22. I was running 2.6.18 because that’s what my VPS offered when I started. Fortunately it was a quick option in the Linode panel, a reboot, and now I’m running 2.6.32.

трапезни масиI’ve started using Git instead of svn. Like many other people, I tried it out to get easier branching, but found it’s just a pleasure to use in general.

One great feature is that git can work with an SVN repository. You use git to check out some SVN code into a local git repository, you work with the repo as if it were git, then you commit your changes back to svn.

The lowdown on this is:

# Check out the svn repo
# It may take a while because it's copying all the changes, not just a checkout
git svn init http://myrepo/myproject/trunk myproject
cd myproject
# do some work
# commit to git
git commit -a -m "did some work"
# send updates to svn
git svn dcommit

But wait, someone else has done some work and you want to pull the latest changes from svn to your git repo

git svn rebase

Nice eh?

So let’s say you’re working in your git repo, and want to branch

git branch somefeature
git checkout somefeature

You’re working along, and you want to pull changes from svn into master and your branch.

git checkout master
git svn rebase
# master is up to date
# so merge master into the branch
git checkout somefeature
git merge master

This is so easy to use! Previously, with SVN, I’d branch, work on the feature, and then merge back to trunk. Given how painful merges were, there was no hope that I’d be merging trunk changes into my feature branch.

I’m sure my tune will change the first time I blow up my repository though

I have been slowly learning how to use JMeter to load test the Small Payroll application. One of the problems has been getting around the CSRF protection that Rails puts in with the authenticity_token parameter.

Each form has a hidden form element:

<div style="margin:0;padding:0;display:inline">
<input name="authenticity_token" type="hidden"
value="16iUP1J2tdSKyvHKgYR/I/og6K7NgPPmTHCZ+idQP4k=" />
</div>

The token is also encrypted in the session so that the response to the form has to match. This value changes every time the form is loaded and can’t be set to a known value, otherwise it would be easy to defeat the CSRF protection. This means that JMeter has to figure out the token and put it into the POST.

Googling around, I found some people that said they did it, and a lot of people who couldn’t get it to work, but no solid walkthroughs.

Here’s my solution:


There are three elements. The first is the HTTP request sampler that gets the login page. Under that is a regular expression extractor post processor that gets the authenticity token. The extractor uses a simple regexp to pull out the value parameter and saves it to the LOGIN_AUTH_TOKEN variable. The login is then done by making reference to the variable – ${LOGIN_AUTH_TOKEN}. Make sure you have the Encode? button checked, as the authenticity_token is not always base-64 friendly!

The final step, not pictured, is that you have an HTTP Cookie Manager in your thread group to take care of cookies. You probably already have one, though.

Sean

I often use the find command as such:

grep foo `find . -name \*.php`

which looks for foo in all the PHP files. If the list of files gets too long for the shell, then xargs is the better option:

find . -name \*.php | xargs grep foo /dev/null

This breaks up the command into manageable chunks. The /dev/null is there in case xargs only passes one file name to the command, this ensures there will be two file names to trigger the printing of the matching file (there’s probably an option, but I like this way better)

The problem with find -print |xargs is that a file or directory with spaces causes problems. For instance “xxx yyy/blah.php” will run

grep foo /dev/null xxx yyy/blah.php

neither of which exist.

My first hack at it was to revert to programmer mode and do a loop:

find . -name \*.php | while read A; do grep foo "$A" /dev/null; done

The best solution I found so far is to use -printf to force find to quote the file when it spits out the file name and get back into the xargs/stream mode of thinking:

find . -name \*.php -printf '"%h/%f"\n' | xargs grep foo /dev/null

Maybe there’s an easier way? (besides shooting people that use spaces in directory or file names)

A comment in a Slashdot article lead me to iploc.org, specifically Country names zone which lets you get the country for a given IP address over DNS.

I wrapped it in a bash function… throw this at the end of your .bashrc:

geo() { dig +short TXT `echo $1 | \
 awk -F. '{print $4"."$3"."$2"."$1".cc.iploc.org"}'`;}

and you can get geo information from the command line:

$ geo 204.187.154.1
"CA"

In 2008 and 2009, my wife and I hired a nanny a few days a week to take care of our three children. Nothing exciting there. Part of hiring a nanny is that you have to deduct and remit taxes, CPP, and EI. All easy stuff; I’ve taken a payroll course before and the calculators were available online.

I wrote some scripts to help me out, which I eventually improved to the point where I had a fully functional payroll system. I put it online at SmallPayroll.ca thinking I’d be able to charge a few bucks for people to use it. Even though our nanny moved on in the summer and we found a day care, I’ve kept on improving the site. I may be biased, but I think it’s a pretty good system for helping people in a similar situation do their payroll.

On Friday I got a piece of mail from the Canada Revenue Agency, saying there was a problem with the T4 I submitted for my nanny in 2008 (2008! It’s almost 2010!). We only started in November so this was before I had written the first script and everything was manual. Of course I couldn’t find any of my paperwork from back then, and with a recent basement flood it could be anywhere.

Fortunately the problem ended up being on CRA’s end, but this incident highlighted how helpful a SaaS such as my own small payroll systemспални комплекти would have been. All the data is there online and I would have known the calculations were correct because software was doing them.

I’m noticing that more and more of the items that pop up in my feed reader are of the “Top N” variety. Some are decent, where the author breaks the topic down into subgroups and then shows one or two tools for each purpose. However some of them are just silly, such as the “top 100 photoshop tutorials”, which is really “the last 50 Photoshop tutorials I bookmarked, and 50 that I just Googled”.

So when I read Top Five Best Database Management Tools, I wondered what the heck the value was. Some of the tools were for Microsoft SQL server, some were for MySQL, and there was no criteria for the list other than “reader favourites”.

So, in an effort to answer the “how do I manage my database” question, here’s the top 1 database management tool, and how to use it.

But first, a disclaimer. It’s only the top tool if:

• You use mySQL
• Your web server can serve PHP
• You’re me

The tool is phpMyAdmin. That should come as no surprise, because it’s on everyone else’s top N list. There may be better, but it’s easy to install and probably handles anything you can throw at it. If you’re not using MySQL, or you have something better, then go ahead and use it!

So go to the homepage and download the software. If you’re in Windows grab the .zip version, if you’re in Linux, grab the .gz version.

First, uncompress the software to somewhere outside your document root. Doing it this way makes management a bit easier in the future, but slightly complicates the installation. On my machine, I’d go

cd /var/www
tar -xzf tar -xzf /tmp/phpMyAdmin-3.2.3-all-languages.tar.gz
ln -s phpMyAdmin-3.2.3-all-languages phpMyAdmin

Line 1 gets you to the parent directory of the web stuff (at least in RedHat/CentOS). Line 2 uncompresses the software. Line 3 creates a symbolic link to the current version.

Next we’ll create an Apache configuration file to handle access to the software. You can do this in the main httpd.conf file, but most distributions have a directory where extra configurations are loaded.

/etc/httpd/conf.d/phpmyadmin.conf:

Alias /phpMyAdmin/ /var/www/phpMyAdmin/

<Location /phpMyAdmin/>
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
    Allow from 192.168.1.0/24
    Allow from 1.2.3.4
</Location>

Line 1 maps anything under /phpMyAdmin/ (note the trailing /) to /var/www/phpMyAdmin/. The latter is a symlink to the current version, so if you want to upgrade you can have both versions uncompressed and use the symlink to change the active version. You can also choose a more friendly name, if you’d like.

The rest of the code restricts access to the localhost (127.0.0.1), the local network, and a certain outside address. If you want everyone to access it, delete the Location stanza and everything inside it (leaving only the Alias)

If you go to the URL, you’ll be able to log in using your username and password.

An interesting thing about phpMyAdmin is that it can manage several servers from one web page. In this case, the login screen will allow you to choose the server. This is great if you have multiple mySQL servers, or if your web server and SQL server are on separate devices.

The easiest way to do this is to use the web based configuration:

mkdir /var/www/phpMyAdmin/config
chmod 777 /var/www/phpMyAdmin/config

Then hop over to http://127.0.0.1/phpMyAdmin/setup/ (or whatever your server is) and follow the instructions to create several servers. Unless you have a specific reason to, you can choose the defaults (except for the server name!)

Once you’re done, protect the directory from writing and copy the generated file to the root:

chmod 755 /var/www/phpMyAdmin/config
cp /var/www/phpMyAdmin/config/config.inc.php /var/www/phpMyAdmin

If you’d rather not use the web interface, copy the config.sample.inc.php to config.inc.php and go from there.