Over at b5media we have some internal code that ties all the blogs together to make us a network. This was initially implemented as a brilliant little REST API, only requiring minor changes in some database tables that could completely alter how the network was arranged. Additional functionality could be added easily to the API, and all the sites could make use of it.

All was good until we started noticing things were slowing down. Web slaves were CPU bound, and we were consumed with hits to the API. Stepping back and looking at what was going on, we soon realized that REST was an inefficient way of achieving what we needed at the volume we needed.

Enter memcached, a distributed hash table of sorts. In a nutshell, a daemon runs on machines with free memory that responds to network requests using a simple protocol. Client libraries connect and can pull data out by its key, or put data in. By hashing the key in an agreed upon manner the load can be evenly distributed across any number of memcached instances (just read how facebook.com uses over 200 instances of memcached).

A server’s profile can be characterized in terms of its CPU, network, disk (I/O and space), and memory usage. Web servers tend to be CPU, network, and disk I/O heavy, so it’s great to be able to make use of the unused memory. Even at peak loads when the server is running busy, memcached barely registers on the charts. It’s light on CPU and uses epoll(4) for efficiency and performance.

So after seeing memcached for the first time, our development ninjas were able to rewrite the template code that calls the API to cache the data, resulting in much faster page loads, reduced CPU on the web slaves, and a drastic cut in database load.

I’m looking forward to exploring other opportunities for caching in the b5 architecture because of the success we’ve seen so far.

Speaking of opportunities, b5 is looking to hire a full time server administrator because the work needed has grown beyond what I can do as a part timer. You’d get to work with a great team who is passionate about what they do, you’d get to work from home, and you’d get the freedom to take the infrastructure wherever you think it should go. I’ll be working with this person to teach them the environment and all the ins and outs before handing over the reins, so it’s not like you’d be thrown to the dogs. Contact me or Aaron with any questions (I’m on Skype as SeanWalberg though usually on text only, or Google Talk as swalberg)