The Blog

Posts from February 2010

Feb 23

Experimenting with Redis

By David Czarnecki

Yesterday I started looking at ways to do inter-application communication. In a number of projects we’ve done here at Agora Games, we’ve used queues to make that happen. Redis has been on my radar for awhile now, but yesterday I drove my Chevy to the levee and guess what? The levee is NOT dry people. I mean, who drinks rye anyway these days? Old people.

Right, Redis.

“Redis is an advanced key-value store. It is similar to memcached but the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists, sets, and ordered sets. All this data types can be manipulated with atomic operations to push/pop elements, add/remove elements, perform server side union, intersection, difference between sets, and so forth. Redis supports different kind of sorting abilities.”

I was particularly interested in the support in Redis for lists, which would allow me to have one application push stuff into the datastore and allow for another application to pull stuff from the datastore. A datastore that could operate as a queue? I guess this is as close as we’re going to get to flying cars in 2010. Redis is also supposed to be wicked fast and there are a number of libraries available in your programming language of choice with which to communicate with Redis.

Redis setup was trivial and “just worked”.

After starting the Redis server, I could just prototype in script/console.

Here we go:

>> redis_queue =
=> #<Redis:0x222df9c @thread_safe=nil, @logger=nil, @password=nil, @timeout=5, @db=0, @sock=nil, @host="", @port=6379>
>> redis_queue.push_tail 'strings', 'string 1'
=> "OK"
>> redis_queue.push_tail 'strings', 'string 2'
>> "OK"
>> redis_queue.list_length('strings')
=> 2
>> redis_queue.list_range('strings', 0, -1)
=> ["string 1", "string 2"]
>> some_string = redis_queue.pop_head('strings')
=> "string 1"
>> r.list_range('strings', 0, -1)
=> ["string 2"]
>> r.list_length('strings')
=> 1
>> quit

There are quite a few commands that make Redis even more awesome, but for now I’m sold. (If using an entire sentence as a link is wrong then I don’t want to be right)

Also, I know of Resque, “a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.” If I ever feel like driving the Mercedes Benz of flying cars, I’ll use it.

Feb 23

Watching Trees

By Jason LaPorte

As a SysAdmin, my job more-or-less exists by knowing miscellaneous arcana that most software engineers aren’t aware of.

When a particular co-worker here at Agora has a problem with his Linux machine, I provide advice and show him how to fix it. Some months after he had joined Agora, I discovered that after each troubleshooting session, he copy-and-pastes the entire text terminal log into a text file that he keeps on his desktop. He has dozens of transcripts at this point; I bet if I were to look through it, it would read something like the Tao te Ching or Bhagavad Gita, only concerning UNIX instead of right living.

(Of course, there’s not much of a difference between UNIX and right living, but that’s a topic for another day.)

In the spirit of allowing you to build your own little collection, here is a simple trick that came in handy to me yesterday.

Last night, the cron jobs powering one of our managed sites stopped running. I had stopped them, cleaned up, and restarted them, but I wanted to watch the system and see if they had started running again so I could monitor that they were doing their job.

UNIX has a well-known command called “ top,” which shows superlative processes running on your system: which ones are consuming the most CPU time, which ones are consuming the most RAM, which ones have been running the longest, and so on. It’s a very useful tool, but if you’re trying to track down processes over time that don’t necessarily consume very many resources, then it’s not the tool you’re looking for.

However, our Linux distribution (and, indeed, most others) provide a useful graphical tool for mapping the current state of the system: “ pstree.” This command will give you a list of processes (much like the “ ps” command), except that instead of generating a list, it will draw a little tree showing the state of the system. This is perfect for seeing exactly what is going on: which daemons are running, who is on the system and what they’re executing, and so on. However, it doesn’t show you what is happening over time; it merely lists what is happening at this moment.

However, our Linux distribution (and, indeed, most others) provides yet another useful tool: “ watch.” It runs a program every 2 seconds (by default) and displays the result to your screen. This allows you to make a ghetto clock by writing something like “watch date;” in our case, we could make a ghetto graphical “top” by writing the simple “watch pstree.” You can now watch your virtual trees rustle in the wind.

It kept us on top of the cron job, and things were back to normal in no time.

For more information, RTFM. :)

Feb 22

And now for some couchdb

By Jason LaPorte

I had occasion tonight to give a quick demo of CouchDB.  This is so simple its almost not worth blogging about, but hey, good software is supposed to be simple.

Install On your Mac:

 sudo port install couchdb
 sudo launchctl load -w /Library/LaunchDaemons/org.apache.couchdb.plist

On Ubuntu:

 sudo apt-get install couchdb

You’re done; now to play.

Futon Admin Interface This allows you to create a DB, create records, etc.

  1. Open http://localhost:5984/_utils/
  2. Prosper

Using our old friend curl

 curl -X PUT http://localhost:5984/mlg/ #create a db
 curl -X GET http://localhost:5984/mlg/ #get a whole bunch of info about the db
 curl -X POST http://localhost:5984/mlg/ -H "Content-Type:application/json" -d '{"body": "Here is a paragraph"}' #create a record