The Blog

Posts from February 2009

Feb 23

Python Is Mocking Me: A Ruby Programmer's Adventures With Python/Mox

By Tim Jones

Overview

Much of Agora’s work over the last few years has been done in Ruby. Recently though, we’re migrating major components to Python in order to enjoy the benefits of cross platform compatibility with Windows. The purpose of this post is to discuss testing in Python; specifically mocking.

Coming from Ruby I’ve so far found the mocking frameworks in Python somewhat lacking. I haven’t found a case where I’ve been unable to implement a particular test in Python but I’ve certainly found myself doing a lot more typing than I would in Ruby/Mocha. Perhaps this is just a sign that the Python libraries need further abstraction to provide higher-level interfaces. I think it also points to the fact that metaprogramming magic is much simpler in Ruby.

A quick and dirty list of some available mocking frameworks can be found here ( http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy#MockTestingTools). We chose Mox primarily because it presented most of the functionality we were used to with Mocha and the interface was concise (relatively speaking…).

The Basics

Mox uses a record/replay format for creating mocks. Mocks are initialized with the system in record mode. All method calls executed on the mocks are recorded in order to be played back later. This is similar to Mocha, although Mocha makes this less explicit and does not enforce order. For comparison:

Ruby/Mocha:

class Foo
  def do_something
    self.first_thing
    self.second_thing
  end
end

f = Foo.new
f.expects(:second_thing).once
f.expects(:first_thing).once

f.do_something

Python/Mox:

class Foo(object):
  def do_something(self):
    self.first_thing()
    self.second_thing()

self.mox = mox.Mox()
f = self.mox.CreateMock(Foo)

f.first_thing()
f.second_thing()
self.mox.ReplayAll()

f.do_something()
self.mox.VerifyAll()

Note, these are not complete examples and they will not run by themselves. We’ll get to that in a bit. The important thing is that, in the Python/Mox version, we’re doing a bit of recording until we get to ReplayAll(). At that point we stop recording and start executing the methods that will cause the recorded portion to be repeated by our running code. Mox checks off each method as it’s called and raises exceptions if a call is not expected.

The Complete Example

Mox provides a handy subclass of unittest.TestCase that takes care of a few things for you. First, self.mox is instantiated in setUp method so you don’t have to worry about doing that yourself. Second, and this is left out of the example above, it introduces some metaclass magic that unsets stubs and executes VerifyAll() to confirm that all expectations were met during execution (see the Mox documentation for an explanation of unsetting stubs). The complete code would look something like this:

import mox
from foo import Foo

class FooTest(mox.MoxTestBase):
  def test_do_something(self):
    f = self.mox.CreateMock(Foo)

    f.first_thing()
    f.second_thing()
    self.mox.ReplayAll()

    f.do_something()

If you want to mock an entire object this is great. It’s relatively clean and gets the job done. But what if I want to stub out a single method of an already instantiated object? Enter StubOutWithMock.

class FooTest(mox.MoxTestBase):
  def test_do_something(self):
    f = Foo()
    self.mox.StubOutWithMock(f, 'second_thing')

    f.second_thing()
    self.mox.ReplayAll()

    f.do_something()

In this example I’ve instantiated my own instance of Foo and allowed the firstthing method to get called normally. The secondthing method is mocked out. I could add return values and whatnot to the mock here if necessary.

These are just two simple examples but they’ll cover a lot of cases. Mox really handles them quite well. Where I think it starts to fall behind Mocha is when it comes to dealing with more complex tasks such as introducing mocks for locally instantiated objects. Ultimately I think the Mox library just needs a bit more abstraction to handle these cases.

The Hard(er) Stuff

Some folks (http://googletesting.blogspot.com/2008/08/by-miko-hevery-so-you-decided-to.html are of the opinion that your code should be written with testing in mind. I couldn’t disagree more (although I think there are many good points that can be taken from that article for reasons outside of testing). I should not have to change the way I write code to make up for flaws in my testing framework. In some languages it may not be possible to support every test case without some extra consideration (compiled, strongly-typed, minimal-reflection languages come to mind although I’ve had excellent luck with MockItNow ( http://code.google.com/p/mockitnow/) for C++ under Visual Studio) but we’re working with highly abstracted scripting languages here! There really shouldn’t be a reason to change the way you code just to support testing. Let’s see what we can do.

Local Object Construction

Note: Since this post was original written I’ve been shown a much more elegant way to mock local object construction. See comment #6 for details.

In the everything’s-an-object land of Python and Ruby objects are instantiated left and right. Most are created in local scopes. What happens when you want to mock out methods of an object that’s instantiated inside of a function? Take the following code:

class Bar(object):
  def do_something(self, something):
  o = SomeOtherClass()
  result = o.crazy_thing_not_to_be_tested(something)
  return self.do_something_else(result)

In our unit test what we really want to do is make sure that the result of crazythingnottobetested is being passed to dosomethingelse. Since the implementation of crazythingnottobetested is crazy (and because this is a unit test, not a functional one) we want to mock it out and return something simple. With Mocha I could use SomeOtherClass.any_instance but there’s no such abstraction in Mox. An alternative might be to change our implementation so that the SomeOtherClass instance is passed to our method instead of being instantiated by it. In some cases this may work but what if the SomeOtherClass object needs to be initialized with private members of Bar? Furthermore this means you’re writing your code for your tests and not vice versa (there are other reasons, unrelated to testing, for why you might be better off passing in the object rather than instantiating it locally but that’s outside the scope of this post). It turns out that the Python solution is to mock out the class method that actually instantiates the object.

Object allocation in Python is a two step process. First the instance is allocated and then the constructor (init) is called. This is similar to Ruby’s allocate/initialize setup. The allocate equivalent for Python is new. This method can be mocked like any other.

 bar = Bar()
 self.mox.StubOutWithMock(bar, 'do_something_else')
 self.mox.StubOutWithMock(SomeOtherClass, '__new__')

 mocked_some_other_class = self.mox.CreateMockAnything()
 SomeOtherClass.__new__(SomeOtherClass, ).AndReturn(mocked_some_other_class)
 mocked_some_other_class.crazy_thing_not_to_be_tested('something').AndReturn('something_else')
 mocked_bar.do_something_else('something_else')
 self.mox.ReplayAll()

 mocked_bar.do_something_else('something')

Skipping Constructors

With our new knowledge of new it becomes easier to take care of a few more problems. Say your constructor does something that you really don’t want to have executed for every test. There’s a good argument here that you shouldn’t be doing real work in your constructor anyway but in some cases it is desirable. At any rate, one solution is to instantiate the object using new without calling init. This could be done in your setUp method so that you have the object available for every test in your test case.

class SomethingTest(mox.MoxTestBase):
  def setUp(self):
    mox.MoxTestBase.setUp(self)

    self.something = Something.__new__(Something)

Note that the new method takes the class object as its first argument. You may need to pass in additional arguments for init (although it’s never called) to match the function definition.

Later, if you actually want the constructor to be called, you can do so explicitly.

def a_test(self):
  self.something.__init__(*args)

Mocking Modules

Modules are nothing special, including those that are part of standard libraries. Mock away!

import os

self.mox.StubOutWithMock(os, 'listdir')
os.listdir(".").AndReturn(["a_dir", "a_link", "cat.py", "dog.py", "goat.txt", ".svn"])

Mocking Builtin Methods

The route to mocking builtin methods is a bit more involved but not terribly so. The first question is how to access the methods to mock them. StubOutWithMock takes an object and the name of the method to mock. We know the name of the method but what’s the object? For those with Python experience it will seem obvious that the answer is the builtin module which houses all of Python’s builtin methods. The way I found to access that module was via sys.modules:

import sys

self.mox.StubOutWithMock(sys.modules['__builtin__'], 'open')

stream = self.mox.CreateMockAnything()
sys.modules['__builtin__'].open('foo.yml', 'r').AndReturn(stream)

Mocking Your Parents

Sometimes its necessary to prevent the super class implementation of a method from being called during a test. This is actually very easy to accomplish and the implementation is relatively intuitive.

To call a parent class implementation in Python you’d do something like this:

class HouseCat(Feline):
  def walk(self):
    self.act_crazy()
    Feline.walk(self)

All we’re doing is passing self directly to the Feline implementation of the walk method. So, all we need to do to mock out the Feline version is:

self.mox.StubOutWithMock(Feline, 'walk')

cat = HouseCat()
Feline.walk(cat)
self.mox.ReplayAll()

cat.walk()

Now self.act_crazy will be executed but the super class implementation of walk will be mocked out.

Sorting Out Errors With MockAnything

There’s a bug of sorts when using the MockAnything class that causes fun problems ( http://code.google.com/p/pymox/issues/detail?id=3) when Mox attempts to print out results of a failed method call. The output from Mox usually consists of a series of “Expected X. Got Y” statements. Well when X or Y includes a MockAnything object we run into a bit of a catch 22. Python attempts to convert the instance to a string using repr but, since we’re mocking anything, we don’t have a real implementation for that method and the object raises an exception to say that the method call for repr was unexpected. Currently the authors of Mox have not submitted a solution to this problem but there’s a relatively naive one you can use yourself… Just implement the repr method explicitly for MockAnything objects. Be aware that this will likely (I haven’t tried it) block attempts to mock the repr method itself and may cause unexpected issues later on.

In the class definition for MockAnything (roughly ln 265 in mox.py) add the following:

    def __str__(self):
      return ""

    def __repr__(self):
      return self.__str__()

MockAnything instances will then be rendered as in your error output.

Misc Trivia/Pointers/Gotchas

  • Everything is an object in Python so there’s probably some way to mock everything
  • Use mox.MoxTestBase as the parent class for all your tests where mocking will be used
  • If you do this and you have your own setUp method declaration be sure to call the parent class implementation!
  • Don’t forget to call self.mox.ReplayAll() before you actually kick off the replay portion of your test
Feb 23

Automagically Managing Merges with SVN

By Ola Mork

For each Guitar Hero game we have a full trunk/branches/tags structure. Most of the trunks are copied off of their predecessor’s trunk. This works pretty well for us, but it can be hairy to merge changes made to the original ( game-1 ) trunk down to each of its descendants. I found that I was often forgetting to merge to *all* of the relevant branches. To mitigate the effects of my amnesia I wrote this, chmod +x ’d it and dropped it into my ~/bin (which happens to be in my PATH ):

#! /usr/bin/env ruby
repository_host = 'your.svn.repository.host'
working_dirs = ['~/Documents/game-2/trunk',
'~/Documents/game-3/trunk',
'~/Documents/game-4/trunk']

p "usage: $ gh-merge revision [revision revision revision ...]" if ARGV.empty?

ARGV.each do |revision|
  working_dirs.each do |working_dir|
    p "merging #{working_dir}, revision #{revision}"
    `svn merge -c #{revision} \
    svn+ssh://#{repository_host}/agora/game-1/trunk \
    #{working_dir}`
  end
end

working_dirs.each do |working_dir|
  p "committing #{working_dir}"
  `svn ci -m "svn merge -c #{ARGV.join(', ')} \
  svn+ssh://#{repository_host}/agora/game-1/trunk" \
  #{working_dir}`
end

The working_dirs represent the paths to my working dirs for each branch I want to merge my trunk changes to.

To use I just pass the revisions I want to push to each branch like so:

ola$ gh-merge 31774 31775 31780

and it merges and commits those revisions to each branch. It assumes you’re caching your authentication for svn.

Feb 23

Automatic Deploy Notifications

By Ola Mork

When we make a deploy to our staging or production environments there are about 5 people who need to know and about 25 who want to know. Often someone is forgotten or the email doesn’t go out at all and we get a phone call (which no one likes).

The solution? Capistrano recipes to the rescue!

before 'deploy', 'deploy:notify'

No matter what happens, if we’re deploying people are going to know about it.

Here’s that task:

namespace :deploy do
  desc "sends a notification email letting everyone know we're deploying"
  task :notify do
    deployment_name = application.to_s.gsub(/(-.*)?/, '').upcase
    stage_name = stage.to_s.upcase
    action_mailer_with_temporary_delivery_method do
      mail = TMail::Mail.new
      mail.subject = "Super Fantastic Project Pending Deploy [#{deployment_name} #{stage}]"
      mail.to      = "superproject-deploy@lists.yourcompany.com"
      mail.from    = ActionMailer::Base.smtp_settings[:user_name]
      mail.date    = Time.now
      mail.body    = ActionMailer::Utils.normalize_new_lines(%Q(
Hello everyone,

We're making a deploy to the #{stage} environment. The #{stage} site may (or may not) be
unavailable for a few minutes sometime within the next 30 minutes (depending on how long
it takes to upload the changes).

This message is automated.

---------------------- debugging information ----------------------
#{variables}
))
      ActionMailer::Base.deliver(mail)
      # puts mail.body
    end
  end
end

Now, what’s going on here? First, apologies to DHH but I’m not smart enough to figure out how to make the Rails email system work so we just use TMail directly. Which is fine, but we use ARMailer which puts all of our email in a queue and it gets delivered by a cron job every 5 minutes or so. This usually works pretty well but in this case we have two problems though. The first is this is run on the deployer’s machine, which may not have access to the database where it needs to queue up the email. Plus, when we deploy we disable the crons and we don’t turn them back on until we’re satisfied everything is working. If we used ARMailer for this particular email, the site may be down and no one would know; defeats the purpose of the notification.

actionmailerwithtemporarydelivery_method addresses this problem:

def action_mailer_with_temporary_delivery_method
  existing_delivery_method = ActionMailer::Base.delivery_method
  existing_smtp_settings = ActionMailer::Base.smtp_settings
  begin
    ActionMailer::Base.delivery_method = :smtp
    ActionMailer::Base.smtp_settings = {
      :address => "smtp.gmail.com",
      :port => "587",
      :domain => "your.domain.com",
      :authentication => :plain,
      :user_name => "do-not-reply@your.domain.com",
      :password => "PASSword"
    }
    yield
  ensure
    ActionMailer::Base.delivery_method = existing_delivery_method
    ActionMailer::Base.smtp_settings = existing_smtp_settings
  end
end

action_mailer_with_temporary_delivery_method temporarily sets up ActionMailer to use SMTP (with gmail) to send out our notification. When we’re done, it sets it back to whatever it was set to before.

We’ve hurdled the actual message delivery, but how can we make that notification message itself more useful to the internal group? We do two things. The first is to sort out exactly what we’re deploying and where we’re deploying it to:

  deployment_name = application.to_s.gsub(/(-.*)?/, '').upcase
  stage_name      = stage.to_s.upcase

application and stage are set in our regular deploy script with:

  set :application, 'superproject-staging'

:stage is more complicated but essentially the same (in this example it would be ‘staging’)

The second pieces is variables . Capistrano 2.3 has an instance variable available to all tasks named @variables . We were just inspecting that for a while but it has a lot of procs for determining some of the values. Now we do this:

def variables
  result = []
  @variables.reject { |k,v| k.to_s.downcase.eql?('password') }.each_pair do |key, value|
    # since we're going to be printing this out, we don't want to see garbage like
    # "#"
value = value.is_a?(Proc) ? value.call : value
value = value.is_a?(Array) ? value.join(", ") : value
result

Which gives you some better output:

---------------------- debugging information ----------------------
 application : superproject-staging
 available_stages : staging, production, qa
 cron_file : config/cron/staging_cron.conf
 cron_role : utility
 cron_role_only_clause :
 current_dir : current
 current_path : /home/www-data/superproject-staging/current
 current_release : /home/www-data/superproject-staging/releases/20090220155729
 current_revision : 32024
 default_environment :
 default_run_options : ptytrue
 deploy_to : /home/www-data/superproject-staging
 deploy_via : checkout
 keep_releases : 7
 latest_release : /home/www-data/superproject-staging/releases/20090220162352
 latest_revision : 32024
 logger : #<Capistrano::Logger:0x1214e30>
 memcache_extra_options :
 memcache_memory : 64
 memcache_pid_file : /home/www-data/superproject-staging/shared/log/memcache.pid
 memcache_port : 20109
 memcache_role : app
 memcache_user : root
 migrate_plugin_role : db
 migrate_plugin_role_only_clause : primarytrue
 mongrel_address : 0.0.0.0
 mongrel_environment : staging
 mongrel_instances : 8
 .
 .
 .
 .
 etc

Prettier examples are available at pastie. Drop that in config/recipes/notification.rb and in your normal config/deploy.rb include this:

load 'config/recipes/notification
Feb 23

Agora in print

By Brian Corrigan

Our CTO Brian Corrigan was quoted in an article in this month’s ComputerWorld magazine. Check it out here!