Mint Digital

The best solution for benchmarking Rails tests is the aptly named test_benchmark. After just one run with BENCHMARK=true¹, it became pretty apparent what the problem was: we had one test file (of out of several dozen total) that was taking nearly a minute to run. That's 25% of total test time being spent in one file. Crikey.

Let me stop right here and spoil the end of this post for you: I rewrote the test from scratch, and I was able to get this specific test down to around 3 seconds. Here's the story:

Mint uses Shoulda for all its tests these days, with much praise about how nicely shoulda tests read. This is the recommended "shoulda way" to write a controller test:

context "on GET to index" do
setup { get :index ; @stuff = Factory(:stuff) }
should_respond_with :success
should_render_template :index
should_assign_to(:stuff) { [@stuff] }
end

When you write a test like this, you need to understand that the macros you see (eg, should_respond_with) each create a separate test. That separate test runs the setup method of its context along with any other setup methods in contexts above it. This means that the above context block results in 3 object creations and three get calls.

Here's the same test, written with Rails' built-in assertions:

test "on GET to index" do
stuff = Factory(:stuff)
get :index
assert_response :success
assert_template :index
assert_equal [@stuff], assigns(:stuff)
end

One object created. One test request made. It may not seem like much, but multiply this behavior over 20 context blocks and it suddenly starts to add up. Does the test/unit version look as nice? Maybe not, but if writing tests like this makes our tests 1600% faster, I'm tempted to overlook the "pretty factor".²

The moral of the story is to be mindful. If you choose to use a third-party testing library, shoulda or otherwise, be aware of how it works and its shortcomings. Be critical about your tests. Benchmark often and consider it a bug if your tests take too long.

1. We added this line to `test.rb` so that benchmarks don't clutter our test output all the time:

   ENV['BENCHMARK'] = 'false' unless ENV['BENCHMARK']

2. In the spirit of full disclosure, shoulda is not completely at fault. In our specific test, this sort of behavior was exacerbated by some bad test design. We had several nested `context` blocks, each with their own setup blocks, along with a global setup method that created loads of objects that were only used in a handful of tests; we were creating 50+ objects for every test. Shoulda didn't write these tests poorly, but it didn't encourage good behavior either.

Original cartoon by xkcd