Cost of stubs in tests

Posted: Dec 9, 2017

A topic about stubs in tests appears now and then. There are people who think that stubs/mocks (there are differences between them, but I will refer to them as stubs. I am sorry, this post isn't about differences between them) are very helpful and there are people who have an opposite opinion. I though everything was settled in a team I work, but recently, this topic's been raised again. So, I decided to outline my thoughts, I might be wrong, please, tell me what I am missing.

A while ago, I used to be a developer who preferred to stub dependencies. It was so easy to write tests, they looked readable and simple.

class Customer
  def order_fee
    if inherit_fee?
      company.fee
    else
      fee
    end
  end
end

class Order
  def total
    subtotal + customer.order_fee
  end
end

As you see fee can be defined for a certain customer or a company they are related to. Here are tests with stubbed dependencies (actually, methods' calls):

describe Order do
  let(:customer) { Customer.new }
  let(:order)    { Order.new(subtotal: 100, customer: customer) }

  describe '#total' do
    context 'a customer has fee' do
      before do
        allow(customer).to receive(:order_fee).and_return(21)
      end

      it 'returns the total price which includes fee' do
        expect(order.total).to eq(121)
      end
    end

    context 'a customer has no fee' do
      before do
        allow(customer).to receive(:order_fee).and_return(0)
      end

      it 'returns the total price without any fee' do
        expect(order.total).to eq(order.subtotal)
      end
    end
  end
end

Instead of setting fee for the customer or the company we just stub the outcome of Customer#order_fee. Some of you might say: "Hey, you should use a stub object instead of the real customer object". Definitely, we can do that.

describe Order do
  let(:customer) { instance_double(Customer, order_fee: 21) }
  # tests
end

But, it doesn't change much. Another way is to use the real object with real methods' calls.

describe Order do
  let(:customer) { Customer.new }
  let(:order)    { Order.new(subtotal: 100, customer: customer) }

  describe '#total' do
    context 'a customer has fee' do
      before do
        customer.fee = 21
      end

      it 'returns the total price which includes fee' do
        expect(order.total).to eq(121)
      end
    end

    context 'a customer has no fee' do
      before do
        customer.fee = 0
      end

      it 'returns the total price without any fee' do
        expect(order.total).to eq(order.subtotal)
      end
    end
  end
end

I prefer the last approach. I would leave these tests as they are. Some of you might say: "Hey, your tests doesn't cover all possible cases, there is a case when fee is inherited from the company". You are right, but Customer#order_fee is covered by tests which are written for the customer class, so there is no reason to repeat them here. Let's imagine a third case gets added to Customer#order_fee, will you come back to tests of Order#total and add another test to cover that new case too? Now, you think that the tests with stubs are better. It looks so easy, you just focus on the returned value of Customer#order_fee and you don't care what happens in the dependency.

Integration/contractor tests safe us

The trickiest part about unit tests using stubs is that there must be integration (contractor?) tests which will make sure real objects works together. Otherwise, unit tests might be green, but code fails in the production. Let's imagine our integration tests cover an action where 8 objects are involved. To be sure, our code works we must be sure that all of them get touched. I am not saying we need to write tests covering all possible cases of those objects, I am saying we need to be sure that objects nicely play together. For example, if there is no integration test which touches the real company object in Customer#order_fee, there is no evidence the customer object works fine with the company object.

Thus, when we stub objects, we simplify unit tests, but we make integration tests more complex. Again, when a new dependency gets added to the customer class, will you check integration tests to be sure they verify how that new dependency works with the customer object? When we use real objects in unit tests, we already check that objects nicely play with their dependencies, so it gives higher confidence. Although, It isn't a silver bullet either.

Missing cases of dependencies

Let's come back to "the missing cases" in the latest set of tests. As I mentioned above, it would be crazy to cover all cases of Customer#order_fee in tests for Order#total. These tests aren't about testing cases of Customer#order_fee, they are about testing cases of Order#total and being sure the order and customer object work together. So, if integration tests doesn't touch some of involved objects in the action, there is lower chance to get a bug in the production, because objects' communication is already covered in unit tests.

Some of you might argue and say that the code can get a different implementation, although, tests will pass:

class Order
  def total
    subtotal + customer.fee
  end
end

Indeed, tests will pass. But, tests will also pass if the subtotal equals 121 (yeah, we might make a silly mistake in setting the object under testing). It is a reason why we write/change tests then write/change code. It is all about trust. If we don't trust tests we wrote for Customer#order_fee, can we trust ActiveRecord#save!? When we use ActiveRecord#save!, do you test the following cases?

• raise an error when there is no DB connection
• raise an error when a table doesn't exist
• raise an error when a field doesn't exist
• whatever you can find in internals of ActiveRecord#save!

When we write code, we need to keep in mind the maintenance cost, eventually, somebody has to pay for that. Integration tests involve lots of objects, it is very easy to forget about a path which touches some of involved objects. Thus, a chance of a bug is higher, again somebody has to pay for fixing bugs. Please, don't forget that some bugs cause loosing real money. For example, a customer cannot confirm an order, it is disaster for a sales team (it should be for you too).

When stubs do make sense

I amn't saying stubs should be avoided everywhere. They are very helpful in certain cases.

You might know SinonJs, this lib provides a way to stub Ajax responses.

this.server.respondWith(
  'GET',
  '/some/article/comments.json',
  [200, { 'Content-Type': 'application/json' }, '[{ "id": 12, "comment": "Hey there" }]']
);

It is a perfect use case for stubs, because it stubs the lowest level, so when you launch tests the request goes through different layers of your code, thus, it gives good confidence that code will work in the production too.

Also, when you write a lib, you might provide a stub for its dependencies. The Faraday gem is a good example applying this approach.

Stubs make tests faster

I know that some people prefer stubs because they make tests faster. In my opinion, if tests are slow, they might be telling: "Hey, the project got performance problems". By using stubs, we hide this feedback we get from tests.

Wrap up

Whatever your preferences are, make sure code base you work on is consistent. Otherwise, new team members will be confused, thus, it will slow them down.