Introduction into Whisperer

Posted: Nov 16, 2014

The project I work on consists from a few parts which are written on PHP, Ruby and Java. To write proper acceptance tests on Ruby part, we have been using VCR which stubs external API. In our case we stubbed all interactions with PHP and Java parts. Everything worked well. We got good written tests which check logic of the application from the lowest level (HTTP interactions with APIs) to the highest (the user interface of the application). But, since we have a lot of API calls, maintenance of VCR cassettes became pain for us. VCR is a very good tool to record the first HTTP interaction. The idea of this library is to record and re-record all API interactions time to time to have up to date stubbs. But, when you have the complicated logic, it isn’t so easy.

Let me give one example, we have the page to manage servers (yes, it is another cloud system). Each server can have one of statuses: running, stopped, destroyed or error. For each status our system allows an user to do various of actions over the server. It sounds like we need 4 VCR cassettes to test those actions. Yes, we have those cassettes, but all actions over servers produce other API interactions which should be stubbed as well. Although it goes even more interesting, each server has dependencies like ip addresses, storages, ISO. To record VCR cassettes for all test cases, we need to keep the test database of third applications (Java and PHP parts) in proper state. Simple list:

This list isn’t completed, it is only for the running servers. Each of these states allows different actions which we have to test.

We tried to keep the database having all states required for tests, but it was very very difficult. Why? It is a lot of manual work. Example, we need to test the attach storage functionality. We can attach a storage to stopped servers only. Ok, lets say we have such server, we wrote the test, we launched it and we got the VCR cassette. Everything looks great, our test uses the generated cassette and this cassette reflects the real response of API. But, lets say tomorrow the API response is extended with a few new fields which we also want to use in our functionality. We cannot use the test database anymore to re-record VCR cassette, because the state was changed while launching our last test. It means we need to go to the database (or admin side) of the external service and detach that storage from the server. Some routine work is required in this case.

We could live with that routine work. The worst issues we got with dependencies. Servers have storages, storages have backups. If somebody change the state of the server your test relies on, your tests will be broken after re-recording cassette. A lot of times we had inconsistency in our VCR cassettes, because the state of test database was changed and after re-recording cassettes we didn’t get what we expected. To break tests for backups, you need to change the server or storage id and you will spend a few hours to find out where the association between entities are incorrect. The situation happened a lot of time to us.

Another issues are constraints. Example from our application, you cannot request another IP address within 30 days if you have returned at least one. It means you cannot re-record cassettes for adding IP addresses. To do that you have to use another user account. To set up such account you need time.

In our application we have about 50 different cassettes. Most of them are identical, only 1-2 attributes in the response are different. We spent a lot of hours on understanding why our tests were red. Everything worked in the browser, but it didn’t in tests. In most cases, some entities were not properly associated with their dependencies. It was hell. I like VCR, it is great tool. But, it doesn’t work when you need to stub a lot of HTTP interactions. We didn’t want to get rid of it, we wanted to have something more powerful to get rid of painful dependencies.

I guess most of Ruby developers know what FactoryGirl is. I guess most of you know why it was invented, because Rails fixtures are painful. I like FactoryGirl, I like VCR and I though why I could not use FactoryGirl with VCR to keep cassettes consistent. Actually, in our application we have 8 entities, mostly we had issues with dependencies and constraints to record cassettes. To solve this problem I created Whisperer gem. This gem offers a few features:

Now the flow in our project is following:

The documentation of the gem describes how to use it. If you are interested, please, check and let me know your opinion. We started using it in our project, now our VCR cassettes look more consistent and we spend less time on writing tests. It means we reduce cost of our application.

Might be interesting too: