The Solution to the Temptation of End-To-End Tests: Integration Testing

The moment that you QA group figures out how to write end-to-end tests or the moment that you boss watches a selenium test, a new fever hits your team to write a test as solution to every problem. Button the wrong color? Write a test. Customer got a 404? Write a test. Need to support IE 8? Write a test.

Next thing you know your team has hundreds of tests that do bunches of things. You also need a small super-computer cluster to run them as well.

The problem is that these end-to-end tests have serious inherent problems. Those problems with end-to-end tests are stability and speed.

Luckily, there is a solution. Let’s define the two problems first.

Lack of Test Stability is a Loss of Test Confidence

Let’s talk about stability first. I have worked on a number of projects where there is an automation test suite. We have our unit, integration, and end-to-end test tests. We usually have a continuous integration (CI) pipeline that will execute our tests on each check in. Things are great for the first couple of weeks. All steps are green and then after that we move to this:

And the failed builds are painful. Everything goes red and stop looking at the build. Then your team doesn’t care.

Then you start hearing these conversations:

The thing is though is that we still care about the first tests: unit. Why is that? Well because they are stable. We make git hooks that prevent people from pushing code to a broken build but our integration and end-to-end tests are still a total blood-bath. Below is a graph the depicts the stability of tests along with team confidence.

One thing to think about. If your test environment works 99% of the time and you have 200+ tests, then you are statistically going to fail almost every build. The law of averages is NOT on your side. Even if you get a good build, how often will that happen?

Solutions to Complexity

First thought: Keep it simple. That seems to be like a obvious solution but try to get your team to focus on integrating the pieces that have the most complex interaction. If two parts connect in one way, then how much benefit do you get putting them together. If you have two units that interact back and forth, then integrate those together. More is not always better.

Second thought: Write a stub for the flakiest part of the system. Do you rely on that third-party processor and it works 80% of the time? Rip out the actual interaction into a stub! It may take a little time but it will make your tests reliable.

A real-world example: I was working on a project where we have a complex front-end web application and then a complex back-end that makes calls to multiple services. Past projects that were somewhat similar had always run into problems with the end-to-end tests where the back-end server would 500 or timeout (always randomly) and the tests would break. So we used sinon.js to fake XMLHttpRequests. You know what? That works 100% of the time. No weird delays, unexpected responses. The stubbing framework was < 200 lines of code and it opened us to up to write several tests on failure scenarios that are very, very difficult to recreate in a true end-to-end test.

Lack of Test Performance = “I don’t run those”

Developers are impatient. They are also easily distracted when they have to do something they don’t want to do. So when you have tests that don’t run quickly, then people don’t run them. One solution that I hear about the “nightly build”. I have never worked on a team that had this, but the problem is that you delay feedback to developers. Why!?! I want to know as quickly as possible that I broke something. First, I don’t want the shame of make a code change that breaks 500+ tests and then have to do the revert of shame.

Plus if you have the central authority run the tests, then your commit messages would look like this:

So how do you make your tests run faster?

Well, there are a couple of things that can be done:

• Choose frameworks that are made to be tested.
• Break your large number of end-to-end tests down into integration tests.
• If a framework has a long setup time, re-use your setup as much as possible.
• Run your tests in parallel.

Integration Testing

Integration tests are interesting because we put together two or more units and verify that their effect and response is correct. An integration test may have more than one side-effect. If you have a user creation controller and a user service, when we test user creation, we send back to the user what their user name is with the controller but we will also verify that the service persists the user details.

Not everything can be integrated and tested. Look for frameworks that let you do this. I like AngularJS with their directives and new component architecture. I also love Spring. You can wire up all sorts of crazy configurations that let you test the interaction of beans. Databases now have DbUnit and Cassandra has CassandraUnit.

Integration tests are very unassuming. They seem like they are more complicated versions of unit tests. Plus, they don’t test the application end-to-end! But, they are fast and stable. When I say fast, I think less than 100ms. An integration tests mocks out the dependencies on the end. This allows the points of instability to become very very stable.

Lastly, integration tests let you build interesting scenarios. Are you worried about a race condition? You can build artificial dependencies that get injected into your integration test to verify how your system will perform. Want to know what happens when the system throws an exception on adding and item to the basket? No problem. Want to make sure that your message processor handles a rare out-of-order scenario? Not difficult.

It’s not the silver bullet, but when used with some end-to-end tests and a bedrock of unit tests; it helps tremendously.