The team I've been part of for the past few months has recently been hit by a long period of random failures on a functional test suite. Eventually we got to the point where confidence in our end-to-end test suite was so low we would just hit rebuild without thinking twice every time it failed and curse at it waiting to pass, often needing several attempts before it turned green again. The worst part of it is that, although it provided a valuable safety net, we hardly ever caught anything with it. The amount of time wasted on it started to hit us, so I embarked on a journey to see what alternatives were out there.
But before that, here is a bit of context...
Our current functional tests are based on Selenium and Nightwatch (and Yadda but that's outside the scope of this post) which allow us to test against a selection of real browsers (even though we don't!). Unfortunately, they are also painfully slow and very hard to maintain, partially because of their nature and partially because everyone in the team had tried to steer away from them. As as result of this, they didn't receive the love and care they needed. Some of the issues with our end-to-end tests are:
- They require an actual browser to run against, whether headless or not. This has the minor overhead of having to install additional binaries on your CI environment and locally but the potential pain of different versions in different environments.
- Flaky tests and random failures: although hitting the rebuild button does not require a massive effort, random test errors undermine the confidence in the build process and cause a great deal of frustration amongst developers, especially when you are rushing to push a bug fix live!
- Debugging and tracing issues: our CI generates screenshots for failing tests but they are often unhelpful and understanding the cause of failures can be a long and painful process. This is not necessarily related to which technologies you are using but some tools offer more than others in terms of debugging capabilities.
Another team at the client I'm consulting for, has been using Nightmare.js for their end-to-end tests, so I decided to give it a go myself to check if it could be a viable replacement for our out of shape functional test suite.
Nightmare.js uses a different approach compared to most of the other end-to-end tools, instead of wrapping Selenium or WebDriver, it's based on a different engine: Electron.
At first sight, Nightmare.js seems to have some nice features up its sleeve:
- Electron claims to be two times faster than PhantomJS which, given how incredibly slow these kind of tests can be, seems an excellent starting point.
- It can be run headlessly in a CI environment or in a window for debugging and troubleshooting purposes locally. You can also breakpoint the tests and look at DevTools to diagnose an issue.
- It provides a nice and extensive browser API, much more concise than PhantomJS.
- It's promise based.
- It offers a Chrome extension for recording interactions with a page.
During the admittedly short trial I gave to Nightmare.js, a few issues stood up as potentially troubling:
- It's based on Node.js and Chromium whereas Selenium allows you to test against a variety of browsers. Nightmare.js doesn't run against a fully functional browser whereas Selenium does, but this might be a compromise you are willing to take in order to gain more reliable and faster end-to-end testing.
- Fancy HTML components (more complex HTML structures hiding basic elements such as checkboxes) didn't seem to play very nicely with the test and I did not manage to find an easy way to set values for some of those form inputs.
- Running tests in interactive mode (i.e. opening the browser window whilst running the test) doesn't show the URL bar.
- The Chrome extension for recording interaction needs some improvements to become really useful: generated CSS selectors were painfully long and didn't capture most of the page interactions.
- How will Nightmare.js play when things starts to get messy? It's all good and well when you run an Hello World example but how will it look when you write complex tests again a real web application?
In order to get a feel on Nightmare.js, let's have a look at some basic examples.
- Test with callbacks.
- Test with promises.
- Test with proxy.
- Interaction with page elements: in this example two types of interaction with input elements are used: type and input. The main difference is type generates keyboard events whereas input doesn't.
- Test with viewport: this allows you to test pages which vary for different types of devices.
- Debugging a test by enabling WebTools and pausing the test execution
Although Nightmare.js seemed like a good testing tool, I haven't been impressed enough to push for replacing our entire functional test suite with it. Having said that, I might consider using it on my next project and withhold further judgment till then!
Regarding web application end-to-end testing in general, no matter what tools you use, doing it in a stable, easy to maintain and reliable way is very difficult to achieve. My advice would be to try and find the right balance of coverage / speed / reliability / maintainability for your end-to-end testing and focus as much as possible on other types of testing such as unit, integration, acceptance, and API testing. Write as few functional tests as you can and use them as smoke tests for the most critical paths of your application or you might end up consuming an entire team on this topic alone.