Visual Regression Testing with PhantomCSS

If you’ve done any serious development in your career you’ve likely reached the point when you realized the importance of automated testing during development. Depending on your experience, this realization might hit you in one big burst or it may gently come to you over time, but it will eventually become second nature. Automatic testing comes in many forms, from unit testing, when you test isolated pieces of code, to integration and functional testing, when you test how different parts of your system behave together. This article is not about an overview on automatic testing in general. It is about a particular and a relatively new niche referred to as visual regression testing.

Visual regression testing takes an alternative approach to testing web pages. Instead of just making sure that some element or a text value is present in the DOM, the test actually opens the page and checks if this specific block looks exactly like you want it to. Just to make sure that you picked up the difference, let me give you an example. Imagine, that you want your website to greet your visitors with a friendly message:

<div>Hello, %username%!</div>

To make sure it works, you can (and should) unit test the piece of code that produces the message, checking that it inserts the correct name. You can also write a functional test using Selenium or Protractor to see if the element is actually present on the page with the correct text. But this is not enough. We want to test not just that the text is generated correctly or appears in the DOM but to make sure that the whole element looks correct, i.e., making sure that the element is not hidden by display: none or that someone hasn’t accidentally overridden the color of the text. There are a number of tools to do that, but today we will be looking at one option in particular — PhantomCSS.

What is PhantomCSS?

PhantomCSS is a Node.js tool to perform visual regression testing. It is open source and developed by the guys at Huddle. PhantomCSS allows you to run a headless browser, open a page and take a screenshot of the whole page or a particular element on the page. This screenshot will be stored as a baseline image for future reference. Whenever you change anything on the website, you can run PhantomCSS again. It will take another screenshot and compare it to the original image. If there are no differences found, the test will pass. If, however, the screenshots don’t match, the test will fail and a new image showing the difference will be created for you to review. This approach makes this tool perfect for testing changes in CSS.

PhantomCSS is built on top of several key components:

CasperJS – a tool for interacting with a PhantomCSS or SlimerJS browser. It allows you to open a page and perform user interactions, such as clicking on buttons or inputting values. Additionally, CasperJS provides its own testing framework and the ability to capture screenshots of a page.
PhantomJS 2 or SlimerJS – two different headless browsers, either of which can be used with PhantomCSS. A headless browser is just like a normal browser without a user interface.
Resemble.js – a library for comparing images.

PhantomCSS can be used together with both PhantomJS and SlimerJS, but in this article, we’ll be using PhantomJS.

Let’s Take It for a Spin

Let’s set up a tiny test project to see how we can use this tool in practice. For that, we’ll need a web page to test and a simple Node.js web server for CasperJS to be able to open the page.

Setting up a Test Project

Create an index.html file with some sample content:

<!doctype html>
<html>
  <head>
    <style>
      .tag {
        color: #fff;
        font-size: 30px;
        border-radius: 10px;
        padding: 10px;
        margin: 10px;
        width: 500px;
      }

      .tag-first {
        background: lightcoral;
      }

      .tag-second {
        background: lightskyblue;
      }
    </style>
  </head>

  <body>
    <div class="tag tag-first">The moving finger writes, and having written moves on.</div>
    <div class="tag tag-second">Nor all thy piety nor all thy wit, can cancel half a line of it.</div>
  </body>
</html>

To install the web server, initialize an npm project and install the http-server package.

npm init
npm install http-server --save-dev

To run the server, let’s define a simple npm script. Just add the following scripts section to package.json

"scripts": {
  "start": "http-server"
},

Now you can run npm start from the project folder and the index page will be accessible on the default address http://127.0.0.1:8080. Start the server and leave it running for now. We’ll need it in a while.

Installing PhantomCSS

Installing PhantomCSS is easy, all you need to do is add a few dependencies to your project:

npm install phantomcss casperjs phantomjs-prebuilt --save-dev

Creating a Test Suite

Now we have everything we need to set up the first test suite. PhantomCSS test suites are created in the form of Node.js scripts where you open the required page of your website, take screenshots and compare them to the images from the previous run. We start with a simple test case based on the demo from PhantomCSS itself.

var phantomcss = require('phantomcss');

// start a casper test
casper.test.begin('Tags', function(test) {

  phantomcss.init({
    rebase: casper.cli.get('rebase')
  });

  // open page
  casper.start('http://127.0.0.1:8080/');

  // set your preferred view port size
  casper.viewport(1024, 768);

  casper.then(function() {
      // take the screenshot of the whole body element and save it under "body.png". The first parameter is actually a CSS selector
      phantomcss.screenshot('body', 'body');
  });

  casper.then(function now_check_the_screenshots() {
    // compare screenshots
    phantomcss.compareAll();
  });

  // run tests
  casper.run(function() {
    console.log('\nTHE END.');
    casper.test.done();
  });
});

The test will open http://127.0.0.1:8080/, take a screenshot of the body element and save it under screenshots/body.png.

Once we have the test itself in place, all that’s left is to define a script to run the test. Let’s add the following script to package.json next to start:

"test": "casperjs test test.js"

You can now run it by executing the following command:

npm test

The output you will see should look something like this:

Test file: test.js                                                              
# Tags
PASS Tags (NaN test)

New screenshot at ./screenshots/body_0.png

Must be your first time?
Some screenshots have been generated in the directory ./screenshots
This is your 'baseline', check the images manually. If they're wrong, delete the images.
The next time you run these tests, new screenshots will be taken.  These screenshots will be compared to the original.
If they are different, PhantomCSS will report a failure.

THE END.
WARN Looks like you didn't run any tests.                                       
npm ERR! Test failed.  See above for more details.

Since you’ve run the test for the first time, it will just create a new baseline screenshot and won’t perform any comparison. Go ahead and peek inside the screenshots folder. You should see an image like this one:

This is the golden standard of how your website is supposed to look and future executions of the test will compare their results to this image.

Introducing a Regression

If you run the same test command again it will report that all tests have passed successfully:

Test file: test.js                                                              
# Tags
PASS Tags (NaN test)


PASS No changes found for screenshot ./screenshots/body_0.png

PhantomCSS found 1 tests, None of them failed. Which is good right?

If you want to make them fail, change some CSS.

THE END.
PASS 1 test executed in 0.827s, 1 passed, 0 failed, 0 dubious, 0 skipped.

This is to be expected since we haven’t changed anything on the website. Let’s break something and re-run the tests again. Try changing some styles in index.html, for example, decrease the size of the blocks to 400px. Now let’s run the test again and see what happens:

Test file: test.js                                                              
# Tags
PASS Tags (NaN test)
Failure! Saved to ./failures/body_0.fail.png


FAIL Visual change found for screenshot ./screenshots/body_0.png (11.41% mismatch)
#    type: fail
#    file: test.js
#    subject: false

PhantomCSS found 1 tests, 1 of them failed.

PhantomCSS has created some images that try to show the difference (in the directory ./failures). Fuchsia colored pixels indicate a difference between the new and old screenshots.

THE END.
FAIL 1 test executed in 1.082s, 0 passed, 1 failed, 0 dubious, 0 skipped.       

Details for the 1 failed test:

In test.js
  Tags
    fail: Visual change found for screenshot ./screenshots/body_0.png (11.41% mismatch)
npm ERR! Test failed.  See above for more details.

Several important things have happened here. First, PhantomCSS reported that the tests failed because of a mismatch for screenshot body_0.png. The mismatch is measured at 11.41%. Second, the difference between the current and the previous version was saved in the failures folder. If you open it, you will see a screenshot like this one:

The screenshot conveniently highlights the areas that have been changed so it’s easy to spot the difference.

Accepting the Changes

Now that the difference has been highlighted, what should we do to accept the change? We should somehow be able to tell the tool that we want to stick with the reduced width of the blocks and accept the current view as the new standard. To do that, you can run the test command with an additional -- --rebase argument:

npm test -- --rebase

Note the two double dashes. It’s npm’s way of passing a parameter to the underlying command. So the following command will result in casperjs test test.js --rebase. Now that we’ve accepted the change, the previous baseline image will be replaced with the new one.

Taking it Further

Now that you’ve got the hang of the basics, you can start thinking about integrating this tool into your own workflow. I won’t get into the details of that since it is pretty project specific, but here are some questions to ponder about:

Are you going to run the tests against the real website, or some kind of style guide, where only separate UI elements are present?
Does your site have dynamic content? If yes, then changes in the content will cause the tests to break. To avoid that, you’ll need to set up a separate version of the website with static context to run the tests against it.
Are you going to add the screenshots into your version control? Yes, you should.
Are you going to take screenshots of whole pages, or separate elements?

Using this tool you can now cover the visual aspects of your website with automated tests. With your unit and functional tests already in place, this new strategy will fill a narrow gap in your testing frontier. Even if you are still new to testing — this is a good place to start!

Frequently Asked Questions (FAQs) about Visual Regression Testing with PhantomCSS

What is PhantomCSS and how does it work?

PhantomCSS is a visual regression testing tool that uses PhantomJS and Resemble.js to compare screenshots of web pages. It works by taking screenshots of your web pages, comparing them to baseline images, and highlighting the differences. This allows developers to quickly identify and fix visual inconsistencies in their web pages. PhantomCSS is particularly useful in large projects where manual testing can be time-consuming and error-prone.

How do I install PhantomCSS?

PhantomCSS can be installed using npm, the Node.js package manager. You can install it globally by running the command npm install -g phantomcss. Alternatively, you can add it as a development dependency to your project by running npm install --save-dev phantomcss.

How do I use PhantomCSS for visual regression testing?

To use PhantomCSS for visual regression testing, you first need to create a test script that tells PhantomCSS what to capture screenshots of. This script can be written in JavaScript or CoffeeScript. Once the script is ready, you can run it using PhantomJS. PhantomCSS will then capture screenshots of the specified elements, compare them to the baseline images, and generate a report showing the differences.

Can I use PhantomCSS with other testing frameworks?

Yes, PhantomCSS can be used with other testing frameworks like Mocha, Jasmine, and QUnit. It can also be integrated with continuous integration systems like Jenkins and Travis CI.

What are the advantages of using PhantomCSS for visual regression testing?

PhantomCSS offers several advantages for visual regression testing. It automates the process of capturing and comparing screenshots, saving developers a lot of time. It also provides a visual report that makes it easy to spot differences between the baseline and test images. Moreover, PhantomCSS supports responsive design testing, allowing developers to test their web pages on different screen sizes.

How do I update the baseline images in PhantomCSS?

To update the baseline images in PhantomCSS, you simply need to delete the old baseline images and run the test script again. PhantomCSS will capture new screenshots and use them as the new baseline images.

Can PhantomCSS handle dynamic content?

Yes, PhantomCSS can handle dynamic content. It allows developers to specify a delay before capturing screenshots, giving the dynamic content enough time to load. It also supports the use of callbacks to wait for specific events before capturing screenshots.

How do I debug tests in PhantomCSS?

PhantomCSS provides several options for debugging tests. It can log messages to the console, save failed tests as image files, and even create a video of the test run. These features make it easier to identify and fix issues in your tests.

Can I customize the comparison process in PhantomCSS?

Yes, PhantomCSS allows you to customize the comparison process. You can set the comparison type, the mismatch tolerance, and the output settings. This gives you more control over the comparison process and allows you to tailor it to your specific needs.

What are the alternatives to PhantomCSS for visual regression testing?

There are several alternatives to PhantomCSS for visual regression testing, including BackstopJS, Wraith, and Gemini. These tools offer similar features to PhantomCSS, but they may have different strengths and weaknesses depending on your specific needs.