Novus Visual Regression Service

Visual regression testing for WebdriverIO

Based on the work of Jan-André Zinser on wdio-visual-regression-service and wdio-screenshot

Installation

You can install wdio-novus-visual-regression-service via NPM as usual:

$ npm install wdio-novus-visual-regression-service --save-dev

Instructions on how to install WebdriverIO can be found here.

Configuration

Setup wdio-novus-visual-regression-service by adding novus-visual-regression to the service section of your WebdriverIO config and define your desired comparison strategy in the service options.

// wdio.conf.js

var path = require('path');
var VisualRegressionCompare = require('wdio-novus-visual-regression-service/compare');

function getScreenshotName(basePath) {
  return function(context) {
    var type = context.type;
    var testName = context.test.title;
    var browserVersion = parseInt(context.browser.version, 10);
    var browserName = context.browser.name;
    var browserViewport = context.meta.viewport;
    var browserWidth = browserViewport.width;
    var browserHeight = browserViewport.height;

    return path.join(basePath, `${testName}_${type}_${browserName}_v${browserVersion}_${browserWidth}x${browserHeight}.png`);
  };
}

exports.config = {
  // ...
  services: [
    [
      'novus-visual-regression',
      {
        compare: new VisualRegressionCompare.LocalCompare({
          referenceName: getScreenshotName(path.join(process.cwd(), 'screenshots/reference')),
          screenshotName: getScreenshotName(path.join(process.cwd(), 'screenshots/screen')),
          diffName: getScreenshotName(path.join(process.cwd(), 'screenshots/diff')),
          misMatchTolerance: 0.01,
        }),
        viewportChangePause: 300,
        viewports: [{ width: 320, height: 480 }, { width: 480, height: 320 }, { width: 1024, height: 768 }],
        orientations: ['landscape', 'portrait'],
      }
    ]
  ],
  // ...
};

Options

Under the key visualRegression in your wdio.config.js you can pass a configuration object with the following structure:

• compare Object
  screenshot compare method, see Compare Methods

• viewportChangePause Number ( default: 100 )
  wait x milliseconds after viewport change. It can take a while for the browser to re-paint. This could lead to rendering issues and produces inconsistent results between runs.

• viewports Object[{ width: Number, height: Number }] ( default: [current-viewport] ) (desktop only)
  all screenshots will be taken in different viewport dimensions (e.g. for responsive design tests)

• orientations String[] {landscape, portrait} ( default: [current-orientation] ) (mobile only)
  all screenshots will be taken in different screen orientations (e.g. for responsive design tests)

Compare Methods

wdio-novus-visual-regression-service allows the usage of different screenshot comparison methods.

VisualRegressionCompare.LocalCompare

As it's name suggests LocalCompare captures screenshots locally on your computer and compares them against previous runs.

You can pass the following options to it's constructor as object:

• referenceName Function
  pass in a function that returns the filename for the reference screenshot. Function receives a context object as first parameter with all relevant information about the command.

• screenshotName Function
  pass in a function that returns the filename for the current screenshot. Function receives a context object as first parameter with all relevant information about the command.

• diffName Function
  pass in a function that returns the filename for the diff screenshot. Function receives a context object as first parameter with all relevant information about the command.

• misMatchTolerance Number ( default: 0.01 )
  number between 0 and 100 that defines the degree of mismatch to consider two images as identical, increasing this value will decrease test coverage.

• ignoreComparison String ( default: nothing )
  pass in a string with value of nothing , colors or antialiasing to adjust the comparison method.

For an example of generating screenshot filenames dependent on the current test name, have a look at the sample code of Configuration.

VisualRegressionCompare.SaveScreenshot

This method is a stripped variant of VisualRegressionCompare.LocalCompare to capture only screenshots. This is quite useful when you just want to create reference screenshots and overwrite the previous one without diffing.

You can pass the following options to it's constructor as object:

• screenshotName Function
  pass in a function that returns the filename for the current screenshot. Function receives a context object as first parameter with all relevant information about the command.

VisualRegressionCompare.Spectre

This method is used for uploading screenshots to the web application Spectre. Spectre is a UI for visual regression testing. It stores the screenshots and compares them which is quite useful for Continuous Integration.

You can pass the following options to it's constructor as object:

• url String
  pass in a spectre webservice url.

• project String
  pass in a name for your project.

• suite String
  pass in a name for your testsuite. One project can contain several suites.

• test Function
  pass in a function that returns the test name for the screenshot. Function receives a context object as first parameter with all relevant information about the command.

• browser Function
  pass in a function that returns the browser for the screenshot. Function receives a context object as first parameter with all relevant information about the command.

• size Function
  pass in a function that returns the size for the screenshot. Function receives a context object as first parameter with all relevant information about the command.

• fuzzLevel Number ( default: 30 )
  number between 0 and 100 that defines the fuzz factor of Spectre's image comparison method. For more details please have a look at Spectre documentation.

Example

// wdio.conf.js

var path = require('path');
var VisualRegressionCompare = require('wdio-novus-visual-regression-service/compare');

exports.config = {
  // ...
  services: [
    [
      'novus-visual-regression',
      {
        compare: new VisualRegressionCompare.Spectre({
          url: 'http://localhost:3000',
          project: 'my project',
          suite: 'my test suite',
          test: function getTest(context) {
            return context.test.title;
          },
          browser: function getBrowser(context) {
            return context.browser.name;
          },
          size: function getSize(context) {
            return context.meta.viewport != null ? context.meta.viewport.width : context.meta.orientation;
          },
          fuzzLevel: 30
        }),
        viewportChangePause: 300,
        viewports: [{ width: 320, height: 480 }, { width: 480, height: 320 }, { width: 1024, height: 768 }],
        orientations: ['landscape', 'portrait'],
      }
    ]
  ],
  // ...
};

Usage

wdio-novus-visual-regression-service enhances an WebdriverIO instance with the following commands:

• browser.checkViewport([{options}]);
• browser.checkDocument([{options}]);
• browser.checkElement(elementSelector, [{options}]);

All of these provide options that will help you to capture screenshots in different dimensions or to exclude irrelevant parts (e.g. content). The following options are available:

• exclude String[]|Object[] (not yet implemented)
  exclude frequently changing parts of your screenshot, you can either pass all kinds of different WebdriverIO selector strategies that queries one or multiple elements or you can define x and y values which stretch a rectangle or polygon

• hide Object[]
  hides all elements queried by all kinds of different WebdriverIO selector strategies (via visibility: hidden)

• remove Object[]
  removes all elements queried by all kinds of different WebdriverIO selector strategies (via display: none)

• viewports Object[{ width: Number, height: Number }] (desktop only)
  Overrides the global viewports value for this command. All screenshots will be taken in different viewport dimensions (e.g. for responsive design tests)

• orientations String[] {landscape, portrait} (mobile only)
  Overrides the global orientations value for this command. All screenshots will be taken in different screen orientations (e.g. for responsive design tests)

• misMatchTolerance Number
  Overrides the global misMatchTolerance value for this command. Pass in a number between 0 and 100 that defines the degree of mismatch to consider two images as identical.

• fuzzLevel Number
  Overrides the global fuzzLevel value for this command. Pass in a number between 0 and 100 that defines the fuzz factor of Spectre's image comparison method.

• ignoreComparison String
  Overrides the global ignoreComparison value for this command. Pass in a string with value of nothing , colors or antialiasing to adjust the comparison method.

• viewportChangePause Number
  Overrides the global viewportChangePause value for this command. Wait x milliseconds after viewport change.

License

MIT