Visual Regression Testing | 6 Tools to Use
A guide on choosing the right visual regression testing tool.
Visual regression testing ensures that your application’s user interface is not adversely affected when you release code changes. In this guide, we’ll discuss some of the many open-source and software as a service (SaaS) visual regression testing tools your team can start using today. Let’s get started!
What is Visual Regression Testing?
Visual regression testing verifies that a web page’s elements are rendered properly from a visual point of view. These tests are meant to make sure that any code changes do not negatively impact the visual appearance of the application’s user interface. They’re automated tests, so they can be run as part of a CI/CD pipeline; with a mature CI/CD pipeline, the correctness of the visual elements is checked with each code change pushed to the code revision server.
Visual regression tests help you catch visual discrepancies in the development stage, before the bug reaches the customer and negatively affects user experience, and before it becomes more costly to fix.
Visual regression testing can also catch bugs that might slip through functional testing. For instance, a functional test will verify the existence of a button and whether it can be clicked. If the button is the same color as the background it’s on, the functional test won’t catch this visual problem, because the element is rendered on the screen and is clickable. On the other hand, a visual regression test will detect that those pixels have changed color in a way that was unintended. This is the power of visual regression testing, compared with other flavors of testing. In the next section, you’ll learn about some useful visual regression testing tools.
Which Visual Regression Testing Will Work Best for Your Team?
Depending on what your team wants to achieve, there are multiple visual regression testing tools to choose from. You can find a tool that meets the requirements of your budget, your need for convenience, and your coding expertise. These tools can be divided into two broad categories: open-source tools and SaaS offerings. (These groups are not mutually exclusive, as an open-source tool might have a cloud-based SaaS offering as well.) Let’s start with the open-source options:
Open-Source Visual Regression Testing Tools
Having a high number of variables like web browsers, screen sizes, operating systems, and responsive design makes visual regression testing complex. With existing open-source projects, this complexity can be managed to a great degree. Below are three popular open-source browser automation and testing tools that support visual testing in some form:
To drive a browser, you need to use Selenium WebDriver. In the context of visual regression testing with Selenium, if you know Java, you can sew up a solution with Selenium and Ashot. As with any other open-source tool, you’ll need to wire multiple projects together and create a solution that fits your needs.
Other open-source projects and tools for visual regression testing include Puppeteer and BackstopJS. They have their own place in the visual regression testing realm; however, the aforementioned three options are the most useful ones. In the next section, we’ll take a look at some SaaS offerings for running visual regression testing.
SaaS Visual Regression Testing Tools
You’ll need to write some code when you use open-source tools for visual regression testing. It will be your team’s responsibility to manage configuration options, decide where to run the tests, and how to execute tests. Similar to other software and tools, you can outsource all of these responsibilities and focus on writing and maintaining top-class visual regression tests; this is where the SaaS offerings shine.
As with any other area of software engineering, testing, and CI/CD, there are many notable players in the visual regression testing game. Below are three that will help you write meaningful visual regression tests for your web applications:
Meticulous is a SaaS visual regression testing tool that helps you catch visual regressions without writing or maintaining UI tests. It replays user traffic to automatically catch visual regressions on your web application, with no side effects. It is the easiest way to create end-to-end tests without writing any code. And no code to write means no code to maintain.
Meticulous is funded by Y Combinator, the CTO of GitHub, the CPO of Adobe, and the founder of Vercel, to name a few. It can be set up in a minute by injecting a code snippet. Meticulous can be easily integrated with your CI/CD pipeline with one line of code and two simple clicks. To learn more about Meticulous, you can book a demo or sign up and take it for a spin.
Meticulous is not a layer on top of existing tools like Playwright or Selenium. It’s a different way of thinking about visual testing. It sets you free from writing code while still providing you with the power of visual tests. It eliminates the need to write or maintain e2e tests in the first-place, by generating tests from recorded user sessions and automatically maintaining them.
Applitools is a SaaS tool for visual testing and monitoring. It helps both development and QA teams to reduce the amount of code used to write tests, and the amount of time spent running and fixing them. It claims to be an AI-powered visual testing tool that is easy to set up and that integrates with all modern test frameworks. It has clear pricing plans that include starter, eyes, and ultrafast cloud, but the prices are not shown on the Applitools site. The free plan gives you one user and 100 checkpoints per month.
Applitools runs tests in a scalable way with reporting for visual tests written in the tool or framework of your choice. It also lets you monitor the results of the tests in a dashboard. Applitools Eyes gives you a screen with the diff in screenshots, and you can accept or reject changes. It covers a widerange of tools, from Cypress and Playwright on the web to Storybook and Appium for Native mobile apps.
As shown in the tutorial for Playwright with Typescript, Applitools’ SDK provides a layer of functionality that helps run the tests on multiple browsers with various resolutions. The SDK also sends the results to the Applitools server. The range of frameworks supported by Applitoos is commendable. If you’re looking for a visual and managed layer on top of your existing open-source visual regression testing tools like Playwright, Puppeteer, Selenium, or Watir, then Applitools is a good option.
Percy is a visual testing tool that supports cross-browser visual testing. It lists the visual changes as screenshot diffs on a dashboard, where they can be approved or denied. It has been acquired by BrowserStack, which provides access to more than 3,000 browsers and iOS and Android devices for cross-browser testing. Percy has a free plan with 5,000 screenshots per month and unlimited team members included. The paid plans start at $149 per month, which includes 25,000 screenshots and other features.
Percy integrates with many frameworks, such as Ember and Rails. It also has integrations with Storybook for React, Angular, Vue.js, and Ember. Percy supports end-to-end testing frameworks including Cypress, TestCafe, Playwright, Selenium, and Nightwatch. The Percy Snapshot CLI tool is the easiest way to get started with Percy. It can also be used with static sites like those built with Jekyll.
As shown in the official documentation for Playwright, the Percy SDK adds an extra layer to the Playwright tests for taking screenshots. If you follow the steps in this tutorial after you run the tests, the screenshots are available on Percy. Then you can go through the review/approval flow.
You can run Percy locally, and it integrates with popular source code management tools like GitHub and GitLab. Similar to Applitools, Percy acts as a bridge between visual testing runs and the evaluation and management of results.
What You Need to Know About Visual Regression Testing
We hope this summary of what visual regression testing is and why to use it is a helpful resource. To recap, we highlighted three popular open-source tools that can do visual regression testing — namely, Selenium, Cypress, and Playwright. We also explored some SaaS offerings that provide an extra layer of test result aggregation and monitoring on top of existing open-source tools.
The different tool in the SaaS bucket is Meticulous, which doesn’t add an extra tier to the existing testing tools but provides a completely different way of doing visual regression testing with configuration files and replaying user traffic.
The choice of the right tool from the open-source or SaaS category will be determined by your team’s preferences. It will also depend on your budget and what your team wants to get out of visual regression testing. If the budget is low, then starting with an open-source tool and integrating it into your current CI/CD pipeline would be an option. If you have some money to spend, then rather than just building a layer on top of existing tools, consider the paradigm shift provided by Meticulous.
Choose a tool or service that fits your needs, fold up your sleeves, and get started with visual regression testing today!
Meticulous is a tool for software engineers to catch visual regressions in web applications without writing or maintaining UI tests.
Inject the Meticulous snippet onto production or staging and dev environments. This snippet records user sessions by collecting clickstream and network data. When you post a pull request, Meticulous selects a subset of recorded sessions which are relevant and simulates these against the frontend of your application. Meticulous takes screenshots at key points and detects any visual differences. It posts those diffs in a comment for you to inspect in a few seconds. Meticulous automatically updates the baseline images after you merge your PR. This eliminates the setup and maintenance burden of UI testing.
Meticulous isolates the frontend code by mocking out all network calls, using the previously recorded network responses. This means Meticulous never causes side effects and you don’t need a staging environment.
Learn more here.
Authored by Geshan Manandhar