Visual Regression Testing in a React App | Step-by-Step Tutorial

Visual regression testing in simple words is a mechanism to make sure that the changes made to a system, primarily code, do not adversely affect the visible appearance of the application’s user interface. In this post, you will learn how to do visual regression testing for a React.js application, let’s get going!

What is Visual Regression Testing

Regression means a return to a former or less developed state. Regression testing means to ensure that system changes in one part do not unintentionally break something that was working in another part of the system.

Along the same lines, visual regression testing means to verify that changes do not negatively impact the visual appearance of the existing user interface of the application. For example, an engineer changed a button used for the Login form but as the button was used for “Saving” things as well unintentionally broke the saving part.

Why do Visual Regression Testing

Similar to any other form of testing like unit testing or smoke testing, setting up visual regression testing helps you to catch any visual discrepancies much earlier in the development phase. Visual regression testing is usually done by comparing two screenshots and figuring out which elements look different than the previous screenshot and if it is intended.

A simple example can be, a profile picture that is expected to be 100x100 px on the desktop web browser, but due to an unintended change in CSS now the profile picture is 150x150 px. These kinds of visual issues can be caught and solved before it reaches the end users which is the main reason to use visual regression testing.

Prerequisites

In the next section, you will start dabbling with some code. So to do that, some of the prerequisites are as follows:

• Some experience with Node.js, React.js, and NPM CLI is expected.

• You should have a working knowledge of Git and GitHub including cloning a repository, creating and switching branches in Git, and a general understanding of pull requests.

• A basic experience with React.js is expected to understand the React.js code snippet.

Given you are aware of the skills needed to proceed, next the React application used to demonstrate visual regression testing in a React.js app will be discussed.

Example App

To show how to do visual regression testing on a React.js app, you will use the Name to Nationality guessing application. You can know more about how it is built in the Jest SpyOn post. It is a simple app where given the first name it tries to guess the Nationality percent of that name using the Nationalize.io API. You can view a quick working example for the name chris below:

It is also hosted on Netlify if you want to play around with the app and the code is also available on GitHub. With that context, next, you will learn how to add visual regression tests on a React.js app with Meticulous.

Visual Regression Testing with Meticulous

Before you proceed further, you should clone the Name Nationality application from GitHub and make sure you are on the apply-meticulous branch. Please also make sure that you have run npm install to install the packages. You can also execute npm start and check the application is running locally on port 3000.

Register on Meticulous

As the next step, register on Meticulous, use your Gmail, GitHub or Email to do so on the screen below:

That should be easy to do, then you will be presented with the onboarding. Select an organization name:

In the above picture, it is Geshan-Meticulous but you can choose it to be what you like as long as it is available. The consequent step in onboarding is to choose a Project name:

The name for the project is chosen as Name-Nationality as the example React app does that mapping with percent of nationality guessed for the name. After that, you will see the project page similar to the one below:

This is where the fun starts, in the next section you will install meticulous and create your first meticulous test. Get those fingers ready to copy paste and type some commands/code.

Install Meticulous and Bootstrap meticulous.json File

To install Meticulous on the Name-Nationality project you have cloned run the “Install Meticulous” command visible on the Project page, which will look like the below:

npm install --save-dev @alwaysmeticulous/cli @alwaysmeticulous/replayer @alwaysmeticulous/record

This example is of the npm version as the Name nationality app uses NPM, not yarn. The above command will install 3 Meticulous packages the CLI, replayer, and record as dev dependencies. These will be used later in this guide.

Next, you will create a meticulous.json file. This file will list all your test cases with their title, sessionId, and baseRepayId. You can create an empty meticulous.js file with the following command:

npx meticulous bootstrap

It will create an empty meticulous.json file and also show an output like the below:

Setting up meticulous.json... 
Setting up test:meticulous script...

The bootstrap command also adds a command to the package.json file which looks like this:

"test:meticulous": "meticulous run-all-tests --headless --parallelize --deflake"

It can run with npm run test:meticulous later to run all the Meticulous tests in headless mode.

Add your First Meticulous Test

To add your first meticulous test first make sure the Name Nationality app is running locally by executing npm start and making sure you see the form with the name field at http://localhost:3000 on your browser of choice.

After you are confirmed that the Name Nationality app is working on your local, create a happy path test by running:

npx meticulous create-test --apiToken=”<get-your-token-from-meticulous-project-page>”

You will find your API token on the meticulous project page, search for “Record a” on that page if you don’t find it. Copy that and replace it in the above command. The above command will open up an instrumented browser (Chromium) controlled by Meticulous. Next, type in https://localhost:3000 and in the name put in chris then

click ‘Get Nationalities”, once you see the flags, close the browser.

Consequently, you will be asked 3 questions, answer them as follows:

? Does the end state screenshot match your expectation? Yes
? Would you like to save this as a test to meticulous.json? Yes
? Test name: Flags show up correctly

The whole output of this command with the browser opening and closing will look like the below:

Creating a new Meticulous test
Step 1: record a new test
Preparing recording...
Opening browser...
Browser ready
Recording session: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/sessions/2022-09-03T12:12:39.095Z_vuQfgdDIxnWihFOLHQpg9
Step 2: validating the new test...
Starting simulation...
Replay done!
Simulation time: 8.508 seconds
Sending simulation results to Meticulous
Simulation artifacts successfully sent to Meticulous
=======
View simulation at: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/simulations/iiqHJzO-S1XFg2frx40Cu
=======
? Does the end state screenshot match your expectation? Yes
? Would you like to save this as a test to meticulous.json? Yes
? Test name: Flags show up correctly
Test saved to meticulous.json.

After this, if you open up the meticulous.json file you will see that the test has been successfully saved with the needed values. If you also visit the “Simulations” and “Session” links on the Meticulous’ project page you will see that these have been sent to Meticulous too as seen below:

At this point, your first meticulous test is saved to the meticulous.json file and also sent to the Meticulous servers as seen on the project’s page. The meticulous.json file looks like the following with the first test added:

{
  "testCases": [
    {
      "title": "Flags show up correctly",
      "sessionId": "2022-09-03T12:12:39.095Z_vuQfgdDIxnWihFOLHQpg9",
      "baseReplayId": "iiqHJzO-S1XFg2frx40Cu"
    }
  ]
}

Now to run the test again, without creating a new now, you can execute:

npx meticulous run-all-tests --apiToken="<get-your-token-from-meticulous-project-page>" --headless

The above command can be found in the “Commands” tab on the project’s page by searching for “Run all tests”. Please make sure the API token is correct. It will yield output like:

Test run URL: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/test-runs/Ex6wxaJiv0z3T1cDpfINC

Starting simulation...
Replay done!
Simulation time: 9.173 seconds
Sending simulation results to Meticulous
Simulation artifacts successfully sent to Meticulous
=======
View simulation at: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/simulations/ptvPReRd1Cq2Yq5lnPWz0
=======
Diffing final state screenshot against replay iiqHJzO-S1XFg2frx40Cu
0% pixel mismatch (threshold is 1%) => PASS
View screenshot diff at https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/simulations/ptvPReRd1Cq2Yq5lnPWz0/diff/iiqHJzO-S1XFg2frx40Cu

Results
=======
URL: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/test-runs/Ex6wxaJiv0z3T1cDpfINC
=======
Flags show up correctly => pass

Hurray! Your first Meticulous test has passed and the good thing is till this point you have not written a single line of code. As seen above the was a 0% pixel mismatch between the saved screenshot of the simulation and your local code. As mentioned by Alister Scott, “Never Trust an Automated Test You Haven’t Seen Fail”. Next, you will make the test fail by making a change. The change is to make the flags smaller. You will make the flags 50px wide.

To do this, you will add width: "50px" on line 58 of src/App.js file which it will look as follows:

<div className="nationalities">
            {Array.isArray(nationalities) && nationalities.map(
              nationality => {
                const flagUrl = `https://flagcdn.com/w160/${nationality.country_id.toLowerCase()}.jpg`;
                const altText = `${nationality.country_id} flag`;
                return <div key={nationality.country_id}><h3>{nationality.country_id} - {(nationality.probability * 100).toFixed(2)}%</h3> <img src={flagUrl} alt={altText} style={{
                  border: "1px solid black",
                  width: "50px"
                }} /></div>
              }
            )}
</div>

After you save the file when you rerun the test with the npx meticulous run-all-tests command in headless mode providing the API key like:

npx meticulous run-all-tests --apiToken="<get-your-token-from-meticulous-project-page>" --headless --appUrl="http://localhost:3000/"

It will fail with the following output:

Test run URL: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/test-runs/oROsG3mdGSyuS69hWYt8D

Starting simulation...
Replay done!
Simulation time: 10.085 seconds
Sending simulation results to Meticulous
Simulation artifacts successfully sent to Meticulous
=======
View simulation at: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/simulations/Izp3KA5jCPUD8gb05gflV
=======
Diffing final state screenshot against replay iiqHJzO-S1XFg2frx40Cu
4% pixel mismatch (threshold is 1%) => FAIL!
View screenshot diff at https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/simulations/Izp3KA5jCPUD8gb05gflV/diff/iiqHJzO-S1XFg2frx40Cu
Screenshots do not match!

Results
=======
URL: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/test-runs/oROsG3mdGSyuS69hWYt8D
=======
Flags show up correctly => fail

Here the main message is 4% pixel mismatch (threshold is 1%) => FAIL! here by the test has failed. If you check the app locally on port 300 after the 50 px style change it will look as follows:

Now it has smaller flags than earlier. So how can this be fixed?

Fixing a Broken Visual Regression Test

To fix the tests, you will not need to write any more code. The test can be fixed easily by running the following command:

npx meticulous update-tests --apiToken="<get-your-token-from-meticulous-project-page>" --testRunId="<test-run-id-from-previous-failing-run>" --acceptAll

The command updates the tests and accepts the new replay result as passed. The testRunId is jZK8Fl_yWjZ-G5WxOJWvU in my case which was taken from the above shown previous run.

It will also replace the “baseReplayId” in the meticulous.json file with the correct simulation id from the new test run. In my case, after running the update-tests with --acceptAll, it looked like the below:

{
  "testCases": [
    {
      "title": "Flags show up correctly",
      "sessionId": "2022-09-03T12:12:39.095Z_vuQfgdDIxnWihFOLHQpg9",
      "baseReplayId": "LYYwD2jJILt-GwnSy98vi"
    }
  ]
}

At this juncture, you can run the tests again to see if it will pass now. To run the tests again, you can execute:

npx meticulous run-all-tests --apiToken="<get-your-token-from-meticulous-project-page>" --headless --appUrl="http://localhost:3000/"

Because the test has been updated, it should pass showing an output like the below:

Test run URL: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/test-runs/KiKq-ZDtEoCTjRE5shDOj

Starting simulation...
Replay done!
Simulation time: 9.579 seconds
Sending simulation results to Meticulous
Simulation artifacts successfully sent to Meticulous
=======
View simulation at: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/simulations/e7aC3WC2DICnORgFX3U4N
=======
Diffing final state screenshot against replay LYYwD2jJILt-GwnSy98vi
0% pixel mismatch (threshold is 1%) => PASS
View screenshot diff at https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/simulations/e7aC3WC2DICnORgFX3U4N/diff/LYYwD2jJILt-GwnSy98vi

Results
=======
URL: https://app.meticulous.ai/projects/Geshan-Meticulous/Name-Nationality/test-runs/KiKq-ZDtEoCTjRE5shDOj
=======
Flags show up correctly => pass

Great! Your failing test is passing and now it can be pushed to your favorite source control management software like GitHub.

Conclusion

In this step-by-step guide, you first learn about what is visual regression testing. After that, you looked at adding a meticulous test to an existing React.js application. Then, you changed the code which failed the added test.

Consequently, you updated the tests to match the code changes that passed the first test. You can look at the steps followed to do it all in this branch comparison on GitHub.

The main takeaway from this post is, that it is a great idea to add visual regression testing on any app React.js included. With Meticulous, changing the test when the code is updated is so easy with the provided commands in the easy-to-use Meticulous CLI.

Meticulous

Meticulous is a tool for software engineers to catch visual regressions in web applications without writing or maintaining UI tests.

Inject the Meticulous snippet onto production or staging and dev environments. This snippet records user sessions by collecting clickstream and network data. When you post a pull request, Meticulous selects a subset of recorded sessions which are relevant and simulates these against the frontend of your application. Meticulous takes screenshots at key points and detects any visual differences. It posts those diffs in a comment for you to inspect in a few seconds. Meticulous automatically updates the baseline images after you merge your PR. This eliminates the setup and maintenance burden of UI testing.

Meticulous isolates the frontend code by mocking out all network calls, using the previously recorded network responses. This means Meticulous never causes side effects and you don’t need a staging environment.

Learn more here.

Authored by Geshan Manandhar