Visual regression testing in simple words is a mechanism to make sure that the changes made to a system, primarily code, do not adversely affect the visible appearance of the application’s user interface. In this post, you will learn how to do visual regression testing for a React.js application, let’s get going!
What is Visual Regression Testing Regression means a return to a former or less developed state. Regression testing means to ensure that system changes in one part do not unintentionally break something that was working in another part of the system.
Along the same lines, visual regression testing means to verify that changes do not negatively impact the visual appearance of the existing user interface of the application. For example, an engineer changed a button used for the Login form but as the button was used for “Saving” things as well unintentionally broke the saving part.
Why do Visual Regression Testing Similar to any other form of testing like unit testing or smoke testing , setting up visual regression testing helps you to catch any visual discrepancies much earlier in the development phase. Visual regression testing is usually done by comparing two screenshots and figuring out which elements look different than the previous screenshot and if it is intended.
A simple example can be, a profile picture that is expected to be 100x100 px on the desktop web browser, but due to an unintended change in CSS now the profile picture is 150x150 px. These kinds of visual issues can be caught and solved before it reaches the end users which is the main reason to use visual regression testing.
Prerequisites In the next section, you will start dabbling with some code. So to do that, some of the prerequisites are as follows:
Some experience with Node.js, React.js, and NPM CLI is expected. You should have a working knowledge of Git and GitHub including forking a repository, creating and switching branches in Git, and a general understanding of pull requests. Having some experience with Vercel will be good to have, it is not compulsary to use Vercel. A basic experience with React.js is expected to understand the React.js code snippet. Given you are aware of the skills needed to proceed, next the React web application used to demonstrate a visual regression test in a React.js app will be discussed.
Example App To show how to do visual regression testing on a React.js app, you will use the Name to Nationality guessing application. You can know more about how it is built in the Jest SpyOn post. It is a simple app where given the first name it tries to guess the Nationality percent of that name using the Nationalize.io API. You can view a quick working example for the name chris below:
It is also hosted on Vercel if you want to play around with the app and the code is also available on GitHub . With that context, next, you will learn how to add visual regression tests on a React.js app with Meticulous. You should know that Meticulous is a useful visual regression testing tool that enables you to add visual regression testing without the need to maintain any code. It also provides an advanced form of browser automation that records your actions and replays them to test for any visual differences catching unwanted visual regressions.
Visual Regression Testing with Meticulous Before you proceed further, you should clone the Name Nationality application from GitHub and make sure you are on the master branch. You can follow the official guide from GitHub to fork the repository to your GitHub account. Please also make sure that you have run npm install to install the packages. You can also execute npm start and check the application is running locally on port 3000.
After that, you can register on Meticulous and follow along with this guide as shown in the next section.
Register on Meticulous As the next step, register on Meticulous, use your Google account, GitHub or Email to do so on the screen below:
That should be easy to do, then you will be presented with the onboarding. Select an organization name:
In the above picture, it is Meticulous-try but you can choose it to be what you like as long as it is available. The consequent step in onboarding is to connect to GItHub or GitLab:
Choose GitHub and add the Meticulous app on GitHub after authenticating it as follows:
After that, use the name-nationality-meticulous repository that you had forked before registering on Meticulous:
The name for the project is chosen as name-nationality-meticulous same as the repo name as seen below: Then you can click “Create Project”.
After that, you will see the page to select the framework, as the app is built with React.js (Create React App) choose the “Another Framework” option as you are using React.js and click “Next”:
Then you will see the following screen to install the Meticulous recorder NPM package and add the script to load the Meticulous recorder. As you have forked and existing codebase, you don’t need to install the @alwaysmeticulous/recorder-loader NPM package. It is already included in the package.json file .
You will only need to copy the projectId and set it as an environment variable named REACT_APP_METICULOUS_PROJECT_ID. For this copy the .env.example file to .env.local and replace the above variable in the new .env.local file.
After you have set the project ID on your .env.local file, you can click the “Next” button on the page.
At this point, you will need to run the app locally with npm start and search for the name “chris” on the app when it runs at https://localhost:3000. It should look something like the below:
Then stop the app by pressing Ctrl+C on the shell/CLI you opened the app in. This will stop the recording. Head back to the Meticulous page, you will see that your first session has been recorded:
In the next section, you will need to configure more things on Meticulous when you click “Next” on the above screen.
More Meticulous configurations After the first recording is added, you will need to configure how to execute your tests. On each new pull request, the above recording will be tested against any code change for any visual regression. When you see a screen like the one below, choose the “Run the tests against my existing preview URLs(recommended)” option. Then click “Next”.
Then you will need to enable the settings page of the project, you can reach that page by clicking the “Go to projects page” button.
On the project overview page, you can click the “Settings” link and check the “Automatically run Meticulous tests whenever a new deployment id is ready” option.
At this juncture, you will need to deploy your app to Vercel to run the visual regression tests on each preview which is automatically deployed for each pull request. You will do that in the next section.
Deploy on Vercel Next, you will deploy the name nationality app to Vercel. Please note that Vercel is not a requirement to use Meticulous, it is used for this example. You can use Meticulous without Vercel too.
To deploy the app to Vercel, you will need to register on Vercel if you don’t have an account. It is very easy to sign up for Vercel as seen below:
Use the “Hobby” option and type in your name. Then click continue, after that, you can continue the registration with GitHub.
Follow the steps and be sure to add the name-nationality-meticulous repository that you had forked in the beginning.
After the registration is done, You can go to the “Overview” and click “Add New…” > “Project”.
Then on the next page, select the repository you have forked name-nationality-meticulous and click the “import” button as seen below:
After that, be sure to add the two environment variable before deploying the project:
REACT_APP_METICULOUS_PROJECT_ID - the same value you used in the .env.local file that you copied from the Meticulous UI. REACT_APP_RUN_VISUAL_TESTS - set this env variable to true as seen below. With this environment variable if you want to test things out on preview without it being recorded on Meticulous you can easily do it if you want to record your activity on preview set it to true. You can set it to false` and your activity on Vercel preview URL will not be recorded. Then you can click the “Deploy” button to deploy your application.
Wait for a couple of minutes and your React App to guess the nationality from the given name will be successfully deployed on Vercel.
At this point, create a new branch from the master branch named readme-change, then make a one-word change to the Readme file and push it to Github. After that open a PR with this readme-change branch. If your recordings are set fine, the test will run for guessing the nationality for chris, and you will see the visual regression tests passing on Meticulous as below:
Hurray! Your first Meticulous test has passed and the good thing is till this point you have not written a single line of code or test code. As seen above there were “zero visual differences” between the saved screenshot of the simulation and the change in your branch.
Add visual difference and check the result As mentioned by Alister Scott , “Never Trust an Automated Test You Haven’t Seen Fail”. Next, you will make the test fail by making a change. The change is to make the flags smaller. You will make the flags 50px wide on a new git branch from the master branch called smaller-flag. To do this, you will add width: ”50px” on line 58 of src/App.js file after which it will look as follows:
<div className="nationalities">
{Array.isArray(nationalities) && nationalities.map(
nationality => {
const flagUrl = `https://flagcdn.com/w160/${nationality.country_id.toLowerCase()}.jpg`;
const altText = `${nationality.country_id} flag`;
return <div key={nationality.country_id}><h3>{nationality.country_id} - {(nationality.probability * 100).toFixed(2)}%</h3> <img src={flagUrl} alt={altText} style={{
border: "1px solid black",
width: "50px"
}} /></div>
}
)}
</div>
After you save the file, you will need to commit and push the smaller-flag branch to GitHub. Then you can open a pull request to master from the smaller-flag branch. The visual testing on Meticulous will fail at this point:
Here the main message is Meticulous spotted visual differences thereby the test has failed. Another important detail to notice here is that the unit tests running with Jest on GitHub actions passed and did not catch the visual difference, whereas the Meticulous tests failed. They failed because they are visual difference tests and the flag becoming smaller is surely a visible difference.
You check the app locally on port 300 after the 50 px style change. First, make sure the REACT_APP_RUN_VISUAL_TESTS variable in the .env.local file is set to false so that you are not recording the actions. When you run your app after that and try chris in the name text field, it will look as follows:
Now it has smaller flags than earlier. So how can this be fixed?
Fixing a broken visual regression test To fix the tests, you will not need to write any more code. The test can be fixed easily by clicking on the view and approve differences detected. link on the comment added by the Meticulous app on your pull request of the smaller-flags branch. That link will show you all the visual differences you have created from the baseline tests as follows:
If this visual difference of making the flag smaller is intended, you can click the “Approve all visual differences” green button and that will change the baseline tests. The baseline tests after this action will be the ones with smaller flags.
As the visual differences are approved, you can merge the Pull request. After that, the pull requests after this one will use the smaller flags as the baseline.
Great! Your failing test is passing and now it can be pushed to your favorite source control management software like GitHub.
Conclusion In this step-by-step guide, you first learn about what visual regression testing is. After that, you looked at adding a meticulous test to an existing React.js application. Then, you changed the code which failed the added test.
Consequently, you updated the tests to match the code changes that passed the first test. You can look at the steps followed to do it all in this branch comparison on GitHub.
The main takeaway from this post is that it is a great idea to add visual regression testing on any app React.js included. With Meticulous, changing the test when the code is updated is so easy with the provided commands in the easy-to-use Meticulous CLI.
Meticulous Meticulous creates and maintains an exhaustive suite of e2e ui tests with zero developer effort.
This quote from the CTO of Traba sums the product up best: "Meticulous has fundamentally changed the way we approach frontend testing in our web applications, fully eliminating the need to write any frontend tests. The software gives us confidence that every change will be completely regression tested, allowing us to ship more quickly with significantly fewer bugs in our code. The platform is easy to use and reduces the barrier to entry for backend-focused devs to contribute to our frontend codebase."
This post from our CTO (formerly lead of Palantir's main engineering group) sets out the context of why exhaustive testing can double engineering velocity. Learn more about the product here.
