Regression testing: the definitive guide

So, set aside 15 minutes to learn about regression testing best practices and how to implement them in a modern Agile context.

Table of contents:

1. What is regression testing?
2. How to do regression testing?
3. Main challenges
4. Regression testing automation
5. Regression test tools

What is regression testing?

Changes to the application code can introduce unintentional side effects called regressions. Often, they can affect the app’s functionality or performance.

Imagine changing a tire on your car making the steering much harder or breaking your air conditioner.

Regression testing is, therefore, aimed at finding issues caused by changes  to the code. It is critical for the overall stability and performance of your app.

In practice, regression testing means re-running a select number of old test cases to verify that the new functionality doesn’t break old features (not to be confused with re-testing which is simply running the same tests for the second time after a bug fix to ensure the defect was indeed removed).

Due to dependencies in the code, changing one part of your app can severely impact other components. A slight change can often cause a domino effect on other features and systems.

Without regression testing, it can be very hard to detect the root cause of a bug and fix it early in development when the cost of change is at its lowest.

This is exactly why regression testing is essential to assuring the quality of your product before its release.

At MindK, we perform regression testing each time we:

• Deploy new functionality;
• Implement a change request;
• Add a new module;
• Fix a defect;
• Improve performance;
• Refactor the code;
• Alter the configuration; or
• Swap the components.

All of these things might result in new bugs or issues.

Regression testing can save as much as one-third of time and money on fixing the bugs

Among other benefits of regression testing are:

+ Earlier detection of bugs caused by changes to the system.

+ Focus on building new features instead of reworking old defects.

+ Better UX without unintended side effects.

+ Less unexpected risks.

+ Improved system performance, etc.

Now let’s get from regression testing definition to its implementation.

Read more: all you need to know about different types of software testing.

How to perform regression testing?

The first step towards software regression testing is impact analysis.

A business analyst or a developer would sit down and look at the recently implemented module. They would examine how it integrates with other parts of the application or affect the current build in other ways.

They will also typically create a risk assessment matrix that evaluates each test case against the probability of regression and bug severity. Assessing the risks and drawing up mitigation plans can help you both prioritize test cases and prepare for possible contingencies.

The impact analysis will determine the test cases to be selected for the current and following regression test cycles. Typically, the regression suite will include cases that:

• Cover the core features;
• Are more apparent to the users;
• Have a high defect rate;
• Deal with integration testing;
• Undergo frequent changes;
• Succeeded previously;
• Failed previously; and
• Include boundary values.

QA specialists will then look at the test scenarios and try to calculate the time needed to run them.

Depending on their limitations, they will choose one of the three approaches to regression testing:

• Re-test everything

This is the most accurate regression method, yet the most expensive.
As re-testing your whole system requires a ton of time, this approach is best reserved for special occasions (or very small projects).

It only makes sense to re-test everything before a major release such as migrating your system to a new platform or rolling out a new localization.

• Select test cases

In most cases, it makes sense to limit your test scope to save time. One way is to re-test only the modified functionality and those parts of your app that are most likely to be affected by the recent changes.

In the process of regression test selection, the team divides all test cases into reusable (i.e. cases that can come in handy in subsequent cycles) and obsolete cases that can only be executed once.

• Prioritize test cases

You can prioritize every case according to their business impact, failure rate, frequency of usage, bugfix cost and start testing your cases one-by-one.

Go through as many top-priority cases as you can within your time limits.This way, you catch most regressions while keeping up with fast-paced Agile development

Regression testing usually starts with basic smoke tests meant to determine that the core functionality of the app is still working.

The QA team can then recede to more rigorous testing. To perform a regression test, you’ll need to:

1. Discover a modification in the app’s code and find the affected modules.
2. Pick the appropriate test cases and tools.
3. Define the entry point to make sure your app meets the entry criteria before the start of regression tests.
4. Determine the exit point with criteria that must be met before finishing the test.
5. Select components to be tested and design a test schedule.

Main challenges

Regression testing takes a lot of time and effort.

This is especially pronounced in Agile environments where changes happen fast and the team releases new updates every other week.

As QA engineers can’t test every single case, they need to be careful about test case selection. With each new update, the regression suite will grow in size as testers have to retest some features again and again while adding new test cases.

At some point, QA engineers can get too overwhelmed to execute the whole suite in time.

If your system experiences frequent changes (e.g. in Agile projects), the cost of manual regression tests will quickly escalate beyond acceptable levels. In such cases, test automation might be your only way to maintain quality without increasing the QA costs.

Automated regression testing

Manual regression testing is used on almost any product, especially early in development. Yet with time, it gets harder to maintain high test coverage.

The solution to this problem is, of course, automation. The larger your project is, the more sense it makes to automate regression testing.

Automated testing can take care of tedious repetitive tasks and free up time for in-depth exploratory testing.

As automation regression testing returns investments with each successive run, reusable test cases are the perfect candidate for automation (in addition to tasks that can’t be accomplished manually such as generating massive amounts of test data).

Regression testing can be accomplished on several different levels:

• Unit testing where developers test the smallest blocks of code called Units. These tests have the highest ROI and should form the backbone of your regression suite.
• Integration/API testing where different pieces of code are tested together to find inconsistent behavior. In modern applications, APIs represent the business logic layer. Automating regression tests at this level has a high ROI and allows you to validate the app’s behavior before its UI is ready.
• GUI testing where QA engineers validate different components by interacting with the user interface. It has a lower ROI and requires careful maintenance as GUI tests tend to break with changes to the app’s UI.

Deciding when to test and how often is one of the main challenges facing the QA team.

Regression testing should be a continuous process where previously completed cases are executed again and the results compared to previous runs. Keeping a regression schedule will allow you to focus on maintaining the high quality of your product instead of rushing to develop new features.

On our projects, regression tests are performed automatically after every commit as a part of a standard Continuous Integration/Continuous Delivery (CI/CD) pipeline.

Continuous Integration vs. Continuous Delivery/Deployment

To make use of CI/CD for regression testing in Agile environments, you need to:

• Impose best coding practices at the Unit level.
• Employ version control systems like Git.
• Set up a CI/CD server for building code.
• Write uniform build scripts you can configure for any machine you need,
• Automatically execute all layers of tests after each successful build.
• Run end-to-end & non-functional tests with automatic deployments to the testing environment.
• Occasionally refactor test scripts, especially at the GUI level.

Setting up a CI/CD pipeline for regression tests cuts down the amount of code rework and reduces the cost of fixing a bug.

Yet, regression tests are only as good as your scripts, automated or not. It’s the QA’s job to make sure they cover every possible variation and use flow. A poor design will likely result in insufficient requirements coverage, builds breaking due to false alarms, and lots of time wasted on adjusting the test cases.

Read more: how to save up to 75% on fixing bugs [7 benefits of automated testing]

Regression testing tools

Among the main criteria for choosing regression testing tools should be your tech stack, budget, ease of use, the tools features, and support options. Also, don’t forget to check whether it integrates with your CI/CD suite and project management software.

With that out of the way, let’s take a look at our toolset for automating regression testing.

From our own experience, we recommend building a custom test framework based on a free open-source tool (e.g. Selenium).

• At the Unit level, Mocha is our preferred JavaScript testing framework.

It supports both front-end and back-end testing and can be used with third-party assertions, mocking, and spying tools.

Mocha has many features that help writing descriptive auto-tests, reports, or even launch an automated regression test each time the file is changed locally.

Among its main benefits are:

+ Support for Behavior-Driven Development (BDD) and Test-Driven Development (TDD).

+ Lots of installation options.

+ Streamlined asynchronous testing.

+ Excellent integration with NodeJs.

+ Lots of assertion libraries.

+ Great variety of reporters and customization.

+ Support for all major browsers.

Read wore: why Mocha is our favorite JavaScript unit testing framework

• At the API level, SoapUI is by far our favorite automated regression testing tool.

It’s an excellent solution if you need advanced automation or complex test scenarios that involve RESTful or SOAP APIs.

The tool is super-easy to learn and has a convenient drag-and-drop UI that allows you to quickly create complex asynchronous tests.

And if you need more advanced features like assertions or workflows configuration, you can write your own code in Groovy.

The tool comes in two versions, a free open-source version and a Pro version that has better support and some additional features like an SQL query builder, a form editor, version control and XPath assertion wizard.

Alternatives: Postman, REST Assured.

Read more: the definitive guide to API testing

• At the GUI layer, we have a great choice of automated regression testing tools: WebDriverIO, Protractor, CodeceptJS, Puppeteer, and Cypress.

WebDriverIO is one of the most popular test automation frameworks for Node.JS. It has excellent stability, support options, and works with all major browsers.

It allows adding helper functions and complicated command sets and combinations in a streamlined fashion.

And if you want some custom functionality, the tool has community-built plugins for every imaginable situation.

Protractor is an end-to-end automation framework for Angular apps. It has multiple support options with tutorials, Q&A, and API docs with examples that are sometimes outdated.

The tool has cross-browser support with WebDriver API that uses native events and browser-specific drivers to simulate user interactions.

Protractor frees testers from the need to add sleeps and waits to their scripts. The tool automatically launches the next test step as soon as the page finishes the current tasks.

Cypress is an open-source front end testing tool that works with pretty much any web tech. It boasts great support, but can only work with the Chrome browser. It also has automatic waiting and a built-in debugger with a GUI.

What’s more, the tool takes snapshots during testing allowing you to hover over commands to view precisely what happened at each step.

Puppeteer is a Node.JS library used for browser manipulation (Chrome only via the DevTools Protocol).

CodeceptJS in a JavaScript end-to-end testing framework. It can work with WebDriver, Puppeteer, or any other driver. The tool has excellent test data management, test parallelization, and great reporting.

We also recommend using a test management tool like TestRail, Jira, or, at least, Excel when dealing with large regression suits.

Wrapping Up

Regression testing is essential in maintaining high quality, especially when the regression scope grows bigger late in development.

That’s why smart test case prioritization and automation are so important for Agile projects.

If you don’t have enough resources to keep up with development or to set up automated tests, it often makes sense to employ a third-party QA automation service.

External partners can shoulder testing for the previous Sprints and set up test automation so that your internal testers can focus on in-depth exploratory testing.

So, if you need experienced automation engineers, you can simply drop us a line.