Codeless Test Automation: Calculating the ROI

So, we decided to experiment with codeless test automation on a live project in the Healthcare domain. Our goal was to assess the effectiveness of the approach and understand its actual value to the company. 

In this article, we’ll concentrate on the more critical metric – Return on Investment (ROI). We’ll also provide an Excel template you can use to evaluate test automation ROI for your organization. We will not be comparing functional aspects of traditional and no-code approaches.

What is automation testing and its benefits?

At its core, test automation involves using specialized software tools to execute pre-scripted tests on software before the release to production. This process validates the software functionality, ensuring that it meets your quality standards.

Test automation is grounded in repeatability, speed, and consistency. By automating repetitive testing tasks, teams can significantly reduce the time and effort required for manual testing. 

This allows the focus to be on more complex scenarios that require human intuition and creativity. Automated tests can be run multiple times during the development cycle. This ensures that new changes do not introduce unexpected issues – often called regression testing.

Test automation improves efficiency and test coverage.  It quickly evaluates a wide range of scenarios and edge cases you might miss in manual testing. This also fosters a more collaborative environment, enabling continuous integration and continuous delivery (CI/CD) practices that are essential for modern software development.

In short, there are four main benefits of automated testing: 

• Cost and resource saving. Replacing routine work for regression testing allows manual testers to get involved in more important and exciting activities.
• Fast feedback about system stability as tests are included in the CI/CD process or can be run manually at any time if needed.
• System stability with fewer regression defects in the prod environment. This also increases the end-user loyalty and reduces the maintenance.
• Efficiency and speed. Automated tests are faster than manual tests, allowing for quicker feedback and more frequent testing cycles, reducing the time to market.

These foundational principles serve as the backbone of both traditional and no-code test automation. Each has its own set of advantages, and understanding these will be key to making informed decisions about your testing strategy.

What is ROI in automation testing?

Return on Investment (ROI) is a fundamental concept in business that measures the profitability of an investment relative to its cost. In simple terms, ROI provides a clear picture of overall value by assessing expected financial gain from a particular investment.

It is particularly relevant when adopting new tools, processes, or methodologies, such as test automation. TA is a significant investment in both time and resources. However, when implemented correctly, it can produce considerable returns.

ROI calculation for automation testing involves weighing these benefits against the initial and ongoing costs. It becomes even more critical when comparing traditional and no-code test automation approaches, as each has its unique cost-benefit profile that can significantly influence the overall ROI.

How to calculate ROI for test automation?

The formula is straightforward:

ROI = (Net Profit / Total Investment Cost) x 100%

Net Profit is the total benefit or returns from the software minus the total costs, including:

• Increased Revenue: additional revenue generated as a result of the software selling.
• Cost Savings: reduction in operational costs due to process automation or improved efficiency;
• Intangible Benefits: improved customer satisfaction, better decision-making, competitive advantage, etc.

Total Investment Cost is the total cost incurred in the software development project, including:

• Development Costs: salaries of developers, costs of tools and technologies, infrastructure costs, etc.
• Operational Costs: costs of deployment, hosting, and maintaining the software.
• Training Costs: costs for training employees to use the new software.
• Opportunity Costs: potential revenue lost while focusing resources on the project.

The ROI test automation formula yields a % representing the investment’s efficiency and profitability. A positive ROI indicates a successful investment, while a negative ROI signals a loss.

Assumptions to consider

Understanding how to calculate ROI for test automation is crucial for making informed decisions that align with your organization’s objectives. The accuracy of these calculations relies on several key assumptions. They shape the analysis and significantly influence the final results.

• Project-related assumptions, such as its scope, complexity, and duration, are critical. The nature and scale of the work, along with the expected outcomes, will directly impact the potential benefits and returns.
• Assumptions about the team’s capabilities and readiness are equally important. A well-prepared and skilled team may lead to smoother implementation and a more rapid outcome.
• Product-market assumptions include potential revenue growth and intangible benefits like brand reputation and customer experience. A product that resonates well with its target market or offers unique value can generate a lot of value that’s not immediately quantifiable.

These assumptions are inherently context-dependent. The parameters used in your ROI calculation – whether they involve costs, savings, or revenue projections – must be tailored to your specific context. Understanding and accurately reflecting these factors ensures a more realistic and meaningful ROI calculation.

Here are the assumptions we used in the article:

• A brand new project in the Healthcare domain. ROI calculation for the first year of development.
• UI and API test automation (TA).
• Full-time Senior TA Engineer for traditional TA vs. a full-time Middle General QA Engineer for codeless test automation approach.
• Full-time involvement of both QA and TA engineers during the entire period.
• QA/TA engineers have enough experience with both technologies to perform testing effectively.
• No additional manual QA engineer involvement is needed for test documentation.
• One manual regression testing iteration and two smoke testing iterations per month to be replaced with automated tests.
• In the Healthcare domain, automated testing should both replace manual work, improve system stability, and secure product delivery.
• The regression testing scope will extend every month for as long as we ship new features.
• Release in six months. The support team will start processing the end user’s requests, and fix found defects.
• TypeScript and Playwright tech stack for the traditional test automation (TA) approach vs. one of the codeless test automation SaaS platforms for the no-code (NC) TA approach.
• Our calculations do not include one-time project set-up activities. Keep in mind that setting up the framework with reporting and CI/CD pipelines can take ~40 hours for TA vs. just a few hours for NC TA.
• Refactoring and maintenance costs are similar for both TA and NC TA approaches.

Test automation ROI calculation and comparison

With these assumptions in mind, we can finally proceed to ROI calculation for automation testing.

Step 1: compare test development efforts

In our pilot project, we developed a scope of UI E2E tests using the NC TA platform. We then estimated how much time it would take to develop this scope with the traditional TA approach.

This allowed us to compare the test development effort of both options.

As you can see, test development for the NC TA approach requires 3.18x less effort vs. the traditional TA approach.

Step 2: calculate the costs

To measure the cost of one hour of test development, we first need to evaluate the time spent on developing new tests and QA engineer activities. Here’s what the approximate numbers look like for an average month of QA work under the Scrum framework. 

In our calculations, we assume that both options require the same amount of time for refactoring and maintenance.

With a meaningful difference in test development time, these numbers might differ in the long run as well. On the other hand, we expect many more tests to be developed within the NC TA approach. This should increase the maintenance time. However, we don’t have enough data to assess this at the moment.

The only difference in our calculations is that Code review is not needed for the NC TA approach in the traditional sense.

You can correct these numbers according to your own development and testing process. Having this information, we can evaluate the actual cost of an hour of new test development.

Legend:

• QA Engineer Salary: the average salary of the Senior TA engineer and Middle General QA from the open sources.
• Rate: the hourly rate based on the salary.
• Tools and Licences Price: The monthly price of the average licenses needed and the NC TA platform;
• Infrastructure Costs: the average price of the pipelines and cloud usage needed for the testing.
• Training Costs: the average costs required for training employees to use the software (e.g., the other TA engineer reviews a pull request).
• Total test development hrs: number of hours in an average month from our previous calculations.
• Total test development hrs with k: number of hours multiplied by the coefficient from previous data (3.18).

Turns out, one hour of developing new tests that extend the testing scope is over 4x more expensive in the traditional TA vs. the NC TA approach.

Step 3: calculate the ROI for test automation

We finally have all the numbers needed to measure the ROI for both approaches and compare them. 

Legend:

• Development Costs: salaries of engineers and costs of tools and licenses.
• Infrastructure Costs: the average price of the pipelines and cloud usage needed for the testing.
• Training Costs: the average costs required for training employees to use the software.
• Cost Savings (QA): costs saved by replacing manual regression testing. This is the approximate number of hours multiplied by the QA engineer’s hourly rate.
• Cost Savings (Maintenance): costs saved on the bugfix during the development phase and customer support after the release.
• Increased Revenue: additional revenue generated as a result of software sales.
• Intangible Benefits: improved customer satisfaction, better decision-making, competitive advantage, etc.

As the results show, the annual ROI of the traditional TA approach is worse than that of the NC TA. 

In our framework, we left the “Increased Revenue” and “Intangible Benefits” categories empty as these numbers depend entirely on your context. Product managers could measure or forecast these numbers based on different metrics. Incorporating this data significantly changes the final ROI, likely making it positive.

Conclusions

Focusing on the ROI provides an assessment of test automation value in a way that is both practical and relevant to your specific needs. Working with either the traditional test automation methods or exploring the potential of no-code test automation tools, the framework explored in this article serves as a guide to help you make informed decisions.

The numbers and assumptions we’ve used can and should be adjusted to reflect your own project parameters, team dynamics, and product goals. This flexibility ensures that the ROI calculations remain accurate and meaningful to your particular situation.

This framework is not just a tool for evaluating current practices. It’s also a means to explore new possibilities, such as codeless test automation. By applying these principles, you can objectively assess whether emerging methods align with your goals and deliver the expected returns.

Want to calculate your project’s ROI? Open our test automation ROI calculation Excel template, copy the file, and adjust it according to your needs.

And if you need any assistance with automation testing, feel free to contact us for a free consultation.