Test automation

strategy in 2021
The ultimate checklist
Test automation is a cornerstone in DevOps, and when implemented correctly, it
helps increase output quality while containing costs. IT departments realize the
importance of having a test automation strategy instead of putting out fires here
and there.

Once you decide to roll out test automation, the next issue presents itself:

How are we actually going to

roll out test automation?
What’s the plan?
The automation test life cycle has the following phases:

Automation Test strategy Environment

feasibility analysis set up

Test result generation Test case execution Test case

and analysis development

Automation feasibility analysis

In this phase, you check the feasibility of automation. This
includes shortlisting the test cases for automation and
selecting the right test tool.

Test strategy
In the test strategy, you select the test automation
framework/tool. In this phase, you create a test plan and
test automation suite in your test management tool.

Test automation strategy in 2021 | 2

 Environment set up
In this phase, you set up the testing environment and
acquire the required hardware and software to execute
the automated test cases.

Test case development

In this phase, you start creating the automation test
scripts/flows. Make sure that your scripts/flows are
reusable, well-structured, and well documented.

Test case execution

In this phase, you execute your test scripts/flows.

Test result generation and analysis

This is the last phase of the test automation life cycle. In
this phase, you analyze the output of test cases and share
the reports with relevant stakeholders.

Remember that, as with anything in IT, you can either ‘save’ a few
days of planning and then spend weeks testing or putting off fires
later. Or you can allow yourself to spend a little time developing a test
automation strategy that will help you save valuable time during sprints.

For this reason, we’ve put together a checklist for creating a test
automation strategy. It consists of eleven items for you to consider as
you head out on your automation journey. You might be able to check
off some of these items already, while others will require some work -
perhaps even help from external consultants.

Test automation strategy in 2021 | 3

 1. Testing tools
Selecting the right tool is the key factor that guarantees
the success of your test automation strategy. It can make
or break your test automation project.

This is because no automation tool in the market works perfectly and

seamlessly for every company. Each organization has a different QA
team, and every QA team has a different set of challenges and works
with different applications in different environments. Therefore, when
choosing a test automation tool, you should pick one that meets the
needs of your specific project and your team.

In order to select the best fit for your project, you have to consider
other parameters such as the licensing cost of the tool, maintenance
cost, training and support, extensibility, and performance and stability.

Training materials

Ease of use
Support
Maintenance
CI/CD plugins

Licensing
REST API
Adoption
Costs Integrations

What to consider when

choosing a test automation tool

Applications supported Usability

Extensibility
Web apps
Mainframe
Performance
Desktop apps
Stability

Test automation strategy in 2021 | 4

 Once you have identified your testing requirements, you can shortlist
the tools that provide your required features.

There are plenty of cost-effective tools available in the market for

automating different kinds of applications like windows applications,
web applications, websites, mobile web applications, and native
mobile apps.

You have both paid and open-source options. The most popular
starting point for many is Selenium. Many people select Selenium
because it is free and readily available. However, Selenium requires a
lot of time, effort, and resources to succeed. The sharp learning curve,
the missing tech support, and the lack of reporting opportunities can
make it a time-consuming and expensive solution.

Read our Test Automation

Tools Comparison
Evaluate tools based on factors such
as adoption time, maintenance load,
supported applications, and ability to
scale.

Download here

Test automation strategy in 2021 | 5

2. Scope
Defining a project’s scope from an automation perspective
includes outlining timelines and milestones for each sprint in
the project. All team members (product owners, developers,
testers, etc.) should be on board with the scope.

At this stage, clearly define which tests to automate and which to keep
doing manually. A smart test manager understands that all test cases
cannot be automated. Instead, he manages the balance between
manual and automation testing. For this purpose, the test manager
needs to thoroughly analyze the test plan and shortlist the test cases
which need automation.

You can use the following criteria to shortlist the test cases for
automation:

• Which scenarios are tedious but can be automated?

• Which regression test cases should be automated?
• Can we automate the test data entry procedure?
• Which test cases are required to perform on cross platforms?
Can we automate those?
• Can we automate all kinds of validations in the application?
• Can we automate all CRUD (create, retrieve, update, delete)
operations in the application?

A rule of thumb is the 80/20 split:

80% Select 80% of the test cases that, if
automated, would reduce the risk of errors
happening during regression testing to an
acceptable level. The remaining 20% can then
be left for manual testing or not considered
part of the current regression suite.

Test automation strategy in 2021 | 6

3. Test automation
approach
When choosing a test automation approach, you must
consider three areas: processes, technology, and roles.

Process
A test automation roll-out must be a well-defined and
structured process. Make sure to cover the following in
your plan:

• When during the sprint should automated test cases

be developed?
• When are features ready for automated testing?
• Which features are tested manually?
• Who takes care of maintenance?
• How do we analyze results?

Technology
Identify the applications to be automated. Figure out
which technology they are based on and whether your
test automation platform supports these technologies.
In most cases, rolling out automation will involve several
application types: web-based, desktop-based, SAP,
mobile apps, etc. Therefore, it’s vital to have a tool that
can handle all your automation requirements.

Make sure that, if end-to-end testing is a requirement

for your company, you pick a tool that supports that.
Otherwise, you can always “code your way through” and
custom-build an integration that gives you the ability to
automate beyond one domain. However, beware that this
solution can sometimes become a complicated solution
that is hard to maintain.

Test automation strategy in 2021 | 7

You should also outline which kind of test automation
is needed. Unit and integration testing are usually an
integrated part of development practices, but there is a
long list of other test activities which can be automated.

Roles
Define the roles for automation in the agile team. Make
sure that all members know who is responsible for which
part of the automation project.

Examples of roles and responsibilities include:

• Automation lead
Responsible for coordinating and managing all
activities regarding automation in the project.
• Test case designer/reviewer
Similar to code reviews among software developers,
it’s important to establish a review process for
automated test cases. This means that a tester will
typically have at least two roles: test case designer
and test case reviewer.

Test automation strategy in 2021 | 8

4. Objectives
You need to clearly define some parameters against
which you can measure the success of your test
automation strategy.

You need to be specific while defining the target. Some common

goals in test automation are:

• Reducing testing time and cost

• Increasing the speed of the testing process
• Improving test coverage
• Reducing redundancy
• Improving quality
• Reducing manual intervention
For example, you can set a target of 30% faster time to market, 40%
reduction in testing cost, 70% faster regression testing. These should
always align with the broader company’s KPIs or OKRs.

The ultimate guide to reducing risk,

lowering costs, and driving value with
test automation

Read the guide

Test automation strategy in 2021 | 9

 5. Risk analysis
Risk analysis is an essential part of project planning, but it is
especially important to consider it within automation.

The analysis is done by creating a list of all identifiable risks qualified

with these details:

• Description and relevant metadata

• Severity: What will happen if the risk becomes a reality? How
hard will it hit the project?
• Probability: What is the likelihood of this happening?
• Mitigation: What can be done to minimize the risk?
• Cost estimate: What is the cost of mitigating the risk – what is the
cost of not doing it?

Note that a risk plan is a dynamic document; risks will be added and
removed to the list as the project evolves.

Examples of identifiable risks

Description Severity Probability Mitigation Estimate

We don’t have enough trained Contact partners or TBD

resources to create test auto- High Medium training providers.
mation cases. This will lead to Alternatively, prioritize
lower test coverage as more self-studies to train the
manual regression testing must team in automation.
be performed before release.
This might delay the release.

Test servers will not be able to Contact Operations and Approx.

keep up with the load from the Medium Medium make sure that the test $10,000
automated regression tests, servers are configured to
which will lead to a high number cope with the expected
of false failures in the reporting. load.

Test automation strategy in 2021 | 10

6. Test automation
environment
Organizations with a software department will more or less
have a well-defined method for how software is released
to production. This process usually includes one or more
test environments.

Test automation is a “deterministic game”; known inputs will produce

predictable outputs. This means that stable and predictable test
environments are a prerequisite for successful test automation.

Also, you must consider the data that is part of the tests:

• Where to store test data?

• Will it work to use a copy of production data?
• Can production data be masked?
• Should the test cases clean up data on their own after use?
Some release pipelines are mature and well-defined (i.e. a DevOps
pipeline), and the work towards fast releases has either already begun
or has been deemed irrelevant. In any case, it is essential to evaluate
the current state of your test environments and consider things such as
environment coverage, usage, agility, and availability.

Where to store
Coverage the test data?

Can you use a copy

Agility of production data?
Test
environment Data

Usage Should tests clean up

data after use?

Availability Can production data

be masked?

Test automation strategy in 2021 | 11

7. Execution plan
An execution plan should outline the day-to-day tasks and
procedures related to automation.

Pick the test cases to be automated based on the approach defined in

step #2. Before any automated test cases are added to the regression
suite, they should be run and verified multiple times to ensure they run
as expected. False failures are time-consuming, so test cases must be
robust and reliable.

Define a set of best practices that make test cases resistant to changes
in the system that’s being automated. These guidelines will depend
on the application in question, but they should cover how test cases
recognize and interact with elements in the application under test.

Execution of test cases should be handled either by the pipeline

orchestrator (Jenkins, TFS, Bamboo, TeamCity, etc.) or by a scheduling
tool. This means that regression tests will run either as part of a build/
deployment event or at a known time during the day.

Also, consider selecting a setup that allows for parallel execution of the
test cases to get the feedback from the regression tests faster back to
the development team.

Join Leapwork’s Best Practices Webinar Series

Once a month, we’ll cover a wide range of strategies to help
your testing efforts across web, desktop, and more.

Watch now

Test automation strategy in 2021 | 12

8. Test naming convention
Test naming convention is the easiest yet most powerful
way to create a good testing framework.

What a test naming convention should contain:

• Test case no/ID
• Feature/Module
• Answer to what does this TC verify
• Positive or Negative Test case (Optional)

Rules of test naming convention:

• A logical name should be given by considering the factors
mentioned above
• A common and standardized naming convention should be used
across all the test suite
• Should not be exhaustively descriptive
• If a few test cases are similar and with a few minutes difference,
then that should be explicitly clear from the name

Benefits of a good test naming convention:

• Easy to find test cases in hierarchy/folder structure by name,
action performed, or test case ID.
• If the test failed during execution, you should understand which
functionality is broken by just checking the test name instead of
reading through to understand the complete test case.

Remember, you can never test too much, and the combination of test
automation, reliable test cases, and scheduled/controlled execution will
always have a positive effect.

Test automation strategy in 2021 | 13

 9. Release control
In a release pipeline, regardless of its complexity and
maturity, there is a point when a team needs to decide
whether to release a build or not.

Parts of this decision-making can be automated, other parts, however,

still require a human touch, so the final release decision will often be
based on a combination of algorithm results and manual inspection.

In any case, make sure that the results from the test automation are
part of the release decision. Either decide only to allow releases if all
regression tests pass or have the lead tester approve the result.

After a complete run of regression tests, consider including the

application logs as part of the release decision. If the regression tests
have good application coverage, then any errors not related to the UI
should reveal themselves in the log files.

Release pipeline

Local Development Test Production

development

The results from both

automated and manual tests
play a key role in the release
decision

Test automation strategy in 2021 | 14

10. Failure analysis
Having a plan for how to analyze failing test cases and
the actions required to take afterward is a critical— and
sometimes neglected—part of a test automation strategy

The time it takes from the moment a tester is notified of a failing

test case until the fix is described, understood, and accepted in the
development backlog, is usually much longer than teams anticipate.
Having a well-defined process for this can save a lot of time and
frustration in a development team.

Also, make sure to have a result-oriented approach. The outcome of

each automation test run should be result-oriented and measurable.
This will give you confidence that you have a robust automation suite.

Different kinds of dashboards will help stakeholders to quantify and

qualify the software by using test metrics. This will also highlight the
quality standards of a particular software before it is launched in the
market.

Outline how different errors are handled, for example

Environment issues Raise a ticket with the DevOps team

A bug in the application under test Flag a bug for development

A bug in the automation scripts Create a task for the test team

Test automation strategy in 2021 | 15

11. Review and feedback
Finally, once you’ve made a draft of a test automation
strategy, make sure to have it reviewed and approved by
all members of the involved development team.

Enforce a culture for continuous learning and improvement. Include

and embrace feedback from stakeholders, peers, and all team
members working with automation. And adjust the strategy if
needed.

Lessons learned while automating software should be captured and

documented for future reference. Continuously improve your test
automation strategy by learning from those lessons.

On-demand webinar

How to build a successful

test automation strategy

Watch now