Agile Methodology: The Complete Guide to

Understanding Agile Testing

Over the past several years, a new way of creating software has taken the software
development and testing world by storm: Agile. In fact, as of 2018, 97% of
organizations practice agile in some form. However, respondents report that this
adoption is not always widespread within their organizations, which means there’s still a
long way to go in terms of adoption and maturity.

So what exactly is agile and why has it become so popular so quickly?

Let’s explore exactly what agile methodologies entail and how to introduce it in your
organization in more detail. Specifically, we’ll cover:
 How does testing fit into agile methodologies?
 What are the different ways to test on an agile team?
 What’s next for the agile movement?

About Agile Methodology

Agile methodology has taken the software development world by storm and rapidly
cemented its place as “the gold standard.” Agile methodologies all started based on four
core principles as outlined in the Agile Manifesto. These methodologies are rooted in
adaptive planning, early delivery and continuous improvement, all with an eye toward
being able to respond to change quickly and easily.

What does it mean to Test on an Agile Team?

Agile principles are all about being collaborative, flexible and adaptive. It’s built on the
premise that the world now changes regularly, and that means software teams no
longer have years to bring new products to market. In that time, competitor offerings or
customer expectations can change, and the team risks irrelevance. Agile minimizes this
risk by helping teams collaborate together more by adapting to what the team needs to
be successful. It does this by encouraging teams to regularly show off their work and
gather feedback so that they can adapt to change quickly. Narrowing in on testing, the
fast pace of agile development has led to several imperatives for testers:
1. Prioritizing requirements based on risk since it’s not possible to test everything
2. Automating tests to increase efficiency
 3. Increasing the use of exploratory testing to accelerate the time from code
delivery to test completion and to emphasize the need to create code that works
4. Adapting to changes from sprint to sprint - The fourth imperative — adaptability
— is particularly important because it requires testers to have broader, cross-
functional testing skills, which represents a departure from the narrower testing
skills often needed in a waterfall environment. Additionally, unlike in a waterfall
environment, testers following an agile methodology need to stay in close contact
with developers to collaborate on testing throughout the entire software
development lifecycle. In waterfall methodologies, there is often a large
requirements document that testers test against. That document doesn’t change
frequently, so testers have been able to work fairly independently of developers.
However, most Agile methodologies are light on documentation and the
requirements for a new feature may only be in a ticket in a requirements tracking
system without all the edge cases listed. The testers in these scenarios need to be
highly communicative with the development and business teams, as tests they
wrote a few weeks ago may become obsolete quickly. To succeed, testers need to
be flexible and able to adapt to moving targets.

To succeed, testers need to be flexible and able to adapt to moving

targets.

In general, there are four central tenets of the Agile Manifesto that are important for
testers to remember:
1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Responding to change over following a plan
4. Collaborating with customers over contract negotiation
The bottom line in all of this is that everyone — testers, developers and beyond — must
evolve in order to embrace an agile style of working.

Agile is Not One-Size-Fits-All

Every organization is unique and faces different internal factors (i.e. organization size
and stakeholders) and external factors (i.e. customers and regulations). To help meet
the varying needs of different organizations, there are various agile methodologies and
several different types of testing you can do while working within one of those agile
methodologies. Which mix is right for your team will depend on your internal and
external factors, needs and goals?

Let’s take a look at what some of the most popular agile methodologies and
testing methods entail, including:
 Agile Methodologies
o Scrum
o Kanban
 Testing Methods
o Behavior Driven Development (BDD)
o Acceptance Test Driven Development (ATDD)
o Exploratory Testing
o Session Based Testing

1 Agile Methodology Types

1.A Scrum
What is it? One of the most popular software testing methodologies, Scrum takes a
highly iterative approach that focuses on defining key features and objectives prior to
each sprint. It is designed to reduce risk while providing value quickly.

Scrum starts with a requirement or user story that outlines how features should perform
and be tested. The team then cycles through a series of sprints to provide small bursts
of value quickly. To help the team work in this flexible way and avoid shifting priorities,
Scrum requires that questions be answered from the very start.

How is it different from Waterfall? Whereas Waterfall includes several testing and
bug fixing cycles before releasing a product, Scrum is far more collaborative and
iterative. One of the biggest differences is that Waterfall calls for heavy documentation
early on. This documentation makes it harder to change features as the process goes
on, which may be a negative in some environments (such as consumer-grade software)
and a positive in others (such as those where the team is trying to launch a rocket,
since no one wants requirements for something dangerous shifting frequently). That
said, you might think of Scrum like many “mini waterfalls,” since requirements are well
defined at the start of each sprint and shouldn’t shift within it. The difference is that
detailed requirements for the next sprint are not set months in advance.

Scrum calls for more regular collaboration between testers, developers and BAs,
typically in the form of daily stand ups and sprint retrospectives, to ensure proper
communication and alignment. Additionally, there is a Scrum Master who helps keep
the project on task by removing blockers from the team to make sure they can be most
effective. The Scrum Master can be anyone on the team, such as a developer or a
tester.

What does adoption entail? Scrum offers one of the easiest transitions for teams
coming from a Waterfall environment because it’s time-based with sprints and releases
can still be planned out in advance. That said, it does call for faster iterations and
stronger collaboration.

Who is it for? Because of its fast iterations, Scrum is best suited for teams whose
customers and stakeholders want to be actively involved by regularly seeing working
products at showcase meetings. This collaboration allows the team to make changes for
upcoming showcases. Key team members who should be involved when taking a
Scrum approach include:
 Product Owner
 Scrum Master
 Developers
 Automation Engineers
 Testers
 Stakeholders

What are best practices? In addition to strong communication, collaboration and

adaptability, other best practices for testers following a Scrum methodology include:
 Determining acceptance criteria based on communication (typically in the form of
a user story) from a sales representative or customer (note: this direct connection
should help reduce miscommunications)
 Using the acceptance criteria to develop code and ensuring team approval of that
code
 Testing the code in sandbox-like environments as well as production-like
environments prior to deploying it to production

1.B Kanban
What is it? Kanban is a very simple Agile based methodology rooted in manufacturing
(it was developed by Toyota to help increase productivity in factories). At its core,
Kanban can be thought of as a large, prioritized to-do list. Like with Scrum,
requirements in Kanban are tracked by their current stage in the process (to-do, in
development, in test, done).
Unlike scrum, Kanban is not time-based. Rather, it is based solely on priority.
When a developer is ready for the next task, he/she pulls it from the to-do list. Since
there are fewer planning meetings, this approach means the team needs to be
extremely close. In this type of environment, if developers work much faster than the
testers, bottlenecks will crop up. In these situations, anyone on the team should jump in
and help in different areas. Of course meeting this need requires a great deal of
flexibility and adaptability.

How is it different from Waterfall? Kanban still has requirements like Waterfall, but
the requirements can change as the testing team doesn’t start to think about testing
each requirement until the developer selects it from the top of the backlog. In contrast,
Waterfall is heavily time-based with a lot of overhead in planning. The heavy planning
that comes in a Waterfall environment is great in some cases, like when building
expensive things, but it’s not always necessary. With Kanban, releases still get planned,
but teams usually don’t promise anyone features by certain dates unless the item in
question is near the top of the backlog.

What does adoption entail? Kanban offers a simple transition for the right teams. To
make a smooth transition to Kanban, business analysts, developers, testers and
stakeholders should sit close together and communicate regularly. When transitioning to
Kanban, it’s important to remember that this methodology offers the quickest way to
bring code to production, but the code is likely to have some technical debt. That’s
because developing without always knowing what’s next doesn’t necessarily lend itself
to producing the most reusable code.

Who is it for? Kanban is best suited for small teams or teams that don’t produce
features for the public and/or promise certain dates for releases. Additionally, it is a top
methodology of choice for any products or teams focused primarily on maintenance
work since bugs are not always straightforward and often require research to resolve,
which makes time management challenging. Teams that cannot minimize the amount of
planning for issues are likely better off following a Scrum or Waterfall methodology.
Key team members who should be involved in a Kanban environment include:
 Product Owner
 Project Manager
 Developers
 Automation Engineers
 Testers
What are best practices? In addition to maintaining visibility and prioritizing
collaboration, best practices for testers following a Kanban methodology include:
 Keeping very open lines of communication between the business owners,
developers and testers
 Ensuring the team has flexibility to take on other roles outside of their core
responsibilities in order to help clear bottlenecks
 Making everyone an owner of the product so that they care fully about the result

2 Agile Testing Methods

2.1 Behavior Driven Development (BDD)
What is it? Many people have heard of or used Test Driven Development (TDD). For
example, developers use TDD to write unit tests to fail before code has been written.
BDD is based on the same principles as TDD, but instead of unit tests, it calls for higher
level tests at the business level. Instead of starting with a technical-facing unit test as
TDD does, BDD starts with an initial requirement based on end user behavior and calls
for tests that are “human readable” and can even replace some requirements
documentation. This requirement is based on behaviors that the product should exhibit,
creating an airtight guide for engineers to use as they develop tests.

Specifically, BDD starts with a functional specification using the Gherkin

Given/When/Then syntax. This specification then guides developers, testers and
product owners who move across features. As they do, they use automated test
functions to determine completeness, refining code until it passes the test, much like in
the TDD approach except at the team level. To ensure the test passes (and it typically
requires multiple tries), the developer should only re-factor code, not add any new
functionality.

In sum, BDD requires a “smart” automation strategy that drives a high level of efficiency.
This strategy sets BDD apart from other Agile methodologies.

How is it different from standard Waterfall testing? BDD is extremely different from
standard Waterfall testing because the former requires test cases to be written early
against requirements and calls for those tests to be executed toward the end of the
development cycle. With BDD in an Agile environment though, tests are not based on
requirements and testing happens in conjunction with development of the features.
Additionally, within a Waterfall methodology, testers are the ones writing the test cases.
A BDD approach, on the other hand, lends itself to the business owners writing the
tests. This switch reduces communication (or miscommunication) between the business
analysts, developers and testers.

What does adoption entail? Changing to a BDD methodology can be challenging

when the team is used to a traditional style of testing. It requires a BA or tester to write
tests up front and for the developers to write the test specification in code to match. It’s
a new type of coordination within the team, but it’s extremely positive in that the team
works together as one unit, business users included.
Who is it for? The BDD methodology is ideal for teams that work on feature-focused
software and/or teams that put the user experience first. Key team members who
should be involved in a BDD environment include:
 Product Owner/Business Analyst
 Project Manager
 Developers
 Automation Engineer/Testers

What are best practices? Best practices for testers following a BDD methodology
include:
 Streamlining documentation in order to keep the entire process lean
 Embracing a “three amigos” model where the product owner, developer and
tester form a cohesive team
 Using a test framework like Cucumber to define criteria
 Building automated tests in a way that make them easy to reuse as much as
possible
 Having business analysts learn the Gherkin syntax and write test cases directly

2.2 Acceptance Test Driven Development (ATDD)

What is it? ATDD is like BDD in that it requires tests to be created first and calls for the
code to be written to pass those tests. However, unlike in TDD where the tests are
typically technical-facing unit tests, in ATDD the tests are typically customer-facing
acceptance tests.
The idea behind ATDD is that user perception of the product is just as important as
functionality, so this perception should drive product performance in order to help
increase adoption. To bring this idea to life, ATDD collects input from customers, uses
that input to develop acceptance criteria, translates that criteria into manual or
automated acceptance tests and then develops code against those tests. Like TDD and
BDD, ATDD is a test-first methodology, not a requirements driven process.

Also like the TDD and BDD methodologies, ATDD helps eliminate potential areas for
misunderstanding by removing the need for developers to interpret how the product will
be used. ATDD goes one step further than TDD and BDD though because it goes
directly to the source (aka the customer) to understand how the product will be used.
Ideally, this direct connection should help minimize the need to re-design features in
new releases.

How is it different from standard Waterfall testing? ATDD is different from standard
Waterfall testing because it is a test-first methodology. Standard Waterfall testing calls
for test cases to be written upfront based on requirements, whereas ATDD is not a
requirements driven testing process.

What does adoption entail? Because ATDD represents such a departure from
traditional methods, going from one to the other isn’t easy for teams to do. To be in the
best position to adopt an ATDD methodology, teams need to get stakeholder buy-in,
which can prove challenging at times.

Who is it for? Due to its emphasis on user perception, ATDD is best suited for teams
that are focused on the user experience, have goals around high rates of adoption and
want to minimize the number of feature changes in future releases. Key team members
who should be involved in an ATDD environment include:
 Customer/Customer Advocate
 Developer
 Product Owner/Business Analyst
 Automation Engineer/Testers
  Project Manager

What are best practices? Best practices for testers following an ATDD Agile
methodology include:
 Interacting closely with customers, for example through focus groups, in order to
determine expectations
 Leaning on customer-facing team members, such as sales representative,
customer service agents and account managers, to understand customer
expectations
 Developing acceptance criteria based on customer expectations
 Prioritizing two questions:
o Will customers use the system if it does X?
o How can we validate if the system does X?

Exploratory Testing
What is it? Next we have exploratory testing, which is actually a type of functional
testing but is important in Agile environments. Exploratory testing gives testers
ownership over the code to test it in an organized, chaotic way. In this case, testers are
not following test steps, but rather using the software in standard or clever ways to try to
break it. Testers will document defects as usual, but detailed documentation of what
and how the application was tested is not always provided.

Exploratory testing is not scripted. Rather, it’s about developing the best tests based on
each unique piece of software. Because of its unscripted approach, exploratory testing
often mimics how users will interact with the software in real life.

Across the board, exploratory testing follows four key principles:

1. Parallel test planning, test design and test execution
2. Specific yet flexible
3. Aligned toward investigation of potential opportunities
4. Knowledge sharing

How is it different from standard Waterfall testing? Exploratory testing can actually
be done in both Waterfall and Agile environments, but the tight integration between
testers and developers in an Agile environment helps ease any bottlenecks that might
surface while running exploratory testing in a Waterfall environment.
Furthermore, in order to run exploratory testing in a Waterfall environment,
documentation on test results is a must and this documentation should be easy to trace
back to requirements. Of course, this type of documentation is always useful in any
environment.

What does adoption entail? Embracing exploratory testing is relatively easy, as it is

quick to launch (and scale), simple to learn and provides benefits for the entire team.
That said, it’s important to keep in mind that it should not be the only form of testing
(rather, it should inform what type of testing happens next). Additionally, even though
it’s unscripted, exploratory testing should not be unstructured (testers still need to set a
goal, log your activities and take on the mindset of a specific user persona).

Who is it for? Exploratory testing can help reduce time spent testing, find more defects
and improve code coverage. As a result, exploratory testing is best suited for teams that
are under time constraints, teams that need help identifying the best types of tests to
run (especially in cases where there are no specifications from developers) and teams
that want to make sure they didn’t miss anything in previous tests. Key team members
who should be involved in exploratory testing include:
 Testers (although everyone on the team should participate in some way)

What are best practices? Best practices for testers using exploratory testing include:
 Organizing the functionality in the application using something like a Mindmap or
a spreadsheet
 Focusing on certain areas or certain scenarios
 Tracking what gets tested to help reproduce any errors
 Documenting results in a tool like qTest Explorer so there is some accountability
for what got tested

Session Based Testing

What is it? Finally, let’s review session based testing. Session based testing builds on
exploratory testing by providing more structure. Because exploratory testing is
completely unscripted, it makes accountability difficult and relies heavily on the skills
and experience of the tester(s) involved. Session based testing aims to ease some of
these shortcomings by bringing more structure to exploratory testing without taking
away from the benefits that exploratory testing provides, such as the ability to better
mimic the user experience and get creative with testing.
Session based testing provides this structure by conducting testing during time-boxed,
uninterrupted sessions, testing against a charter and requiring testers to report on the
testing that took place during each session. Additionally, session based testing should
be capped off with a “debrief” between the tester(s) and the manager that covers the
five PROOF points: What happened (Past), what was achieved (Results), what got in
the way (Obstacles), what still needs to be done (Outlook) and how does the tester feel
about it (Feelings).

How is it different from standard Waterfall testing? The same as exploratory testing,
session based testing can be run in both Agile and Waterfall environments, but it is
more conducive to the tight collaboration between testers and developers that is
typically found in Agile environments.

What does adoption entail? Much like with exploratory testing, adopting session
based testing proves relatively easy because it is easy to pick up and launch quickly.
For testers already accustomed to exploratory testing, the biggest hurdle is embracing
the additional structure for which session based testing calls. Also like with exploratory
testing, teams running session based testing should remember that it is not a final stop,
but rather a method to help determine the best type of testing to conduct next.

Who is it for? Session based testing can help reduce testing time while increasing
defect discovery and code coverage, making it ideal for teams that face time constraints
and need more guidance to determine what types of tests to run. It is also ideal for
teams that have found benefits in exploratory testing but need to improve accountability
throughout the process. Key team members who should be involved in session based
testing include:
 Testers
 Managers

What are best practices? Best practices for testers using session based testing
include:
 Outlining a mission so that testers are clear about the software they’re testing
  Developing a clear charter that indicates the mission, the areas of the software to
test, which tester(s) will run the session, when the session will take place and for
how long, the tests designed and executed, the bugs found and overall notes (like
with exploratory testing, a documentation tool like qTest Explorer can help here)
 Running the testing session without any interruptions
 Clearly documenting activities and notes during the session in a session report
 Holding a debrief between the tester(s) and the manager to review findings from
the session and discuss next steps for testing

How to Align Testing with an Agile Delivery Process

Once you determine which testing methodology is right for your organization, you’re not
quite done yet. You still need to align testing with the delivery. To achieve this goal we
recommend a three-pronged approach:

1) Get involved in the development process as early as possible

The sooner testers can get involved, the better. Ideally, testers should be present from
day one. That’s because giving testers a seat at the table every step of the way
provides a higher level of insight into requirements and goals, encourages collaboration
and helps hammer home the need to conduct frequent (if not continuous) testing.

2) Test frequently, but thoughtfully

As more and more teams adopt Agile methodologies, efficiency is everything. This need
for speed has led teams to embrace DevOps and continuous integration as well in order
to keep things moving, and that requires testing more frequently. But in the midst of an
efficiency and frequency focused shakeup, testers need to remain thoughtful so as not
to create more overhead and run unnecessary tests that actually slow down the
process.

3) Hit the ground running with test creation

Keeping in mind the need for speed in today’s Agile, DevOps driven world, testers need
to hit the ground running when it comes to getting tests created. Specifically, the more
testers can reduce the time from requirements gathering to test creation, the better.
Having a seat at the table for all conversations from the very beginning should help in
this regard.
What’s Next for Agile Testing?
Although Agile has already made significant inroads into the software development
lifecycle, there’s still a long way to go, especially among testing teams.
Going forward, more widespread adoption and greater maturity of Agile methodologies
will require testers to go beyond test creation and execution and begin to focus on code
delivery and integration too. At the same time, testers will need to hone their automation
skills, become more involved in the entire software development process and continue
to develop a collaborative relationship with developers. Ultimately, these changes will
also require testers to become experts at development and product usage in order to
provide more holistic testing strategies and take on the role of “quality champions.”
In the future, three key tenets will become particularly important for testers working in
Agile environments:

1) Communication
Agile requires a tight collaboration between testers and developers, and that
collaboration makes communication a top priority for testers. Additionally, in a world
where quality becomes everyone’s responsibility, testers will become “quality
champions” that serve as internal experts, which will put their ability to clearly
communicate testing needs and reasoning under the spotlight.

2) Skill Diversity
In an Agile environment, everything can change on a dime, and that requires testers to
be adaptable. Part of this adaptability is having a diverse skillset so that testers can
change course as needed. For instance, functional testers need to expand their skills
beyond manual scripted execution. This diverse skillset will be a must as different
sprints require different types of testing to be executed in a short timeframe.

3) Business Mindset
Finally, Agile takes on a very customer-centric approach in order to ensure customers
receive as much value as possible as quickly and as early on as possible. Testers have
a big role to play in delivering this value, but it requires them to take on a business
mindset so that they can understand customer expectations, desires and concerns and
develop their testing strategies accordingly.
Why Leading Companies are Embracing Agile with qTest
Over 300 leading companies have chosen to improve their software testing processes
and embrace Agile with qTest. Here’s what a few of these leaders had to say about why
they made the choice to go with qTest:

We were able to quickly mitigate all of our test cases from HP Quality Center to qTest
and get the team up and running in just a few weeks. The implementation process was
excellent”
-Radka Iordanova, Director, E-Commerce, Office Depot, Inc.

 “When you transition to Agile, it’s not well enough to only change your
development methodology, you must upgrade your software tools as well…
QASymphony was exactly what we were looking for.”
-Alex Bantz, Director of Quality Engineering, Salesforce Marketing Cloud

 “We’ve seen a large reduction in tester errors and now we have history
on our defects. We can see where the problems are. The partnership between
American Equity and QASymphony has been wonderful.”
-Dennis Young, Assistant Vice President of QA, American Equity

 “We evaluated a lot of other testing platforms out there and found
qTest to be the best and by far the most intuitive…we’ve improved efficiency by at least
50% since first implementing qTest.’
-Jesse Reynosa, Senior Quality Engineer, Zappos