SE401 - Software Quality Assurance and Testing

SE401 - Software Quality Assurance
and Testing
Introduction
Thought for the Day
Software Testing takes at least twice as long as estimated.
Software is Everywhere
Outlines
 What is Software Testing?

 Why is Software Testing Important?
 Testing Myths
 Terminology- Failures, Defects, and Errors
 Principles of Software Testing
 Categories of Software Testing
 SDLC and Testing
College of Computer and Information Systems - Prince Sultan University These slides are prepared from: https://www.guru99.com/software-testing.html and Software Testing and Analysis: 4
Process, Principles and Techniques (2008)
What is Software Testing?
▪ It is a systematic process used to identify the correctness,

completeness, and quality of developed software. It includes a set
of activities conducted with the intent of finding errors in a software
so it can be corrected before the product is released to the end
users
▪ In simple words: Software testing is an activity that ensures the
software system is defect free.
▪ It can be either done manually or using automated tools.
College of Computer and Information Systems - Prince Sultan University These slides are prepared from: https://www.guru99.com/software-testing.html and 5
Software Testing and Analysis: Process, Principles and Techniques (2008)
Why Software Testing is Important?
▪ China Airlines Airbus A300
Crashed due to a software bug on April 26, 1994, killing 264 innocent live
▪ Canada's Therac-25 radiation therapy machine
Malfunctioned in 1985 due to software bug and delivered lethal radiation doses to
patients, leaving 3 people dead and critically injuring 3 others.
▪ $1.2 billion military satellite launch
Failed In April of 1999 because of a software bug. The costliest accident in history
▪ US Bank Accounts
A software bug caused 823 customers to be credited $920 million
Economic Impact
▪ NIST, “The Economic Impacts of Inadequate Infrastructure for
Software Testing”
− Inadequate software testing costs the US alone around $59.5
billion annually
− Better approaches could cut this amount by $22.2 billion
▪ We want our programs to be reliable
− Testing is how, in most cases, we find out if they are
Launch of MS Windows 98
▪ Bill Gates and Blue Screen of Death
Launch of MS Windows 98
▪ Bill Gates and Blue Screen of Death
A Case Study: Ariane 5 Launch Vehicle
▪ European expendable launch system
▪ Double the payload capacity of its
predecessor Ariane 4
− 113 successful launches, 3 failures
▪ Flight 501, maiden launch,
June 4, 1996,12:34pm
Case Study  Ariane 5: Why Did It Happen?
▪ Not a programming error
− unprotected conversion = design decision (~1980)
▪ Not a design error
− Justified against Ariane 4 trajectory & RT constraints
▪ Problem with integration testing
− Theoretically detectable.
− But huge test space vs. limited resources
− Furthermore, Inertial Reference System (SRI) useless at this stage of the flight!
▪ Reuse of a component with a hidden constraint
− Precondition : abs(BH) < 32768.0
− Valid for Ariane 4, but no longer for Ariane 5
o More powerful rocket
Case Study  Ariane 5:
Lessons Learned in Software Eng.
▪ Test! Test! Test!
▪ Test! Even when the code is reused.
▪ When reuse, ensure the assumptions are still valid.
▪ When write reusable code, document the assumptions.
▪ Write fail-safe code.
▪ Do not propagate errors.
▪ History is full of such examples. Look for more examples!

▪ Testing is important because software bugs could be expensive or
even dangerous.
▪ Software bugs can potentially cause monetary and human loss,
Why Programs fail
▪ In Federalist 51, James Madison wrote: “If men were angels, no
government would be necessary.”
▪ If he lived today, Madison might have written: “If software
developers were angels, debugging would be unnecessary.”
Why Programs fail
Congratulations!
▪ Your code is complete. It compiles. It runs …
▪ Your program fails. How can this be?
▪ There is a defect in the code. When the code is executed, the defect causes bad behavior, which
later becomes visible as a failure.
▪ Before a program can be debugged, we must set it up such that it can be tested — that is,
executed with the intent to make it fail.
▪ The first step in debugging is to reproduce the problem in question — that is, to create a test case
that causes the program to fail in the specified way.
− The first reason is to bring it under control, such that it can be observed.
− The second reason is to verify the success of the fix.
▪ Now you will have the fun of testing and debugging!
Why Do We Test?
▪ Testing is expensive.
− So are failures!
▪ What do we gain from that cost?
− Finding bugs
− Leading to
o Fixing bugs
o Raising the quality of the program or system we are testing
Trade-Offs of Cost and Failures
Total Cost of Quality (CoQ) =
Cost of Conformance (CoC) +
Cost of Non-Conformance (CoNC)
▪ Cost of Conformance
− Prevention: quality planning, investment in tools, quality training
− Appraisal: testing, inspection
▪ Cost of Non-Conformance
− Internal failures: rework
− External failure: liability, loss of properties, loss of lives
Trade-Offs of Cost and Failures
▪ Testing adds to Cost of Conformance
▪ It must directly reduce Cost of Non-Conformance
Software Testing
Myths and Facts
Myth #1 in Software Testing
Q: What is the objective of software testing?

A: Testing is to show that there are no
errors/bugs/defects in the software.
 Fact:
 No!! The main objective of testing is to discover defects.
 Testing is a destructive activity.
Q: What is the objective of software testing?

A: Testing is to ensure that the software does what it
is supposed to do.
 Fact:
 Only partly true.
 Testing is also to ensure the software does not do what it is not supposed
to do.
Q: How challenging is software testing?

A: Testing is easier than design and
implementation.
 Fact:
 Must consider all possible scenarios.
 Implied and unstated requirements and threats.
 Must be imaginative and creative.
Q: How challenging is software testing?

A: Testing is an extremely creative and
intellectually challenging task.
A Self Assessment Test
Test the Following Program
▪ The program reads in 3 integer values that represent the lengths of the
sides of a triangle.
▪ The program prints a message that states whether the triangle is
− Equilateral (all 3 sides are equal)
− Isosceles (exactly 2 of the 3 sides are equal)
− Scalene (all 3 sides are of a different length)
Write a set of test cases that you feel would adequately test this
program.
How Will You Do It?
Write a set of test cases that you feel would adequately test this
program.
▪ How many cases are needed?
▪ How do you plan to test the program?
− Run it once and manually enter the values?
− Run it many times with different inputs?
− Run it with a file containing a set of lines with test values?
How will you do it?
▪ “Smoke test.”, aka “Hello World!”
▪ Check handling of inputs
− Illegal inputs – text instead of integers
− Impossible inputs – floating vs. integer
− Outrageous values – infinities, max and min values
− Not in domain – negative numbers,
values outside specifications
− Input errors – wrong input, e.g. mis-spellings
▪ Stress test
− multiple inputs without restart
− run program for long periods of time
Software Testing
Terminologies
The Very First Software Bug
▪ A moth found trapped

between points at Relay #
70, Panel F
− Mark II Aiken Relay Calculator
− Harvard University
− September 9, 1945
On display in Smithsonian
Failures
▪ Failures are
− deviation of the observed behavior of a system from its
specification, i.e., its expected behavior.
▪ Failures can only be determined with respect to the
specifications.
▪ Failures are concerned with the observed behavior
and outcome of the system.
Defects
▪ Defects are
− flaws in a system that can cause the system to fail to
perform its required function
oe.g. an incorrect condition or statement.
▪ Defects are concerned with specific parts or

components of the system.
▪ Defects are synonymous with faults, bugs.
Errors
▪ Errors are
− human actions that result in a fault or defect in the
system.
▪ Errors are concerned with the underlying causes of
the defects.
▪ Errors are synonymous with mistakes.
The Relations among Failures, Defects, and Errors
▪ A human being makes an error (mistake)

− can occur in design, coding, requirements, even testing.
▪ An error can lead to a defect (fault)

− can occur in requirements, design, or program code.
▪ If a defect in code is executed, a failure may occur.

− Failures only occur when a defect in the code is executed.
− Not all defects cause failures all the time.
▪ Defects occur because human beings are fallible

▪ Failures can be caused by environmental conditions as well.
Failures, Defects, and Errors
▪ An example: For any integer n, square (n) = n*n.
Error:
should be int square (int x)
x^2 square (3) = 6
{
return x*2;
}
A defect A failure
Failures, Defects, and Errors
▪ An example: For any integer n, square (n) = n*n.
int square (int x)

square (2) = 4
{
return x*2;
}
A defect Correct result

Not a failure
Testing: Concepts
Testing: Concepts
• Test case (or, simply test)

• An execution of the software with a given test input, including:
• Input values
• Sometimes include execution steps
• Expected outputs
int actual_output=sum(1,2)
assertTrue(actual_output==3);
Example JUnit test case for testing “sum(int a, int b)”
Testing: Concepts
• Test oracle
• The expected outputs of software for given input
• A part of test cases
• Hardest problem in auto-testing: test oracle generation
int actual_output=sum(1,2)
assertTrue(actual_output==3);
Example JUnit test case for testing “sum(int a, int b)”
Testing: Concepts
• Test fixture: a fixed state of the software under test used as a baseline for
running tests; also known as the test context, e.g.,
• Loading a database with a specific, known set of data

• Preparation of input data and set-up/creation of fake or mock objects
Testing: Concepts
• Test suite
• A collection of test cases
• Usually these test cases share similar pre-requisites and configuration
Usually can be run together in sequence
•
Different test suites for different purposes
•Certain platforms, Certain feature, performance, …
• Test Script
• A script to run a sequence of test cases or a test suite automatically
Testing: Concepts
• Test driver
• A software framework that can load a collection of test cases or a test
suite
• It can also handle the configuration and comparison between
expected outputs and actual outputs
Testing: Concepts
▪Test adequacy
− We can’t always use all test inputs, so which do we use and when do we stop?
− We need a strategy to determine when we have done enough
− Adequacy criterion: A rule that lets us judge the sufficiency of a set of test data
for a piece of software
Testing: Concepts
• Test adequacy example: test coverage

• A measurement to evaluate the percentage of tested code
• Statement coverage
• Branch coverage, ...
Cobertura tool
Test Plan and Testing Process
▪ Test plan
− A document that specifies how a system will be tested,
including criteria for success, i.e., the exit criteria.
▪ Testing process
− The testing process involves developing test plans,
designing test cases, running the test cases, and
evaluating the results
Granularity of Testing
▪Unit Testing System&

− Test of each single module Acceptance
Testing
▪Integration Testing
− Test the interaction between modules
▪System Testing Integration Testing
− Test the system as a whole, by developers
▪Acceptance Testing
− Validate the system against user requirements,
by customers, without formal test cases
Unit Testing
Software Testing
1. Software testing ≠ Debugging.

2. Software testing ≠ Quality assurance
Software Testing vs. Quality Assurance (QA)
▪ Software testing is a planned process that is used to

identify the correctness, completeness, security and
quality of software.
▪ Quality Assurance (QA) is planned and systematic way
to evaluate quality of process used to produce a quality
product.
▪ The goal of a QA is to provide assurance that a product
is meeting customer’s quality expectations.
▪ Testing is necessary, but not sufficient for quality

assurance
− Testing contributes to improve quality by identifying
problems.
▪ Quality assurance sets the standards for the
team/organization to build better software.
Techniques for
Software Testing
Testing
▪ Testing “Phases” ▪ Static vs. Dynamic Testing
− Unit ▪ Integration: 3 types
− Integration − Top down
− Bottom up
− System
− Sandwich
− User Acceptance Testing
▪ Automated Testing
▪ Testing Types − Pros and cons
− Black-box
− White-box ▪ Defect tracking
Ad Hoc Testing
▪ Ad hoc testing
− Testing carried out informally
− No formal test preparation
− No recognized test design technique is used
− Expected results not defined
− Arbitrariness guides the test execution activity
▪ Ad hoc testing is not sufficient

Black Box Testing
▪ Treats a program or system as a black box

▪ Design test cases without the knowledge of internal structure and
design of the system
▪ Design test cases based on the functional requirements of the system
− a.k.a: functional testing
▪ Send the system inputs, observe the outputs, decide if the system
passed or failed the test
▪ Sometimes you don’t have access to source code
White Box Testing
▪ Open up the box!

− a.k.a: glass box, clear box, or structural testing
▪ Design test cases by examining the internal design and
source code of the program
▪ Require detailed knowledge of its structure
▪ Introduce the idea of measuring coverage:
− How adequate, or thorough, is the test suite?
Software Analysis
▪ Software analysis is
− the process of finding defects in a program (or parts of a program) without
executing the program.
− a.k.a. static analysis
▪ Complementary to software testing

− Same purpose, different techniques
▪ Software analysis can be
− automated, i.e., using tools, static analyzers
− manual, e.g., inspection, code review
Categories of Software Testing
▪ Typically Testing is classified into three categories.

− Functional Testing
− Non-Functional Testing or Performance Testing
− Maintenance (Regression and Maintenance)
Categories of Software Testing
Testing Category Types of Testing (examples)
Functional Testing Unit Testing  This is not the

Integration Testing complete list of
Smoke
testing types. There
UAT ( User Acceptance Testing)
Localization are much more
Globalization types and still
Interoperability and So on adding.
Non-Functional Testing Performance  Also, note that not

Endurance all testing types are
Load applicable to all
Volume
Scalability projects but depend
Usability and So on on nature & scope of
the project.
Maintenance Regression
Maintenance
64
Testing vs. Verification
TESTING
▪ Goal: find evidence for presence of failures
− Testing: execute a program with the intent of detecting failure
− Testing cannot guarantee correctness, i.e., absence of failures
▪ Related techniques: code reviews, program inspections
VERIFICATION
▪ Goal: find evidence for absence of failures
− Verification guarantees correctness
▪ Related techniques: code generation, program synthesis (from spec)
Testing vs. Verification
▪ Both, testing and verification attempts to exhibit new failures
▪ Debugging is a systematic process that finds and eliminates the
defect that led to an observed failure
▪ Programs without known failures may still contain defects:
− If they have not been verified
− If they have been verified, but the failure is not covered by the specification
Fundamental Principles of
Software Testing
Fundamental Principles of Software Testing
▪ It is important that you achieve an optimum test results

▪ But how do you determine that you are following the right
strategy for testing?
▪ For that, you need to stick to some basic testing principles to
optimize the testing effort
▪ Here are some common testing principles that are widely
practiced in the software industry
Testing shows presence of defects
▪ Testing talks about the presence of defects and don’t talk

about the absence of defects.
▪ i.e. Software Testing reduces the probability of undiscovered
defects remaining in the software but even if
no defects are found, it is not a proof
of correctness.
▪ Despite the thorough testing of a product
or application, no one can ensure the
product is 100% defect free
Exhaustive testing is not possible
▪ It is impossible to test all the combinations of data, inputs, and

test scenarios since it requires lots of time.
▪ Instead, we need the optimal amount of testing based on the
risk assessment of the application
▪ Prioritize test cases
Exhaustive Testing is not possible
loop < 20 iterations
▪ Can we exhaustively test a

program?
▪ Let’s consider this simple program
▪ There are 1014 possible paths!
− If we execute one test per millisecond,
− it would take 3,170 years to test this program!!
▪ Exhaustive testing is impossible!
Early testing
▪ Testing should start as early as possible in the Software
Development Life Cycle.
▪ This allows any defects in the requirements or design phase to be
captured in early stages.
▪ It is much cheaper to fix a Defect in early
stages of testing.
▪ But how early one should start testing?
− It is recommended that you start finding the bug
the moment the requirements are defined.
(More on this principle is discussed later.)
Defect clustering
▪ Defect Clustering states that a small number of modules contain

most of the defects detected. (approximately 80% of the problems
are found in 20% of the modules.)
▪ By experience, you can identify such risky modules. But this
approach has its own problems:
− If the same tests are repeated over and over again,
eventually the same test cases will no longer find
new bugs.
Pesticide Paradox
▪ If the same set of repetitive tests are conducted, the method will be
useless for discovering new defects.
▪ To overcome this, the test cases need to be regularly reviewed &
revised, adding new & different test cases to help find more defects.
▪ Testers cannot simply depend on existing test techniques. They must
aim continually to improve the existing methods to make testing
more effective.
▪ But even after all this testing, you can never claim your product is bug
free.
Testing is basically context-dependent
▪ Various products or projects contain different elements, features, and
requirements.
− Hence, testers cannot apply the same test approach for different
projects. For instance, an application in banking industry should require
different testing techniques than an entertainment software.
75
College of Computer and Information Systems - Prince Sultan University These slides are prepared from: https://www.guru99.com/software-testing.html and Software
Testing and Analysis: Process, Principles and Techniques (2008)
The absence of Errors Fallacy
▪ The software product should not be tested only on the technical

side but also on the expectations and needs of users .
▪ Although the testing didn’t show up any bugs in the software, it
would not mean that the product is ready to be released since it
must be confirmed the testing is conducted thoroughly
with the right requirements.
▪ So .. Absence of Error is a Fallacy
i.e. Finding and fixing defects does not help
if the system build does not fulfill the user’s
needs & requirements.
Make No Assumptions
Do not assume that no new

defects will be found.
Define Expected Output
▪ Each test case must include the expected output.

▪ The output of each test must be thoroughly inspected.
− Preferably: automated inspection, assertions
?
Actual output
Expected output
Test the Invalid and Unexpected
▪ Test input conditions that are

a) valid and expected
b) invalid and unexpected
▪ Test a program to see whether it
a) does not do what it is supposed to do
o perform some but not all the required functions
b) does what it is not supposed to do
o perform unnecessary functions, undesired side-effects
Test Independently
A program should be tested by independent

testers.
developer independent tester

Understands the system Must learn about the system,
but, will test "gently" but, will attempt to break it
and, is driven by "delivery" and, is driven by quality
Tool Support is Essential
▪ Some Reasons for Using Tools
− Automate repetitive tasks
− Avoid typos, etc.
− Cope with large programs
▪ Tools Used
− Test script
− Automated compilation, build and testing: Ant
− Automated running of tests: JUnit
− Debugging: Eclipse debugger
− Code Coverage: Cobertura, EclEmma, JaCoCo
Fundamental Principles Myth
▪ Myth: "Principles are just for reference. I will not use them in
practice .“
▪ Fact:
− This is so very untrue. Test Principles will help you create an effective Test
Strategy and draft error catching test cases.
− Experienced testers have internalized these principles to a level that they apply
them without even thinking.
− Hence the myth that the principles are not used in practice is simply not true.
Limitations of
Software Testing
Exhaustive Testing
loop < 20 iterations
▪ Can we exhaustively test a

program?
▪ Let’s consider this simple program
▪ There are 1014 possible paths!
− If we execute one test per millisecond,
− it would take 3,170 years to test this program!!
▪ Exhaustive testing is impossible!
Absence of Defects
“Program testing can be used to show the presence of

bugs, but never their absence.”
(Dijkstra, 1969)
Testing reduces the probability of undiscovered defects remaining

in the software but, even if no defects are found, it is not a proof of
absence of defects.
When to Stop Testing?
▪ One of the most difficult problems in testing is not

knowing when to stop.
▪ You can use metrics to make a guess. Keep track of

the defect rate. When it goes towards zero you can
use it as an indicator. Or, …
▪ Even if you do find the last bug, you’ll never know it.
How Much Testing is Enough?
▪ Take into account
− the level of risk, including technical, safety, and business risks, and
− project constraints such as time and budget.
▪ Testing should provide feedback to stakeholders to make informed decisions

− about the release of the software,
− for the next development step, or
− handover to customers.
▪ Later, we will discuss some techniques for making this decision.

Testing and SDLC
88
SDLC – Waterfall
▪ Software Development
Life Cycle (SDLC), the
waterfall model
As you may observe,

that testing starts only after
implementation is done.
Discuss: what is the affect of delaying testing until after

implementation?
Cost of Fixing Defects across the SDLC
▪ The costs of fixing a defect
increases across the
development life cycle. The
earlier in life cycle a defect is
detected, the cheaper it is to fix it.
▪ To address this concern, the V
model of testing was developed
where, for every phase in the
Development life cycle, there is a
corresponding Testing phase
V-Model of Testing
▪ The left side of the model is
Software Development Life
Cycle - SDLC
▪ The right side of the model is
Software Test Life Cycle
- STLC
▪ The entire figure looks like a V,
hence the name V – model
▪ Achieves the principle “Early
Testing”
Other Development Models
planning
estimation
scheduling
risk analysis
communication
modeling
analysis
design
start
deployment
construction
delivery
code
feedback test
RAD Model Spiral Model
Other Development Models
Prototype Model Agile Model
SDLC and Testing
▪ It is important to understand that there are various development
life cycle models.
− Selecting a development model for a project depends on the aims and goals of
that project.
▪ Testing is not a stand-alone activity, and it has to adapt to the

development model chosen for the project.
▪ In any model, testing should be performed at all levels i.e. right
from requirements until maintenance.
▪ Refer to this link for more details:
https://www.guru99.com/testing-methodology.html
Summary – Key Concepts
▪ Objective and principles of testing
▪ Cost of quality
▪ Fundamentals and basic terminologies in testing
▪ Failures, defects, faults, errors, mistakes
▪ Testing, analysis, black-box and white-box testing
▪ Test cases, test suites, test plan, test process
▪ Testing principles
▪ Independent testing
▪ Limitations of testing, exhaustive testing

SE401 - Software Quality Assurance and Testing

Uploaded by

Copyright:

Available Formats

SE401 - Software Quality Assurance and Testing

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SE401 - Software Quality Assurance and Testing

Uploaded by

Copyright:

Available Formats

SE401 - Software Quality Assurance

 What is Software Testing?

▪ It is a systematic process used to identify the correctness,

A software bug caused 823 customers to be credited $920 million

▪ History is full of such examples. Look for more examples!

Q: What is the objective of software testing?

Q: What is the objective of software testing?

Q: How challenging is software testing?

Q: How challenging is software testing?

▪ A moth found trapped

▪ Defects are concerned with specific parts or

▪ A human being makes an error (mistake)

▪ An error can lead to a defect (fault)

▪ If a defect in code is executed, a failure may occur.

▪ Defects occur because human beings are fallible

int square (int x)

A defect Correct result

• Test case (or, simply test)

• Loading a database with a specific, known set of data

• Test adequacy example: test coverage

▪Unit Testing System&

1. Software testing ≠ Debugging.

▪ Software testing is a planned process that is used to

▪ Testing is necessary, but not sufficient for quality

▪ Ad hoc testing is not sufficient

▪ Treats a program or system as a black box

▪ Open up the box!

▪ Complementary to software testing

▪ Typically Testing is classified into three categories.

Functional Testing Unit Testing  This is not the

Non-Functional Testing Performance  Also, note that not

▪ It is important that you achieve an optimum test results

▪ Testing talks about the presence of defects and don’t talk

▪ It is impossible to test all the combinations of data, inputs, and

▪ Can we exhaustively test a

▪ Defect Clustering states that a small number of modules contain

▪ The software product should not be tested only on the technical

Do not assume that no new

▪ Each test case must include the expected output.

▪ Test input conditions that are

A program should be tested by independent

developer independent tester

▪ Can we exhaustively test a

“Program testing can be used to show the presence of

Testing reduces the probability of undiscovered defects remaining

▪ One of the most difficult problems in testing is not

▪ You can use metrics to make a guess. Keep track of

▪ Testing should provide feedback to stakeholders to make informed decisions

▪ Later, we will discuss some techniques for making this decision.

As you may observe,

Discuss: what is the affect of delaying testing until after

RAD Model Spiral Model

Prototype Model Agile Model

▪ Testing is not a stand-alone activity, and it has to adapt to the

You might also like