INTRODUCTION

SOME SOFTWARE FAILURES

 The Explosion of the Ariane 5 Rocket
 The Y2K Problem
 The USA Star-Wars Program
Testing is an important aspect of the software development life
cycle. It is basically the process of testing the newly developed
software, prior to its actual use. The program is executed with
desired input(s) and the output(s) is/are observed accordingly. The
observed output(s) is/are compared with expected output(s). If both
are same, then the program is said to be correct as per
specifications, otherwise there is something wrong somewhere in
the program. Testing is a very expensive process and consumes one-
third to one-half of the cost of a typical development project.

WHAT IS SOFTWARE TESTING?

 There are 8 sets of inputs in Table 1.1. We may feel that these 8 test
cases are sufficient for such a trivial program. In all these test cases,
the observed output is the same as the expected output. We may
also design similar test cases to show that the observed output is
matched with the expected output. There are many definitions of
testing. A few of them are given below:
 Testing is the process of demonstrating that errors are not
present.
 The purpose of testing is to show that a program performs its
intended functions correctly.
 Testing is the process of establishing confidence that a program
does what it is supposed to do.
“Testing is the process of executing a program with the intent of
finding faults.”
We again consider the program ‘Minimum’ (given in Figure 1.1) and
concentrate on some typical and critical situations as discussed
below:
 A very short list (of inputs) with the size of 1, 2, or 3 elements.
 An empty list i.e. of size 0.
 A list where the minimum element is the first or last element.
 A list where the minimum element is negative.
 A list where all elements are negative.
 A list where some elements are real numbers.
 A list where some elements are alphabetic characters.
 A list with duplicate elements.
 A list where one element has a value greater than the
maximum permissible limit of an integer.
Software testing is a very expensive and critical activity; but releasing
the software without testing is definitely more expensive and
dangerous. No one would like to do it. It is like running a car without
brakes. Hence testing is essential; but how much testing is required?
Do we have methods to measure it? Do we have techniques to
quantify it? The answer is not easy. All projects are different in
nature and functionalities and a single yardstick may not be helpful
in all situations. It is a unique area with altogether different
problems.
We shall try to find more errors in the early phases of software
development. The cost of removal of such errors will be very
reasonable as compared to those errors which we may find in the
later phases of software development. The cost to fix errors
increases drastically from the specification phase to the test phase
and finally to the maintenance phase as shown in Figure.

 
There are seven principles in software testing:
1. Testing shows presence of defects
2. Exhaustive testing is not possible
3. Early testing
4. Defect clustering
5. Pesticide paradox
6. Testing is context dependent
7. Absence of errors fallacy

WHO SHOULD WE DO THE TESTING?

Testing a software system may not be the responsibility of a single
person. Actually, it is a team work and the size of the team is
dependent on the complexity, criticality and functionality of the
software under test. The software developers should have a reduced
role in testing, if possible.
The testing persons must be cautious, curious, critical but non-
judgmental and good communicators. One part of their job is to ask
questions that the developers might not be able to ask themselves or
are awkward, irritating, insulting or even threatening to the
developers. Some of the questions are:
(i) How is the software?
(ii) How good is it?
(iii) How do you know that it works? What evidence do you have?
(iv) What are the critical areas?
(v) What are the weak areas and why?
(vi) What are serious design issues?
(vii) What do you feel about the complexity of the source code?
Roles of the persons involved during development and testing are
given in Table
WHAT SHOULD WE TEST?
Is it possible to test the program for all possible valid and invalid
inputs? The answer is always negative due to a large number of
inputs. We consider a simple example where a program has two 8 bit
integers as inputs. Total combinations of inputs are 28 28. If only one
second is
required (possible only with automated testing) to execute one set of
inputs, it may take 18 hours to test all possible combinations of
inputs. Here, invalid test cases are not considered which may also
require a substantial amount of time. In practice, inputs are more
than two and the size is also more than 8 bits. What will happen
when inputs are real and imaginary numbers? We may wish to go for
complete testing of the program, which is neither feasible nor
possible.
What is the bottom line for testing? At least, we may wish to touch
this bottom line, which may incorporate the following:
(i) Execute every statement of the program at least once.
(ii) Execute all possible paths of the program at least once.
(iii) Execute every exit of the branch statement at least once.

SOME TERMINOLOGIES

PROGRAM AND SOFTWARE

Both terms are used interchangeably, although they are quite
different. The software is the superset of the program(s). It consists
of one or many program(s), documentation manuals and operating
procedure manuals.
 

 
VERIFICATION AND VALIDATION
Verification is the process of checking that a software achieves its
goal without any bugs. It is the process to ensure whether the
product that is developed is right or not. It verifies whether the
developed product fulfills the requirements that we have.
Verification is static testing.
Verification means Are we building the product right?
Validation is the process of checking whether the software product is
up to the mark or in other words product has high level
requirements. It is the process of checking the validation of product
i.e. it checks what we are developing is the right product. it is
validation of actual and expected product. Validation is the dynamic
testing.
Validation means Are we building the right product?
 
 
VERIFICATION VALIDATION
It includes checking documents, It includes testing and validating
design, codes and programs. the actual product.
VERIFICATION VALIDATION
Verification is the static testing. Validation is the dynamic testing.
It does not include the execution It includes the execution of the
of the code. code.
Methods used in validation are
Methods used in verification are
Black Box Testing, White Box
reviews, walkthroughs,
Testing and non-functional
inspections and desk-checking.
testing.
It checks whether the software
It checks whether the software meets the requirements and
conforms to specifications or not. expectations of a customer or
not.
It can only find the bugs that
It can find the bugs in the early
could not be found by the
stage of the development.
verification process.
The goal of verification is
The goal of validation is an actual
application and software
product.
architecture and specification.
Validation is executed on
Quality assurance team does
software code with the help of
verification.
testing team.
It comes before validation. It comes after verification.
Hence, testing includes both verification and validation. Thus
Testing = Verification + Validation

FAULT, ERROR, BUG AND FAILURE

All terms are used interchangeably although error, mistake and
defect are synonyms in software testing terminology. When we
make an error during coding, we call this a ‘bug’. Hence, error /
mistake / defect in coding is called a bug.
A fault is the representation of an error where representation is the
mode of expression such as data flow diagrams, ER diagrams, source
code, use cases, etc. If fault is in the source code, we call it a bug.
A failure is the result of execution of a fault and is dynamic in nature.
When the expected output does not match with the observed
output, we experience a failure. The program has to execute for a
failure to occur. A fault may lead to many failures. A particular fault
may cause different failures depending on the inputs to the program.
 

TEST, TEST CASE AND TEST SUITE

Test and test case terms are synonyms and may be used
interchangeably. A test case consists of inputs given to the program
and its expected outputs. Inputs may also contain pre- condition(s)
(circumstances that hold prior to test case execution), if any, and
actual inputs identified by some testing methods. Expected output
may contain post-condition(s) (circumstances after the execution of a
test case), if any, and outputs which may come as a result when
selected inputs are given to the software. Every test case will have a
unique identification number. When we do testing, we set desire
pre-condition(s), if any, given selected inputs to the program and
note the observed output(s). We compare the observed output(s)
with the expected output(s) and if they are the same, the test case is
successful. If they are different, that is the failure condition with
selected input(s) and this should be recorded properly in order to
find the cause of failure. A good test case has a high probability of
showing a failure condition. Hence, test case designers should
identify weak areas of the program and design test cases accordingly.
The template for a typical test case is given in Table
The set of test cases is called a test suite. We may have a test suite of
all test cases, test suite of all successful test cases and test suite of all
unsuccessful test cases. Any combination of test cases will generate a
test suite.

DELIVERABLES AND MILESTONES

Different deliverables are generated during various phases of the
software development. The examples are source code, Software
Requirements and Specification document (SRS), Software Design
Document (SDD), Installation guide, user reference manual, etc.
The milestones are the events that are used to ascertain the status of
the project. For instance, finalization of SRS is a milestone;
completion of SDD is another milestone. The milestones are essential
for monitoring and planning the progress of the software
development.
ALPHA, BETA AND ACCEPTANCE TESTING
Acceptance Testing: This term is used when the software is
developed for a specific customer. The customer is involved during
acceptance testing. He/she may design adhoc test cases or well-
planned test cases and execute them to see the correctness of the
software. This type of testing is called acceptance testing. The
discovered errors are fixed and modified and then the software is
delivered to the customer.
Alpha and Beta Testing: These terms are used when the software is
developed as a product for anonymous customers. Therefore,
acceptance testing is not possible. Some potential customers are
identified to test the product. The alpha tests are conducted at the
developer’s site by the customer. These tests are conducted in a
controlled environment and may start when the formal testing
process is near completion. The beta tests are conducted by
potential customers at their sites. Unlike alpha testing, the developer
is not present here. It is carried out in an uncontrolled real life
environment by many potential customers. Customers are expected
to report failures, if any, to the company.

QUALITY AND RELIABILITY

Software reliability is one of the important factors of software

quality. Other factors are understandability, completeness,
portability, consistency, maintainability, usability, efficiency, etc.
These quality factors are known as non-functional requirements for a
software system.

Software reliability is defined as “the probability of failure free

operation for a specified time in a specified environment” [ANSI91].
Although software reliability is defined as a probabilistic function and
comes with the notion of time, it is not a direct function of time. The
software does not wear out like hardware during the software
development life cycle. There is no aging concept in software and it
will change only when we intentionally change or upgrade the
software.

Software quality determines how well the software is designed

(quality of design), and how well the software conforms to that
design (quality of conformance).

Some software practitioners also feel that quality and reliability is the
same thing. If we are testing a program till it is stable, reliable and
dependable, we are assuring a high quality product. Unfortunately,
that is not necessarily true. Reliability is just one part of quality. To
produce a good quality product, a software tester must verify and
validate throughout the software development process.
TESTING, QUALITY ASSURANCE AND QUALITY CONTROL

The purpose of the testing team and Quality Assurance (QA) team.
As we have seen in the previous section (1.2.1), the purpose of
testing is to find faults and find them in the early phases of software
development. We remove faults and ensure the correctness of
removal and also minimize the effect of change on other parts of the
software.

The purpose of QA activity is to enforce standards and techniques to

improve the development process and prevent the previous faults
from ever occurring. A good QA activity enforces good software
engineering practices which help to produce good quality software.
The QA group monitors and guides throughout the software
development life cycle. This is a defect prevention technique and
concentrates on the process of the software development. Examples
are reviews, audits, etc.

Quality control attempts to build a software system and test it

thoroughly. If failures are experienced, it removes the cause of
failures and ensures the correctness of removal. It concentrates on
specific products rather than processes as in the case of QA. This is a
defect detection and correction activity which is usually done after
the completion of the software development. An example is software
testing at various levels.

STATIC AND DYNAMIC TESTING

Static testing refers to testing activities without executing the source

code. All verification activities like inspections, walkthroughs,
reviews, etc. come under this category of testing. This, if started in
the early phases of the software development, gives good results at a
very reasonable cost. Dynamic testing refers to executing the source
code and seeing how it performs with specific inputs. All validation
activities come in this category where execution of the program is
essential.
LIMITATIONS OF TESTING

We want to test everything before giving the software to the

customers. This ‘everything’ is very illusive and has many meanings.
What do we understand when we say ‘everything’? We may expect
one, two or all of the following when we refer to ‘everything’:

(i)  Execute every statement of the program

(ii)  Execute every true and false condition
(iii)  Execute every condition of a decision node
(iv)  Execute every possible path
(v)  Execute the program with all valid inputs
(vi)  Execute the program with all invalid inputs

These six objectives are impossible to achieve due to time and

resource constraints as discussed in section 1.2.4. We may achieve a
few of them. If we do any compromise, we may miss a bug. Input
domain is too large to test and there are too many paths in any
program. Hence ‘Everything’ is impossible and we have to settle for
‘less than everything’ in real life situations. Some of the other issues
which further make the situation more complex and complicated are
given in the subsequent sub-sections.

DIFFICULT TO MEASURE THE PROGRESS OF TESTING

How to measure the progress of testing? Is experiencing more

failures good news or bad news? The answer could be either. A
higher number of failures may indicate that testing was thorough
and very few faults remain in the software. Or, it may be treated as
an indication of poor quality of the software with lots of faults; even
though many have been exposed, lots of them still remain.

This difficulty of measuring the progress of testing leads to another

issue i.e. when to stop testing and release the software to the
customer(s)? This is a sensitive decision and should be based on the
status of testing. However, in the absence of testing standards,
‘economics’, ‘time to market’ and ‘gut feeling’ have become
important issues over technical considerations for the release of any
software.
Software companies are facing serious challenges in testing their
products and these challenges are growing bigger as the software
grows more complex. Hence, we should recognize the complex
nature of testing and take it seriously. The gap between standards
and practices should be reduced in order to test the software
effectively which may result in to good quality software.

SOFTWARE QUALITY

Software quality product is defined in term of its fitness of purpose.

That is, a quality product does precisely what the users want it to do.
For software products, the fitness of use is generally explained in
terms of satisfaction of the requirements laid down in the SRS
document. Although "fitness of purpose" is a satisfactory
interpretation of quality for many devices such as a car, a table fan, a
grinding machine, etc.for software products, "fitness of purpose" is
not a wholly satisfactory definition of quality.

Example: Consider a functionally correct software product. That is, it

performs all tasks as specified in the SRS document. But, has an
almost unusable user interface. Even though it may be functionally
right, we cannot consider it to be a quality product.

The modern view of a quality associated with a software product

several quality methods such as the following:

Portability: A software device is said to be portable, if it can be freely

made to work in various operating system environments, in multiple
machines, with other software products, etc.

Usability: A software product has better usability if various

categories of users can easily invoke the functions of the product.

Reusability: A software product has excellent reusability if different

modules of the product can quickly be reused to develop new
products.

Correctness: A software product is correct if various requirements as

specified in the SRS document have been correctly implemented.
Maintainability: A software product is maintainable if bugs can be
easily corrected as and when they show up, new tasks can be easily
added to the product, and the functionalities of the product can be
easily modified, etc.

SOFTWARE QUALITY MANAGEMENT SYSTEM

A quality management system is the principal methods used by

organizations to provide that the products they develop have the
desired quality.

A quality system subsists of the following:

Managerial Structure and Individual Responsibilities: A quality

system is the responsibility of the organization as a whole. However,
every organization has a sever quality department to perform
various quality system activities. The quality system of an
arrangement should have the support of the top management.
Without help for the quality system at a high level in a company,
some members of staff will take the quality system seriously.

Quality System Activities: The quality system activities encompass

the following:

Auditing of projects

Review of the quality system

Development of standards, methods, and guidelines, etc.

Production of documents for the top management summarizing the

effectiveness of the quality system in the organization.
SOFTWARE QUALITY ASSURANCE
Software Quality Assurance (SQA) is simply a way to assure quality
in the software. It is the set of activities which ensure processes,
procedures as well as standards suitable for the project and
implemented correctly.
Software Quality Assurance is a process which works parallel to
development of a software. It focuses on improving the process of
development of software so that problems can be prevented before
they become a major issue. Software Quality Assurance is a kind of
an Umbrella activity that is applied throughout the software process.
Software Quality Assurance have:
1. A quality management approach
2. Formal technical reviews
3. Multi testing strategy
4. Effective software engineering technology
5. Measurement and reporting mechanism
Major Software Quality Assurance Activities:
1. SQA Management Plan:
Make a plan how you will carry out the sqa through out the
 project. Think which set of software engineering activities are
the best for project.check level of sqa team skills.
2. Set The Check Points:
SQA team should set checkpoints. Evaluate the performance of
the project on the basis of collected data on different check
points.
3. Multi testing Strategy:
Do not depend on single testing approach. When you have lot of
testing approaches available use them.
4. Measure Change Impact:
The changes for making the correction of an error sometimes re
introduces more errors keep the measure of impact of change
on project. Reset the new change to change check the
compatibility of this fix with whole project.
5. Manage Good Relations:
In the working environment managing the good relation with
other teams involved in the project development is mandatory.
Bad relation of sqa team with programmers team will impact
directly and badly on project. Don’t play politics.
Benefits of Software Quality Assurance (SQA):
1. SQA produce high quality software.
2. High quality application saves time and cost.
3. SQA is beneficial for better reliability.
4. SQA is beneficial in the condition of no maintenance for long
time.
5. High quality commercial software increase market share of
company.
6. Improving the process of creating software.
7. Improves the quality of the software.
Disadvantage of SQA:
There are a number of disadvantages of quality assurance. Some of
them include adding more resources, employing more workers to
help maintain quality and so much more.
WHAT IS SOFTWARE QUALITY MANAGEMENT?
Software Quality Management ensures that the required level of
quality is achieved by submitting improvements to the product
development process. SQA aims to develop a culture within the
team and it is seen as everyone's responsibility.
Software Quality management should be independent of project
management to ensure independence of cost and schedule
adherences. It directly affects the process quality and indirectly
affects the product quality.

ACTIVITIES OF SOFTWARE QUALITY MANAGEMENT:

 Quality Assurance - QA aims at developing Organizational
procedures and standards for quality at Organizational level.
 Quality Planning - Select applicable procedures and standards
for a particular project and modify as required to develop a
quality plan.
 Quality Control - Ensure that best practices and standards are
followed by the software development team to produce
quality products.

Quality software refers to a software which is reasonably bug or

defect free, is delivered in time and within the specified budget,
meets the requirements and/or expectations, and is maintainable.
In the software engineering context, software quality reflects
both functional quality as well as structural quality.
 Software Functional Quality − It reflects how well it satisfies a
given design, based on the functional requirements or
specifications.
 Software Structural Quality − It deals with the handling of non-
functional requirements that support the delivery of the
functional requirements, such as robustness or maintainability,
and the degree to which the software was produced correctly.
 Software Quality Assurance − Software Quality Assurance
(SQA) is a set of activities to ensure the quality in software
 engineering processes that ultimately result in quality software
products. The activities establish and evaluate the processes
that produce products. It involves process-focused action.
 Software Quality Control − Software Quality Control (SQC) is a
set of activities to ensure the quality in software products.
These activities focus on determining the defec

THE COST OF QUALITY

Cost of Quality concept has been continuously improved into a fully

developed financial model that has many strategic benefits.

This Chapter will break down The Cost of Quality into its key
concepts, which include:

 The Total Cost of Quality & the 4 Quality Cost Categories

 The Quality Cost data collection methods for each category
 The reporting and interpreting of Quality Cost results
 The COQ Benefits & Limitations

4 QUALITY COST CATEGORIES – TOTAL QUALITY COST

Categorization of Quality Costs

The cost of quality is generally classified into four categories:

1. External Failure Cost

2. Internal Failure Cost
3. Inspection (appraisal) Cost
4. Prevention Cost

1. External Failure Cost: Cost associated with defects found after the

customer receives the product or service. Example: Processing customer
complaints, customer returns, warranty claims, product recalls.

2. Internal Failure Cost: Cost associated with defects found before the

customer receives the product or service. Example: Scrap, rework, re-
inspection, re-testing, material review, material downgrades
3. Inspection (appraisal) Cost: Cost incurred to determine the degree of
conformance to quality requirements (measuring, evaluating or auditing).   
Example: Inspection, testing, process or service audits, calibration of
measuring and test equipment.

4. Prevention Cost: Cost incurred to prevent (keep failure and appraisal cost

to a minimum) poor quality.  Example: New product review, quality planning,
supplier surveys, process reviews, quality improvement teams, education and
training.

Cost of Quality (COQ) = Cost of Control + Cost of Failure of Control

where

Cost of Control = Prevention Cost + Appraisal Cost

and
Cost of Failure of Control = Internal Failure Cost + External Failure
Cost