0 All Slides

22/08/2011
Software testing
Nguyen Thanh Binh IT Faculty, Danang University of Technology Email: [email protected] Website: www.dut.edu.vn/itf/ntb
Prerequisites
No testing experience needed Some familiarity with the development phases of a project would be helpful. Some mathematics would help
References
1. 2. 3. 4. 5. 6. 7. 8. 9.
10.
Paul Jorgensen, Software Testing-A Craftsman's Approach, CRC Press, 1995. Spyos Xanthakis, Pascal Rgnier, Constantin Karapoulios, Le test des logiciels, Hermes Science, 2000. Hung Q. Nguyen and al., Testing application on the Web, John Wiley & Sons, 2004. Ilene Burnstein, Practical Software Testing, Springer, 2003. Glenford J. Myers, The art of software testing, Wiley, 2004. Cem Kaner, Jack Falk, Hung Q. Nguyen, Testing Computer Software, 2nd Edition, John Wiley & Sons, 1999. Boris Beizer, Software Testing Techniques, International Thomson Computer Press, Second Edition, 1990. Neil Bitzenhofer, Software Testing and Verification, Course, MSSE, 2008. Paul Ammann and Jeff Offutt, Introduction to Software Testing, Cambridge University Press, Cambridge, UK, ISBN 0-52188-038-1, 2008. Mauro Pezz, Michal Young, Software Testing and Analysis: Process, Principles, and Techniques, John Wiley & Sons.
3
22/08/2011
Course description
Cover both theoretical and practical aspects of testing software The student will participate in the entire range of test activities:
Analyze a requirements document for test conditions Write a test plan Design, create and execute test cases using various testing approaches Record defects Write a test report
4
Contents
Session 1: Introductory lecture
Introductions and expectations Course overview Contents
Contents
Session 2: Introduction to Software Testing
Definitions, Principles, Axioms Stages of testing Perspectives on Software Testing
22/08/2011
Contents
Session 3: Requirements analysis
Software Development Life Cycle (SDLC) Software Development stage
Requirements
Testing and requirements

Learn to think like a tester
Some examples
Contents
Session 4
Exercise 1: Examining requirements
Contents
Session 5: Structural Testing
White box testing / Structural testing Graph Theory Control flow criteria Data flow criteria Graph Coverage for Source Code
Testing State Behavior
22/08/2011
Contents
Session 6: Static Testing
Reviews and the test process Types of review Static analysis
10
Contents
Session 7: Functional Testing
Introduction to functional testing Functional testing techniques
Boundary Value testing Equivalence Class testing Special Value testing Decision Tables
11
Contents
Session 8: Test Documentation
Test Plan
The need for test plans The structure of test plans A Test Plan Template A Test Plan example Testing on a large project
Test Cases
Test Case Design Test Case Examples
12
22/08/2011
Contents
Session 9 Exercise 2: Writing a test plan and test cases
13
Contents
Session 10: Levels of Testing
Levels of Testing
Integration Testing System Testing
Additional System Test Categories
14
Contents
Session 11: Defect Reports/Test Reports
Handling Defects Bug Tracking System Test Reports Examples
15
22/08/2011
Contents
Session 12: Test Automation and Tools
Test Automation Test tools
16
Why test?
List of 107 software failures that should have been caught by testing http://www.cs.tau.ac.il/~nachumd/verify/horro r.html One vital consideration from Myers book The Art of Software Testing Mars Climate Orbiter Mars Polar Lander
17
Software Errors
1. 2.
3. 4.
5.
6. 7.
The Mars Climate Orbiter crashed in September 1999 because of a "silly mistake": wrong units in a program. The 1988 shooting down of the Airbus 320 by the USS Vincennes was attributed to the cryptic and misleading output displayed by the tracking software. Death resulted from inadequate testing of the London Ambulance Service software. Several 1985-7 deaths of cancer patients were due to overdoses of radiation resulting from a race condition between concurrent tasks in the Therac-25 software. Errors in medical software have caused deaths. Details in B.W. Boehm, "Software and its Impact: A Quantitative Assessment," Datamation, 19(5), 48-59(1973). An Airbus A320 crashes at an air show. A China Airlines Airbus Industrie A300 crashes on April 26, 1994 killing 264. Recommendations include software modifications.
18
22/08/2011
Software Errors
The Explosion of the Ariane 5
On June 4, 1996 an unmanned Ariane 5 rocket launched by the European Space Agency exploded just forty seconds after its lift-off from Kourou, French Guiana. The rocket was on its first voyage, after a decade of development costing $7 billion. The destroyed rocket and its cargo were valued at $500 million. A board of inquiry investigated the causes of the explosion and in two weeks issued a report. It turned out that the cause of the failure was a software error in the inertial reference system. Specifically a 64 bit floating point number relating to the horizontal velocity of the rocket with respect to the platform was converted to a 16 bit signed integer. The number was larger than 32,767, the largest integer storeable in a 16 bit signed integer, and thus the conversion failed.
19
Software Errors
The Patriot Missile Failure
On February 25, 1991, during the Gulf War, an American Patriot Missile battery in Dharan, Saudi Arabia, failed to track and intercept an incoming Iraqi Scud missile. The Scud struck an American Army barracks, killing 28 soldiers and injuring around 100 other people. A report of the General Accounting office, GAO/IMTEC-92-26, entitled Patriot Missile Defense: Software Problem Led to System Failure at Dhahran, Saudi Arabia reported on the cause of the failure. It turns out that the cause was an inaccurate calculation of the time since boot due to computer arithmetic errors. Specifically, the time in tenths of second as measured by the system's internal clock was multiplied by 1/10 to produce the time in seconds. This calculation was performed using a 24 bit fixed point register. In particular, the value 1/10, which has a non-terminating binary expansion, was chopped at 24 bits after the radix point. The small chopping error, when multiplied by the large number giving the time in tenths of a second, led to a significant error. Indeed, the Patriot battery had been up around 100 hours, and an easy calculation shows that the resulting time error due to the magnified chopping error was about 0.34 seconds. A Scud travels at about 1,676 meters per second, and so travels more than half a kilometer in this time. This was far enough that the incoming Scud was outside the "range gate" that the Patriot tracked. Ironically, the fact that the bad time calculation had been improved in some parts of the code, but not all, contributed to the problem, since it meant that the inaccuracies did not cancel.
20
Software Errors
Y2K
Spent some billions dollars
21
22/08/2011
Chapter 1 of the textbook A Perspective on Testing

Basic Definitions
Error a mistake in design, coding, requirements, even testing Fault the representation of the error Failure what happens when the fault executes
22
A Perspective on Testing
More Definitions
Testing the process of finding errors and of validating the program/system Test Case a test case has Inputs Steps Outputs Expected results Process Test plan, Write test cases Run the test cases Evaluate results
23
Test Cases
Test Cases will be discussed in detail in Session 7 and throughout the course. Testing entails establishing the environment, providing the inputs (running the test case), observing outputs, and comparing to expected outputs. Test Cases are developed, reviewed, used, managed, and saved and hopefully reused!
24
22/08/2011
Identifying Test Cases
Functional Testing
The program is a function that maps input values to output values The only information used is the software specification In our Venn diagram, the Functional Test Cases are a subset of S Further elaborated on in Part II Math background: Chapter 3 We will discuss in Session 6
25
Structural Testing
Uses the information inside the black box the actual implementation In our Venn diagram, the Structural Test Cases are a subset of P. Further elaborated on in Part III of the text Math background: Chapter 4 We will discuss this in Session 4 Main method: Test coverage metrics
26
Comparing the two (Functional vs Structural)
We will discuss this in Sessions 4 and 6 If all specified behaviors have not been implemented, structural test cases will never be able to recognize this. Conversely, if the program implements behaviors that have not been specified, this will never be revealed by functional test cases.
27
22/08/2011
Levels of Testing
Again, this will be covered in detail in Session 2.
Requirements Specification System Testing
Preliminary Design
Integration Testing
Detailed Design
Unit Testing
Coding
28
Testing a Program
A program that we want to test reads in 3 integer values these 3 values are interpreted as the lengths of the sides of a triangle. The program prints a message that states whether the triangle is
Equilateral (all 3 sides equal) Isosceles (exactly 2 of the 3 sides are equal) Scalene (all 3 sides are of a different length)
On a sheet of paper, write specific sets of test data that you feel would adequately test this program.
29
Introduction to software testing
10
22/08/2011
Contents
Definitions, Principles, Axioms Stages of testing Perspectives on Software Testing
31
Definitions, Principles, Axioms
32
Definitions
Testing Verification Validation Error/Defect Black box testing White box testing
33
11
22/08/2011
Definitions of Testing (1)

The process of executing a program (or part of a program) with the intention of finding errors (Myers, Humphrey) The purpose of testing is to find errors. Testing is the process of trying to discover every conceivable fault or weakness in a work product (Myers, Kit) The process of searching for errors (Kaner)
34
Definitions of Testing (2)

Testing the process of finding errors and of validating the program/system (Jorgensen) The purpose of software testing is to find errors and to get them fixed (Bitzenhofer) The purpose of software testing is to reduce risk (E&M)
35
What Testing is Not

Testing is not just another phase of the project Testing is not just finding defects Testing is not debugging Testing is not a final exam
36
12
22/08/2011
Verification and Validation: Myers

Verification: An attempt to find errors by executing a program in a test or simulated environment Validation: An attempt to find errors by executing a program in a real environment
37
Verification and Validation: IEEE

Verification: The process of evaluating a system or component to determine whether the productssatisfy the conditions imposed Validation: The process of evaluating a system or componentto determine whether it satisfies specified requirements.
38
Verification and Validation: Kaner

Verification: Checking a program against the most closely related design documents or specifications (Design Verification Testing) Validation: Checking the program against the published user or system requirements (System Validation Testing)
39
13
22/08/2011
Software Fault Terminology

Error a mistake in design, coding, requirements, even testing Fault the representation of the error Failure what happens when the fault executes
40
Software Fault Terminology: Examples

Error
A wrong loop index Programmer-to-device pointer error
Fault the representation of the error

A loop executes one too many times A programmed parameter gets written to the wrong address
Failure what happens when the fault executes

A table overflows; memory gets overwritten The physician only thinks he/she changed that heart parameter
41
Black and White Box

Black box testing
Designed without knowledge of the programs internal structure and design Based on functional requirements Also called Functional Testing
White box testing

Examines the internal design of the program Requires detailed knowledge of its structure Also called Structural Testing
42
14
22/08/2011
Six Essentials of Software Testing (1)

1.The quality of the test process determines the success of the test effort. 2.Prevent defect migration by using early lifecycle testing techniques. 3.The time for software testing tools is now.
Adapted from Software Testing in the Real World, Edward Kit; AddisonWesley, 1995
43
Six Essentials of Software Testing (2)

4.A real person must take responsibility for improving the testing process. 5.Testing is a professional discipline requiring trained, skilled people. 6.Cultivate a positive team attitude of creative destruction.
Adapted from Software Testing in the Real World, Edward Kit; AddisonWesley, 1995
44
Some Principles and Axioms of Testing (1)

Program testing can be used to show the presence of bugs, but never their absence. (Dijkstra, 1969) One of the most difficult problems in testing is knowing when to stop. It is impossible to test your own program.
45
15
22/08/2011
Some Principles and Axioms of Testing (2)

As the number of detected defects in a piece of software increases, the probability of the existence of more undetected defects also increases. Testing is an extremely creative and intellectually challenging task (Myers again) Exhaustive testing is impossible.
46
Stages of testing
47
The Test Process

Requirements phase Planning phase Design and development phase Execution phase Reporting phase
48
16
22/08/2011
Software Test Execution Stages

Unit Test Design Verification Test (DVT)
Interface, Integration, Exit/Formal
System Validation Test (SVT)

Main, Regression, Acceptance
Customer Acceptance Test
49
Unit Test
Performed by the developers White-box testing Should be done with each release to testing Results should be communicated to the Test Group
50
Typical Entry Criteria (Unit Test)

Module compiles successfully (no errors, approved warnings) Unit Test Plan complete Unit Test Cases complete Tools are available
51
17
22/08/2011
Typical Unit Test Activities

Source level debugging if it wont compile Design and review the Unit Test cases Track test defects (maybe) Measure Unit Test coverage Make sure it builds
52
Typical Exit Criteria (Unit Test)

All tests attempted 99% of the tests executed successfully 85% statement coverage Test cases have been stored under Configuration Management
53
Software Test Stages

Unit Test Design Verification Test
Interface, Integration, Exit/Formal Test Phase
System Validation Test

54
18
22/08/2011
Design Verification Test (DVT)

Performed by the test organization Black box testing Verification of engineering requirements Verification of the software design
55
Three Components of DVT

Interface Testing Integration Testing DVT Exit/Formal Test Phase
56
Typical Entry Criteria (DVT)

Unit Test complete (for all code being tested) Engineering requirements and program design are complete The code is under Configuration Management A current build is available that contains all necessary integrated code Interface and Integration test cases have been written
57
19
22/08/2011
Typical DVT activities

Design, write and review the Test Plan Write the DVT Test Cases Execute Interface/Integration test cases Track defects to the code and to the tests
58
Interface Testing
How does a user interact with the program? What input variables are required? What outputs are produced? Extremely important for e-business applications
59
Integration Testing
Integration between software components Integration with the front-end GUIs Integration between a server and its clients Integration between software and hardware Integration between Web services
60
20
22/08/2011
DVT Exit or Formal Test Phase

This is a requirement for System Test Probably involves running a suite of Regression Tests Defect requirements of some sort must be present The whole point: Are we ready for System Test?
61
Typical Exit Criteria (DVT)

All test cases executed 98% of the test cases executed successfully High severity defects have been fixed and verified
62

Interface, Integration, Exit/Formal

63
21
22/08/2011
System Validation Test (SVT)

Performed by the test organization Validation of customer requirements Three components
Main Test Regression Test SVT Acceptance Test
64
Typical Entry Criteria (SVT)

DVT Exit completes successfully Test Plan has been updated and frozen All test cases have been finalized No high severity problems are open Code is frozen The install works
65
SVT Main Test

Verify the functional requirements of the product from the Customers point of view Includes usability, reliability, performance, and installation Also has its own Entry and Exit criteria Use Cases form a big part of this testing Marketing issues are addressed here too
66
22
22/08/2011
SVT Regression Test

Verify that all defects identified during SVT have been addressed; i.e. all tests run OK
67
System Acceptance Test

Verify final manufacturing build Verify all installation scenarios Verify delivery of all media
68
Typical Exit Criteria (SVT)

All test cases attempted All test cases executed successfully, OR every defect that has been submitted has been addressed No high severity defects are open Test Cases are under Configuration Management Final Test Report has been written
69
23
22/08/2011

Interface, Integration, Exit

Main, Regression, Exit
70
Customer Acceptance Testing

Usually performed at the customer site Must be well-written and simple, but meaningful May be dictated by the customer
71
Perspectives on Software Testing
72
24
22/08/2011
The Objectives and Limits of Testing (1)

Limits
You cannot test a program completely Even if you do find the last bug, youll never know it It takes more time than you have to test less than youd like You will run out of time before you run out of test cases
73

Limits
You cannot You cannot You cannot You cannot isnt!) test every path test every valid input test every invalid input prove a program correct (because it
74

Objectives
The purpose of testing is to find problems The purpose of finding problems is to get them corrected The purpose of testing is to reduce identifiable risks
75
25
22/08/2011
Tests and Test Cases (1)

Test a collection of Test Cases Test Case a test case has inputs, steps, and outputs, as well as expected results Process
write the Test Plan write test cases run the test cases evaluate results
76
Tests and Test Cases (2)

We will discuss Test Cases in detail in Session 7 and throughout the course. Testing entails establishing the environment, providing the inputs (running the test case), observing outputs, and comparing to expected outputs. (a paraphrase) Test Cases are developed, reviewed, used, managed, and saved and hopefully reused!
77
What is Quality?
Quality from the Producers viewpoint: Does the product meet the requirements? Quality from the Customers viewpoint: Fitness for use (reliability, usability, portability, etc.) Quality (one strongly held view): The measure of a products quality is the satisfaction of the customers, not whether it meets a specification
78
26
22/08/2011
Quality Assurance vs Quality Control

Quality Assurance is the activity that establishes and evaluates the processes that produce products
QA improves the product by improving the process
Quality Control is the activity that evaluates the product produced

QC improves the product by finding defects
79
A Generic Testing Approach (SPRAE)

Specification
A written statement of expected software behavior
Premeditation
Written test plans, test scripts, test data, test schedules
Repeatability
Same results every time a result of a mature process
Accountability
Test logs; analysis and interpretation of results
Economy
The cost effectiveness of testing
80
The Testers Toolkit

Static testing Structural testing methods Functional testing methods Non-functional techniques
81
27
22/08/2011
Requirements analysis
Contents
Software Development Life Cycle (SDLC) Software Development stage
Requirements

Some examples Writing test requirements

83
Software Development Life Cycle (SDLC)
84
28
22/08/2011

SDLC is a disciplined and systematic approach that divides the software development process into various phases, such as requirement, design, and coding The phase-wise software development process helps you track schedule, cost, and quality of the software projects
85

There are six phases of SDLC
Feasibility analysis phase includes the analysis of project requirement. Requirement analysis and specification phase includes gathering, analyzing, validating, and specifying requirements. Design phase includes translation of requirements specified in the SRS into a logical structure that can be implemented in a programming language. Coding phase includes implementation of design specified in the design document into executable programming language code. Testing phase includes detection of errors in the software. Maintenance phase includes implementation of changes and new requirements in the software at the customer location.
86
SDLC models
Are a tailored form of SDLC phases. Provide a basis for categorizing and controlling the various activities required to develop and maintain the software system Typical types of SDLC models are
Linear model Iterative model
87
29
22/08/2011
Linear Models
Linear models are suitable for projects where all the requirements are identified and well understood before the design of the software begins. The two types of linear models are:
Waterfall model Prototyping model
88
Waterfall model
Describes the software development process in a linear sequential flow. Defines the software development process in seven phases:
Conception Initiation Analysis Design Construction Integration and testing Implementation and maintenance
89
Waterfall model
90
30
22/08/2011
Prototyping model
Is a sample implementation of the system that shows limited and main functional capabilities of the proposed system. Helps the customer determine how the features will function in the final software.
91
Prototyping model
92
Iterative model
Iterative model is used when the requirements for the software are likely to evolve throughout the development process. The various types of iterative models are
Spiral model Component-based development model Unified process
93
31
22/08/2011
Spiral model
Spiral model
Includes the iterative nature of the prototyping model and the linear nature of the waterfall model. Is ideal for developing software that are released in various versions.
The six phases of spiral model are:

Customer communication Planning Risk analysis Engineering Construction and release Customer evaluation
94
Spiral model
95
Component-Based Development Model

In a component-based development model:
Components are reused and combined with other components in the same or other computers in a distributed network to form an application. All related modules that form a component are tested to ensure that they work together
96
32
22/08/2011
Advantages: Linear Models

Requirements are in the hands of the testers early The project is better designed, with better focus Late changes to requirements or design are limited Test tasks of scheduling, planning, etc., are made easier
97
Advantages: Iterative Models

A shippable product is available early Progress (Where are we?) is much easier to track Many opportunities to reappraise requirements or rethink the design Easy to drop features
98
Disadvantages: Linear Models

Big decisions must be made early It is unknown what will be difficult to implement Changing a specification affects everyone Tremendous pressure on testers to prove that a product is NOT ready for release
99
33
22/08/2011
Disadvantages: Iterative Models

Product design may become a bottleneck Products often ship with half the features missing Big temptation for developers to keep writing new code rather than fix whats broken Lack of early planning may cause problems when later features are added How does Marketing market the product? Constant re-running of the same tests Potentially longer product cycles
100
Effects on Testing: Linear Models

Requirements must be well-reviewed early Test plan can be written early When System Testing does start, you are on the critical path
101
Effects on Testing: Iterative Models

Testing starts as soon as the first level of functionality is reached Test plan is written as you go Requests for design changes can be more easily made
102
34
22/08/2011
Software Development stage

Requirements
103
Software Development Stages

Project/Product Planning Requirements and Design Coding and Documentation Testing and fixing bugs Maintenance and Enhancement
104
Why Projects are Cancelled

Project Impaired Factors % of Responses 1. Incomplete Requirements 13.1% 2. Lack of User Involvement 12.4% 3. Lack of Resources 10.6% 4. Unrealistic Expectations 9.9% 5. Lack of Executive Support 9.3% 6. Changing Requirements & Specifications 8.7% 7. Lack of Planning 8.1% 8. Didn't Need It Any Longer 7.5% 9. Lack of IT Management 6.2% 10. Technology Illiteracy 4.3% Other 9.9%
105
35
22/08/2011
Why Projects are Challenged

Project Challenged Factors % of Responses 1. Lack of User Input 12.8% 2. Incomplete Requirements & Specifications 12.3% 3. Changing Requirements & Specifications 11.8% 4. Lack of Executive Support 7.5% 5. Technology Incompetence 7.0% 6. Lack of Resources 6.4% 7. Unrealistic Expectations 5.9% 8. Unclear Objectives 5.3% 9. Unrealistic Time Frames 4.3% 10. New Technology 3.7% Other 23.0%
106
Why Projects Succeed

Project Success Factors 1. User Involvement 2. Executive Management Support 3. Clear Statement of Requirements 4. Proper Planning 5. Realistic Expectations 6. Smaller Project Milestones 7. Competent Staff 8. Ownership 9. Clear Vision & Objectives 10. Hard-Working, Focused Staff Other % of Responses 15.9% 13.9% 13.0% 9.6% 8.2% 7.7% 7.2% 5.3% 2.9% 2.4% 13.9%
107
Software Requirements
According to software engineering theory, software requirements should contain a finite list of behaviors and features, and each requirement should be written to be verifiable. Given a finite list of requirements and a set of completion criteria, requirements-based testing becomes a feasible process.
108
36
22/08/2011
Typical Requirements (FURPS)

Functionality does it do what I need? Usability is it easy to do what I need? Reliability can it do what I need when I need it? Performance does it do what I need in a reasonable amount of time? Supportability what if is DOESNT do what I need?
109
Functionality Requirements
Correctness
Extent to which a program satisfies its specifications and meets customer expectations
Features/Capabilities Generality Security

Extent to which access to software or data by unauthorized persons can be controlled
110
Usability Requirements
Ease of use
Measures the effort required to learn, operate, prepare input for, and interpret the output of, a program
Intuitive Human factors Consistency Documentation
111
37
22/08/2011
Reliability Requirements
Frequency and severity of failure - MTBF Recoverability Predictability Accuracy
112
Performance Requirements
Speed Efficiency Resource consumption Throughput Response time
113
Supportability Requirements
Testability Adaptability Maintainability
Effort required to locate and fix errors
Serviceability
Effort required to modify/update a program
Installability Portability
114
38
22/08/2011
Contents

115
Testing Requirements
Usually done by review Are they complete? Are they reasonable? Can they be tested?
116
Requirements Testing
Testing of the requirements Determine all testable requirements Determine all non-testable requirements Write and execute the tests
117
39
22/08/2011
Reviewing Software Requirements

Learn to recognize what has not been specified (limits, consequences) Learn to recognize what is vague or ambiguous (sometimes, faster) As a tester:
Make sure testers are included in the reviews! Push back to get requirements clarified Always be thinking And just how am I going to test this?
118
Reviewing the Design

Helps determine how to test Shows what to look for Do you need to read the code?
119
Some examples
Some requirements statements Some examples of Test Cases
120
40
22/08/2011
Requirements Example 1
After a high temperature is detected, an alarm must be raised quickly
121
Testable Requirements: Ex. 1

If the threshold temperature is reached, an alarm is triggered. As long as the temperature remains below the threshold temperature, no alarm is triggered. If the temperature dips below the threshold (having gone above it), the alarm does not cease.
122
Novice users should be able to learn the interface with little training When thinking about a test for this, ask yourself: What would be the criterion for saying the test passed? failed?
123
41
22/08/2011
There was a hidden requirement in this feature change: The menu option 6. Utilities | 1. Security will now display a new submenu rather than a Security dialog box. The submenu will have 3 items - 1. CSM, 2. CIS, and 3. Options.
124
How many testable requirements are here? The Access Profile choices for 2. CIS and 3. Options will be grayed out unless those options have been purchased and installed.
125
Testable Requirements for Ex. 4

The Access Profile choice for CIS is available if and only if the CIS feature has been installed. The Access Profile choice for Options is available if and only if the Options feature has been installed.
126
42
22/08/2011
Some Test Cases for Example 4

1. 2. 3. 4.
If the CIS option is installed, Access Profile choices will be available for CIS If the Options option is installed, Access Profile choices will be available for Options If the CIS option is not installed, Access Profile choices for CIS will not be available If the Options option is not installed, Access Profile choices for Options will not be available
127
An upgraded security system was being installed with enhanced password protection. Here is one requirement: Existing passwords which do not conform to the new restrictions will be reset to the User ID, and a list of those users whose passwords were reset will be generated.
128
Untestable requirement Number of invalid attempts before a user is suspended: This count is reset when a successful logon is completed for the user. The default is that the User will never be suspended. The valid range is from 0 (never) to 10 attempts.
129
43
22/08/2011
Testable Requirements for Example 6

Requirement: The number of invalid attempts before suspension may be set to 2 Test Case 1: After 2 bad attempts to log on, the Users USERID is suspended. Test Case 2: A bad attempt, followed by a good attempt, can then be followed by 2 bad attempts again before suspension. The same holds for 1, 3-10 Requirement: The number of invalid attempts before suspension cannot be set to 11 Test Case 3: Attempt to set invalid attempts to 11, or verify it cant be done.
130
Vagueness in different disguises: The Advanced Color Module will improve the print quality of the module. The Advanced Color Module firmware will increase the number of color shades per panel from 4 - 9 shades to 25 - 30 shades. The Advanced Color Module will reduce the tiling effects associated with previous modules.
131
A lot of test cases from just one sentence: All new and modified screens will provide online help.
132
44
22/08/2011
The user interface must be easy to use. In case of failure, the system must be easy to recover and must suffer minimum loss of important data.
133
Lessons Learned
Watch for words like always, never, every, etc. They are difficult to verify. Words like must, should, shall, will, has to, etc. signal potential requirements. Watch for hidden or unstated requirements. Watch for vague requirements: high, quickly, better, faster, should be able to. For negative testing, ask What happens if the test case fails? Ask yourself the question: How will I know if the requirement is not met?
134
Structural testing
45
22/08/2011
Contents
White box testing / Structural testing Graph Theory Control flow criteria Data flow criteria Graph Coverage for Source Code
136
White box testing / Structural testing
137
What is white-box software testing?

What is white-box software testing?
Basic idea is to test a program based on the structure of a program
Pseudonyms: Glass Box, Clear Box, Crystal Box, Structural Testing To distinguish from functional (requirements-based, black-box testing) What do you need for white-box testing?
A white-box testing model and test criteria A white-box test design and generation method Program source code
138
46
22/08/2011
White-Box Testing Objectives

The major objective of white-box testing is to focus on internal program structure, and discover all internal program errors
Errors made by developers
The major testing focuses

Program structures Program statements and branches Various kinds of program paths Program internal logic and data structures Program internal behaviors and states
139
Traditional White-Based Software Testing Methods

Test Model: control flow graph Test case design
Various white-box testing methods generate test cases based on a given control flow graph for a program
The goal is to
Guarantee that all independent paths within a module have been exercised at least once Exercise all logical decisions one their true and false sides Execute all loops at their boundaries and within their operational bounds Exercise internal data structures to assure their validity Exercise all data define and use paths
140
No guarantees
Executing all control flow elements does not guarantee finding all faults
Execution of a faulty statement may not always result in a failure The state may not be corrupted when the statement is executed with some data values Corrupt state may not propagate through execution to eventually lead to failure
What is the value of structural coverage?

Increases confidence in thoroughness of testing Removes some obvious inadequacies
141
47
22/08/2011
Graph theory
142
Definition of a Graph
A set N of nodes, N is not empty A set N0 of initial nodes, N0 is not empty A set Nf of final nodes, Nf is not empty A set E of edges, each edge from one node to another
( ni , nj ), i is predecessor, j is successor
143
Three Example Graphs

0 0 1 2 0
Not a valid graph
3 N0 = { 0 } Nf = { 3 }
8 N0 = { 0, 1, 2 } Nf = { 7, 8, 9 }
3 N0 = { } Nf = { 144 3}
48
22/08/2011
Paths in Graphs
Path : A sequence of nodes [n1, n2, , nM]
Each pair of nodes is an edge
Length : The number of edges

A single node is a path of length 0
Subpath : A subsequence of nodes in p is a subpath of p Reach (n) : Subgraph that can be reached from n
0 1 2
Paths [ 0, 3, 7 ] 6
Reach (0) = { 0, 3, 4, 7, 8, 5, 1, 9 } Reach ({0, 2}) = G Reach([2,6]) = {2, 6, 9}

145
[ 1, 4, 8, 5, 1 ] [ 2, 6, 9 ]
Test Paths and SESEs

Test Path : A path that starts at an initial node and ends at a final node Test paths represent execution of test cases
Some test paths can be executed by many tests Some test paths cannot be executed by any tests
SESE graphs : All test paths start at a single node and end at another node
Single-entry, single-exit N0 and Nf have exactly one node
1 0 2 3
4 6 5
Double-diamond graph Four test paths [ 0, 1, 3, 4, 6 ] [ 0, 1, 3, 5, 6 ] [ 0, 2, 3, 4, 6 ] [ 0, 2, 3, 5, 6 ]

146
Tests and Test Paths

path (t) : The test path executed by test t path (T) : The set of test paths executed by the set of tests T Each test executes one and only one test path A location in a graph (node or edge) can be reached from another location if there is a sequence of edges from the first location to the second
Syntactic reach : A subpath exists in the graph Semantic reach : A test exists that can execute that subpath
147
49
22/08/2011
Testing and Covering Graphs

We use graphs in testing as follows :
Developing a model of the software as a graph Requiring tests to visit or tour specific sets of nodes, edges or subpaths
Test Requirements (TR) : Describe properties of test paths Test Criterion : Rules that define test requirements Satisfaction : Given a set TR of test requirements for a
criterion C, a set of tests T satisfies C on a graph if and only if for every test requirement in TR, there is a test path in path(T) that meets the test requirement tr
Control flow (Structural) Coverage Criteria : Defined on a
graph just in terms of nodes and edges

Data Flow Coverage Criteria : Requires a graph to be
annotated with references to variables
148
Control flow test criteria
149
Node and Edge Coverage

The first (and simplest) two criteria require that each node and edge in a graph be executed Node Coverage (NC) : Test set T satisfies node coverage on graph G if only if for every syntactically reachable node n in N, there is some path p in path(T) such that p visits n.
We can abbreviate it in terms of the set of test requirements
Node Coverage (NC) : TR contains each reachable node in G.

150
50
22/08/2011
Node and Edge Coverage

Edge coverage is slightly stronger than node coverage Edge Coverage (EC) : TR contains each reachable path of length up to 1, inclusive, in G.
The length up to 1 allows for graphs with one node and no
edges
NC and EC are only different when there is an edge and
another subpath between a pair of nodes (as in an if-else statement)

0 1 2 Node Coverage : TR = { 0, 1, 2 } Test Path = [ 0, 1, 2 ] Edge Coverage : TR = { (0,1), (0, 2), (1, 2) } Test Paths = [ 0, 1, 2 ] [ 0, 2 ]
151
Covering Multiple Edges

Edge-pair coverage requires pairs of edges, or subpaths of length 2 Edge-Pair Coverage (EPC) : TR contains each reachable Edgepath of length up to 2, inclusive, in G.
The length up to 2 is used to include graphs that have less
than 2 edges
The logical extension is to require all paths
Complete Path Coverage (CPC) : TR contains all paths in G.

Unfortunately, this is impossible if the graph has a loop, so a
weak compromise is to make the tester decide which paths: Specified Path Coverage (SPC) : TR contains a set S of 152 test paths, where S is supplied as a parameter. parameter.
Structural Coverage Example

Node Coverage TR = { 0, 1, 2, 3, 4, 5, 6 } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] 0 1 2 3 4 5 6 Edge Coverage TR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ] Edge-Pair Coverage TR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6], [2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] } Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 5, 4, 6 ] Complete Path Coverage Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 5, 4, 6 ] 153
51
22/08/2011
Loops in Graphs
If a graph contains a loop, it has an infinite number of paths Thus, CPC is not feasible SPC is not satisfactory because the results are subjective and vary with the tester Attempts to deal with loops:
1970s : Execute cycles once ([4, 5, 4] in previous example, informal) 1980s : Execute each loop, exactly once (formalized) 1990s : Execute loops 0 times, once, more than once (informal description) 2000s : Prime paths
154
Simple Paths and Prime Paths

Simple Path : A path from node ni to nj is simple if no node appears more than once, except possibly the first and last nodes are the same
No internal loops Includes all other subpaths A loop is a simple path
Prime Path : A simple path that does not appear as a proper subpath of any other simple path
Simple Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ], [ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ], [ 2, 3, 0, 1 ], [ 0, 1, 3 ], [ 0, 2, 3 ], [ 1, 3, 0 ], [ 2, 3, 0 ], [ 3, 0, 1 ], [3, 0, 2 ], [ 0, 1], [ 0, 2 ], [ 1, 3 ], [ 2, 3 ], [ 3, 0 ], [0], [1], [2], [3] Prime Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ], [ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ], 155 [ 2, 3, 0, 1 ]
0 1 3 2
Prime Path Coverage

A simple, elegant and finite criterion that requires loops to be executed as well as skipped
Prime Path Coverage (PPC) : TR contains each prime path in G.
Will tour all paths of length 0, 1, That is, it subsumes node, edge, and edge-pair coverage
156
52
22/08/2011
Round Trips
Round-Trip Path : A prime path that starts and ends at the same node Simple Round Trip Coverage (SRTC) : TR contains at least one roundround-trip path for each reachable node in G that begins and ends a roundround-trip path path. . Complete Round Trip Coverage (CRTC) : TR contains all round-trip paths for each reachable node in G. round These criteria omit nodes and edges that are not in round trips That is, they do not subsume edge-pair, edge, or node coverage
157
Prime Path Example

The previous example has 38 simple paths Only nine prime paths
0 1 2 3 4 5 6 Prime Paths [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ] [ 0, 2, 4, 5] [ 0, 2, 4, 6 ] [ 5, 4, 6 ] [ 4, 5, 4 ] [ 5, 4, 5 ] Execute loop 0 times Execute loop once Execute loop more than once
158
! means path terminates
Simple & Prime Path Example

Simple paths 0 1 2 3 4 5 6 Len 4 [0, 1, 2, 3, 6] ! [0, 1, 2, 4, 6] ! [0, 1, 2, 4, 5] ! Len 0 [0] [1] [2] [3] [4] [5] [6] ! Len 1 [0, 1] [0, 2] [1, 2] [2, 3] [2, 4] [3, 6] ! [4, 6] ! [4, 5] [5, 4] Len 2 [0, 1, 2] [0, 2, 3] [0, 2, 4] [1, 2, 3] [1, 2, 4] [2, 3, 6] ! [2, 4, 6] ! [2, 4, 5] ! [4, 5, 4] * [5, 4, 6] ! [5, 4, 5] * Len 3 [0, 1, 2, 3] [0, 1, 2, 4] [0, 2, 3, 6] ! [0, 2, 4, 6] ! [0, 2, 4, 5] ! [1, 2, 3, 6] ! [1, 2, 4, 5] ! [1, 2, 4, 6] ! * means path cycles
Prime Paths
159
53
22/08/2011
Data flow test criteria
160
Data Flow Criteria

Goal: Try to ensure that values are computed and used correctly Definition (def) : A location where a value for a variable is stored into memory Use : A location where a variables value is accessed def (n) or def (e) : The set of variables that are defined by node n or edge e use (n) or use (e) : The set of variables that are used by node n or edge e
Z = X*2 X = 42 0 1 3 2 5 Z = X-8 4 6 Defs: def (0) = {X} def (4) = {Z} def (5) = {Z} Uses: use (4) = {X} use (5) = {X}
161
DU Pairs and DU Paths

DU pair : A pair of locations (li, lj) such that a variable v is defined at li and used at lj Def-clear : A path from li to lj is def-clear with respect to variable v if v is not given another value on any of the nodes or edges in the path Reach : If there is a def-clear path from li to lj with respect to v, the def of v at li reaches the use at lj du-path : A simple subpath that is def-clear with respect to v from a def of v to a use of v du (ni, nj, v) the set of du-paths from ni to nj du (ni, v) the set of du-paths that start at ni
162
54
22/08/2011
Data Flow Test Criteria

Three criteria
Use every def Get to every use Follow all du-paths
163
Data Flow Test Criteria

First, we make sure every def reaches a use
AllAll -defs coverage (ADC) : For each set of du du-paths S = du (n, v), TR contains at least one path d in S.
Then we make sure that every def reaches all possible uses
All-uses coverage (AUC) : For each set of duAlldu-paths to uses S = du (ni, nj, v), TR contains at least one path d in S.
Finally, we cover all the paths between defs and uses
AllAll -dudu-paths coverage (ADUPC) : For each set S = du (ni, nj, v), TR contains every path d in S. 164
Data Flow Testing Example

Z = X*2 X = 42 0 1 3 2 5 Z = X-8 4 6
All-defs for X [ 0, 1, 3, 4 ]
All-uses for X [ 0, 1, 3, 4 ] [ 0, 1, 3, 5 ]
All-du-paths for X [ 0, 1, 3, 4 ] [ 0, 2, 3, 4 ] [ 0, 1, 3, 5 ] [ 0, 2, 3, 5 ]
165
55
22/08/2011
Graph Coverage Criteria Subsumption
Complete Path Coverage CPC Prime Path Coverage PPC
All-DU-Paths Coverage ADUP All-uses Coverage AUC All-defs Coverage ADC
Edge-Pair Coverage EPC Edge Coverage EC Node Coverage NC
Complete Round Trip Coverage CRTC Simple Round Trip Coverage SRTC
166
Graph Coverage for Source Code
167
Graph Coverage for Source Code

The most common application of graph criteria is to program source Graph : Usually the control flow graph (CFG) Node coverage : Execute every statement Edge coverage : Execute every branch Loops : Looping structures such as for loops, while loops, etc. Data flow coverage : Augment the CFG defs are statements that assign values to variables uses are statements that use variables
168
56
22/08/2011
Control Flow Graphs

A CFG models all executions of a method by describing control structures Nodes : statements or sequences of statements (basic blocks) Edges : Transfers of control Basic Block : A sequence of statements such that if the first statement is executed, all statements will be (no branches) CFGs are sometimes annotated with extra information
branch predicates defs uses
Rules for translating statements into graphs

169
CFG : The if Statement

if (x < y) { y = 0; x = x + 1; } else { x = y; } 1
x<y y=0 x=x+1 x >= y
2 4
x=y
if (x < y) { y = 0; x = x + 1; }
1
x<y y=0 x=x+1
x >= y
3
170
CFG : The if-Return Statement

if (x < y) { return; } print (x); return; 1
x<y return
x >= y
print (x) return
No edge from node 2 to 3. The return nodes must be distinct.
171
57
22/08/2011
Loops
Loops require extra nodes to be added Nodes that do not represent statements or basic blocks
172
CFG : while and for Loops

x = 0; while (x < y) { y = f (x, y); x = x + 1; }
x=0
1 dummy node 2
x<y
x >= y
3
y =f(x,y) x=x+1
implicitly initializes loop
x=0
2
x<y x >= y
for (x = 0; x < y; x++) { y = f (x, y); }
y = f (x, y)
x=x+1
implicitly increments loop
173
CFG : The case (switch) Structure

read ( c) ; switch ( c ) { case N: y = 25; break; case Y: y = 50; break; default: y = 0; break; } print (y);
1
c == N
read ( c );
c == Y default
y = 25; break;
3
y = 50; break;
y = 0; break;
5
print (y);
174
58
22/08/2011
Example Control Flow

public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } med = numbers [ length / 2 ]; mean = sum / (double) length; varsum = 0; for (int i = 0; i < length; i++) { varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println System.out.println System.out.println System.out.println System.out.println } ("length: " + length); ("mean: " + mean); ("median: " + med); ("variance: " + var); ("standard deviation: " + sd);
175
Control Flow Graph for Stats

public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } med = numbers [ length / 2 ]; mean = sum / (double) length;
i=0 i >= length
varsum = 0; i < length for (int i = 0; i < length; i++) i++ 4 { varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println System.out.println System.out.println System.out.println System.out.println } ("length: " + length); ("mean: " + mean); ("median: " + med); ("variance: " + var); ("standard deviation: " + sd);
5 i=0
6 i < length i >= length 7 8

176
Control Flow TRs and Test Paths PPC

1
3 4 5
Prime Path Coverage TR Test Paths A. [ 3, 4, 3 ] i. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ] B. [ 4, 3, 4 ] ii. [ 1, 2, 3, 4, 3, 4, 3, C. [ 7, 6, 7 ] 5, 6, 7, 6, 7, 6, 8 ] D. [ 7, 6, 8 ] iii. [ 1, 2, 3, 4, 3, 5, 6, 8 ] E. [ 6, 7, 6 ] iv. [ 1, 2, 3, 5, 6, 7, 6, 8 ] F. [ 1, 2, 3, 4 ] v. [ 1, 2, 3, 5, 6, 8 ] G. [ 4, 3, 5, 6, 7 ] H. [ 4, 3, 5, 6, 8 ] I. [ 1, 2, 3, 5, 6, 7 ] J. [ 1, 2, 3, 5, 6, 8 ]
59
22/08/2011
Data Flow Coverage for Source

def : a location where a value is stored into memory
x appears on the left side of an assignment (x = 44;) x is an actual parameter in a call and the method changes its value x is a formal parameter of a method (implicit def when method starts) x is an input to a program
use : a location where variables value is accessed

x appears on the right side of an assignment x appears in a conditional test x is an actual parameter to a method x is an output of the program x is an output of a method in a return statement
If a def and a use appear on the same node, then it is only a DUpair if the def occurs after the use and the node is in a loop
178
Example Data Flow

public static void computeStats (int [ ] numbers) { int length = numbers.length; double med, var, sd, mean, sum, varsum; sum = 0; for (int i = 0; i < length; i++) { sum += numbers [ i ]; } med = numbers [ length / 2 ]; mean = sum / (double) length; varsum = 0; for (int i = 0; i < length; i++) { varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean)); } var = varsum / ( length - 1.0 ); sd = Math.sqrt ( var ); System.out.println System.out.println System.out.println System.out.println System.out.println } ("length: " + length); ("mean: " + mean); ("median: " + med); ("variance: " + var); ("standard deviation: " + sd);
179
Control Flow Graph for Stats

1
( numbers ) sum = 0 length = numbers.length
i=0
3
i < length
i >= length med = numbers [ length / 2 ] mean = sum / (double) length; varsum = 0 i=0 i >= length var = varsum / ( length - 1.0 ) sd = Math.sqrt ( var ) print (length, mean, med, var, sd)
4
sum += numbers [ i ] i++
6
i < length varsum = i++
180
60
22/08/2011
CFG for Stats With Defs & Uses 1 def (1) = { numbers, sum, length }
2
def (2) = { i }
3
use (3, 4) = { i, length }
use (3, 5) = { i, length }
4
def (4) = { sum, i } use (4) = { sum, numbers, i }
def (5) = { med, mean, varsum, i } use (5) = { numbers, length, sum }
6
use (6, 7) = { i, length }
use (6, 8) = { i, length }
7 def (7) = { varsum, i } use (7) = { varsum, numbers, i, mean }
8
181
Defs and Uses Tables for Stats

Node 1 2 3 4 5 6 7 8 { varsum, i } { var, sd } { varsum, numbers, i, mean } { varsum, length, var, mean, med, var, sd } { sum, i } { med, mean, varsum, i } { numbers, i, sum } { numbers, length, sum } Def { numbers, sum, length } {i} Use Edge (1, 2) (2, 3) (3, 4) (4, 3) (3, 5) (5, 6) (6, 7) (7, 6) (6, 8) { i, length }
182
Use
{ i, length } { i, length } { i, length }
DU Pairs for Stats

variable numbers length med var sd mean sum varsum i (1, 4) (1, 5) (1, 7) (1, 5) (1, 8) (1, (3,4)) (1, (3,5)) (1, (6,7)) (1, (6,8)) (5, 8) (8, 8) (8, 8) (5, 7) (5, 8) (1, 4) (1, 5) (4, 4) (4, 5) (5, 7) (5, 8) (7, 7) (7, 8) (2, 4) (2, (3,4)) (2, (3,5)) (2, 7) (2, (6,7)) (2, (6,8)) (4, 4) (4, (3,4)) (4, (3,5)) (4, 7) (4, (6,7)) (4, (6,8)) (5, 7) (5, (6,7)) (5, (6,8)) (7, 7) (7, (6,7)) (7, (6,8)) DU Pairs
61
22/08/2011
DU Paths for Stats No Duplicates

There are 38 DU paths for Stats, but only 12 unique
[ 1, 2, 3, 4 ] [ 1, 2, 3, 5 ] [ 1, 2, 3, 5, 6, 7 ] [ 1, 2, 3, 5, 6, 8 ] [ 2, 3, 4 ] [ 2, 3, 5 ] [ 4, 3, 4 ] [ 4, 3, 5 ] [ 5, 6, 7 ] [ 5, 6, 8 ] [ 7, 6, 7 ] [ 7, 6, 8 ]
5 expect a loop not to be entered 5 require at least one iteration of a loop 2 require at least two iteration of a loop
184
Test Cases and Test Paths

Test Case : numbers = (44) ; length = 1 Test Path : [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ] Additional DU Paths covered
[ 1, 2, 3, 4 ] [ 2, 3, 4 ] [ 4, 3, 5 ] [ 5, 6, 7 ] [ 7, 6, 8 ] The five stars that require at least one iteration of a loop
Test Case : numbers = (2, 10, 15) ; length = 3 Test Path : [ 1, 2, 3, 4, 3, 4, 3, 4, 3, 5, 6, 7, 6, 7, 6, 7, 6, 8 ] DU Paths covered
[ 4, 3, 4 ] [ 7, 6, 7 ] The two stars that require at least two iterations of a loop
Other DU paths require arrays with length 0 to skip loops But the method fails with divide by zero on the statement
mean = sum / (double) length; A fault was found
185
186
62
22/08/2011

A finite state machine (FSM) is a graph that describes how software variables are modified during execution Nodes : States, representing sets of values for key variables Edges : Transitions, possible changes in the state
switch up Off switch down
187
On
Finite State Machine Two Variables

Tropical Depression
circulation = yes windspeed < 39mph
Tropical Storm
circulation = yes windspeed = 39..73 mph
Something Else
circulation = no windspeed = 0..38 mph
Major Hurricane
circulation = yes windspeed >= 110 mph
Hurricane
circulation = yes windspeed 74..109 mph
Other variables may exist but not be part of state
188
Finite State Machines are Common

FSMs can accurately model many kinds of software
Embedded and control software (think electronic gadgets) Abstract data types Compilers and operating systems Web applications
Creating FSMs can help find software problems Numerous languages for expressing FSMs
UML statecharts Automata State tables (SCR) Petri nets
Limitations : FSMs are not always practical for programs that have lots of states (for example, GUIs)
189
63
22/08/2011
Annotations on FSMs
FSMs can be annotated with different types of actions
Actions on transitions Entry actions to nodes Exit actions on nodes
Actions can express changes to variables or conditions on variables These slides use the basics:
Preconditions (guards) : conditions that must be true for transitions to be taken Triggering events : changes to variables that cause transitions to be taken
This is close to the UML Statecharts, but not exactly the same
190
Example Annotations
open elevator door Closed pre: elevSpeed = 0 trigger: openButton = pressed Open
191
Covering FSMs
Node coverage : execute every state (state coverage) Edge coverage : execute every transition (transition coverage) Edge-pair coverage : execute pairs of transitions (transition-pair) Data flow:
Nodes often do not include defs or uses of variables Defs of variables in triggers are used immediately (the next state) Defs and uses are usually computed for guards, or states are extended FSMs typically only model a subset of the variables
Generating FSMs is often harder than covering them

192
64
22/08/2011
Deriving FSMs
With some projects, a FSM (such as a statechart) was created during design
Tester should check to see if the FSM is still current with respect to the implementation
If not, it is very helpful for the tester to derive the FSM Strategies for deriving FSMs from a program:
1. 2. 3. 4.
Combining control flow graphs Using the software structure Modeling state variables Using implicit or explicit specifications Class Watch uses class Time
Discussion of these strategies based on a digital watch
193
Class Watch
class Watch
// Constant values for the button (inputs) private static final int NEXT = 0; private static final int UP = 1; private static final int DOWN = 2; // Constant values for the state private static final int TIME = 5; private static final int STOPWATCH = 6; private static final int ALARM = 7; // Primary state variable private int mode = TIME; // Three separate times, one for each state private Time watch, stopwatch, alarm;
class Time ( inner class )

private int hour = 0; private int minute = 0; public void changeTime (int button) public String toString ()
public Watch () // Constructor public void doTransition (int button) // Handles inputs public String toString () // Converts values
194
// Takes the appropriate transition when a button is pushed. public void doTransition (int button) { switch ( mode ) { case TIME: if (button == NEXT) mode = STOPWATCH; else watch.changeTime (button); break; case STOPWATCH: if (button == NEXT) mode = ALARM; else stopwatch.changeTime (button); break; case ALARM: if (button == NEXT) mode = TIME; else alarm.changeTime (button); break; default: break; } } // end doTransition()
// Increases or decreases the time. // Rolls around when necessary. public void changeTime (int button) { if (button == UP) { minute += 1; if (minute >= 60) { minute = 0; hour += 1; if (hour >= 12) hour = 0; } } else if (button == DOWN) { minute -= 1; if (minute < 0) { minute = 59; hour -= 1; if (hour <= 0) hour = 12; } } } // end changeTime()
195
65
22/08/2011
1. Combining Control Flow Graphs

The first instinct for inexperienced developers is to draw CFGs and link them together This is really not a FSM Several problems
Methods must return to correct callsites built-in nondeterminism Implementation must be available before graph can be built This graph does not scale up
Watch example
196
CFGs for Watch

doTransition () 1 2 2 3 4 6 8 11 12 13 10 14 12
197
changeTime () 1 2 4 6 8 10
4 10
2. Using the Software Structure

A more experienced programmer may map methods to states These are really not states Problems
Subjective different testers get different graphs Requires in-depth knowledge of implementation Detailed design must be present
Watch example
198
66
22/08/2011
Software Structure for Watch

inMain
button button==NEXT / change mode button==UP or button==DOWN ??? Should inChangeTime transit back to inMain??? button==UP / change hour, minute
inDoTransition
inChangeTime
??? Not clear what triggers this button==DOWN / change hour, minute
199
3. Modeling State Variables

More mechanical State variables are usually defined early First identify all state variables, then choose which are relevant In theory, every combination of values for the state variables defines a different state In practice, we must identify ranges, or sets of values, that are all in one state Some states may not be feasible
200
State Variables in Watch

Constants NEXT, UP, DOWN Not relevant, really TIME, STOPWATCH, ALARM just values Non-constants int mode Time watch, stopwatch, alarm Time class variables int hour Merge into the three Time variables int minute
201
67
22/08/2011
State Variables and Values

Relevant State Variables
These three ranges actually represent quite a bit of thought and semantic domain knowledge of the program
mode : TIME, STOPWATCH, ALARM watch : 12:00, 12:01..12:59, 01:00..11:59 stopwatch : 12:00, 12:01..12:59, 01:00..11:59 alarm : 12:00, 12:01..12:59, 01:00..11:59
Total 3*3*3*3 = 81 states But the three watches are independent, so we only care about 3+3+3 = 9 states (still a messy graph )
202
State Variable Model for Watch

mode = TIME watch = 12:00
next next next
mode = TIME watch = 12:01.. 12:59

next next next
mode = TIME watch = 01:00..12:59

next next next
mode = STOP stopw = 12:00

next next next
mode = STOP stopw = 12:01.. 12:59

next next next
mode = STOP stopw = 01:00..12:59

next next next
mode = ALARM alarm = 12:00
next nextnext
mode = ALARM alarm = 12:01.. 12:59

next next next next
mode = ALARM alarm = 01:00..12:59

next next
203
NonDeterminism in the State Variable Model

Each state has three outgoing transitions on next This is a form of non-determinism, but it is not reflected in the implementation Which transition is taken depends on the current state of the other watch The 81-state model would not have this non-determinism This situation can also be handled by a hierarchy of FSMs, where each watch is in a separate FSM and they are organized together
204
68
22/08/2011
4. Using Implicit or Explicit Specifications

Relies on explicit requirements or formal specifications that describe software behavior These could be derived by the tester These FSMs will sometimes look much like the implementationbased FSM, and sometimes much like the state-variable model
For watch, the specification-based FSM looks just like the statevariable FSM, so is not shown
The disadvantage of FSM testing is that some implementation decisions are not modeled in the FSM
205
Tradeoffs in Applying Graph Coverage Criteria to FSMs

There are two advantages
1. 2.
Tests can be designed before implementation Analyzing FSMs is much easier than analyzing source
There are three disadvantages

1. 2. 3.
Some implementation decisions are not modeled in the FSM There is some variation in the results because of the subjective nature of deriving FSMs Tests have to mapped to actual inputs to the program the names that appear in the FSM may not be the same as the names in the program
206
Static testing
69
22/08/2011
Contents
Reviews and the test process Types of review Static analysis
208
Reviews and the test process
209
Static testing
Static: Not Dynamic, i.e. you dont execute the code This is the process of testing some documentation products The primary goal of Static Testing is to reduce defects in the software by reducing defects in the documentation from which the software is developed
210
70
22/08/2011
Static techniques
Do not execute code
People techniques
Specific techniques
Desk checking Developer tests his/her own documentation Walkthroughs Developer walks the participants through the documentation Peer Reviews/Inspections More formal documentation is distributed beforehand
211
Benefits of reviews
Development productivity improvement Reduced development timescales Reduced testing time and cost Lifetime cost reductions Reduced fault levels Improved customer relations
212
Reviews are cost-effective

10 times reduction in faults reaching test, testing cost reduced by 50% to 80%
Freedman & Weinberg, Handbook of Walkthroughs, Inspections & Technical Reviews
reduce faults by a factor of 10

Yourdon, Structured Walkthroughs
25% reduction in schedules, remove 80% 95% of faults at each stage, 28 times reduction in maintenance cost, many others
Gilb & Graham, Software Inspection
213
71
22/08/2011
What can be Inspected?

Anything written down can be Inspected
policy, strategy, business plans, marketing or advertising material, contracts system requirements, feasibility studies, acceptance test plans test plans, test designs, test cases, test results system designs, logical & physical software code user manuals, procedures, training material
214
What can be reviewed?

anything which could be Inspected
i.e. anything written down
plans, visions, big picture, strategic directions, ideas project progress

work completed to schedule, etc.
Should we develop this marketing options
215
What to review / Inspect?

Tests Requirements Tests Functions Tests Design Tests Code Unit Test
216
Accept. Test
System Test
Integration T
72
22/08/2011
Costs of reviews
Rough guide: 5%-15% of development effort
half day a week is 10%
Effort required for reviews

planning (by leader / moderator) preparation / self-study checking meeting fixing / editing / follow-up recording & analysis of statistics / metrics process improvement (should!)
217
Types of review
218
Types of review of documents

Informal Review (undocumented)
widely viewed as useful and cheap (but no one can prove it!). A helpful first step for chaotic organisations.
Technical Review (or peer review)

includes peer and technical experts, no management participation. Normally documented, fault-finding. Can be rather subjective.
Decision-making Review
group discusses document and makes a decision about the content, e.g. how something should be done, go or no-go decision, or technical comments
219
73
22/08/2011
Types of review of documents

Walkthrough
author guides the group through a document and his or her thought processes, so all understand the same thing, consensus on changes to make
Inspection
formal individual and group checking, using sources and standards, according to generic and specific rules and checklists, using entry and exit criteria, Leader must be trained & certified, metrics required
220
Reviews in general 1
Objectives / goals
validation & verification against specifications & standards achieve consensus (excluding Inspection) process improvement (ideal, included in Inspection)
221
Activities
planning overview / kickoff meeting (Inspection) preparation / individual checking review meeting (not always) follow-up (for some types) metrics recording & analysis (Inspections and sometimes reviews)
222
74
22/08/2011
Roles and responsibilities
Leader / moderator - plans the review / Inspection, chooses participants, helps & encourages, conducts the meeting, performs follow-up, manages metrics Author of the document being reviewed / Inspected Reviewers / Inspectors - specialised fault-finding roles for Inspection Managers - excluded from some types of review, need to plan project time for review / Inspection Others: e.g. Inspection/ review Co-ordinator
223
Deliverables
Changes (edits) in review product Change requests for source documents (predecessor documents to product being reviewed / Inspected) Process improvement suggestions
to the review / Inspection process to the development process which produced the product just reviewed / Inspected
Metrics (Inspection and some types of review)

224
Pitfalls (they dont always work!)
Lack of training in the technique (especially Inspection, the most formal) Lack of or quality of documentation - what is being reviewed / Inspected Lack of management support Failure to improve processes
225
75
22/08/2011
Inspection is different
the document to be reviewed is given out in advance typically dozens of pages to review instructions are "please review this" not just product, sources chunk or sample training, roles
226
Inspection is different
some people have time to look through it and make comments before the meeting (which is difficult to arrange) the meeting often lasts for hours entry criteria to meeting, may not be worth holding
2 max., often much shorter
227
The Inspection Process

Software Development Stage
Change Request
Process Improvement
. Entry Planning Kick off Ind Chk Meet
Exit Edit
.
Next Software 228 Development Stage
76
22/08/2011
Static analysis
229
What can static analysis do?

Remember: static techniques do not execute the code A form of automated testing
check for violations of standards check for things which may be a fault
Descended from compiler technology

a compiler statically analyses code, and knows a lot about it, e.g. variable usage; finds syntax faults static analysis tools extend this knowledge can find unreachable code, undeclared variables, parameter type mis-matches, uncalled functions & procedures, array bound violations, etc.
230
Data flow analysis

This is the study of program variables
variable defined* where a value is stored into it variable used where the stored value is accessed variable is undefined before it is defined or when it goes out of scope
x is defined, y and z are used x=y+z IF a > b THEN read(S) a and b are used, S is defined
*defined should not be confused with declared
231
77
22/08/2011
Data flow analysis faults

n := 0 read (x) n := 1 while x > y do begin read (y) write( n*y) x := x - n end
232
Data flow anomaly: n is re-defined without being used Data flow fault: y is used before it has been defined (first time around the loop)
Control flow analysis

Highlights:
nodes not accessible from start node infinite loops multiple entry to loops whether code is well structured, i.e. reducible whether code conforms to a flowchart grammar any jumps to undefined labels any labels not jumped to cyclomatic complexity and other metrics
233
Cyclomatic complexity
cyclomatic complexity is a measure of the complexity of a flow graph
(and therefore the code that the flow graph represents)
the more complex the flow graph, the greater the measure it can most easily be calculated as:
complexity = number of decisions + 1
234
78
22/08/2011
Which flow graph is most complex?

1
What is the cyclomatic complexity?
3
235
Example of control flow graph

Pseudo-code:
Result = 0 Right = 0 DO WHILE more Questions IF Answer = Correct THEN Right = Right + 1 ENDIF END DO Result = (Right / Questions) IF Result > 60% THEN Print "pass" ELSE Print "fail ENDIF
init do if r=r+1
end
res
if
pass
fail
end
236
Other static metrics

lines of code (LOC) operands & operators (Halsteads metrics) fan-in & fan-out nesting levels function calls OO metrics: inheritance tree depth, number of methods, coupling & cohesion symbolic evaluation
237
79
22/08/2011
Limitations and advantages

Limitations
cannot distinguish "fail-safe" code from programming faults or anomalies (often creates overload of spurious error messages) does not execute the code, so not related to operating conditions
Advantages
can find faults difficult to "see" gives objective quality assessment of code
238
Summary
Reviews help to find faults in development and test documentation, and should be applied early Types of review: informal, walkthrough, technical / peer review, Inspection Static analysis can find faults and give information about code without executing
239
Functional testing
80
22/08/2011
Contents
Introduction to functional testing Functional testing techniques
241
Introduction to functional testing
242
Functional testing
Functional testing: Deriving test cases from program specifications
Functional refers to the source of information used in test case design, not to what is tested
Also known as:

specification-based testing (from specifications) black-box testing (no view of the code)
Functional specification = description of intended program behavior

either formal or informal
243
81
22/08/2011
Systematic vs Random Testing

Random (uniform):
Pick possible inputs uniformly Avoids designer bias A real problem: The test designer can make the same logical mistakes and bad assumptions as the program designer (especially if they are the same person) But treats all inputs as equally valuable
Systematic (non-uniform):
Try to select inputs that are especially valuable Usually by choosing representatives of classes that are apt to fail often or not at all
Functional testing is systematic testing

244
Why Not Random?

Non-uniform distribution of faults Example: Java class roots applies quadratic equation
Incomplete implementation logic: Program does not properly handle the case in which b2 - 4ac < 0 and a=0 Random sampling is unlikely to choose a=0.0 and b=0.0
245
Systematic Partition Testing

Failure (valuable test case) No failure Failures are sparse in the space of possible inputs ... ... but dense in some parts of the space
The space of possible input values (the haystack)
If we systematically test some cases from each part, we will include the dense parts
Functional testing is one way of drawing pink lines to isolate regions with likely failures 246
82
22/08/2011
Functional testing: exploiting the specification

Functional testing uses the specification (formal or informal) to partition the input space
E.g., specification of roots program suggests division between cases with zero, one, and two real roots
Test each category, and boundaries between categories

No guarantees, but experience suggests failures often lie at the boundaries (as in the roots program)
247
Why functional testing?

Timely Often useful in refining specifications and assessing testability before code is written Effective finds some classes of fault (e.g., missing logic) that can elude other approaches Widely applicable to any description of program behavior serving as spec at any level of granularity from module to system testing. Economical typically less expensive to design and execute than structural (code-based) test cases
248
Early functional test design

Program code is not necessary
Only a description of intended behavior is needed Even incomplete and informal specifications can be used Although precise, complete specifications lead to better test suites
Early functional test design has side benefits

Often reveals ambiguities and inconsistency in spec Useful for assessing testability And improving test schedule and budget by improving spec Useful explanation of specification or in the extreme case (as in XP), test cases are the spec
249
83
22/08/2011
Functional versus Structural: Classes of faults

Different testing strategies (functional, structural) are most effective for different classes of faults Functional testing is best for finding design faults Structural testing is best for finding programming faults
250
Functional vs structural test: granularity levels

Functional test applies at all granularity levels:
Unit Integration System Regression
(from module interface spec) (from API or subsystem spec) (from system requirements spec) (from system requirements + bug history)
Structural (code-based) test design applies to relatively small parts of a system:

Unit Integration
251
Steps: From specification to test cases

1. Decompose the specification
If the specification is large, break it into independently testable features to be considered in testing
2. Select representatives
Representative values of each input, or Representative behaviors of a model
Often simple input/output transformations dont describe a system. We use models in program specification, in program design, and in test design
3. Form test specifications

Typically: combinations of input values, or model behaviors
4. Produce and execute actual tests

252
84
22/08/2011
From specification to test cases
253
Simple example: Postal code lookup
Input: ZIP code (5-digit US Postal code) Output: List of cities What are some representative values (or classes of value) to test?
254
Example: Representative values

Simple example with one input, one output
Correct zip code

With 0, 1, or many cities
Malformed zip code
Note prevalence of boundary values (0 cities, 6 characters) and error cases
Empty; 1-4 characters; 6 characters; very long Non-digit characters Non-character data
255
85
22/08/2011
Functional testing techniques

256
Examples
257
The Triangle Program

In order for 3 integers a, b, and c to be the sides of a triangle, we must have c1 a + b > c c2 a + c > b c3 b + c > a A triangle is:
Equilateral if all 3 sides are equal Isosceles if 2 sides are equal Scalene if no two sides are equal
a
258
86
22/08/2011
The Triangle Program

The Triangle Program reads in 3 integers and decides if they form an equilateral triangle, an isosceles triangle, a scalene triangle, or if they dont form a triangle at all The logic of the program is clear, but complex, due to the relationships between the inputs and the outputs We assume that each side length is between 1 and 200
259
NextDate Function
This program reads in a date in a certain format and prints out the next days date. For example, an input of 03 31 1998 gives an output of 04 01 1998. The year is constrained to lie between 1812 and 2012 inclusive
260
The Commission Problem

A salesman sells locks, stocks and barrels On a monthly basis, the salesperson reports how many of each he or she has sold Due to manufacturing constraints, we have: 1 #locks 70 1 #stocks 80 1 #barrels 90 Goal: Determine total sales and the commission
261
87
22/08/2011
The Commission Problem

The parts sell for the following: Locks $45 each Stocks $30 each Barrels $25 each Commission is computed on sales: 10% on sales up to and including $1000; 15% on the next $800; and 20% on any amount over $1800 This program has calculation logic to test inputs and outputs are straightforward
262
Boundary Value testing
263
Boundary/Limits Testing
Test values, sizes or quantities near the design limits Value limits Length limits Volume limits Null strings vs Empty strings Errors tend to occur near the extreme values of inputs Robustness: How does the software react when boundaries are exceeded?
264
88
22/08/2011
Single Fault Assumption

The Single Fault assumption from reliability states: Failures are rarely the result of the simultaneous occurrence of two (or more) faults.
265
Extensions of Boundary Testing

Boundary value testing
Stay within the limits
Robustness testing
Exceed the limits
Worst-case testing
Like boundary value testing above, but discard Single Fault Assumption
Robust worst-case testing

Like robustness testing above, but discard Single Fault Assumption
266
Boundary Value Testing

The basic idea of boundary value analysis is to use input variable values at
their minimum, just above the minimum, at a nominal value, just below the maximum, and at the maximum
267
89
22/08/2011
Input Boundary Value Testing

a b x
x(min)
x(min+)
x(nom)
x(max -)
x(max)
Test cases for a variable x, where a x b Experience shows that errors occur more frequently for extreme values of a variable
268
Input Boundary Value Testing (2 Variables)

x2
c x1 a
Test cases for variables x1 and x2, where a x1 b and c x2 d
As in reliability theory, two variables rarely assume their extreme values simultaneously.
269
Robustness Testing
An extension of Boundary Value testing: Include values below the minimum value and above the maximum value This is a form of what we call negative testing -- how does the program handle errors in the input? This type of testing may not be possible with some strongly-typed languages, GUIs with fixed palette values or drop-down lists, etc.
270
90
22/08/2011
Robustness Testing
a b x
Test cases for a variable x, where a x b
1. Stress input boundaries 2. Acceptable response for invalid inputs? 3. Leads to exploratory testing (test hackers)
271
Robustness Testing (2 Variables)

x2
c x1 a b
272
Worst-Case Testing
Discard the Single Fault Assumption Allow the input values to simultaneously approach their boundaries
273
91
22/08/2011
Worst Case Testing (2 Variables)

x2
c x1 a b Eliminate the "single fault" assumption. Murphy's Law

274
Robust Worst-Case Testing

Discard the Single Fault Assumption Allow the input values to simultaneously approach and exceed their boundaries
275
Robust Worst Case (Paranoid) Testing with 2 Variables

x2
c x1 a b
276
92
22/08/2011
Limitations of Boundary Value Testing

Doesnt require much thought May miss internal boundaries Usually assumes the variables are independent Values at the boundary may not have any more meaning than non-boundary values
277
Boundary Value Test Cases for the
examples
278
Boundary Value Test Cases for the Triangle Program

Side lengths are integers ranging from 1 to 200 Typical boundary value analysis requires 5 values for each variable: min, min+, nominal value, max-, max Input Boundary Value testing for 3 variables requires 4x3 + 3 = 15 test cases. Our values: 1, 2, 100, 199, 200
279
93
22/08/2011
Input Boundary Value Analysis
280
Worst Case Boundary Value Analysis

There should be 53 = 125 test cases Side a = 1, 2, 100, 199, 200 times Side b = 1, 2, 100, 199, 200 times Side c = 1, 2, 100, 199, 200 = 125 test cases
281
Test Cases for the NextDate & Commission

Design boundary value test case for
NextDate function Commission problem
282
94
22/08/2011
Equivalence Classes Testing
283
Equivalence Classes
Idea
divide the input domain of a program into classes of data derive test cases based on these partitions
Determine the boundariesthen determine the equivalencies Motivation

Gain a sense of complete testing and avoid redundancy
If a given set of values all cause the program to behave in exactly the same way, then thats a candidate for an Equivalence Class
284
Equivalence Classes
An equivalence class represents a set of valid or invalid states for input condition An input condition is:
a specific numeric value, a range of values a set of related values, or a Boolean condition
Valid inputs
system invalid inputs partition outputs All Rights Reserved
285
95
22/08/2011
Determining equivalence classes

Equivalence classes can be defined using the following guidelines:
If an input condition specifies a range, one valid and two invalid equivalence class are defined If an input condition requires a specific value, one valid and two invalid equivalence classes are defined If an input condition specifies a member of a set, one valid and one invalid equivalence classes are defined If an input condition is Boolean, one valid and one invalid classes are defined
286
Examples
Area code
input condition, Boolean: the area code may or may not be present input condition, range: value defined between 200 and 900
Password
input condition, Boolean: a password may or may not be present input condition, value: six character string
287
Determining equivalence classes

Look for ranges of numbers or values Look for memberships in groups Some may be based on timing Include invalid inputs Include internal boundaries Dont worry if they overlap each other better to be redundant than to miss something However, these test cases will almost surely overlap with boundary value test cases
288
96
22/08/2011
Types of Equivalence Classes (two variables)

Weak normal
one data point from each valid class for each variable
Strong normal
one data point from each class in the cross product of the valid classes
Traditional Equivalence Class

For valid inputs, use one value from each valid class For invalid inputs, a test case will have one invalid value and the remaining values will all be valid
These usually apply when the variables have internal boundaries

289
Types of Equivalence Classes

Input domain Output range
290
Example for input domain

Suppose a program has 2 input variables, x and y Suppose x can lie in 3 possible domains: a x < b, b x < c, or c x d Suppose y can lie in 2 possible domains: e y < f or f y g
291
97
22/08/2011
Weak Equivalence Class Test Cases

y
g f
e d x
292
Strong Equivalence Class Test Cases

y
g f
e d x
293
Equivalence Classes Test Cases for the examples
294
98
22/08/2011
Triangle Program Input Equivalence Classes

Equivalence classes based on input: We get at least 9 classes (5 triangle, 3 not a triangle, 1 garbage): a=b=c * a b, a c, b c * a=bc * ab+c a=cb * b a+c b=ca * ca+b garbage: (0, 1,2), (-7, 4, 8), (x, &, #) * but still a triangle!
295
Output Equivalence Classes for the Triangle Program

There are 4 output equivalence classes:
Equilateral Isosceles Scalene Not a triangle
296
Equivalence Classes testing for NextDate

The classes for the Day variable could be {1 Day 28, Day = 29, Day = 30, Day = 31} The classes for the Month variable could be {Month has 30 days, Month has 31 days, Month = February} The classes for the Year variable could be {Year is not a leap year, Year is a leap year}
297
99
22/08/2011

Traditional Equivalence Class Test Cases
Valid inputs are mechanically selected from the approximate middle of the valid range
298

Weak Equivalence Class Test Cases
Inputs are mechanically selected from the approximate middle of the corresponding class
299

Strong Equivalence Class Test Cases
36 strong equivalence class test cases
300
100
22/08/2011

36 strong equivalence class test cases (cond)
301
Equivalence Classes testing for Commission Problem

Variable Input Domain Equivalence Classes
302

Weak Input Domain Equivalence Class Test Cases
303
101
22/08/2011

Traditional Input Domain Equivalence Class Test Cases
304

Traditional Input Domain Equivalence Class Test Cases
305

Output Range Equivalence Classes
306
102
22/08/2011
Notes on Equivalence Classes

Check for boundaries in the inputs or in the outputs Always include a garbage class If a variable can take on the value 0, it is usually best to have {0} as its own Equivalence Class Use your experience and domain knowledge to determine the best classes
307

The traditional form of equivalence class testing is generally not as thorough as weak equivalence class testing, which in turn, is not as thorough as the strong form of equivalence class testing The only time it makes sense to use the traditional approach is when the implementation language is not strongly type If error conditions are a high priority, we could extend strong equivalence class testing to include invalid classes Equivalence class testing is appropriate when input data is defined in terms of ranges and sets of discrete values 308

Equivalence class testing is strengthened by a hybrid approach with boundary value testing Equivalence class testing is indicated when the program function is complex Strong equivalence class testing makes a presumption that the variables are independent when the Cartesian Product is taken
If there are any dependencies, these will often generate "error" test cases, as they did in the NextDate function
The decision table technique in Chapter 7 resolves this problem
309
103
22/08/2011
Special Value Testing
310
Special Value Testing

One author calls this the most widely practiced form of functional testing The tester uses his or her domain knowledge, experience, or intuition to probe areas of probable errors Other terms: hacking, out-of-box testing, ad hoc testing, seat of the pants testing, guerilla testing
311
Uses of Special Value Testing

1.
2.
3. 4.
Complex mathematical (or algorithmic) calculations Worst Case situations (similar to robustness) Problematic situations from past experience Second guess the likely implementation
312
104
22/08/2011
Examples of Special Value Testing

When dates are involved, check Feb. 28, Feb. 29 and leap years When strings are involved, check the empty string and the single-character blank string Arithmetic: Always try 0 Any ad hoc hacking
313
Characteristics of Special Value Testing

1. 2. 3. 4. 5. 6.
Experience really helps Done by the customer or user frequently Defies measurement Highly intuitive Seldom repeatable Often, very effective
314
Decision Tables Based Testing
315
105
22/08/2011
Decision Tables
Inputs are interpreted as conditions Outputs are interpreted as actions The columns in a table are rules they show which actions result from which conditions Every rule then becomes a test case Dashes represent dont care conditions
316
Decision Table Terminology

Stub c1 Condition c2 c3 True T F Entry True False True T F False False
a1 a2 Action a3 a4
X X
X X X X X X
317
Rule
One Decision Table for the Triangle Program

c1: a, b, c are a triangle? c2: a = b? c3: a = c? c4: b = c? a1: not a triangle a2: Scalene a3: Isosceles a4: Equilateral a5: Impossible X X X X
318
F T T T F
T F T F T F F T F
T F X
X X X X
106
22/08/2011
Decision Table-Based Testing

Most rigorous functional method enforces logical rigor Used to represent and analyze complex logical relationships Decision Tables are declarative no particular order among the conditions is implied Decision Tables support consistency and completeness Dependencies can yield impossible combinations, so we usually have an impossible action
319
Refined Decision Table for the Triangle Problem

c1: a < b+c? c2: b < a+c? c3: c < a+b? c4: a = b? c5: a = c? c6: b = c? a1: Not a triangle a2: Scalene a3: Isosceles a4: Equilateral a5: Impossible X X X X
320
F T T T T T T T T T T -- F T T T T T T T T T -- -- F T T T T T T T T -- -- -- T T T T F F F F -- -- -- T T F F T T F F -- -- -- T F T F T F T F X X X X X X X
A Redundant Decision Table

Conditions 1-4 5 c1: T F c2: c3: a1: a2: a3: --X -X T T X X -6 F T F X X X 7 F F T -X X 8 F F F --X 9 T F F X -X
Rule 9 is identical to Rule 4 (T, F, F) Since the action entries for rules 4 and 9 are identical, there is no ambiguity, just redundancy.
321
107
22/08/2011
An Inconsistent Decision Table

Conditions 1-4 5 c1: T F c2: c3: a1: a2: a3: --X -X T T X X -6 F T F X X X 7 F F T -X X 8 F F F --X 9 T F F -X --
Rule 9 is identical to Rule 4 (T, F, F) Since the action entries for rules 4 and 9 are different there is ambiguity. This table is inconsistent, and the inconsistency implies non-determinism cant tell which rule to apply!
322
Procedure for Decision-Table Based Testing

1. Determine conditions and actions.
2. Develop the Decision Table, watching for completeness don't care entries redundant and inconsistent rules 3. Each rule defines a test case.
323
Decision Tables Based Testing Test cases for the examples
324
108
22/08/2011
Test Cases for the Triangle Problem
325
Exercises
Test cases for NextDate function Test cases for Commission problem
326
Test documentation
109
22/08/2011
Contents
Test Plan
The need for test plans The structure of test plans A Test Plan Template A Test Plan example Testing on a large project
Test Cases
Test Case Design Test Case Examples
328
Test plan
329
The need for a good test plan

Helps to organize, schedule and manage the testing effort Helps in writing test cases Improves communication with developers and with management If done correctly, it may help minimize the documentation required May be required by the customer
330
110
22/08/2011
Validation Planning Issues

Testing is intended to measure software quality and must be planned It takes careful planning to select a good set of tests Questions that must be answered: What to test and when to stop
331
Benefits of Test Planning

Measure the cost of testing vs the confidence gained Forces you to work out expected results before you run the test cases You can actually find errors before running any tests!
332
Components of a Test Plan

Testing approach Schedule Resources Entry and Exit criteria Requirements Matrix What will NOT be tested Test Cases
333
111
22/08/2011
Other Issues to Consider

Test methods Facilities for test execution Test automation Testing tools Support software Configuration management Risks (budget, schedule, etc.)
334
Test Plan activities

Use a Test Plan template, or design one List what cannot be tested Write only what you need
335
Test Plan activities

Have the Test Plan reviewed Test while you write Make it a living document
336
112
22/08/2011
Concerns of Test Planning

Not enough training Lack of test tools Lack of management support Lack of customer/user involvement Not enough time Rapid change Its hard to say No
337
Example of a Test Plan

Test Plan template Test Plan example
338
A Partial List for Your Test Plans

Title page of some sort Description of the testing to be done Updated Requirements Matrix List of test cases Traceability Matrix 3 detailed test cases
339
113
22/08/2011
Testing on a large project

Roles/Responsibilities
Project Mgr/Test Mgr/Development Test Team Lead/Lead Test Designer Testers Test Technicians Build Manager
340
Test Planning Tips

Meet regularly with the Project Manager Plan for the test team to attend a lot of reviews Make assignments early Assign one tester to know everything Have only one Test Plan Biggest headache: Builds
341
Test case
342
114
22/08/2011
Terminology
Test Case
Documentation specifying inputs, predicted results, and a set of execution conditions for a test item
Test Suite
a collection of test cases that are intended to be used to test a software program
343
Characteristics of good Test Cases

Reasonable probability of catching an error
Recall: Testing is the process of finding errors and of validating the program/system
Non-redundant Efficient Medium complexity Repeatable Specific Always list expected results
344
Test Case caveats

Errors can mask other errors Make sure you know how to tell whether a test case passes or fails Bug fixes can cause more bugs
345
115
22/08/2011
Where do you get your Test Cases?

Test Requirements Matrix Your experience Your learning The developer(s) Marketing The customer: Use Cases/Scenarios Defects from the field or previous releases
346
Test Case Examples

Test Case Template Test Case Example
347
To read more
Cem Kaner, What Is a Good Test Case?
348
116
22/08/2011
Test Case Generation

Case-insensitive Passwords State look-up Simple program
349
Case-insensitive Passwords
Write the test cases to verify that Passwords are case insensitive when logging onto a secure system
350
An example
Test Case: Verify that passwords are case insensitive Test Procedure:
1. 2. 3. 4. 5. 6.
Create a User with Password ABCDE. Log off the system. Log on using Password ABCDE (should be successful) Log off the system. Log on using Password abcDe (should be successful). Repeat Steps 4 and 5 using abcde, ABCDe, etc.
351
117
22/08/2011
State Lookup
Input: 2-character state abbreviation The program looks up the state in a table and displays the full name Discuss all considerations for testing such a function
352
Simple program
The program accepts as input 2 integers and outputs their sum Assume arbitrary limits, say -100 to +100 Design the test cases
353
Levels of Testing
118
22/08/2011
Contents
Levels of Testing Additional System Test Categories
355
Levels of Testing
356
Levels of Testing
Waterfall model
Requirements Specification System Testing
Preliminary Design
Integration Testing
Detailed Design
Unit Testing
Coding
357
119
22/08/2011
Levels of Testing
Unit testing Integration Testing System Testing Acceptance Testing Regression Testing
358
Unit testing
Lowest Level of Testing Individual unit of the software is tested in isolation from other parts of a program What can be a unit?
Function / method / module / class / component Screen / Program Back-end related to a Screen Screen + back end
359
Unit testing
Uncover errors in design and implementation, including
data structure in a unit program logic and program structure in a unit unit interface functions and operations of a unit
Unit testers
developers of the units
360
120
22/08/2011
Integration Testing
Intermediate level of testing Group of dependent components are tested together Test that evaluates the interaction and consistency of interacting components Uncover errors in:
Design and construction of software architecture Integrated functions or operations at sub-system level Interfaces and interactions between them Resource integration and/or environment integration
Integration testers
either developers and/or test engineers
361
System Testing
Test the whole system Test end to end functionality Testing against complete requirement Requirement focus
Test cases derived from specification
System testers
test engineers
362
Acceptance Testing
User (or customer) involved Environment as close to field use as possible Focus on:
Building confidence Compliance with defined acceptance criteria in the contract
363
121
22/08/2011
Regression Testing
Testing activities occur after software changes Regression testing usually refers to testing activities during software maintenance phase Retest changed components (or parts) Check the affected parts (or components) Who perform regression testing
Developers - regression testing at the unit level or integration Test engineers - regression testing at the function level QA and test engineers - regression testing at the system level
364
System Test categories

Functional testing Load/Stress/Volume testing Destructive testing/Recoverability Configuration Security Performance Compatibility Usability Reliability Installability Documentation
365
Stress Testing
Also called Load Testing or Volume Testing Test the extremes of real-world workloads Large data volumes Repetitive operations Operation for extended periods: Endurance Testing
366
122
22/08/2011
Destructive Testing
This is what we have called Error Testing The product should
Detect, diagnose and perhaps inhibit unexpected conditions Correct the conditions where possible Report on the unexpected conditions Recover gracefully in any case
367
Thinking Destructively
The most disastrous failure for most software is the loss or corruption of data Think like a hacker, especially when it comes to security issues Destructive Testing is usually unstructured, but document what you do Be creative! Think about the consequences if you succeed
368
Configuration Testing
Software components Hardware components Software with hardware
369
123
22/08/2011
Security Testing
Password usage Access rights Protection Encryption
370
Performance Testing
Usually pertains to hardware Software can be the bottleneck Generally, how fast a task can be done Can also be measured in volume
371
Compatibility Testing
Shared data Shared memory Compatibility with previous versions
operating system, utilities, setups, etc.
Portability might fall here
372
124
22/08/2011
Usability Testing
Human factors Accessibility Ease of use (product) Availability of documentation Can a non-familiar user perform a task?
373
Reliability Testing
Low if a bug is likely to be found High if bugs are unlikely Failures and Time
Time when the failure occurs Time interval between failures Cumulative failures up to a given time Number of failures in a time interval
374
User Manual Testing

Installation Documentation On-line Help
375
125
22/08/2011
Installability
Installing a product and its options De-installing (or Uninstalling) a product or its options Re-installing a product or its options Doing upgrade installs vs doing clean installs
376
Documentation
Reviews Inspections Walkthroughs
377
Documentation
Keep a copy of your comments This is a possible place for new testers to start Manuals can be task-oriented or featureoriented Tutorials are very helpful, so test them out, too
378
126
22/08/2011
Documentation
Installation instructions If task oriented, do the tasks! Other documentation (README, Utilities, etc.) Ease of use of the documentation Missing features Feature compatibility
379
Documentation
On-line Help
Availability Relevance Accuracy Completeness Links Controls
380
Test Reports
127
22/08/2011
Contents
Handling Defects Bug Tracking System Test Reports Examples
382
Handling Defects
383
Handling a defect
Analyze the defect Report the defect Track the defect Retest the defect Close the defect
384
128
22/08/2011
Analyzing a defect
Is it the test, the tester, or the product? Reproducible vs Repeatable Can it be isolated? Find related problems Look for alternate paths to the problem Is it worth reporting?
385
Handling a defect
386
Reporting the defect

Look for duplicates Talk with the developer Enter it into the system Make sure it will get noticed!
387
129
22/08/2011
Characteristics of an effective defect report

Tells how to reproduce the problem
in a minimum number of steps
The report is complete and understandable Written Simple
388
Components of a defect report

Identifying information Description of the problem Various status indicators Comments Supporting information Miscellaneous information
389

Identifying information
Unique number Submitter Submit date Program or product this is against Version or revision of the product
390
130
22/08/2011

Description of the problem
Title
Must convey in a limited number of words enough information that the problem can be understood Any abbreviations must be generally acknowledged or explained in the Description
Description
391

Description of the problem
Contents
The problem itself
What did happen What should have happened
The test case used Any other helpful information whatsoever

In particular, any attachments
Keep it to 3 or 4 sentences if you can Avoid vague or confusing terms (frequently, sometimes) Avoid uncommon abbreviations Use any standard terminology Pay attention to spelling and grammar
392

Status indicators
Report status Severity and priority Resolution status
393
131
22/08/2011

Status indicators
Report Status
Open Closed
394

Status indicators
Severity and Priority
Severity: How bad is it?

Generally related to the defects effect on testing
High -- critical defect; test cannot continue Medium -- critical defect; test can continue with a workaround Low -- its a problem but we can keep testing Really low -- documentation, help files, spelling mistakes
Priority: How urgent is it?

When does this have to be fixed?
395

Status indicators
Examples of Resolution status
None In Process Fixed Ready for Test Retest Not repeatable Not a problem Do Not Fix Duplicate Multiple Deferred
396
132
22/08/2011

Comments/Notes
Investigation Notes Resolution Notes Tester Notes Modification Notes Whatever-works-for-you Notes
397

Miscellaneous fields (1)
Steps to Reproduce
Include setup information Anyone with the appropriate setup should be able to reproduce the problem Could be as simple as a Test Case ID, or a lengthy procedure Pay attention to problems that are merely repeatable or not reproducible
398

Miscellaneous fields (2)
Environment Target Release Closed Release Closed Date Owner Tester Discovered by Defect type Software component affected Fix hours Test hours Workarounds
399
133
22/08/2011

Supporting information
Error printouts Screen shots The test case itself A diskette with data or files Memory dump Trace files, error logs, etc.
400
Handling a defect
401
Tracking a Defect
Must have a process in place Generally done by some sort of Review Board The lead tester should be on that board The purpose is to make sure the defect progresses from being submitted to getting closed
402
134
22/08/2011
Retesting a Defect
Retest can have 3 outcomes
The problem has indeed been fixed The problem remains A different problem shows up
403
Closing a Defect
Add Tester Notes or Verification Notes Make sure that anyone else with an interest in the problems resolution has a chance to test it too Close it, or recommend closure
404
Life cycle of a defect report

Entered by a tester Review Board evaluates and assigns information Assigned programmer fixes the problem A build containing the fix becomes available The tester retests the problem The tester or board closes the defect
405
135
22/08/2011
Bug Tracking System
406
The need for a Defect Tracking System

Facilitates clear communication about defects Allows developers to fix bugs based on priority Allows for analysis and reporting options Helps to manage the bugs Keeps a history
407
Items for your Defect Reports

Identifiers: title, number, submitter, submit date Program name and version Description of the problem The Test Case used
408
136
22/08/2011
Test Reports
409
Test Reports
Your chance to evaluate how testing went List what was tested List what was not tested, and why List still-open defects (some ask for all defects) Show the actual schedule (Test Plan) Lets management know the status of that product, from a testing standpoint, as it went out the door
410
Your Test Report

A brief description of what you did, when An attached list of all the Test Cases with an indication of which ones passed and which ones failed (and by default, then, which ones didnt get run) A list of all the defects found (a list of Brief Descriptions)
411
137
22/08/2011
Examples
412
Examples
Test Report template Test Report example
413
Test Automation and Tools
138
22/08/2011
Contents
Test Automation Testing Tools
415
Test Automation
416
What is Test Automation?

Test Automation
refers to the activities and efforts that intend to automate engineering tasks and operations in a software test process
using well-defined strategies and systematic solutions
assists software testers and software testing managers with different aspects of a testing project
417
139
22/08/2011
Objectives of Test Automation?

to free engineers from tedious and redundant manual testing operations to speed up a software testing process to reduce software testing cost and time during a software life cycle to increase the quality and effectiveness of a software test process by achieving pre-defined adequate test criteria in a limited schedule The major key to the success of software automation
to reduce manual testing activities and redundant test operations using a systematic solution to achieve a better testing coverage
418
Different Maturity Levels of Software Test Automation

Level 4: Optimal Systematic Test Measurement & Optimization Level 3: Automatic Systematic Test Generation Level 2: Repeatable Systematic Test Execution Control Level 1: Initial Systematic Test Information Management
419

Level 1: Initial
A software test process at this level provides engineers with systematic solutions and tools to create, update, and manage all types of software test information, including test requirements, test cases, test data, test procedures, test results, test scripts, and problem reports No systematic solutions and tools are available to support engineers test design, test generation, and test executions
Level 2: Repeatable
A software test process at this level not only provides engineers with tools to manage diverse software testing information, but also provides systematic solutions to execute software tests in a systematic manner. These solutions allow engineers to use a systematic approach to run tests and validate test results However, no systematic solutions and tools are available to assist test engineers in test design, test generation, and test coverage 420 measurement
140
22/08/2011

Level 3: Automatic
Besides the test management and test execution tools, a software test process at this level is supported with additional solutions to generate software tests using systematic methods They could be useful to generate black box or white-box software tests However, no systematic solutions are available to measure the test coverage of a test process
Level 4: Optimal
At this level, systematic solutions are available to manage test information, execute tests, and generate tests, and measure test coverage The primary benefit of achieving this level is to help engineers understand the current coverage of a test process, and identify the test coverage issues
421
Essential Needs of Test Automation

A dedicated work force for test automation The commitment from senior managers and engineers The dedicated budget and project schedule A well-defined plan and strategy Talent engineers and cost-effective testing tools Maintenance of automated software tests and tools
422
Types of Automation
Automated test case generation Automated test data generation Automated test execution Applications
Client-server architecture Networks GUI Regression Testing
423
141
22/08/2011
False expectations (1)

You can go from test planning to test execution with no manual intervention One tool can support all test requirements: Not so as the book points out, no tool offers assistance with all tester activities There will be an immediate reduction in test effort and test schedule Automation will reduce testing resources
424
False expectations (2)

Test automation tools are easy to use Everything can be tested Automation compensates for bad requirements or bad design
425
Benefits of Automation
Reduce manual software testing operations and eliminate redundant testing efforts Produce more systematic repeatable software tests, and generate more consistent testing results Execute much more software tests and achieve a better testing coverage in a very limited schedule.
426
142
22/08/2011
Is Automation right for me?

You must have a Development Process in place You should have a Testing Process in place - mostly manual You have many tests that get executed many times Consider the costs Consider the time it takes to implement
427
Some Costs of Automation

The cost of the tool itself, including licensing Any additional computer equipment or test environment enhancements Test tool installation The time it takes to learn the tool Training costs
428
The Automated Test Lifecycle Management

1. 2. 3. 4. 5. 6.
Decision to automate testing Test tool acquisition Automated testing introduction process Test planning, design and development Execution and management of tests Test program review and assessment
429
143
22/08/2011
Testing Tools
430
Selecting a Testing Tool

Ease of use Customizable Platform support (Hardware and OS) Multi-user Defect tracking Functionality Reporting capability Performance and stress testing Version control Pricing Vendor qualifications
431
Types of tools
Requirements Management
Generate (or capture) requirements quickly and efficiently, and manage them Basic notification maybe even determine which test procedures are affected when a requirement changes Examples
TBIs Caliber-RM, QSSs DOORS, Rationals RequisitePro, Integrated Chipwares RTM Workshop
432
144
22/08/2011
Types of tools
Configuration Management
Controls software code, documentation, hardware, firmware, and system environment configuration Examples
CVS, Subversion, Visual SourceSafe http://www.daveeaton.com/scm/CMTools.html
433
Types of tools
Test Planning and Test Case Management
Create and manage your test projects Import test cases from other file formats Tracks history and status of test cases Examples
TestLog, Salome-TMF, IBM Rational Quality Manager http://www.opensourcetesting.org/testmgt.php
434
Types of tools
Test Data Generators
Generate test data from specifications / design / code Examples DataGen, Tnsgen, forSQL Data Generator
GUI testing
Record and Playback a test Rerun repetitive test cases Regression testing Examples
Segue SilkTest, Compuware QA Run, QuickTestPro
435
145
22/08/2011
Types of tools
Load/Performance/Stress Testing Tools
Simulate thousands of users
Specially client-server systems
Examples
Empirix Eload, XML Load Testing, Segue SilkPerformers, IBM Rational Suite
436
Types of tools
Code Coverage
Determining a quantitative measure of code coverage
Statement, Decision, Condition, Path
Finding areas of a program not exercised by a set of test cases Examples

http://csharp-source.net/open-source/code-coverage http://java-source.net/open-source/code-coverage
437
Types of tools
Rule-Based Analyzers
Compare the code written with coding standards or languages-specific rules Support Review / Inspection Examples
Intelli J, FlexeLint for C++, SSW Code Auditor
438
146
22/08/2011
Types of tools
Defect Tracking
Communicate and mange defect-related issues Examples
BugTracker, TrackGear, Bugtrack http://www.testingfaqs.org/t-track.html
439
Types of tools
Debuggers
Allow inspection of program state Pause execution at selected points (breakpoints) when certain conditions occur (watchpoints) after a fixed number of execution steps Provide display and control at the level of program source code
Examples
DDD, Debuggy, Valgrind
440
147

0 All Slides

Uploaded by

Copyright:

Available Formats

0 All Slides

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

0 All Slides

Uploaded by

Copyright:

Available Formats

22/08/2011

Testing and requirements

Additional System Test Categories

Chapter 1 of the textbook A Perspective on Testing

Introduction to software testing

Definitions, Principles, Axioms

Definitions of Testing (1)

Definitions of Testing (2)

What Testing is Not

Verification and Validation: Myers

Verification and Validation: IEEE

Verification and Validation: Kaner

Software Fault Terminology

Software Fault Terminology: Examples

Fault the representation of the error

Failure what happens when the fault executes

Black and White Box

White box testing

Six Essentials of Software Testing (1)

Six Essentials of Software Testing (2)

Some Principles and Axioms of Testing (1)

Some Principles and Axioms of Testing (2)

The Test Process

Software Test Execution Stages

System Validation Test (SVT)

Customer Acceptance Test

Typical Entry Criteria (Unit Test)

Typical Unit Test Activities

Typical Exit Criteria (Unit Test)

Software Test Stages

System Validation Test

Customer Acceptance Test

Design Verification Test (DVT)

Three Components of DVT

Typical Entry Criteria (DVT)

Typical DVT activities

DVT Exit or Formal Test Phase

Typical Exit Criteria (DVT)

Software Test Stages

System Validation Test

Customer Acceptance Test

System Validation Test (SVT)

Typical Entry Criteria (SVT)

SVT Main Test

SVT Regression Test

System Acceptance Test

Typical Exit Criteria (SVT)

Software Test Stages

System Validation Test

Customer Acceptance Test

Customer Acceptance Testing

Perspectives on Software Testing

The Objectives and Limits of Testing (1)

The Objectives and Limits of Testing (2)

The Objectives and Limits of Testing (3)

Tests and Test Cases (1)

Tests and Test Cases (2)

Quality Assurance vs Quality Control

Quality Control is the activity that evaluates the product produced

A Generic Testing Approach (SPRAE)