Testing Object-Oriented Programs

Dr. RAJIB MALL

Professor Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur, India

Plan of the Talk

Introduction Challenges in testing OO programs Test suite design using UML models Test design patterns Conclusion
2

Introduction
More than 50% of development effort is being spent on testing. Quality and effective reuse of software depend to a large extent: on thorough testing. It was expected during initial years that OO would help substantially reduce testing effort: But, as we find it out today --- it only complicates testing.
3

Todays Focus
Most reported research on OO paradigms focus on: Analysis and design. We discuss some important issues in testing object-oriented systems.

Fault Model
Different types of faults in a program: Infinite for practical purposes . A fault model: A map of possible types of faults. Necessary to guide any rational testing strategy. A fault model may be constructed from analysis of failure data.
5

Places To Look For Faults

Some places (error types) can trivially be omitted from a fault model: I Cannot make English grammar mistakes when I am writing a C program. Cannot commit errors due to side effects (change global variables) while writing a Java program. A test strategy: Yields a test suite when applied to a unit under test.
6

Challenges in OO Testing
What is an appropriate unit for testing? Implications of OO features: Encapsulation Inheritance Polymorphism & Dynamic Binding, etc. State-based testing Test coverage analysis Integration strategies Test process strategy
7

 What is the fundamental unit of testing for conventional programs? A function. However, as far as OO programs are concerned: Methods are not the basic unit of testing. Weyukar's Anticomposition axiom: Any amount of testing of individual methods can not ensure that a class has been satisfactorily tested.
8

What is a Suitable Unit for Testing?

Suitable Unit for Testing OO Programs

Class level: Testing interactions between attributes and methods must be addressed. State of the object must be considered.

Suitable Unit of Testing

Cluster level:

cont

Tests the interactions among a group of cooperating classes. A sequence of interactions is typically required to implement a visible behavior (i.e. a use case). System level: Tests an entire operational system.
10

Encapsulation
Encapsulation is not a source of errors: However, an obstacle to testing. It prevents accessing attribute values by a debugger. While testing: Precise information about current state is necessary.
11

Solving EncapsulationRelated Problems

Several solutions are possible: Built-in or inherited state reporting methods. Low level probes to manually inspect object attributes. Proof-of-correctness technique (formal).
12

Solving Encapsulation-Related Problems

Most feasible way: State reporting methods. Reliable verification of state reporting methods is a problem.

13

Inheritance
Should inherited methods be retested? Retesting of inherited methods is the rule, rather than an exception. Retesting required: Because a new context of usage results when a subclass is derived. Correct behavior at an upper level: Does not guarantee correct behavior at a lower level.
14

Inheritance --- Overriding

In case of method overriding: Need to retest the classes in the context of overriding. An overridden method must be retested even when only minor syntactic changes are made.
15

Deep Inheritance Hierarchy

A subclass at the bottom of a deep hierarchy: May have only one or two lines of code. But may inherit hundreds of features. This situation creates fault hazards: Similar to unrestricted access to global data in procedural programs. Inheritance weakens encapsulation.
16

Deep Inheritance Hierarchy cont...

A deep and wide inheritance hierarchy can defy comprehension: Lead to bugs and reduce testability. Incorrect initialization and forgotten methods can result. Class flattening may increase understandibility. Multiple Inheritance: Increases number of contexts to test.
17

Abstract and Generic Classes

Unique to OO programming: Provide important support for reuse. Must be extended and instantiated to be tested. May never be considered fully tested: Since need retesting when new subclasses are created.
18

Polymorphism
Each possible binding of a polymorphic component requires separate testing: Often difficult to find all bindings that may occur. Increases the chances of bugs . An obstacle to reaching coverage goals.

19

Polymorphism
Polymorphism complicates integration planning:

cont

Many server classes may need to be integrated before a client class can be tested.
C S1 S2
20

S3

Dynamic Binding
Dynamic binding implies: The code that implements a given function is unknown until run time. Static analysis cannot be used to identify the precise dependencies in a program. It becomes difficult to identify all possible bindings and test them.
21

State-Based Testing

The concept of control flow of a conventional program : Does not map readily to an OO program. In a state model: We specify how the object's state would change under certain conditions.

22

State-Based Testing
Flow of control in OO programs: Message passing from one object to another. Causes the receiving object to perform some operation, can lead to an alteration of its state.
23

State-Based Testing
The state model defines the allowable transitions at each state. States can be constructed: Using equivalence classes defined on the instance variables. Jacobson's OOSE advocates: Design test cases to cover all state transitions.
24

cont...

State-Based Testing
cont the Test cases can be derived from... state machine model of a class: Methods result in state transitions. Test cases are designed to exercise each transition at a state. However, the transitions are tied to user-selectable activation sequences: Use Cases
25

Difficulty With State Based Testing

The locus of state control is distributed over an entire OO application. Cooperative control makes it difficult to achieve system state and transition coverage. A global state model becomes too complex for practical systems. Rarely constructed by developers. A global state model is needed to show how classes interact.
26

Test Coverage
Test coverage analysis: Helps determine the thoroughness of testing achieved. Several coverage analysis criteria for traditional programs have been proposed: What is a coverage criterion? Tests that are adequate w.r.t a criterion: Cover all elements of the domain determined by that criterion.
27

cont... But, what are the elements that characterize an object-oriented program? Certainly different from procedural programs. For example: Statement coverage is not appropriate due to inheritance and polymorphism. Appropriate test coverage criteria are needed.
28

Test Coverage
Criterion

Integration Strategies
Essentially two approaches exist: Thread-based Use-based

29

Thread-based integration Strategies

A thread consists of: All the classes needed to respond to a single external input. Each class is unit tested, Then each thread is exercised.
30

Use Case-based Integration Strategies

Use case-based integration: Begins by testing classes that use services of none (or a few) of other classes. Next, classes that use the first group of classes are tested.

Followed by classes that use the second group, and so on.

31

Test Process Strategy

Object-oriented development tends toward: Shorter incremental cycles. Development characterized as: Design a little, code a little, test a little. The distance between OO specification and implementation, therefore: Typically small compared to conventional systems.
32

Test Process Strategy

cont... The gap between white-box/black-box test is diminished. Therefore lower importance of codebased testing. Model-based testing has assumed importance (also called grey box testing). Conventional white-box testing can be used for testing individual methods.
33

 UML has become the de facto standard for OO modeling and design: Though UML is intended to produce rigorous models: Does incorporate many flexibilities. Incomplete, inconsistent, and ambiguous models often result. Never the less: UML models are an important source of information for test design.
34

Testing Based on UML Models

Test Approaches for UML-Based Designs

Two Approaches: 1 Interpret the generic test strategy to develop the test suite. 2 Apply the related test design pattern.

35

 Relations among classes are:

Testing Relations Among Classes

Inheritance, Association, Composition, and Dependency. Each relationship has corresponding generic test requirements: For example, a generalization relation is:
not Reflexive(R) and not Symmetric(S) but is Transitive(T)
36

Testing Relations

For any relation, we can ask which of the three (R, S, T) properties is required, excluded, or irrelevant. The answers leads to a simple but effective test suite.

cont...

37

Testing Relations

cont...

For example, when reflexivity is excluded, Any x that would make xRx true should be rejected. Example: Student is member of DeptLibrary and Dept-Library is a part of Dept. The relation is member of is not R, not S but is T.
38

Use Case Diagram

Use cases roughly capture system level requirements . A collection of use cases defines: Complete functionality of the system.
39

Use Case Diagram

cont... Several general kinds of tests can be derived from use cases: 1. Tests for basic courses-- the expected flow of events mainline sequence. 2. Tests of other courses-- all other flows of events alternate sequences. 3. Tests of features described in user documentation traceable to each use case.

40

Use Case Diagram

cont...

Generic test requirements include: At least one test case should exercise:

Every Use Case when actor communicates with Use Case. Every extension combination such as Use Case 1 extends Use Case 2. Every uses combination such as Use Case 1 uses Use Case 2.
41

Class Diagram
Documents the structure of a system: Represents the entities and their interrelationships. Test cases should exercise: Each association. Independent creation and destruction of the container and the components in an aggregation relation. Each inherited method should be retested.
42

Sequence Diagram
A sequence diagram documents message exchanges in time dimension. Generic tests include: Message coverage: At least one test case should exercise each message. Message path coverage: All end-to-end paths should be identified and exercised.
43

Difficulties with Sequence Diagrams

Representing complex control is difficult: For example: notation for selection and iteration is clumsy. It becomes difficult to determine whether all scenarios are covered. Dynamic binding and the consequent superclass/subclass behavior : Cannot be represented directly.
44

Difficulties with Sequence

A Sequence diagram shows only a single collaboration. Different bindings are shown on separate sheets.

Diagrams

cont

45

Activity Diagram
Activity diagrams are based on: Flow charts, state transition diagrams, flow graphs and Petri nets. Can be considered to be an extension of flow chart: Can represent two or more threads of concurrent execution. Supports all features of a basic flow graph: Can be used to develop test models for control flow based techniques. 46

Activity Diagram
Decision tables

cont...

Activity diagram can be used to construct: Composite control flow graph representing a collection of sequence diagrams. Control flow graph at method scope:
This may be useful to analyze path coverage.
47

Activity Diagram
Generic test requirements:

cont...

At least one test case should exercise each different path:

Action 1 is followed by Action 2

Synch Point is followed by Action Action is started after Signal Synch Point is reached after
Action
48

Statechart Diagram
Can be used to represents statebased behavior of : a class, subsystem, or system. Statechart model: Provides most of the information necessary for class-level state-based testing.
49

Collaboration Diagram
Represents interactions among objects. A collaboration diagram specifies: The implementation of a use case. May also depict the participants in design patterns.
50

Collaboration Diagram

Each collaboration diagram represents only one slice of the IUT. A composite collaboration diagram would be necessary: To develop a complete test suite for an implementation.
51

cont...

Component Diagram
Shows the dependency relationships among: Components, Physical containment of components, Interfaces and calling components. The basic testing practice with respect to component diagrams: All call paths should be identified and exercised.
52

Deployment Diagram

Represents hardware, software, and network architecture. Useful in integration planning. When a component runs on a node, Test cases should exercise whether a component can be loaded and run on each designated host node. When a node communicates with another node, Test cases should exercise open, transmit, and close communication for each remote component.

53

Test Coverage Criteria Based on UML Models

Generalization criterion Class attribute criterion Condition coverage criterion Full predicate coverage criterion Each message on link criterion All message paths criterion Collection coverage criterion
54

Generalization (GN) Criterion

Given a test set T and a system model SM, T must cause:

Every specialization defined in a generalization relationship to be created.

GN criterion fault model: Likely to reveal faults that can arise from violation of the substitutability principle.
55

Substitutability Principle

An instance of a subclass can be used anywhere an instance of its superclass is expected.

56

Class Attribute Criterion

Given a test set T , a system model SM, and a class C, T must cause

A set of representative attribute value combinations in each instance of class C must be created.

The value space of attributes provides an opportunity to develop test criteria. The value space of an attribute are usually restricted by OCL constraints. Example: age attribute in Customer class can have a constraint age >18

57

Condition Coverage Criterion

Each message in a collaboration diagram occurs only under certain conditions. Given a test set T and collaboration diagram CD, T must cause: Each condition in each decision to evaluate to both TRUE and FALSE.
58

 Given a test set T and collaboration diagram CD, T must cause: Each clause in every condition in CD to take the values of TRUE and FALSE. A condition may consist of more than one clause connected by Boolean operators (e.g., AND, OR)
59

Full Predicate Coverage Criterion

 Given a test set T and collaboration diagram CD, T must cause:

Message On Each Link Criterion

Message on each link connecting two objects in CD to be exercised at least once. This criterion ensures: All possible messages between two objects occur during tests.
60

All Message Paths Criterion

Given a test set T and collaboration diagram CD, T must cause:

Each possible message path in CD to be taken at least once.

A message path is a sequence of messages. All message paths is a stronger criterion than message on each link criterion.

61

Collection Coverage Criterion

Given a test set T and collaboration diagram CD, T must test: Each interaction with a collection of objects of various representative sizes at least once. Example: An object Staff can be an aggregation of Office staff, Faculty and Technical Staff
62

Test Design Pattern

A design pattern: A generalized (reusable) solution to a recurring problem. It is based on common sense and some good practices. It provides a concise description of: common elements, context, and essential requirements for a solution. Test design patterns are to testing what software design patterns are to programming.

63

Test Design Pattern

cont...

Patterns developed under different scope of testing: Scope Example Pattern Method scope Polymorphic Message Test Class Scope Modal Class Test Reusable Components Abstract Class Test Subsystem Round-trip Scenario Test Integration Collaboration Integration Application Scope Extended Use Case Pattern Regression Test Retest Within Firewall
64

Extended Use Case (EUC)Test Pattern

Difficulties with use case : No variable definitions required No business rules required No input/output constraints Narrative can be ambiguous In a test-ready model we need to Define all input/output relationships Define all operational variables
65

EUC Intent
Model the input output relationships in a use case using a decision table. It then becomes possible to develop test cases: By considering specific inputs for a use case and corresponding expected results.
66

EUC Context

Unless test points are systematically selected based on a use cases implicit constraints and relationships:

cont...

All fundamental relationships would not be exercised. Example: A Use Case Pay-Fees can be a generalization of Pay-by-Credit-Card and Pay-by-Cash Use Cases
67

EUC Fault Model

What kind of bugs can EUCs help detect?

Example: Customer registers with a Supermarket:

Domain bugs. Logic bugs.

Domain error: Incorrect Pin code Logic Error: Sales registered against an invalid customer code.

68

EUC Fault Model

Incorrect output. Abnormal termination. Inadequate response time. Omitted capability. Extra capability.

cont...

Generic system-scope faults:

69

EUC Test Model

An Extended Use Case consists of the following: A complete inventory of operational variables. A complete specification of domain constraints for each operational variable. An operational relation for each use case The relative frequency of each use case (optional).

70

EUC Test Procedure

A test suite is developed in 4 Steps 1. Identify operational variables. 2. Define the domains of the operational variables. 3. Develop operational relations. 4. Develop Test cases.
71

Operational variables:

1. Identify Operational Variables

Factors which vary from one scenario to the next. Determines different system responses. Explicit inputs and outputs. Environmental conditions which result in significantly different behavior. Abstractions of the state of the system. 72

Operational variables include:

2. Define the Domains of the Operational Variables

The domains are developed by defining: The valid and invalid values for each variable.

73

 Operational variables:

3. Develop the Operational Relation

May determine distinct classes of system responses. Express them in the form of a decision table: When all the conditions in a row are true, the expected action is to be produced.
74

4. Develop Test Cases

Every variant is made true once and false once. Oracle: Expected results are typically developed by inspection.

75

EUC Test Example

76

Some classes can accept any message in any state:

Modal Class Test (MCT) Pattern

But others restrict based on past messages received or current values (domain).

A modal class is a special class :

Places domain constraint as well as sequence constraint while accepting messages.

Domain Constraint: An object of class Account will not accept a withdrawal message if the balance is less than 0. Sequence Constraint: In the frozen state, an Account object can accept a credit or debit message only after an unfreeze message.

77

Modal Class Test (MCT)

cont...

78

MCT Fault Model

A class may fail to implement its state model in five ways: 1. Missing transition: A valid message is rejected in a valid state. 2. Incorrect action: A wrong response for an accepting state and valid message. 3. Invalid resultant state: A message results in transition to a wrong state. 4. Invalid state: An invalid state is produced. 5. Sneak path: A sneak path allows a wrong message to be accepted.
79

MCT Test Strategy

The behavior of a class is represented using a state model. Generate the transition tree. Perform conditional transition test. Tabulate events and actions along each path to form test cases. Perform illegal transitions and sneak path test.
80

MCT Transition Tree

For each transition out of the root state , A branch is drawn to a node that represents the resultant state. This step is repeated for each resultant state node. A node is marked terminal : if no transition to a new state (yet to be drawn) exists. Each branch in the tree (an event path) becomes a test case.
81

MCT Transition Tree Testing: Example

82

 Each conditional transition:

MCT Conditional Transition Test

Analyzed to identify additional test cases.

For each conditional transition : Develop a truth table for the variables in the condition. Develop an additional test case: For each entry in the truth table that is not already exercised.
83

MCT Sneak Path Test

Sneak Path is a bug that allows an illegal message to be accepted, Resulting in an illegal transition. Illegal Transition is present: When a valid state of CUT accepts a message not specified for that state. Illegal Message: Results in an illegal transition.
84

MCT Sneak Path

cont... To test sneak paths send illegal messages:
Example: In the overdrawn state, messages: open, freeze, close etc can lead to possible sneak paths. To confirm that no sneak paths exist: Attempt each type of illegal message. The expected response: The message should be rejected. Also, the state of the object should be unchanged after rejecting the illegal message. 85

Collaboration Integration Pattern(CIP)

CIP chooses the order of integration: According to collaborations and dependencies. Classes to be integrated are tested by testing one collaboration at a time. A system consists of many collaborations. Their sequence is determined by dependencies and sequential activation constraints. Integration by collaboration: Exercises interfaces between the participants of a collaboration.
86

CIP Test Procedure

1. Develop a dependency tree for the SUT. 2. Map collaborations onto tree until all components and interfaces are covered. 3. Choose a sequence in which to apply the collaborations. 4. Several heuristics are available for choosing a sequence.
87

Example Heuristics:

CIP Test Procedure cont..

1. Begin with the simplest and finish with the most complex. 2. Begin with the collaboration that requires the fewest stubs . 3. Test in order of risk of disruption of system testing.
88

CIP Test Procedure

cont...

4. Develop test suite for each collaboration 5. Run the test suite and debug until first collaboration passes 6. Continue until all collaborations have been exercised
89

Research Challenge 1: Testing Based on Precode Artifacts

Precode artifacts: Design, Requirements, Architecture specifications. Software architecture involves description of elements from which systems are built, Interactions among those elements, Patterns that guide their composition, Constraints on these patterns.
90

Research Challenge 2:Testing Evolving Software

Regression testing is used to test software that undergoes evolution due to: Technology changes, New or modified components. Regression testing remains one of the most expensive activities performed during a software lifecycle.
91

Research Challenge 3: Measuring Effectiveness Of Testing Techniques

Quantitative values of effectiveness of a test-set design in revealing faults: Analytical, statistical, or empirical. We also need to understand the classes of faults for which the different criteria are useful.
92

Research Challenge 4: Test Data Generation

Generating test data (inputs for test cases) is often a labour-intensive process. To date, available techniques generating test data automatically work for toy systems: Do not scale to large systems. Also, it should be possible to automatically execute test cases: The test output should also be automatically analyzed.
93

Research Challenges 5: Methods and Tools

Along with fundamental research and empirical methods for testing: Tool development is also needed. An important requirement: Methods and tools be scalable to large systems. Many of the tools and methods developed so far work only on toy systems.
94