 My Expectations:

 Overview of different types of testing

 Where and how they fit into the overall SDLC
 Manual Testing using Test Cases / Scripts
 Automating unit testing using JUnit Framework
 Test Coverage Analysis

 Your Expectations ?

 Software testing is a critical element of software quality assurance and represents

the ultimate process to ensure the correctness of the product.
 A quality product which enhances the customer's confidence also increases the
business economics (and hence improves the economics of consulting companies
and in turn developers / testers etc)
 Testing is the process of executing a program with the intent of finding errors
 Testing is:
 Process of demonstrating that errors are not present
 Done to demonstrate that a program performs its intended functions
correctly
 Process of establishing confidence that a program does what is supposed
to do

Types of Testing

Activities Based:
 Unit Testing
 Functional Testing
 Regression Testing
 Functional Regression
 Unit Regression
 Break Testing / Monkey Testing
 Usability Testing
 UI Testing
 OS/Browser Testing
 Performance Testing
Project Phase Based:
 Build Testing
 Integration Testing
 Stabilization Testing
 User Acceptance Testing
 Pilot / Beta
 Types of Testing … (continued)

Nov Dec Jan Feb Mar Apr May Jun Jul

Project
Design
Live !!
Integration /
Development
Stabilization
UAT Pilot / Beta
Build Testing
Warranty Support

Unit Testing
Functional Testing

Regression Testing (Both Unit and Functional level)

Performance Testing

NOTE: Diagram not to Scale

Unit Testing

 Test at a ‘unit / building block’ level

 At a single class level in OO terms
 Written by developers during implementation
 Unit tests are written to test the fundamental building blocks of the system from
the inside out
 Can be manual or automated
 Helps developers in certifying that their code is written to a given specification
 Gives a safety net to developers – reduces the risk of breaking the application
because of a code change
 Cost of fixing bugs increases exponentially as time goes by… Unit testing helps
in bringing this cost down

Functional Testing

 Treats the system as a black box and ensure that the software works as a whole
 Test the system by providing know inputs and ensure that it returns
expected outputs
 Tests are done at a scenario / flow level
 Done post implementation by a QA team
 Can be manual or automated
 The ‘scenarios’ can be at different levels (depending on the phase of the project)
 Interaction of all layers in a given module
 Interaction between two / three related modules
 Business flows / scenarios involving multiple modules in the system
Regression Testing

 Bugs have a way of creeping into the systems and there are many ways and many
reasons why they occur
 When a particular bug gets fixed….there is always a chance that this particular fix
could potentially break some other code (either in the same module or other
related module)
 This could happen because the developer who fixes a bug might not
understand all the intricacies of the system
 Or developer could be careless while fixing
 The testing done by a developer after fixing a bug, to ensure that he/she has not
‘introduced’ a bug else where… is called Regression Testing
 Note, Regression testing can be done a ‘Unit’ level and/or at a ‘Functional’ level
 It is usually a combination of these two types of tests

Build Testing

 Testing that is done right after a regular build is completed and released to a ‘test’
environment is called Built Testing
 Build testing may include unit testing / functional testing / regression testing
 Testing done by the Infrastructure / CM team, before, releasing the new build to
other is called ‘Sanity Test’
 The purpose of this test is to ensure that the build is done correctly and
there are no configuration issues
 Infra team does some sanity tests like logging into the site, app can access
the DB, browsing to see if main pages are accessible, test if emails are
being sent etc
Break Testing

 Also called as ‘Monkey Testing’

 Usually done without any script/scenario to tell the user as to what to test (very
similar to ‘Free hand’ testing)
 Idea of this testing is to ‘monkey around’ to try and break the system by any
means possible
 Do things that you do not expect to do in a regular flow
 Enter huge amounts of data into text fields
 Change the query string parameters and their values
 In a wizard, go back and forth using browser back button multiple times
 Go to ‘random’ links from one module to the other
Usability Testing
 Usability testing is a means for measuring how well people can use some human-
made object (such as a web page, a computer interface, a document, or a device)
for its intended purpose, i.e. usability testing measures the usability of the object.
 Usability testing usually involves a controlled experiment to determine how well
people can use the product
 Rather than showing users a rough draft and asking, "Do you understand this?",
usability testing involves watching people trying to use something for its intended
purpose
 For example, to test the attachment function of an e-mail program, a scenario
would describe a situation where a person needs to send an e-mail attachment, and
ask him or her to undertake this task.
 http://en.wikipedia.org/wiki/Usability_testing

UI and OS/Browser Testing

UI Testing:
 User Interface Testing tests the system to ensure that it meets all the UI
specifications/standards
 Just like code has coding standards, applications do have UI standards for
the look and feel of the site
 CSS, font size, colours, lengths, button style, language etc

OS / Browser Testing:
 Testing the application on different OS / Browser combinations to ensure that the
app works from a functionality as well as a look and feel perspective

Performance Testing

 Performance parameters are defined as a NFR (Non Functional Requirement)

 Parameters that define how ‘fast’ a given page should be served back to the user,
when there is a ‘expected load’ on the system
 There are many different types of performance tests:
 Load Testing
 Stress Testing
 Endurance / Soak Testing
 Phase Based Testing
Nov Dec Jan Feb Mar Apr May Jun Jul

Project
Design
Live !!
Integration /
Development
Stabilization
UAT Pilot / Beta
Build Testing
Warranty Support

Unit Testing
Functional Testing

Regression Testing (Both Unit and Functional level)

Performance Testing

NOTE: Diagram not to Scale

Why so many types of testing?

V - Model
Requirements
Gathering
User Acceptance
Requirements Testing

Business Design
Functional Functional
Specification Testing

Tech Design
Technical Integration
Specification Testing

Detailed Tech
Program
Design
Unit Testing
Specification
Development

CODE

Manual Testing

How is manual testing done?

 Free hand testing:

  The tester is assumed to know the system (at least at a high level) so that
he can test what he wants
 Done without any ‘structure’ being provided … either in terms of flow of
app or what test data to use etc
 Tester will open new bugs that he finds on the system

 Script based testing:

 The user is provided with a ‘script’ that tells what exactly to do, what test
data to enter and what to expect from the system after each step
 Tester then captures his findings in the script
 Bugs can get opened
 Feedback about how the script was / changes to the script are also
captured

Test Scripts

 The general term used is ‘Test Scripts’ and any given test script might have
multiple ‘Test Cases’

Functional Verification Scripts (FVS):

 Usually written at a ‘screen’ level
 Used to verify that a completed screen and all its underlying components are
working as per the specifications
Scenario Scripts:
 These are scripts written to test the app across screens and multiple modules
 They are used to test if the build app handles a ‘business scenario’ correctly i.e as
designed
End to End Scripts:
 They are scripts written to test the app across a given set of applications …. i.e
interaction of one application with another new/existing application

JUNIT FRAME WORK

EXERCISE 1

 Writing a java class to manually test the following classes that have been
provided:
 IAccount.java
 Account.java
 AccountInsufficientFundsException.java
 How was the unit test ?
 What are its drawback ?

 How do we improve on it ….. use JUnit Framework.
 
What is JUnit ?

Open source regression testing framework for Java, written by Erich Gamma and Kent
Beck
 Part of XUnit family (HTTPUnit, CppUnit, Cactus etc)
 Consists of
 Base classes for definition of tests and test suites
 Tools for running tests (graphical and text-based)
 Used to write and run repeatable unit tests for Java objects ( Not EJB, JSP,
Servlets)

 Supported by many IDEs and build tools

 Eclipse, Visual Age, JDeveloper, ANT

Why JUnit ?

 JUnit tests allow you to write code faster while increasing quality
 JUnit is elegantly simple
 JUnit tests check their own results and provide immediate feedback (Green / Red
Status Bar)
 JUnit tests can be composed into a hierarchy of test suites
 Writing JUnit tests is inexpensive
 JUnit tests increase the stability of software
 JUnit tests are developer tests
 JUnit tests are written in Java
 JUnit is free !!

What does JUnit Provide?

 Framework provides tools for:
 Base classes for definition of test cases and suite
 Ability to execute these tests
 Assertions
 Aggregating tests (suites)
 Reporting results

 Philosophy :
 Let developers write unit tests
 Make it easy and painless
 So that they test early and test often !!!

assertion in JPL
  An assertion is a statement that enables you to test your assumptions about your
program
 Each assertion contains a boolean expression that you believe will be true when
the assertion executes.
 If it is not true, the system will throw an error
 By verifying that the boolean expression is indeed true, the assertion confirms
your assumptions about the behavior of your program, increasing your confidence
that the program is free of errors.
 Experience has shown that writing assertions while programming is one of the
quickest and most effective ways to detect and correct bugs.
 As an added benefit, assertions serve to document the inner workings of your
program, enhancing maintainability.
 Syntax:
 assert <boolean expression>
 OR
 assert <boolean expression> : <string expression>

 In the first statement, you only get a AssertError, where as in the second one, the
AssertError that is generated will contain the <string_expression>
 You will have to enable assertion via a jvm argument:
 “-enableassertions” or “-ea”

 Demo AssertionDemo.java

 http://java.sun.com/j2se/1.4.2/docs/guide/lang/assert.html

TestCase

 Any JUnit test should extend TestCase (provided by framework)

 Naming Convention: Test<ClassName>.java
 Each single test is contained in a single method
 Naming Convention: test<MethodName>
 A single test case can contain multiple testXYZ() methods
 Ideally every public method in the class that is being tested should have
1..n testXYZ()
 Each testXYZ() method should have at least one assertXYZ() method call

assertXYZ()

 assertEquals(double expected, double actual, double delta)

 assertEquals(java.lang.String message, double expected, double actual,
double delta)
  assertEquals(float expected, float actual, float delta)
 assertEquals(java.lang.String message, float expected, float actual, float delta)

 assertEquals(java.lang.Object[] expecteds, java.lang.Object[] actuals)

 assertEquals(java.lang.String message, java.lang.Object[] expecteds,
java.lang.Object[] actuals)

 assertEquals(java.lang.Object expected, java.lang.Object actual)

 assertEquals(java.lang.String message, java.lang.Object expected,
java.lang.Object actual)
 assertFalse(boolean condition)
 assertFalse(java.lang.String message, boolean condition)

 assertTrue(boolean condition)
 assertTrue(java.lang.String message, boolean condition)

 assertNotNull(java.lang.Object object)
 assertNotNull(java.lang.String message, java.lang.Object object)

 assertNull(java.lang.Object object)
 assertNull(java.lang.String message, java.lang.Object object)

 assertNotSame(java.lang.Object unexpected, java.lang.Object actual)

 assertNotSame(java.lang.String message, java.lang.Object unexpected,
java.lang.Object actual)

 assertSame(java.lang.Object expected, java.lang.Object actual)

 assertSame(java.lang.String message, java.lang.Object expected,
java.lang.Object actual)

 fail()
 fail(java.lang.String message)

EXERCISE 2

 Creating the first TestCase to test the Account.java class

Errors vs. Failures

 Assertions are used to check for the possibility of success and failures, therefore
success and failures within actual code should be anticipated and tested
appropriately
 Positive vs. Negative testing
  Failed assertions will result in JUnit failures
 Uncaught errors/exceptions result in JUnit errors
 Errors are unanticipated problems (within actual code or in a test case) resulting
in uncaught exceptions being propagated to a JUnit test method
 Both failures and errors will cause the test to fail, the behaviour of a failed test
within a test suite is up to the developer
 Ideally there should be zero errors and failures in the entire test suite
Fixtures

 Using a test fixture avoids duplicating the test code that is necessary to initialize
and cleanup objects that are common for each test within a given TestCase

 A test fixture is useful if you have two or more tests that use a common set of
objects
 Used to insure there are no side effects between tests
 Enforce the test independence rule, test execution order is not guaranteed
 Override setUp() method to initialize the variables
 Override tearDown() to release any permanent resources allocated in setUp()
 File Pointers, DB connections, sockets etc

EXERCISE 3

 Removing redundant set up operations in TestAccount1.java

EXERCISE 4

 Handling Exceptions that are thrown in code by writing positive and a negative
test cases

Test Suites

 TestCase instances can be composed into TestSuite hierarchies that automatically

invoke all the testXYZ() methods defined in each TestCase instance
 The composite behavior exhibited by the TestSuite allows you to assemble test
suites of test suites of tests, to an arbitrary depth, and run all the tests
automatically and uniformly to yield a single pass or fail status.

EXRCISE 5

 Creating Test Suite to include multiple TestCases

JUnit – Best Practices

 One Test Case for each Java Class and each public method in the class should
have 1…n textXYZ methods
 TestClass naming: Test<className>.java
 Test method naming convention: textXYZ()
 Test each class in isolation. Each test should test only one class, not indirectly its
collaborators.
 Do not use the constructor of the Test Case to set up data for the tests…use the
setUp() method
 Each test method should be independent of each other. One should not assume a
certain ‘state’ that will be build by the other test method
 Each test method should test only one specific flow through a method… if the test
method tests multiple flows, then it is really not unit testing
 Tests should be self verifying …. each and every test should run automatically
and should assert ….no manual intervention
 Be careful while writing test cases that changes the state of data, which might
affect other test cases
 Changing System data by a test case
 Do not load data from hard-coded locations on a filesystem
 FileInputStream inp ("C:\\TestData\\dataSet1.dat"); in the setUp() method
 As far as possible, use the assertXYZ() methods with the message option, so that
you get a good user friendly message in case of an error
 Running the entire suite of test cases for any given system should not take
hours….keep them simple and make them fast

JUnit – What to Test and What not to?

 It is not necessary to test the setters and getters of a class…..else we will be

testing the java compiler
 Every public method should be tested
 Test positive case (success scenario)
 Test negative case (i.e error conditions…and business exception being
thrown)
 Every exit point within a method should be tested
 If a method has a switch case with 3 switch statements…then you
got to have 3 test methods
 Exit points could be either positive or negative
 Always test for boundary conditions
Limitations of JUnit

 Can only test Java Objects

 Cannot test server side objects like servlets, EJB’s, JSP
 Use framework like Cactus, Mock Objects
 Cannot test Framework based code like Struts
 Use framework like Struts JUnit

 Good news is that, all the other frameworks mentioned above are build on top of
JUnit….so if you know JUnits then you already have a head start in understanding
other frameworks

Test Coverage

 To talk about test coverage, you must have two things:

1. A software product
2. A automated test suite used to verify that the product is working correctly

 Test coverage provides a measure of how well your test suite actually test the
product i.e if each and every line of the actual code that has been written is
executed, by running the entire test suite, then you have 100% coverage

 Procedure-Level Test Coverage:

 Probably the most basic form of test coverage is to measure what
procedures were and were not executed during the test suite
 In java this translates to which objects and which of its methods were
executed during the test suite
 Line-Level Test Coverage:
 The basic measure of a dedicated test coverage tool is tracking which lines
of code are executed, and which are not. This result is often presented in a
summary at the procedure, file, or project level giving a percentage of the
code that was executed

 Higher the % of test coverage, better the ‘quantity’ of testing and hopefully
quality also increases
 Test coverage tools give you these metrics…using which you can decide if you
need to write more test cases….and for which part/modules do you need to write
them or which parts of the code are being tested multiple times

Java Test Coverage Tools

  Open Source Code Coverage Frameworks:
 Coverlipse (Exercise - Eclipse Plugin … given to you -
http://coverlipse.sourceforge.net/howto.php)
 EMMA (Demo using ECLEMMA - updates from the internet -
http://www.eclemma.org/installation.html)
 NoUnit
 Jester
 InsECT
 Hensel
 GroboCodeCoverage
 Jcoverage/gpl
 Cobertura

Resources

 JUnit
 http://www.junit.org/index.htm

 Cactus
 http://jakarta.apache.org/cactus

 Code Coverage Analysis

 http://www.bullseye.com/coverage.html

 EMMA – Java Code Coverage Tool

 http://emma.sourceforge.net/

JUnit – TextUI and SwingUI

 Installation
 unzip the junit.zip file
 add junit.jar to the CLASSPATH. For example: set classpath=%classpath
%;INSTALL_DIR\junit3\junit.jar
 Testing
Test the installation by using either the batch or the graphical TestRunner tool to run the
tests that come with this release. All the tests should pass OK.
 for the batch TestRunner type:
java junit.textui.TestRunner junit.samples.AllTests
 for the graphical TestRunner type:
java junit.awtui.TestRunner junit.samples.AllTests
 for the Swing based graphical TestRunner type:
java junit.swingui.TestRunner junit.samples.AllTests
 Notice:
 The tests are not contained in the junit.jar but in the installation directory
directly. Therefore make sure that the installation directory is on the class
path
 Important:
 Don't install the junit.jar into the extension directory of your JDK
installation.
 If you do so the test class on the files system will not be found.