Software Testing Techniques with Test

Case Design Examples

What is Software Testing Technique?
Software Testing Techniques help you design better test cases. Since
exhaustive testing is not possible; Manual Testing Techniques help reduce the
number of test cases to be executed while increasing test coverage. They
help identify test conditions that are otherwise difficult to recognize.

Boundary Value Analysis (BVA)

Boundary value analysis is based on testing at the boundaries between
partitions. It includes maximum, minimum, inside or outside boundaries,
typical values and error values.

It is generally seen that a large number of errors occur at the boundaries of

the defined input values rather than the center. It is also known as BVA and
gives a selection of test cases which exercise bounding values.

This black box testing technique complements equivalence partitioning. This

software testing technique base on the principle that, if a system works well
for these particular values then it will work perfectly well for all values which
comes between the two boundary values.

Guidelines for Boundary Value analysis

 If an input condition is restricted between values x and y, then the test

cases should be designed with values x and y as well as values which
are above and below x and y.
 If an input condition is a large number of values, the test case should be
developed which need to exercise the minimum and maximum
numbers. Here, values above and below the minimum and maximum
values are also tested.
 Apply guidelines 1 and 2 to output conditions. It gives an output which
reflects the minimum and the maximum values expected. It also tests
the below or above values.

Example:
Input condition is valid between 1 to 10

Boundary values 0,1,2 and 9,10,11

Equivalence Class Partitioning
Equivalent Class Partitioning allows you to divide set of test condition into a
partition which should be considered the same. This software testing method
divides the input domain of a program into classes of data from which test
cases should be designed.

The concept behind this technique is that test case of a representative value
of each class is equal to a test of any other value of the same class. It allows
you to Identify valid as well as invalid equivalence classes.

Example:

Input conditions are valid between

1 to 10 and 20 to 30

Hence there are five equivalence classes

--- to 0 (invalid)
1 to 10 (valid)
11 to 19 (invalid)
20 to 30 (valid)
31 to --- (invalid)

You select values from each class, i.e.,

-2, 3, 15, 25, 45

Decision Table Based Testing.
A decision table is also known as to Cause-Effect table. This software testing
technique is used for functions which respond to a combination of inputs or
events. For example, a submit button should be enabled if the user has
entered all required fields.

The first task is to identify functionalities where the output depends on a

combination of inputs. If there are large input set of combinations, then divide
it into smaller subsets which are helpful for managing a decision table.
For every function, you need to create a table and list down all types of
combinations of inputs and its respective outputs. This helps to identify a
condition that is overlooked by the tester.

Following are steps to create a decision table:

 Enlist the inputs in rows

 Enter all the rules in the column
 Fill the table with the different combination of inputs
 In the last row, note down the output against the input combination.

Example: A submit button in a contact form is enabled only when all the
inputs are entered by the end user.

State Transition
In State Transition technique changes in input conditions change the state of
the Application Under Test (AUT). This testing technique allows the tester to
test the behavior of an AUT. The tester can perform this action by entering
various input conditions in a sequence. In State transition technique, the
testing team provides positive as well as negative input test values for
evaluating the system behavior.
Guideline for State Transition:

 State transition should be used when a testing team is testing the

application for a limited set of input values.
 The technique should be used when the testing team wants to test
sequence of events which happen in the application under test.

Example:

In the following example, if the user enters a valid password in any of the first
three attempts the user will be able to log in successfully. If the user enters
the invalid password in the first or second try, the user will be prompted to re-
enter the password. When the user enters password incorrectly 3rd time, the
action has taken, and the account will be blocked.

State transition diagram

In this diagram when the user gives the correct PIN number, he or she is
moved to Access granted state. Following Table is created based on the
diagram above-
State Transition Table
Correct PIN Incorrect PIN

S1) Start S5 S2

S2) 1st attempt S5 S3

S3) 2nd attempt S5 S4

S4) 3rd attempt S5 S6

S5) Access Granted - -

S6) Account blocked - -

In the above-given table when the user enters the correct PIN, the state is
transitioned to Access granted. And if the user enters an incorrect password,
he or she is moved to next state. If he does the same 3rd time, he will reach
the account blocked state.

Error Guessing
Error Guessing is a software testing technique based on guessing the error
which can prevail in the code. The technique is heavily based on the
experience where the test analysts use their experience to guess the
problematic part of the testing application. Hence, the test analysts must be
skilled and experienced for better error guessing.

The technique counts a list of possible errors or error-prone situations. Then

tester writes a test case to expose those errors. To design test cases based
on this software testing technique, the analyst can use the past experiences to
identify the conditions.

Guidelines for Error Guessing:

  The test should use the previous experience of testing similar
applications
 Understanding of the system under test
 Knowledge of typical implementation errors
 Remember previously troubled areas
 Evaluate Historical data & Test results

Boundary Value Analysis & Equivalence

Partitioning with Examples
Practically, due to time and budget considerations, it is not possible to perform
exhausting testing for each set of test data, especially when there is a large
pool of input combinations.

 We need an easy way or special techniques that can select test cases
intelligently from the pool of test-case, such that all test scenarios are
covered.
 We use two techniques - Equivalence Partitioning & Boundary Value
Analysis testing techniques to achieve this.

What is Boundary Testing?

Boundary testing is the process of testing between extreme ends or
boundaries between partitions of the input values.

 So these extreme ends like Start- End, Lower- Upper, Maximum-

Minimum, Just Inside-Just Outside values are called boundary values
and the testing is called "boundary testing".
 The basic idea in boundary value testing is to select input variable
values at their:

1. Minimum
2. Just above the minimum
3. A nominal value
4. Just below the maximum
5. Maximum
  In Boundary Testing, Equivalence Class Partitioning plays a good role
 Boundary Testing comes after the Equivalence Class Partitioning.

Equivalence Partitioning
Equivalence Partitioning or Equivalence Class Partitioning is type of black
box testing technique which can be applied to all levels of software testing like
unit, integration, system, etc. In this technique, input data units are divided
into equivalent partitions that can be used to derive test cases which reduces
time required for testing because of small number of test cases.

 It divides the input data of software into different equivalence data

classes.
 You can apply this technique, where there is a range in the input field.

Example 1: Equivalence and Boundary Value

 Let's consider the behavior of Order Pizza Text Box Below
 Pizza values 1 to 10 is considered valid. A success message is shown.
 While value 11 to 99 are considered invalid for order and an error
message will appear, "Only 10 Pizza can be ordered"

Submit
Order Pizza:      

Here is the test condition

1. Any Number greater than 10 entered in the Order Pizza field(let say 11)
is considered invalid.
 2. Any Number less than 1 that is 0 or below, then it is considered invalid.
3. Numbers 1 to 10 are considered valid
4. Any 3 Digit Number say -100 is invalid.

We cannot test all the possible values because if done, the number of test
cases will be more than 100. To address this problem, we use equivalence
partitioning hypothesis where we divide the possible values of tickets into
groups or sets as shown below where the system behavior can be considered
the same.

The divided sets are called Equivalence Partitions or Equivalence Classes.

Then we pick only one value from each partition for testing. The hypothesis
behind this technique is that if one condition/value in a partition passes all
others will also pass. Likewise, if one condition in a partition fails, all
other conditions in that partition will fail.
Boundary Value Analysis- in Boundary Value Analysis, you test boundaries
between equivalence partitions

In our earlier example instead of checking, one value for each partition you
will check the values at the partitions like 0, 1, 10, 11 and so on. As you may
observe, you test values at both valid and invalid boundaries. Boundary
Value Analysis is also called range checking.

Equivalence partitioning and boundary value analysis(BVA) are closely related

and can be used together at all levels of testing.

Example 2: Equivalence and Boundary Value

Following password field accepts minimum 6 characters and maximum 10
characters

That means results for values in partitions 0-5, 6-10, 11-14 should be
equivalent

Submit
Enter Password:       

Test Scenario # Test Scenario Description Expected Outcome

1 Enter 0 to 5 characters in password field System should not accep

2 Enter 6 to 10 characters in password field System should accept

3 Enter 11 to 14 character in password field System should not accep

Examples 3: Input Box should accept the Number 1 to 10

Here we will see the Boundary Value Test Cases

Test Scenario Description Expected Outcome

Boundary Value = 0 System should NOT accept

Boundary Value = 1 System should accept

Boundary Value = 2 System should accept

Boundary Value = 9 System should accept

Boundary Value = 10 System should accept

Boundary Value = 11 System should NOT accept

Why Equivalence & Boundary Analysis Testing

1. This testing is used to reduce a very large number of test cases to
manageable chunks.
2. Very clear guidelines on determining test cases without compromising
on the effectiveness of testing.
3. Appropriate for calculation-intensive applications with a large number of
variables/inputs

Summary:
  Boundary Analysis testing is used when practically it is impossible to
test a large pool of test cases individually
 Two techniques - Equivalence Partitioning & Boundary Value Analysis
testing techniques are used
 In Equivalence Partitioning, first, you divide a set of test condition into a
partition that can be considered.
 In Boundary Value Analysis you then test boundaries between
equivalence partitions
 Appropriate for calculation-intensive applications with variables that
represent physical quantities

Decision Table Testing: Learn with

Example

Decision Table
A Decision Table is a tabular representation of inputs versus rules/cases/test
conditions. It is a very effective tool used for both complex software testing
and requirements management. Decision table helps to check all possible
combinations of conditions for testing and testers can also identify missed
conditions easily. The conditions are indicated as True(T) and False(F)
values.

What is Decision Table Testing?

Decision table testing is a software testing technique used to test system
behavior for different input combinations. This is a systematic approach where
the different input combinations and their corresponding system behavior
(Output) are captured in a tabular form. That is why it is also called as
a Cause-Effect table where Cause and effects are captured for better test
coverage.

Let's learn with an example.

Example 1: How to make Decision Base Table for
Login Screen

The condition is simple if the user provides correct username and password
the user will be redirected to the homepage. If any of the input is wrong, an
error message will be displayed.

Conditions Rule 1 Rule 2 Rule 3 Rule 4

Username (T/F) F T F T

Password (T/F) F F T T

Output (E/H) E E E H

Legend:

 T – Correct username/password
 F – Wrong username/password
 E – Error message is displayed
 H – Home screen is displayed

Interpretation:
  Case 1 – Username and password both were wrong. The user is shown
an error message.
 Case 2 – Username was correct, but the password was wrong. The
user is shown an error message.
 Case 3 – Username was wrong, but the password was correct. The
user is shown an error message.
 Case 4 – Username and password both were correct, and the user
navigated to homepage

While converting this to test case, we can create 2 scenarios ,

 Enter correct username and correct password and click on login, and
the expected result will be the user should be navigated to homepage

And one from the below scenario

 Enter wrong username and wrong password and click on login, and the
expected result will be the user should get an error message
 Enter correct username and wrong password and click on login, and the
expected result will be the user should get an error message
 Enter wrong username and correct password and click on login, and the
expected result will be the user should get an error message

As they essentially test the same rule.

Example 2: How to make Decision Table for Upload

Screen
Now consider a dialogue box which will ask the user to upload photo with
certain conditions like –

1. You can upload only '.jpg' format image

2. file size less than 32kb
3. resolution 137*177.

If any of the conditions fails the system will throw corresponding error
message stating the issue and if all conditions are met photo will be updated
successfully
Let's create the decision table for this case.

Conditio Case 1 Case 2 Case 3 Case 4 Case 5 Case 6 Case 7 Case 8

ns

Format .jpg .jpg .jpg .jpg Not .jpg Not .jpg Not .jpg Not .jpg

Size Less Less >= >= 32kb Less Less >= >= 32k
than than 32kb than than 32kb
32kb 32kb 32kb 32kb

resolutio 137*17 Not 137*17 Not 137*17 Not 137*17 Not

n 7 137*17 7 137*17 7 137*17 7 137*17
7 7 7 7

Output Photo Error Error Error Error Error Error Error

upload messag messag messag messag messag messag messa
ed e e size e size e for e e for e for
resoluti mismat and format format format format,
on ch resoluti mismat and and size,
mismat on ch resoluti size and
ch mismat on mismat resolut
ch mismat ch on
ch misma
ch

For this condition, we can create 8 different test cases and ensure complete
coverage based on the above table.

1. Upload a photo with format '.jpg', size less than 32kb and resolution
137*177 and click on upload. Expected result is Photo should upload
successfully
 2. Upload a photo with format '.jpg', size less than 32kb and resolution not
137*177 and click on upload. Expected result is Error message
resolution mismatch should be displayed
3. Upload a photo with format '.jpg', size more than 32kb and resolution
137*177 and click on upload. Expected result is Error message size
mismatch should be displayed
4. Upload a photo with format '.jpg', size more than equal to 32kb and
resolution not 137*177 and click on upload. Expected result is Error
message size and resolution mismatch should be displayed
5. Upload a photo with format other than '.jpg', size less than 32kb and
resolution 137*177 and click on upload. Expected result is Error
message for format mismatch should be displayed
6. Upload a photo with format other than '.jpg', size less than 32kb and
resolution not 137*177 and click on upload. Expected result is Error
message format and resolution mismatch should be displayed
7. Upload a photo with format other than '.jpg', size more than 32kb and
resolution 137*177 and click on upload. Expected result is Error
message for format and size mismatch should be displayed
8. Upload a photo with format other than '.jpg', size more than 32kb and
resolution not 137*177 and click on upload. Expected result is Error
message for format, size and resolution mismatch should be displayed

Why Decision Table Testing is Important?

Decision Table Testing is Important because it helps to test different
combinations of conditions and provide better test coverage for complex
business logic. When testing the behavior of a large set of inputs where
system behaviour differs with each set of input, decision table testing provides
good coverage and the representation is simple so it is easy to interpret and
use.

In Software Engineering, boundary value and equivalent partition are other

similar techniques used to ensure better coverage. They are used if the
system shows the same behavior for a large set of inputs. However, in a
system where for each set of input values the system behavior is different,
boundary value and equivalent partitioning technique are not effective in
ensuring good test coverage.

In this case, decision table testing is a good option. This technique can make
sure of good coverage, and the representation is simple so that it is easy to
interpret and use.
This table can be used as the reference for the requirement and for the
functionality development since it is easy to understand and cover all the
combinations.

The significance of this technique becomes immediately clear as the number

of inputs increases. Number of possible Combinations is given by 2 ^ n ,
where n is the number of Inputs. For n = 10, which is very common in the web
based testing, having big input forms, the number of combinations will be
1024. Obviously, you cannot test all but you will choose a rich sub-set of the
possible combinations using decision based testing technique.

Advantages of Decision Table Testing

 When the system behavior is different for different input and not same
for a range of inputs, both equivalent partitioning, and boundary value
analysis won't help, but decision table can be used.
 The representation is simple so that it can be easily interpreted and is
used for development and business as well.
 This table will help to make effective combinations and can ensure a
better coverage for testing
 Any complex business conditions can be easily turned into decision
tables
 In a case we are going for 100% coverage typically when the input
combinations are low, this technique can ensure the coverage.

Disadvantages of Decision Table Testing

The main disadvantage is that when the number of input increases the table
will become more complex

What is State Transition Testing?

Diagram, Technique, Example

What is State Transition Testing?

State Transition Testing is a black box testing technique in which changes
made in input conditions cause state changes or output changes in the
Application under Test(AUT). State transition testing helps to analyze
behaviour of an application for different input conditions. Testers can provide
positive and negative input test values and record the system behavior.

It is the model on which the system and the tests are based. Any system
where you get a different output for the same input, depending on what has
happened before, is a finite state system.

State Transition Testing Technique is helpful where you need to test

different system transitions.

When to Use State Transition?

 This can be used when a tester is testing the application for a finite set
of input values.
 When the tester is trying to test sequence of events that occur in the
application under test. I.e., this will allow the tester to test the application
behavior for a sequence of input values.
 When the system under test has a dependency on the events/values in
the past.

When to Not Rely On State Transition?

 When the testing is not done for sequential input combinations.
 If the testing is to be done for different functionalities like exploratory
testing

Four Parts Of State Transition Diagram

There are 4 main components of the State Transition Model as below
1) States that the software might get
2) Transition from one state to another

3) Events that origin a transition like closing a file or withdrawing money

4) Actions that result from a transition (an error message or being given the
cash.)

State Transition Diagram and State Transition Table

There are two main ways to represent or design state transition, State
transition diagram, and state transition table.

In state transition diagram the states are shown in boxed texts, and the
transition is represented by arrows. It is also called State Chart or Graph. It is
useful in identifying valid transitions.

In state transition table all the states are listed on the left side, and the events
are described on the top. Each cell in the table represents the state of the
system after the event has occurred. It is also called State Table. It is useful in
identifying invalid transitions.

How to Make a State Transition (Examples of a State

Transition)
Example 1:
Let's consider an ATM system function where if the user enters the invalid
password three times the account will be locked.

In this system, if the user enters a valid password in any of the first three
attempts the user will be logged in successfully. If the user enters the invalid
password in the first or second try, the user will be asked to re-enter the
password. And finally, if the user enters incorrect password 3rd time, the
account will be blocked.

State transition diagram

In the diagram whenever the user enters the correct PIN he is moved to
Access granted state, and if he enters the wrong password he is moved to
next try and if he does the same for the 3rd time the account blocked state is
reached.

State Transition Table

Correct PIN Incorrect PIN

S1) Start S5 S2

S2) 1st attempt S5 S3

S3) 2nd attempt S5 S4

S4) 3rd attempt S5 S6
 S5) Access Granted - -

S6) Account blocked - -

In the table when the user enters the correct PIN, state is transitioned to S5
which is Access granted. And if the user enters a wrong password he is
moved to next state. If he does the same 3rd time, he will reach the account
blocked state.

In the flight reservation login screen, consider you have to enter correct agent
name and password to access the flight reservation application.

State Transition Graph

It gives you the access to the application with correct password and login
name, but what if you entered the wrong password.

The application allows three attempts, and if users enter the wrong password
at 4th attempt, the system closes the application automatically.

The State Graphs helps you determine valid transitions to be tested. In this
case, testing with the correct password and with an incorrect password is
compulsory. For the test scenarios, log-in on 2nd, 3rd and 4th attempt anyone
could be tested.

You can use State Table to determine invalid system transitions.

In a State Table, all the valid states are listed on the left side of the table, and
the events that cause them on the top.

Each cell represents the state system will move to when the corresponding
event occurs.

For example, while in S1 state you enter a correct password you are taken to
state S6 (Access Granted). Suppose if you have entered the wrong password
at first attempt you will be taken to state S3 or 2nd Try.

Likewise, you can determine all other states.

Two invalid states are highlighted using this method. Suppose you are in state
S6 that is you are already logged into the application, and you open another
instance of flight reservation and enter valid or invalid passwords for the same
agent. System response for such a scenario needs to be tested.
Advantages and Disadvantages of State Transition
Technique
Advantages Disadvantages

This testing technique will provide a pictorial The main disadvantage of this testing
or tabular representation of system behavior technique is that we can't rely in this
which will make the tester to cover and technique every time. For example, if the
understand the system behavior effectively. system is not a finite system (not in
sequential order), this technique cannot be
used.

By using this testing, technique tester can Another disadvantage is that you have to
verify that all the conditions are covered, define all the possible states of a system.
and the results are captured While this is all right for small systems, it
soon breaks down into larger systems as
there is an exponential progression in the
number of states.

Summary:
 State Transition testing is defined as the testing technique in which
changes in input conditions cause's state changes in the Application
under Test.
 In Software Engineering, State Transition Testing Technique is helpful
where you need to test different system transitions.
 Two main ways to represent or design state transition, State transition
diagram, and State transition table.
 In state transition diagram the states are shown in boxed texts, and the
transition is represented by arrows.
 In state transition table all the states are listed on the left side, and the
events are described on the top.
 This main advantage of this testing technique is that it will provide a
pictorial or tabular representation of system behavior which will make
the tester to cover and understand the system behavior efficiently.
 The main disadvantage of this testing technique is that we can't rely in
this technique every time.
What is Use Case Testing? Technique,
Examples
What is Use Case in Testing?
A Use Case in Testing is a brief description of a particular use of the
software application by an actor or user. Use cases are made on the basis of
user actions and the response of the software application to those user
actions. It is widely used in developing test cases at system or acceptance
level.

Use Case Testing

Use Case Testing is a software testing technique that helps to identify test
cases that cover entire system on a transaction by transaction basis from start
to end. Test cases are the interactions between users and software
application. Use case testing helps to identify gaps in software application that
might not be found by testing individual software components.

How to do Use Case Testing: Example

In a use-case, an actor is represented by "A" and system by "S". We create
Use for a login functionality of a Web Application as shown below

Main Success Scenario Step Description

 A:Actor 1 A: Enter Agent Name & Password
S:System

2 S: Validate Password

3 S: Allow Account Access

Extensions 2a Password not valid

S: Display Message and ask for re-try 4 times

2b Password not valid 4 times

S: Close Application

 Consider the first step of an end to end scenario for a login functionality
for our web application where the Actor enters email and password.
 In the next step, the system will validate the password
 Next, if the password is correct, the access will be granted
 There can be an extension of this use case. In case password is not
valid system will display a message and ask for re-try four times
 If Password, not valid four times system will ban the IP address.

Here we will test the success scenario and one case of each extension.

Test Plan Template: Sample Document

with Web Application Example
What is test plan template?
TEST PLAN TEMPLATE is a detailed document that describes the test
strategy, objectives, schedule, estimation and deliverables, and resources
required for testing. Test Plan helps us determine the effort needed to validate
the quality of the application under test. The test plan serves as a blueprint to
conduct software testing activities as a defined process which is minutely
monitored and controlled by the test manager.

Creating a Test Plan is mandatory to ensure success of your Software testing

project.If you are new to Test Planning refer this tutorial on How to Create a
Test Plan

Download Sample Test Plan Template

1) Introduction
Brief introduction of the test strategies, process, workflow and methodologies
used for the project

1.1) Scope

1.1.1) In Scope
Scope defines the features, functional or non-functional requirements of the
software that will be tested

1.1.2) Out of Scope

Out Of Scope defines the features, functional or non-functional requirements
of the software that will NOT be tested

1.2) Quality Objective

Here make a mention of the overall objective that you plan to achive with your
manual testing and automation testing.

Some objectives of your testing project could be

  Ensure the Application Under Test conforms to functional and non-
functional requirements
 Ensure the AUT meets the quality specifications defined by the client
 Bugs/issues are identified and fixed before go live

1.3) Roles and Responsibilities

Detail description of the Roles and responsibilities of different team members

like

 QA Analyst
 Test Manager
 Configuration Manager
 Developers
 Installation Team

Amongst others

2) Test Methodology
2.1) Overview

Mention the reason of adopting a particular test methodology for the project.
The test methodology selected for the project could be

 WaterFall
 Iterative
 Agile
 Extreme Programming

The methodology selected depends on multiple factors. You can read about
Test Methodology here

2.2) Test Levels

Test Levels define the Types of Testing to be executed on the
Application Under Test (AUT). The Testing Levels primarily depends on the
scope of the project, time and budget constraints.

2.3) Bug Triage

The goal of the triage is to

 To define the type of resolution for each bug

 To prioritize bugs and determine a schedule for all "To Be Fixed Bugs'.

2.4) Suspension Criteria and Resumption Requirements

Suspension criteria define the criteria to be used to suspend all or part of the
testing procedure while Resumption criteria determine when testing can
resume after it has been suspended

2.5) Test Completeness

Here you define the criterias that will deem your testing complete.

For instance, a few criteria to check Test Completeness would be

 100% test coverage

 All Manual & Automated Test cases executed
 All open bugs are fixed or will be fixed in next release

3) Test Deliverables
Here mention all the Test Artifacts that will be delivered during different
phases of the testing lifecycle.

Here are the simple deliverables

  Test Plan
 Test Cases
 Requirement Traceability Matrix
 Bug Reports
 Test Strategy
 Test Metrics
 Customer Sign Off

4) Resource & Environment Needs

4.1) Testing Tools

Make a list of Tools like

 Requirements Tracking Tool

 Bug Tracking Tool
 Automation Tools

Required to test the project

4.2) Test Environment

It mentions the minimum hardware requirements that will be used to test the

Application.

Following software's are required in addition to client-specific software.

 Windows 8 and above

 Office 2013 and above
 MS Exchange, etc.

5) Terms/Acronyms
Make a mention of any terms or acronyms used in the project

TERM/ACRONYM DEFINITION
 API Application Program Interface

AUT Application Under Test

Download the above Test Plan Template Format

Sample Test Plan Document Banking Web

Application Example
1 Introduction
The Test Plan is designed to prescribe the scope, approach, resources, and
schedule of all testing activities of the project Guru99 Bank.

The plan identify the items to be tested, the features to be tested, the types of
testing to be performed, the personnel responsible for testing, the resources
and schedule required to complete testing, and the risks associated with the
plan.

1.1 Scope
1.1.1 In Scope
All the feature of websiteGuru99 Bank which were defined in software
requirement specs are need to bested

Module Name Applicable Description

Roles

Balance Enquiry Manager Customer: A customer can have multiple bank

Customer accounts. He can
view balance of his accounts only
Manager: A manager can view balance of all the
customers who
 come under his supervision

Fund Transfer Manager Customer: A customer can have transfer funds from
Customer his “own”
account to any destination account.
Manager: A manager can transfer funds from any
source bank
account to destination account

Mini Statement Manager A Mini statement will show last 5 transactions of an

Customer account
Customer: A customer can see mini-statement of
only his “own”
accounts
Manager: A manager can see mini-statement of any
account

Customized Manager A customized statement allows you to filter and

Statement Customer display
transactions in an account based on date,
transaction value
Customer: A customer can see Customized-
statement of only
his “own” accounts
Manager: A manager can see Customized
-statement of any
account

Change Manager Customer: A customer can change password of

Password Customer only his account.
Manager: A manager can change password of only
his account.
He cannot change passwords of his customers

New Customer Manager Manager: A manager can add a new customer.

  Manager Manager: A manager can edit details like address,
email,
telephone of a customer.

New Account Manager Currently system provides 2 types of accounts

• Saving
• Current
A customer can have multiple saving accounts (one
in his name,
other in a joint name etc).
He can have multiple current accounts for different
companies
he owns.
Or he can have a multiple current and saving
accounts.
Manager: A manager can add a new account for an
existing
customer.

Edit Account Manager Manager: A manager can add a edit account details
for an existing account

Delete Account Manager Manager: A manager can add a delete an account

for a customer.

Delete Customer Manager A customer can be deleted only if he/she has no

active current or saving accounts
Manager: A manager can delete a customer.

Deposit Manager Manager: A manager can deposit money into any

account.
Usually done when cash is deposited at a bank
branch.

Withdrawal Manager Manager: A manager can withdraw money from any

account.
 Usually done when cash is withdrawn at a bank
branch.

1.1.2 Out of Scope

These feature are not be tested because they are not included in the software
requirement specs

 User Interfaces
 Hardware Interfaces
 Software Interfaces
 Database logical
 Communications Interfaces
 Website Security and Performance

1.2 Quality Objective

The test objectives are to verify the Functionality of website Guru99 Bank, the
project should focus on testing the banking operation such as Account
Management, Withdrawal, and Balance…etc. to guarantee all these
operation can work normally in real business environment.

1.3 Roles and Responsibilities

The project should use outsource members as the tester to save the project
cost.

No Member Tasks
.

1. Test Manager Manage the whole project

Define project directions
Acquire appropriate resources

2. Test Identifying and describing appropriate test

techniques/tools/automation architecture Verify and assess the Tes
Approach Execute the tests, Log results, Report the defects.
 Outsourced members

3. Developer in Implement the test cases, test program, test suite etc.
Test

4. Test Builds up and ensures test environment and assets are managed
Administrator and maintained
Support Tester to use the test environment for test execution

5. SQA Take in charge of quality assurance

members Check to confirm whether the testing process is meeting specified
requirements

2 Test Methodology
2.1 Overview
2.2 Test Levels
In the project Guru99 Bank, there're 3 types of testing should be conducted.

 Integration Testing (Individual software modules are combined and

tested as a group)
 System Testing: Conducted on a complete, integrated system to
evaluate the system's compliance with its specified requirements
 API testing: Test all the APIs create for the software under tested

2.3 Bug Triage

2.4 Suspension Criteria and Resumption Requirements
If the team members report that there are 40% of test cases failed, suspend
testing until the development team fixes all the failed cases.

2.5 Test Completeness

 Specifies the criteria that denote a successful completion of a test
phase
 Run rate is mandatory to be 100% unless a clear reason is given.
 Pass rate is 80%, achieving the pass rate is mandatory
2.6 Project task and estimation and schedule
Task Members Estimate effort

Create the test specification Test Designer 170 man-hour

Perform Test Execution Tester, Test Administrator 80 man-hour

Test Report Tester 10 man-hour

Test Delivery 20 man-hour

Total 280 man-hour

Schedule to complete these tasks

3 Test Deliverables
Test deliverables are provided as below

 Test plans document.

 Test cases documents
 Test Design specifications.

During the testing

- Test Tool Simulators.

- Test Data

- Test Trace-ability Matrix - Error logs and execution logs.

After the testing cycles is over

 Test Results/reports
 Defect Report
 Installation/ Test procedures guidelines
  Release notes

4 Resource & Environment Needs

4.1 Testing Tools
No Resources Descriptions
.

1. Server Need a Database server which install MySQL server Web server which
install Apache Server

2. Test tool Develop a Test tool which can auto generate the test result to the
predefined form and automated test execution

3. Network Setup a LAN Gigabit and 1 internet line with the speed at least 5 Mb/s

4. Computer At least 4 computer run Windows 7, Ram 2GB, CPU 3.4GHZ

4.2 Test Environment

Test Environment to be setup as per figure below

Test Environment for Software Testing

What is a Test Environment?

A testing environment is a setup of software and hardware for the testing
teams to execute test cases. In other words, it supports test execution with
hardware, software and network configured.

Test bed or test environment is configured as per the need of the Application
Under Test. On a few occasion, test bed could be the combination of the test
environment and the test data it operates.
Setting up a right test environment ensures software testing success. Any
flaws in this process may lead to extra cost and time to the client

Key areas to set up in Test Environment

For the test environment, a key area to set up includes

 System and applications

 Test data
 Database server
 Front-end running environment
 Client operating system
 Browser
 Hardware includes Server Operating system
 Network
 Documentation required like reference documents/configuration
guides/installation guides/ user manuals

Process of Software Test environment setup

Tests are limited to what can be tested and what not should be tested.

Following people are involved in test environment setup

 System Admins,
 Developers
 Testers
 Sometimes users or techies with an affinity for testing.

The test environment requires setting up of various number of distinct areas

like,

Setup of Test Server

Every test may not be executed on a local machine. It may need establishing
a test server, which can support applications.

For example, Fedora set up for PHP, Java-based applications with or without
mail servers, cron set up, Java-based applications, etc.

Network
Network set up as per the test requirement. It includes,

 Internet setup
 LAN Wifi setup
 Private network setup

It ensures that the congestion that occurs during testing doesn't affect other
members. (Developers, designers, content writers, etc.)

Test PC setup

For web testing, you may need to set up different browsers for different
testers. For desktop applications, you need various types of OS for different
testers PCs.

For example, windows phone app testing may require

 Visual Studio installation

 Windows phone emulator
 Alternatively, assigning a windows phone to the tester.

Bug Reporting

Bug reporting tools should be provided to testers.

Creating Test Data for the Test Environment

Many companies use a separate test environment to test the software

product. The common approach used is to copy production data to test. This
helps the tester, to detect the same issues as a live production server, without
corrupting the production data.

The approach for copying production data to test data includes,

 Set up production jobs to copy the data to a common test environment

 All PII (Personally Identifiable Information) is modified along with other
sensitive data. The PII is replaced with logically correct, but non-
personal data.
 Remove data that is irrelevant to your test.

Testers or developers can copy this to their individual test environment. They
can modify it as per their requirement.
Privacy is the main issue in copy production data. To overcome privacy issues
you should look into obfuscated and anonymized test data.

For Anonymization of data two approaches can be used,

 BlackList: In this approach, all the data fields are left unchanged. Except
those fields specified by the users.
 WhiteList: By default, this approach, anonymizes all data fields. Except
for a list of fields which are allowed to be copied. A whitelisted field
implies that it is okay to copy the data as it is and anonymization is not
required.

Also, if you are using production data, you need to be smart about how to
source data. Querying the database using SQL script is an effective approach.

Test Environment Management

Test Environment Management deals with the maintenance and upkeep of the
test bed.

List of activities by the Test environment management function include,

1. Maintenance of a central repository with all the updated version of test

environments.
2. Test environment management as per the test team demands.
3. As per the new requirements creating new environments
4. Monitoring of the environments
5. Updating/deleting outdated test-environments
6. Investigation of issues on the environment
7. Co-ordination till an issue resolution.

Test Environment Checklist

Hardware

1 Check whether required If this is not the case, analyze the supply time!
equipment for testing is
available?

  Check whether peripheral Such as scanners, special printers, handhelds, etc.

 equipment is available?

Software / connections

2 Are the needed applications An application such as excel, word, drawings, etc.
specified?

  For the new software does the Has the organization experience with use and
test environment exist for the maintenance of the software?
organization?

Environmental data

3 Check whether the standard With the regression test set, consider
test data sets are available? the Defect administration to collect test data.

  Do agreements with the test Consider functional maintenance.

data owners about the test
data exist?

Maintenance tools/processes

4 Check whether a single point If no, prepare a list of all possible members involved in
of contact exists for test keeping the test environment running. It should includ
environment maintenance? their contact information as well.

  Does the agreement reached For instance, acceptance criteria, maintenance

about the readiness and requirements, etc. Also, check whether other/extra
quality of the test quality attributes for environments are there in
environment? agreement.

  Do all members involved in  

the maintenance process are
 known?

Besides these, there are a few more questions to answer before setting up the
test environment.

 Whether to develop an internal Test Environment or to outsource?

 Whether to follow an internal company standard or follow any External
(IEE, ISO, etc.)?
 How long the test environment is required?
 Differences between the test and production systems and their impact
on test validity must be determined.
 Can you re-use an existing setup for other projects in the company?

Challenges in setting up Test Environment

Management
1. Proper planning on resource usage

Ineffective planning for resource usage can affect the actual output.
Also, it may lead to conflict between teams.

2. Remote environment

It is possible that a Test environment is located geographically apart. In

such a case, the testing team has to rely on the support team for
various test assets. (Software, hardware, and other issues).

3. Elaborate setup time

Sometimes test set up gets too elaborated in cases of Integration

Testing.

4. Shared usage by teams

If the testing environment is used by development & testing team

simultaneously, test results will be corrupted.

5. Complex test configuration

 Certain test requires complex test environment configuration. It may
pose a challenge to the test team.

Best practices for setting up a Test Environment

Management
1. Understand the test requirements thoroughly and educate the test team
members.
2. Connectivity should be checked before the initiation of the testing
3. Check for the required hardware and software, licenses
4. Browsers and versions
5. Planning out the Scheduled use of the test environment.
6. Automation tools and their configurations.

What is Test Bed in Software Testing?

A Test Bed in Software Testing is a software development environment. It
allows developers to test their modules without affecting the live production
servers. Test bed is not only confined to developers but also used by testers.
It is referred to as a test environment for rigorous and transparent testing of
new technologies.

Summary:

 A testing environment is a setup of software and hardware on which the

test team will conduct the testing
 For the test environment, a key area to set up includes
o System and applications
o Test data
o Database server
o Front-end running environment, etc.
 Few challenges while setting up a test environment include,
o Remote environment
o Combined usage between teams
o Elaborate setup time
o Ineffective planning for resource usage for integration
o Complex test configuration
Defect/Bug Life Cycle in Software
Testing
What is Defect Life Cycle?
Defect Life Cycle or Bug Life Cycle in software testing is the specific set of
states that defect or bug goes through in its entire life. The purpose of Defect
life cycle is to easily coordinate and communicate current status of defect
which changes to various assignees and make the defect fixing process
systematic and efficient.

Defect Status
Defect Status or Bug Status in defect life cycle is the present state from
which the defect or a bug is currently undergoing. The goal of defect status is
to precisely convey the current state or progress of a defect or bug in order to
better track and understand the actual progress of the defect life cycle.

The number of states that a defect goes through varies from project to project.
Below lifecycle diagram, covers all possible states

 New: When a new defect is logged and posted for the first time. It is
assigned a status as NEW.
 Assigned: Once the bug is posted by the tester, the lead of the tester
approves the bug and assigns the bug to the developer team
 Open: The developer starts analyzing and works on the defect fix
 Fixed: When a developer makes a necessary code change and verifies
the change, he or she can make bug status as "Fixed."
 Pending retest: Once the defect is fixed the developer gives a
particular code for retesting the code to the tester. Since the software
testing remains pending from the testers end, the status assigned is
"pending retest."
 Retest: Tester does the retesting of the code at this stage to check
whether the defect is fixed by the developer or not and changes the
status to "Re-test."
 Verified: The tester re-tests the bug after it got fixed by the developer. If
there is no bug detected in the software, then the bug is fixed and the
status assigned is "verified."
 Reopen: If the bug persists even after the developer has fixed the bug,
the tester changes the status to "reopened". Once again the bug goes
through the life cycle.
 Closed: If the bug is no longer exists then tester assigns the status
"Closed." 
 Duplicate: If the defect is repeated twice or the defect corresponds to
the same concept of the bug, the status is changed to "duplicate."
 Rejected: If the developer feels the defect is not a genuine defect then
it changes the defect to "rejected."
 Deferred: If the present bug is not of a prime priority and if it is expected
to get fixed in the next release, then status "Deferred" is assigned to
such bugs
 Not a bug:If it does not affect the functionality of the application then
the status assigned to a bug is "Not a bug".
Defect Life Cycle Explained

1. Tester finds the defect

2. Status assigned to defect- New
3. A defect is forwarded to Project Manager for analyze
4. Project Manager decides whether a defect is valid
5. Here the defect is not valid- a status is given "Rejected."
6. So, project manager assigns a status rejected. If the defect is not
rejected then the next step is to check whether it is in scope.
Suppose we have another function- email functionality for the
same application, and you find a problem with that. But it is not a
 part of the current release when such defects are assigned as
a postponed or deferred status.
7. Next, the manager verifies whether a similar defect was raised
earlier. If yes defect is assigned a status duplicate.
8. If no the defect is assigned to the developer who starts fixing the
code. During this stage, the defect is assigned a status in-
progress.
9. Once the code is fixed. A defect is assigned a status fixed
10. Next, the tester will re-test the code. In case, the Test
Case passes the defect is closed. If the test cases fail again, the
defect is re-opened and assigned to the developer.
11. Consider a situation where during the 1st release of Flight
Reservation a defect was found in Fax order that was fixed and
assigned a status closed. During the second upgrade release the
same defect again re-surfaced. In such cases, a closed defect will
be re-opened.

That's all to Bug Life Cycle

This training video describes the various stages in a bug aka defect life cycle
and its importance with the help of an example

Top 50 Software Testing Tools in 2020

What are Testing Tools?
TESTING TOOLS in software testing can be defined as products that support
various test activities starting from planning, requirement gathering, build
creation, test execution, defect logging and test analysis. These testing tools
are mainly used for testing software firmness, thoroughness, and other
performance parameters.

There are tons of software testing tools available in the market, and with the
plethora of choices it becomes difficult to zero in on the best testing tools for
your project. The following list categorizes, ranks and grades the different
software testing tools in the market. For each tool - important features, USP
and download links are mentioned.

Test Management Tool:

These tools help organize the end to end test Cycle
1) Xray

Xray is the #1 Manual & Automated Test Management App for QA. It’s a full-
featured tool that lives inside and seamlessly integrates with Jira. Its aim is to
help companies improve the quality of their products through effective and
efficient testing.

Features:

 Traceability between requirements, tests, defects, executions

 Define reusable preconditions and associate to tests
 Organize tests in folders and test sets
 Test plans for tracking progress
 Test environments
 BDD - Write Cucumber scenarios in Jira
 Integrates with test automation frameworks (Selenium, JUnit, Nunit,
Robot, ...)
 Built-in REST API
 CI integrations (Bamboo, Jenkins)
 Built-in reports

2) Testpad
Testpad is a simpler and more accessible manual test tool that prioritises
pragmatism over process. Instead of managing cases one at a time, it uses
checklist-inspired test plans that can be adapted to a wide range of styles
including Exploratory testing, the manual side of Agile, syntax highlighted
BDD, and even traditional test case management.
Key features:

 Guest testers, invited by email, who don’t need accounts

 Simple enough to use by non-testers; get everyone to help at release
time
 Keyboard-driven editor with a javascript-powered (i.e. responsive) UI
 Drag'n'drop organisation of test plans
 Add new tests during testing, as you think of new ideas
 Lightweight integration with issue trackers, including JIRA

3) Practitest
PractiTest is an end-to-end test management tool. A common meeting ground
for all QA stakeholders, it enables full visibility into the testing process and a
deeper broader understanding of testing results.

Features:

 A vast array of third-party integrations with common bug trackers,

automation tools, and robust API for the rest.
 Fully customizable & flexible for the ever-changing needs of QA teams:
customize fields, views, permissions, issue workflows and more
 Reuse tests and correlate results across different releases and
products.
  Unique hierarchical filter trees - organize everything and find anything
quickly.
 Never work twice - with anti-bug duplicates, permutations, step
parameters and call to test
 Visualize data with advanced dashboards and reports
 Fast professional and methodological support

4) TestRail
TestRail is your source for scalable, customizable, web-based test case
management. Set up in just minutes with our cloud-based/SaaS solution, or
install on TestRail your own server.

 Efficiently manage manual and automated test cases, plans, and runs.
 Get real-time insights into testing progress with informative dashboards,
metrics, and activity reports.
 Boost efficiency with milestones, personal to-do lists, and email
notifications.
 Document test cases with screenshots and expected results. Use the
flexible built-in templates or create your own custom templates.
 Integrate with tools in your CI/CD/DevOps pipeline including JIRA,
Bugzilla, Jenkins, TFS and more.
 Enterprise edition designed for large teams & mission-critical projects.
 Support for Docker containers.
5) TestMonitor
TestMonitor is an end-to-end test management tool for every organization. A
simple, intuitive approach to testing. Whether you’re implementing enterprise
software, need QA, building a quality app or just need a helping hand in your
test project, TestMonitor has you covered.

Features

 Requirement and risk-based testing.

 Advanced test case design capable of supporting thousands of cases.
 Robust planning tools with multi-tester runs and milestone cloning.
 Comprehensive result tracking.
 Integrated issue management.
 Smart reporting with many filter and visualization options.
 Revolutionary simple UI.
 Third-party integrations featuring Jira, DevOps, and Slack. REST API
included.
 Professional support with quick response time.

Automated Testing Tools

This category of tools helps automate functional and Regression Testing of
your application under test.

6) 21
21 is a sophisticated, self-learning test automation and analytics platform for
iOS and Android applications.
Features:

 Fast and intelligent authoring - AI assisted authoring empowers users to

create automated functional and UI tests in minutes.
 Results you trust - Seamless algorithmic locators system ensures stable
results across all frameworks.
 Eliminate maintenance and flaky results - self-learning maintenance
autonomously updates tests and assures your team can focus on
developing new features, while relying on test results.
 Release with confidence - Production integration closes the feedback
loop and analyze actual coverage. Use data when releasing.
 Fully SaaS, requires no installation or devices to create or execute
tests. Offers access to dozens of devices seamlessly.

7) Parasoft Selenic
Parasoft Selenic leverages AI to address your problems with Selenium so you
don’t have to get rid of Selenium. It works directly with your existing Selenium
tests, so you're not locked into a proprietary platform.

Key Features:
  Self-healing at runtime and AI-driven recommendations post-execution.
You can efficiently maintain existing Selenium scripts, reducing the time
spent maintaining, repairing, and fixing broken tests.
 Create Selenium test suites with the Page Object Model. Teams can
rapidly build a maintainable testing practice from the start.
 Optimize test execution with Test Impact Analysis. Cross-correlating
test execution with underlying code changes gives you quicker feedback
from your CI/CD pipeline.

8) Squish
Squish is the GUI Test Automation tool of choice for more than 3000
companies worldwide to automate the functional regression tests and system
tests of their graphical user interface (GUIs) and Human Machine Interfaces
(HMIs). The Squish GUI testing tool, a 100% cross-platform tool, features

 In-depth support for all major GUI technologies

 Full support for all desktop, mobile, web and embedded platforms
 Test script recording
 Powerful and reliable object identification and verifications (object-based
and image based)
 No dependency on screenshots or visual appearance
 Powerful integrated development environment (IDE)
 A wide range of popular script languages for test scripting
 Full support for Behavior Driven Development (BDD)
 Full control via command line tools
 Integrations with Test Management and CI-Systems
9) Ranorex
Over 14,000 users worldwide accelerate testing with Ranorex Studio, an all-
in-one tool for cross-platform test automation. Ranorex is easy for beginners
with a codeless click-and-go interface, but powerful for automation experts
with a full IDE.

Features include:

 Robust object identification for reliable tests and reduced maintenance

 Shareable object repository and reusable code modules
 Cross-browser testing
 Automates challenging interfaces like SAP, ERP, Delphi and legacy
applications
 Run tests in parallel or distribute on a Selenium Grid
 Built-in reporting

Ranorex integrates with solutions for BDD, CI/CD, source control, test
management, defect-tracking, and more for a complete test automation
toolchain

10) Selenium:
Selenium is one of the most popular software testing tools. It specifically
designed to support Automation Testing of functional aspects of web based
applications, wide range of platforms and browsers.

Features:

 It is one of the best qa tools which offers the support for parallel test
execution that reduce the time taken in executing parallel tests.
 Selenium needs very lesser resources when compared to other manual
testing tools.
 Test cases prepared using this testing tool can be executed on any OS
 It supports the many known programming languages like Java, Python,
C#, Perl, PHP, and JavaScript.

Download Link: http://www.seleniumhq.org/download/

11) QTP:

Quick Test Professional (QTP) now called as Micro Focus UFT (Unified
Functional Testing) is an automated functional GUI testing tool which allows
the automation of user actions on a web or client based computer application.
It is widely used for functional regression test automation. It is one of the
manual testing tools which uses a scripting language to manipulate the
objects and controls of the application under test.

Features:

 QTP is one of the software testing tools that allows beginner testers to
learn this tool in the few minutes.
 The tool is very easy to understand. It presents a test case in a simple
workflow to the tester.
 The tool allows complete validation of applications through a full
complement of checkpoints
Download Link: https://www.microfocus.com/en-us/products/uft-
one/overview

12) Watir:

Waitr is an open-source cross-platform web application testing tool. It is most

reliable and flexible automation tool of Ruby libraries for web browsers
automation. This cross browser testing tool interacts with a browser just like
human being so it clicks links, filling out forms and validating text.

Features:

 It is one of the best testing tools which is free, so there is no costs to

use the tool.
 This automated testing tool is supported by very active and growing
community.
 It is one of the qa tools which supports multiple browsers on different
platforms.
 It is very powerful and lightweight tool.

Download Link: http://watir.com/guides/downloads/

13) Testim:
Testim is a modern day automated testing tool. It uses machine learning to
speed the authoring, execution, and maintenance of automated tests. The
tester can be analysis any test case in few minutes and execute them on
multiple web and mobile platforms.

Features:

 It is one of the qa testing tools that can easily add annotations to find
out what's wrong in the system.
 Quickly view and share easy-to-understand annotated screenshots
using the bug tracker.
 Developers just need to click the automated bug test to reproduce in the
browser automatically.

Download link: https://www.testim.io/

14) AppliTools:

Applitools is an automated testing tool which automatically validates the look

and feels and user experience of the apps and sites. It is designed is such a
way that it easily integrates with the existing tests instead of requiring to
create a new test.

Features:

 It is one of the qa tools which allows cross browser test in various

devices
 Provide Interactive visual test reports to the user
 Robust user access management
 It is available as a cloud service or on premise

Download Link: https://applitools.com/users/register
15) TestComplete:

TestComplete is an automated test management tool which helps to increase

efficiency and reduce the cost of the testing process. It's very easy-to-use
interface helps QA teams to implement an automation solution in very less
amount of time.

Features:

 It is one of the qa tools that supports multiple scripting languages

 Allows to record robust automated tests without scripting knowledge
 It offers data-driven testing
 Allows user to create Customize plugins and extensions

Download Link: https://support.smartbear.com/downloads/testcomplete/

Cross-browser Testing Tools

This category of tool help in Cross Browser Testing of your site across
Chrome, Firefox, IE, Edge, Safari, and other browsers.

16) LambdaTest
LambdaTest is a scalable cloud-based cross browser testing platform
designed to offer all website or web app testing need to cloud infrastructure.
LambdaTest platform helps you to ensure your website pr web app render
seamlessly across every desktop and mobile browser with support of manual,
visual, and automated testing. With LambdaTest, you can access up to 2000+
combinations of desktop and mobile browsers on the cloud.

Features

 Selenium Web Testing Automation

 Live Interactive Browser Compatibility Testing
  Faster Automated Screenshot Testing
 Parallel Testing For Faster Goto Market Launch
 API For Continuous Testing
 Continuous Testing with Continuous Integration tools

17) Browsera:

This is one of the best browser compatibility testing software which allows
testing website and its elements in multiple browsers. This tool also used to
test website and all web pages for scripting and layout errors.

Features:

 Browsera can quickly detect cross-browser layout problems

automatically by comparing each browser's output.
 JavaScript errors from every browser are collected and reported after
the every test.
 With its site crawling feature, it's easy to test all the web pages of the
single site.

Download Link: http://www.browsera.com/web_sites
18) CrossBrowser Testing:

Cross-browser testing helps to ensure that website or web application

functions correctly in various web browsers. With the help of this tool, it is
possible to run parallel automated tests, compare screenshots, and remotely
debug real desktop and mobile browsers.

Features:

 It is one of the qa testing tools that can run Numerous Tests on Multiple
Devices At One Time
 Easily run automated tests against real iOS, Androids, & other desktop
browsers
 It allows running testing frameworks like WebDriver.IO, and Nightwatch
for superior quality and speed

Download link: https://crossbrowsertesting.com/freetrial

19) SauceLabs:

Sauce Labs is a Selenium cloud-based solution that supports automated

cross-browser testing. It can perform testing in any OS and platform and
browser combination.

Features:

 It is possible to test the web and mobile apps with the highest number of
platforms, browsers, and OS combinations.
  The tool ensures web applications work efficiently with all the latest
Android and iOS mobile devices.
 Run JavaScript unit tests for web apps to make sure that code works in
every environment.

Download Link: https://saucelabs.com/signup/trial

20) Ghostlab:

Ghostlab is a Mac based testing app that allows test out responsive design
across a variety of devices and browsers. It is a tool for synchronized browser
testing. It synchronizes scrolls, clicks, reloads and form input across all
connected clients to test a full user experience.

Features:

 No setup required for installation

 This mobile testing tool can Synchronised clicks, events and scrolls
 It provides support for remote debugging for all types of browser

Download Link: https://saucelabs.com/signup/trial
21) Browsershots:

Browsershots is a cross-browser testing tool allows testing website which is

running on Windows, MacOS, Android or iOS. It provides screenshots of a
website across 200+ browsers. This tool provides screenshots of websites in
a testing environment which is helpful for finding all the major HTML or CSS
faults.

Features:

 Browsershots is a completely free tool, and it provides support for 200

different browser versions to capture screenshots
 Browsers supported range from Arora, Chrome, Dillo, Firefox,
Iceweasel, Konqueror, Links, SeaMonkey, and Safari.
 It allows to disable JavaScript, Change Color Depth, and disable/enable
Java and Flash.

Download link: http://browsershots.org/

Load Testing Tools

These tools help performance/load test a site or application.

22) Webload:
WebLOAD is an excellent testing tool which offers many powerful scripting
capabilities, that is helpful for testing complex scenarios. The tool supports
hundreds of technologies from Selenium to mobile, enterprise application to
web protocols. It is possible to generate load both in the cloud and on-premise
using this tool.

Features:

 Building load test scenarios more easily and efficiently with WebLOAD
 It helps you identify performance bottlenecks in your system with more
than 80 reports types and graphs.
 Performance Testing can run from the cloud by generating load from
Amazon EC2.

Download Link: http://www.radview.com/webload-download/

23) Loadrunner:

It is a load testing tool for Windows and Linux, which allows testing the web
application efficiently. It helpful testing tool to determining the performance
and result of the web application under heavy load.

Features:

 It offers supports for various types of Apps

 This testing tool can work on several enterprise environments.
  All the Vusers can be controlled with just a single dashboard.
 The LoadRunner provide support for several types of protocols.
 The monitoring and analysis is very user-friendly and easy to grasp.

Download Link: https://www.microfocus.com/en-us/products/loadrunner-
professional/free-trial

24) Wapt:

Wapt is a load, and stress testing tool works for all Windows. It provides an
easy and cost-effective way to test all types of websites. It works with the
same efficiency on secure HTTPS websites, dynamic content and RIA
applications under data-driven mode. This testing tool also provides supports
for RIA applications in the data-driven mode.

Features:

 Allows to include different types of users in a single test

 Provide most flexible load level definitions
 The tool uses a large number of advanced techniques to simulate real
load conditions.
 WAPT supports testing of websites and applications secured by SSL

Download Link: https://www.loadtestingtool.com/download.shtml

25) LoadUI Pro:

LoadUI is an open source load testing tool that allows performing complex
load tests by simply dragging the different components. It also allows to create
and update test cases while it's executing which make it a worthy tool to use.

Features:

 Allows to create multiple performance strategies.

 Reuse of existing SoapUI Pro functional tests.
 Real-time feedback about load testing results.
 Simultaneous load test even in the complex scenarios.

Download Link: https://www.loadui.org/downloads/download-loadui-pro.html

26) Silk Performer:

Silk Performer is the cost-effective load testing tool to meet all the critical
applications, performance expectations, and service-level requirements. It
also supports cloud integration which means that it is easy to simulate
massive loads without a need to invest in hardware setup.

Features:

 Quickly find the root cause of performance issues with in-depth analysis
 It can be used anywhere with unlimited scalability from the Cloud
 Authenticate real time user experience by transaction monitoring and
performance tracking

Download Link: https://www.microfocus.com/products/silk-portfolio/silk-
performer/trial/
27) Jmeter:

Apache JMeter is one of the open source testing tools for load testing. It is a
Java desktop application, designed to load test functional behavior and
measure performance of websites. The tool was developed for the purpose of
load testing web applications, but it is now expanded to other test functions.

Features:

 JMeter allows performing load and performance test for various server
types.
 This load testing tool store its test plans in XML format which allows
users to generate the test plan using a text editor.
 It is one of the manual testing tools that can be also used to perform
automated and functional testing of the applications.

Download link: http://jmeter.apache.org/download_jmeter.cgi

28) Agileload:

AgileLoad is a load testing tool to enhance the performance of business

critical applications. With the help of this load testing tool, it is easy to test the
performance of all kind of web and mobile application.

Features:

 Full support of dynamic web and mobile technologies

 Excellent monitoring and comprehensive analysis diagnostics
  It provides support for creating customizable test reports
 Take the right decisions to improve the application performance rapidly

Download link: http://www.agileload.com/signup-download/getting-started-
with-agileload

31) Loadfocus

LoadFocus is the best cloud testing tool for Load Testing and Performance
Testing. It also offers cloud testing services like Website Speed Testing,
Mobile Applications, APIs testing, and Mobile Emulation on different types of
Mobile Devices.

Features:

 Cloud Load Test Website and RESTful APIs

 Load tests can run in the cloud or on secure servers
 Allows number of repeats to inspect application behavior
 It offers Website Speed Testing and Insight Analytics

Download link: https://loadfocus.com/

32) BlazeMeter:

BlazeMeter is a Load testing tool which ensures delivery of high-performance

software to quickly run performance tests for mobile apps, website or API to
check the performance at every stage of its development.

Features:
  It allows to test website and integrate multi-geo locations results into
single report
 Retrieve site data from Google Analytics account and integrate them
into a new test setup.
 Use VPN credentials to integrate a series of load servers into the private
network.

Download Link: http://info.blazemeter.com/live-request-a-demo

33) LoadImpact:

Load Impact is the best cloud-based load testing system which widely used by
enterprises all over the world to develop their websites, mobile applications,
web-based apps, and APIs by performing all types of test.

Features:

 It simulates traffic exactly happen in real life. No hidden caching or

unreal user behaviour.
 Generating load from 10 different locations simultaneously is possible.
 Allows to Record an HTTP session using a proxy recorder and allow to
perform the same actions during the test.

Download Link: https://saucelabs.com/signup/trial

Defect Tracking Tools

This category of tools help in defect/bug management.
34) JIRA:

JIRA is a defect tracking tool which is used for defect/issue tracking as well as
project management. This tool is not only used for recording, reporting but
also integrated directly with code development environment.

Features:

 JIRA Query Language helps to create quick filters with a single click
 Possible to create custom workflows of any size which is helpful to
build, test, and release software.
 Install plug-and-play add-ons from The Atlassian Marketplace to fit all
types of cases.

Download Link: https://www.atlassian.com/software/jira/try

35) Mantishub:

Mantis is an open source defect tracking tool that provides a great balance
between simplicity and power. The users can easily get started with this tool
for managing their teammates and clients effectively.

Features:

 Improve workflow and efficiency of the organisation

 Support for internal issue tracking
 This tool allows Single-sign-on using Bitbucket and GitHub
 MantisHub has its own inbuilt time tracking feature which is helpful for
effectively report on time spent on the specific issues.
Download Link: https://www.mantishub.com/signup/gold

36) FogBugz:

The FogBugz is a tracking tool which can be used to track the status of
defects and changes in ongoing software projects, such as application
development and deployment. It is specifically helpful for organizations to
keep track of bugs for multiple projects.

Features:

 Fogbugz provide flexibility to find the information in no time.

 It provide support for Agile project management
 Notifications and emails keep team members updated about changes to
the projects and cases

Download Link: https://www.fogcreek.com/fogbugz

37) Bugzilla:

Bugzilla is one of the best defect Tracking System. The tool allows individual
or groups of developers to keep track of outstanding bugs in their system. It is
the best open source software used in the market by small scale as well as
large- scale organizations.

Features:
  Optimized database structure to enhance performance and scalability
 Advanced query tool that remembers customized searches of the user
 Editable user profiles and comprehensive email preferences
 Extension Mechanism for Highly Customizable Installations

Download Link: https://www.bugzilla.org/download/

38) BugNet:

BugNet is open source Bug Finding Tool. It is a cross-platform application that

is written using an ASP.NET platform, and it needs MySQL database as
backend tool. The main objective of this defect tracking tool is to make
codebase simple and easy to deploy.

Features:

 It is open source web base application under GPL license

 BugNET tool makes it simple to file, manage and report bugs
 The tool support multiple databases
 Offer an easy navigation and easy administration

Download Link: https://bugnet.codeplex.com/releases

39) Bug Genie:

It is an open source, web-based bug tracking software. Its main functionality
includes issue tracking, bug reporting, and project management. It also
includes a feedback publishing system.

Features:

 Easy and efficient source code management

 Helpful for Interactive project planning
 It offers Powerful command line tools
 Provide support for Incoming and outgoing email

Download Link: http://www.thebuggenie.com/

40) RedMine:

Redmine is another important defect tracing tool. The basic version of this tool
is open-source and it can work on any machine that supports Ruby. It takes
more time for installation, but once installed it runs smoothly.

Features:

 Issue creation via e-mail

 Offers support for multiple databases
 Flexible issue tracking system
 Flexible role based access control

Download Link: http://www.redmine.org/
Mobile Testing Tools
These tools help to automate testing of your Android or iOS applications.

41) Appium:

Appium is one of the open source testing tools for automation of mobile
applications. It allows users to test all kind of native, mobile, web, and hybrid
apps. It also supports the automated tests on emulators and simulators.

Features:

 Test the same application which is going to the marketplace.

 It is a simple application which needs a very little memory of for test
process.
 Testing Native apps do not need SDK, it offers standard automation
APIs which can be used on for all types of platforms.

Download Link: http://appium.io/
42) Espresso:

Espresso is a mobile testing tool for the enterprises. As an open-source tool,

it's very easy to use, and it can extend within that working environment. This
test recorder tool allows creating UI tests for the mobile app without writing a
single line of the test code.

Features:

 Espresso Workflow Is Simple to Use

 Less Mobile Testing Flakiness
 Fast and consistent feedback to developers
 Developing Espresso test automation is quite easy.

Download link: https://developer.android.com/training/testing/espresso/

43) Perfecto:

Perfecto is a SaaS platform that allows app developers to perform web,

mobile & IoT software testing. This tool also allows customers to select the
deployment options that are best for the specific project.

Features:
  Cloud base mobile application testing
 It provides support for Agile Environment
 The tool can integrate with other qa testing tools like Selenium and
Appium

Download link: https://www.perfecto.io/

44) ExperiTest:

Experitest is a quality assurance tools for mobile developers. The tool

includes Manual Testing, Performance Testing, and load testing for mobile
applications. This testing tool supports all mobile OS, iOS, Android, Windows
Phone, and Blackberry.

Features:

 Create and execute automated tests on simulators or emulators hosted

in Experitest data centers
 Record and play tests, and automatically recorded test to code and use
the same test script for different mobile OS.
 It offers reports with video or Screen HTML-based reporting
 View mobile apps element structure and generate identifiers with ease

Download Link: https://experitest.com
45) Robotium:

Robotium is an open-source test automation tool mainly use for Android UI

testing. It supports both native and hybrid applications. It is possible to
perform System Testing, functional testing, and user acceptance testing over
Android-based apps with the help of this tool.

Features:

 Record Android UI test cases in minutes

 Provide support to records from emulators and actual devices
 Automatically detects resource ID:s
 Complete support for native and hybrid Android apps

Downloadlink: https://robotium.com/pages/free-trial

API Testing Tools

These tools help in testing REST/SOAP protocols
46) SoapUI:

SoapUI is one of the best testing tools which is cross-platform open source
tool for functional testing of SOAP and REST, written use the Java language.
It is primarily used to perform functional and load testing on API.

Features:

 The GUI of the software is easy to handle and use

 Vulnerability testing feature helps to secure website from hackers and
viruses.
 It is possible to do the detailed analysis using its reporting feature.
 SQL Injection feature provide some standard SQL queries and methods
to identify the weak areas of the application.

Download Link: https://www.soapui.org/downloads/download-soapui-pro-
trial.html

47) SOAPSonar:
SOAPSonar is an Api Testing tool which focuses on reducing the time and
complexity to develop and maintain test cases. It supports testing every
individual service independently of the client application and yet groups the
test workflow for automation. Moreover, the creation and execution of these
test cases require no programming or scripting skills.

Features:

 SOAP, XML, and REST service validation

 Functional Testing with Success Rule Framework
 Performance Profiling and Concurrent Client Load Testing
 Web Service Security Testing with Risk Mediation

Download Link: http://www.crosschecknet.com/products/soapsonar.php

48) WebInject:

WebInject is the best Api Testing tool for automated testing of web
applications and web services. It can also test individual system components
which have HTTP interfaces and can be used to perform automated
functional, regression and acceptance tests.

Features:

 HTTP response times can be monitored in real-time at the time of test

execution.
 Combine mobile and desktop GUI tests with web testing
 Timer statistics are calculated and displayed during the runtime.

Downloadlink: http://www.webinject.org/download.html
49) Tricentis:

Tricentis is an Api Testing tool which helps to manage test cases reduces
testing time, manual effort and costs by building up and executing test cases.

Features:

 It offers Autonomous SAP Testing

 Mature, Robust SAP Test Automation Capabilities
 Solution Manager Integration

Download link: https://www.tricentis.com/automated-software-testing-tool-
trial/

Security Testing Tools

These tools detect security vulnerabilities in your Application Under Test.

50) NetSparker:

NetSparker is a security testing tool which automatically scans websites, web

applications and web services for vulnerabilities. It is the only scanner that
automatically identified and verifies vulnerabilities with a proof of concept.

Features:

 Affordable and Maintenance Free Web Application Security Solution

 It is fully Configurable Online Web Vulnerability Scanner
 This security testing tool easily Integrate Web Security Scanning In
SDLC
 It supports Enterprise Level Collaboration
Download Link: http://browsershots.org/

51) OWASP:

The Open Web Application Security Project is known as OWASP is a tool that
helps organizations to develop, purchase, and maintain web and software
which are reliable and trusted.

Features:

 Verify for Security Early and Often

 Validate all types of Inputs
 Implement Identity and Authentication Controls
 Implement all the appropriate Access Controls

Download Link: https://github.com/zaproxy/zaproxy/wiki/Downloads

52) Acunetix Vulnerability Scanner:

Acunetix Web Vulnerability Scanner is the leading security testing tool. It can
find various types of security vulnerabilities than any other scanners and
displays the fewest number of false positives.

Features:
  Most advanced Cross site scripting testing and in-depth SQL injection
 Comprehensive scanning of Single Page Applications and JavaScript-
based websites
 Detection of Blind XSS vulnerabilities
 Automated detection of DOM-based XSS vulnerabilities

Download Link: https://www.acunetix.com/vulnerability-scanner/download/

CSS Validator Tool

53) W3C CSS validator:

It is a free software developed by the W3C to help Web designers and Web
developers to check their CSS. With the help of this validation tool. It also
helps users to find errors or incorrect uses of CSS.

Features:

 W3C CSS validator recognize many browser extensions and prefixes

 It includes an integrated web browser. Therefore, websites can be
browsed and checked at the same time.
 It checks for and alerts accessibility issues to prevent people from using
that page.

Download Link: https://jigsaw.w3.org/css-validator/DOWNLOAD.html

54) Telerik studio:

Telerik Studio is a software testing tool to test web and desktop applications of
all Windows OS. It is a useful tool to test functionality, load and the
performance of the web and mobile apps. It also offers a plugin to check
Cross-browsing issues.

Features:

 Test Automation for AJAX Applications

 Native in build Support for the Telerik UI Controls
 Testing HTML Popups and Browser Dialogs
 Support for JavaScript Invocation and Logging
 Allows Continuous Integration use the Build Server

Download Link: http://www.telerik.com/download/teststudio

FAQ
❓ what is Software Testing?
Software Testing is a method to check whether the actual software product
matches expected requirements and ensures that it is Defect free.

⚡ Which factors should you consider while selecting a

Software Testing Tool?
You should consider the following factors before selecting a tool.

 Compatibility across platforms.

 Easy maintenance.
 Quality of Customer support.
 Ease of creating test scripts.
 License cost, if applicable.
 In the case of an outsourcing project, you need to factor customer/Client
Preference of the software testing tool.
 The cost involved in training employees on a tool.
 Hardware/Software requirements of the software testing tool.
 Support and Update policy of the tool.
 Reviews of the company.
Agile Methodology: What is Agile
Software Development Model?
What is Agile Methodology?
AGILE methodology is a practice that promotes continuous iteration of
development and testing throughout the software development lifecycle of the
project. In the Agile model, both development and testing activities are
concurrent, unlike the Waterfall model.

What is Agile Software Development?

The Agile software development methodology is one of the simplest and
effective processes to turn a vision for a business need into software
solutions. Agile is a term used to describe software development approaches
that employ continual planning, learning, improvement, team collaboration,
evolutionary development, and early delivery. It encourages flexible
responses to change.

The agile software development emphasizes on four core values.

1. Individual and team interactions over processes and tools

2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

Agile Model Vs Waterfall Model

Agile and Waterfall model are two different methods for software development
process. Though they are different in their approach, both methods are useful
at times, depending on the requirement and the type of the project.

Agile Model Waterfall Model

 Agile method proposes incremental and iterative  Development of the software flows
 approach to software design sequentially from start point to end point.

 The agile process is broken into individual models that  The design process is not broken into an
designers work on individual models

 The customer has early and frequent opportunities to  The customer can only see the product at th
look at the product and make decision and changes to end of the project
the project

 Agile model is considered unstructured compared to the  Waterfall model are more secure because
waterfall model they are so plan oriented

 Small projects can be implemented very quickly. For  All sorts of project can be estimated and
large projects, it is difficult to estimate the development completed.
time.

 Error can be fixed in the middle of the project.  Only at the end, the whole product is tested.
If the requirement error is found or any
changes have to be made, the project has to
start from the beginning

 Development process is iterative, and the project is  The development process is phased, and the
executed in short (2-4) weeks iterations. Planning is very phase is much bigger than iteration. Every
less. phase ends with the detailed description of
the next phase.

 Documentation attends less priority than software  Documentation is a top priority and can even
development use for training staff and upgrade the
software with another team

 Every iteration has its own testing phase. It allows  Only after the development phase, the testin
implementing regression testing every time new phase is executed because separate parts ar
functions or logic are released. not fully functional.

 In agile testing when an iteration end, shippable  All features developed are delivered at once
 features of the product is delivered to the customer. after the long implementation phase.
New features are usable right after shipment. It is useful
when you have good contact with customers.

 Testers and developers work together  Testers work separately from developers

 At the end of every sprint, user acceptance is performed  User acceptance is performed at the end of
the project.

 It requires close communication with developers and  Developer does not involve in requirement
together analyze requirements and planning and planning process. Usually, time delays
between tests and coding

Agile Process
Check below Agile process model to deliver successful systems quickly.

There are various Agile methods present in agile testing, and those are listed
below:
Scrum
SCRUM is an agile development method which concentrates specifically on
how to manage tasks within a team-based development environment.
Basically, Scrum is derived from activity that occurs during a rugby match.
Scrum believes in empowering the development team and advocates working
in small teams (say- 7 to 9 members). It consists of three roles, and their
responsibilities are explained as follows:

 Scrum Master
o Master is responsible for setting up the team, sprint meeting and
removes obstacles to progress
 Product owner
o The Product Owner creates product backlog, prioritizes the
backlog and is responsible for the delivery of the functionality at
each iteration
 Scrum Team
o Team manages its own work and organizes the work to complete
the sprint or cycle

Product Backlog
This is a repository where requirements are tracked with details on the no of
requirements(user stories) to be completed for each release. It should be
maintained and prioritized by Product Owner, and it should be distributed to
the scrum team. Team can also request for a new requirement addition or
modification or deletion

Scrum Practices
Practices are described in detailed:

Process flow of Scrum Methodologies:

Process flow of scrum testing is as follows:

 Each iteration of a scrum is known as Sprint

 Product backlog is a list where all details are entered to get the end-
product
 During each Sprint, top user stories of Product backlog are selected and
turned into Sprint backlog
 Team works on the defined sprint backlog
 Team checks for the daily work
 At the end of the sprint, team delivers product functionality
eXtreme Programming (XP)
Extreme Programming technique is very helpful when there is constantly
changing demands or requirements from the customers or when they are not
sure about the functionality of the system. It advocates frequent "releases" of
the product in short development cycles, which inherently improves the
productivity of the system and also introduces a checkpoint where any
customer requirements can be easily implemented. The XP develops software
keeping customer in the target.

Business requirements are gathered in terms of stories. All those stories are
stored in a place called the parking lot.

In this type of methodology, releases are based on the shorter cycles called
Iterations with span of 14 days time period. Each iteration includes phases like
coding, unit testing and system testing where at each phase some minor or
major functionality will be built in the application.
Phases of eXtreme programming:
There are 6 phases available in Agile XP method, and those are explained as
follows:

Planning
 Identification of stakeholders and sponsors
 Infrastructure Requirements
 Security related information and gathering
 Service Level Agreements and its conditions

Analysis
 Capturing of Stories in Parking lot
 Prioritize stories in Parking lot
 Scrubbing of stories for estimation
 Define Iteration SPAN(Time)
 Resource planning for both Development and QA teams

Design
 Break down of tasks
 Test Scenario preparation for each task
 Regression Automation Framework

Execution
 Coding
 Unit Testing
 Execution of Manual test scenarios
 Defect Report generation
 Conversion of Manual to Automation regression test cases
 Mid Iteration review
 End of Iteration review

Wrapping
 Small Releases
 Regression Testing
  Demos and reviews
 Develop new stories based on the need
 Process Improvements based on end of iteration review comments

Closure
 Pilot Launch
 Training
 Production Launch
 SLA Guarantee assurance
 Review SOA strategy
 Production Support

There are two storyboards available to track the work on a daily basis, and
those are listed below for reference.

 Story Cardboard
o This is a traditional way of collecting all the stories in a board in
the form of stick notes to track daily XP activities. As this manual
activity involves more effort and time, it is better to switch to an
online form.
 Online Storyboard
o Online tool Storyboard can be used to store the stories. Several
teams can use it for different purposes.

Crystal Methodologies
Crystal Methodology is based on three concepts

1. Chartering: Various activities involved in this phase are creating a

development team, performing a preliminary feasibility analysis,
developing an initial plan and fine-tuning the development methodology
2. Cyclic delivery: The main development phase consists of two or more
delivery cycles, during which the
1. Team updates and refines the release plan
2. Implements a subset of the requirements through one or more
program test integrate iterations
3. Integrated product is delivered to real users
4. Review of the project plan and adopted development
methodology
 3. Wrap Up: The activities performed in this phase are deployment into
the user environment, post- deployment reviews and reflections are
performed.

Dynamic Software Development Method (DSDM)

DSDM is a Rapid Application Development (RAD) approach to software
development and provides an agile project delivery framework. The important
aspect of DSDM is that the users are required to be involved actively, and the
teams are given the power to make decisions. Frequent delivery of product
becomes the active focus with DSDM. The techniques used in DSDM are

1. Time Boxing
2. MoSCoW Rules
3. Prototyping

The DSDM project consists of 7 phases

1. Pre-project
2. Feasibility Study
3. Business Study
4. Functional Model Iteration
5. Design and build Iteration
6. Implementation
7. Post-project

Feature Driven Development (FDD)

This method is focused around "designing & building" features. Unlike other
agile methods, FDD describes very specific and short phases of work that has
to be accomplished separately per feature. It includes domain walkthrough,
design inspection, promote to build, code inspection and design. FDD
develops product keeping following things in the target

1. Domain object Modeling

2. Development by feature
3. Component/ Class Ownership
4. Feature Teams
5. Inspections
6. Configuration Management
7. Regular Builds
 8. Visibility of progress and results

Lean Software Development

Lean software development method is based on the principle "Just in time
production". It aims at increasing speed of software development and
decreasing cost. Lean development can be summarized in seven steps.

1. Eliminating Waste
2. Amplifying learning
3. Defer commitment (deciding as late as possible)
4. Early delivery
5. Empowering the team
6. Building Integrity
7. Optimize the whole

Kanban
Kanban originally emerged from Japanese word that means, a card containing
all the information needed to be done on the product at each stage along its
path to completion. This framework or method is quite adopted in software
testing method especially in agile testing.

Scrum Vs Kanban
Scrum Kanban

 In scrum technique, test must be broken down so that  No particular item size is prescribed
they can be completed within one sprint

 Prescribes a prioritized product backlog  Prioritization is optional

 Scrum team commits to a particular amount of work for  Commitment is optional

the iteration

 Burndown chart is prescribed  No particular item size is prescribed

 Between each sprint, a scrum board is reset  A Kanban board is persistent. It limits the
 number of items in workflow state

 It cannot add items to ongoing iteration  It can add items whenever capacity is availab

 WIP limited indirectly  WIP limited directly

 Timeboxed iterations prescribed  Timeboxed iterations optional

Agile metrics:
Metrics that can be collected for effective usage of Agile is:

 Drag Factor
o Effort in hours which do not contribute to sprint goal
o Drag factor can be improved by reducing number of shared
resources, reducing the amount of non-contributing work
o New estimates can be increased by percentage of drag factor
-New estimate = (Old estimate+drag factor)
 Velocity
o Amount of backlog(user stories) converted to shippable
functionality of sprint
 No of Unit Tests added
 Time interval taken to complete daily build
 Bugs detected in an iteration or in previous iterations
 Production defect leakage

What is Agile Testing? Methodology,

Process & Life Cycle
What is Agile Testing?
AGILE TESTING is a testing practice that follows the rules and principles of
agile software development. Unlike the Waterfall method, Agile Testing can
begin at the start of the project with continuous integration between
development and testing. Agile Testing methodology is not sequential (in the
sense it's executed only after coding phase) but continuous.
Agile Test Plan
Agile test plan includes types of testing done in that iteration like test data
requirements, infrastructure, test environments, and test results. Unlike the
waterfall model, in an agile model, a test plan is written and updated for every
release. Typical test plans in agile includes

Agile Testing Strategies

Agile testing life cycle spans through four stages

(a) Iteration 0

During the first stage or iteration 0, you perform initial setup tasks. It includes
identifying people for testing, installing testing tools, scheduling resources
(usability testing lab), etc. The following steps are set to achieve in Iteration 0

a) Establishing a business case for the project

b) Establish the boundary conditions and the project scope

c) Outline the key requirements and use cases that will drive the design trade-
offs

d) Outline one or more candidate architectures

e) Identifying the risk

f) Cost estimation and prepare a preliminary project

(b) Construction Iterations

The second phase of agile testing methodology is Construction Iterations, the

majority of the testing occurs during this phase. This phase is observed as a
set of iterations to build an increment of the solution.  In order to do that,
within each iteration, the team implements a hybrid of practices from XP,
Scrum, Agile modeling, and agile data and so on.

In construction iteration, the agile team follows the prioritized requirement

practice: With each iteration, they take the most essential requirements
remaining from the work item stack and implement them.

Construction iteration is classified into two, confirmatory testing and

investigative testing.  Confirmatory testing concentrates on verifying that
the system fulfills the intent of the stakeholders as described to the team to
date, and is performed by the team.  While the investigative testing detects
the problem that confirmatory team has skipped or ignored.  In Investigative
testing, tester determines the potential problems in the form of defect stories.
Investigative testing deals with common issues like integration testing,
load/stress testing, and security testing. 

Again for, confirmatory testing there are two aspects developer

testing and agile acceptance testing. Both of them are automated to
enable continuous regression testing throughout the lifecycle.  Confirmatory
testing is the agile equivalent of testing to the specification. 

Agile acceptance testing is a combination of traditional functional testing and

traditional acceptance testing as the development team, and stakeholders are
doing it together.  While developer testing is a mix of traditional unit testing
and traditional service integration testing.  Developer testing verifies both the
application code and the database schema.

(c) Release End Game Or Transition Phase

The goal of “Release, End Game” is to deploy your system successfully into
production.  The activities include in this phase are training of end users,
support people and operational people.  Also, it includes marketing of the
product release, back-up & restoration, finalization of system and user
documentation.

The final agile methodology testing stage includes full system testing and
acceptance testing. In accordance to finish your final testing stage without any
obstacles, you should have to test the product more rigorously while it is in
construction iterations. During the end game, testers will be working on its
defect stories.

(d) Production

After the release stage, the product will move to the production stage.

The Agile Testing Quadrants

The agile testing quadrants separate the whole process in four Quadrants and
help to understand how agile testing is performed.

a) Agile Quadrant I – The internal code quality is the main focus in this
quadrant, and it consists of test cases which are technology driven and are
implemented to support the team, it includes

1. Unit Tests

2.Component Tests
b) Agile Quadrant II – It contains test cases that are business driven and
are implemented to support the team.  This Quadrant focuses on the
requirements. The kind of test performed in this phase is

1. Testing of examples of possible scenarios and workflows

2. Testing of User experience such as prototypes

3. Pair testing

c) Agile Quadrant III – This quadrant provides feedback to quadrants one

and two.  The test cases can be used as the basis to perform automation
testing.  In this quadrant, many rounds of iteration reviews are carried out
which builds confidence in the product.  The kind of testing done in this
quadrant is

1. Usability Testing

2. Exploratory Testing

3. Pair testing with customers

4. Collaborative testing

5. User acceptance testing

d) Agile Quadrant IV – This quadrant concentrates on the non-functional

requirements such as performance, security, stability, etc.  With the help of
this quadrant, the application is made to deliver the non-functional qualities
and expected value.

1. Non-functional tests such as stress and performance testing

2. Security testing with respect to authentication and hacking

3. Infrastructure testing

4. Data migration testing

5. Scalability testing

6. Load testing
QA challenges with agile software development
a) Chances of error are more in agile, as documentation is given less priority,
eventually puts more pressure on QA team

b) New features are introduced quickly, which reduces the available time for
test teams to identify whether the latest features are according to the
requirement and does it truly address the business suits

c) Testers are often required to play a semi-developer roled

d) Test execution cycles are highly compressed

e) Very less time to prepare test plan

f) For regression testing, they will have minimal timing

g) Change in their role from being a gate-keeper of quality to being a partner

in Quality

h) Requirement changes and updates are inherent in an agile method,

becoming the biggest challenge for QA

Risk of Automation in Agile Process

 Automated UI provides a high level of confidence, but they are slow to
execute, fragile to maintain and expensive to build. Automation may not
significantly improve test productivity unless the testers know how to
test
 Unreliable tests are a major concern in automated testing. Fixing failing
tests and resolving issues related to brittle tests should be a top priority
in order to avoid false positives
 If the automated test are initiated manually rather than through CI
(Continuous Integration) then there is a risk that they are not regularly
running and therefore may cause failing of tests
 Automated tests are not a replacement for an exploratory manual
testing. To obtain the expected quality of the product, a mixture of
testing types and levels is required
 Many commercially available automation tools provide simple features
like automating the capture and replay of manual test cases. Such tool
encourages testing through the UI and leads to an inherently brittle and
 difficult to maintain tests. Also, storing test cases outside the version
control system creates unnecessary complexity
 In order to save time, much times the automation test plan is poorly
planned or unplanned which results in the test fail
 A test set up and tear down procedures are usually missed out during
test automation, while Performing manual testing, a test set up and tear
down procedures sounds seamless
 Productivity metrics such as a number of test cases created or executed
per day can be terribly misleading, and could lead to making a large
investment in running useless tests
 Members of the agile automation team must be effective consultants:
approachable, cooperative, and resourceful, or this system will quickly
fail
 Automation may propose and deliver testing solutions that require too
much ongoing maintenance relative to the value provided
 Automated testing may lack the expertise to conceive and deliver
effective solutions
 Automated testing may be so successful that they run out of important
problems to solve, and thus turn to unimportant problems.

Conclusion
Agile methodology in software testing involves testing as early as possible in
the software development lifecycle. It demands high customer involvement
and testing code as soon as it becomes available. The code should be stable
enough to take it to system testing. Extensive regression testing can be done
to make sure that the bugs are fixed and tested. Mainly, Communication
between the teams makes agile model testing success!!!

Scrum Testing Methodology Tutorial:

What is, Process, Artifacts, Sprint
Scrum in Software Testing
Scrum in Software Testing is a methodology for building complex software
applications. It provides easy solutions for executing complicated tasks.
Scrum helps the development team to focus on all aspects of the software
product development like quality, performance, usability and so on. It provides
with transparency, inspection and adaptation during the software development
to avoid complexity.

Scrum Testing
Scrum Testing is a testing done in scrum methodology to verify the software
application requirements are met. It involves checking non-functional
parameters like security, usability, performance etc. There is no active role of
tester in the process so it is usually performed by developers with Unit Test.
Sometimes dedicated test teams are needed depending on nature &
complexity of project.

Key Features of Scrum Methodology

Following are Key Features of Scrum-

 Scrum has a short fixed schedule of release cycles with adjustable

scope known as sprints to address rapidly changing development
needs. Each release could have multiple sprints. Each Scrum Project
could have multiple Release Cycles.
 A repeating sequence of meetings, events, and milestones
 A practice of testing and implementing new requirements, known
as stories, to make sure some work is released ready after each sprint

Scrum is based on the following 3 Pillars-

Let's look at the one by one

1. Roles in Scrum
There are three chief roles in Scrum Testing – Product Owner, Scrum Master
and The Development Team. Let's study them in detail

Product Owner Scrum Master The Team

 He/She manages the team
 He/She defines features of the  The team is usually
and look after the team's
product. about 5-9 members
productivity

 He/She maintains the block  It includes developers,

 Product Owner decides the release
list and removes barriers in designer and sometimes
date and corresponding features
the development testers, etc.

 They prioritize the features  The team organizes and

 He/She coordinates with all
according to the market value and schedule their work on
roles and functions
profitability of the product their own

 Has right to do
everything within the
 He/She is responsible for the  He/She shields team from
boundaries of the
profitability of the product external interferences
project to meet the
sprint goal
 Product Owner Scrum Master The Team
 Invites to the daily scrum,
 He/She can accept or reject work  Actively participate in
sprint review and planning
item result daily ceremonies
meetings

2. Scrum Artifacts

A scrum process includes

 User stories: They are a short explanation of functionalities of the

system under test. Example for Insurance Provider is – "Premium can
be paid using the online system."
 Product Backlog: It is a collection of user stories captured for a scrum
product. The product owner prepares and maintains the product
backlog. It is prioritized by the product owner, and anyone can add to it
with approval from the product owner.
 Release Backlog: A release is a time frame in which the number of
iterations is completed. The product owner co-ordinates with the
scrum master to decide which stories should be targeted for a release.
Stories in the release backlog are targeted to be completed in a release.
 Sprints: It is a set period of time to complete the user stories, decided
by the product owner and developer team, usually 2-4 weeks of time.
 Sprint Backlog: It's a set of user stories to be completed in a sprint.
During sprint backlog, work is never assigned, and the team signs up for
work on their own. It is owned and managed by the team while the
estimated work remaining is updated daily. It is the list of task that has
to be performed in Sprint
  Block List: It is a list of blocks and unmade decisions owned by scrum
master and updated daily
 Burndown chart: Burn-down chart represents overall progress of the
work in progress and work completed throughout the process. It
represents in a graph format the stories and features not completed

3. Ceremonies (Processes) in Scrum

 Sprint Planning: A sprint begins with the team importing stories from
the release backlog into the sprint backlog; it is hosted by scrum master.
The Testers estimate effort to test the various stories in the Sprint
Backlog.
 Daily Scrum: It is hosted by scrum master, it last about 15 minutes.
During Daily Scrum, the members will discuss the work completed the
previous day, the planned work for the next day and issues faced during
a sprint. During daily stand-up meeting team progress is tracked.
 Sprint Review/ Retrospective: It is also hosted by scrum master, it last
about 2-4 hours and discuss what the team has accomplished in the last
sprint and what lessons were learned.

Role of Tester in Scrum

There is no active role of Tester in the Scrum Process. Usually, testing is

carried out by a developer with Unit Test. While product owner is also
frequently involved in the testing process during each sprint. Some Scrum
projects do have dedicated test teams depending on the nature &
complexity of the project.

The next question is, what tester do in a scrum? Following note will answer

Testing Activities in Scrum

Testers do following activities during the various stages of Scrum-

Sprint Planning
  In sprint planning, a tester should pick a user-story from the product
backlog that should be tested.
 As a tester, he/she should decide how many hours (Effort Estimation) it
should take to finish testing for each of selected user stories.
 As a tester, he/she must know what sprint goals are.
 As a tester, contribute to the prioritizing process

Sprint

 Support developers in unit testing

 Test user-story when completed. Test execution is performed in a lab
where both tester and developer work hand in hand. Defect are logged
in Defect Management tool which are tracked on a daily basis. Defects
can be conferred and analyzed during the scrum meeting. Defects are
retested as soon as it is resolved and deployed for testing
 As a tester, he/she attends all daily standup meeting to speak up
 As a tester, he/ she can bring any backlog item that cannot be
completed in the current sprint and put to the next sprint
 Tester is responsible for developing automation scripts. He schedules
automation testing with Continuous Integration (CI) system. Automation
receives the importance due to short delivery timelines. Test Automation
can be accomplished by utilizing various open source or paid tools
available in the market. This proves effective in ensuring that everything
that needs to be tested was covered. Sufficient Test coverage can be
achieved with a close communication with the team.

 Review CI automation results and send Reports to the stakeholders

 Executing non-functional testing for approved user stories
 Coordinate with customer and product owner to define acceptance
criteria for Acceptance Tests
 At the end of the sprint, the tester also does acceptance testing(UAT) in
some case and confirms testing completeness for the current sprint

Sprint Retrospective

 As a tester, he will figure out what went wrong and what went right in
the current sprint

 As a tester, he identifies lesson learned and best practices

Test Reporting
Scrum Test metrics reporting provides transparency and visibility to
stakeholders about the project. The metrics that are reported allow a team to
analyze their progress and plan their future strategy to improve the product.
There are two metrics that are frequently used to report.

Burn down chart: Each day, Scrum Master records the estimated remaining
work for the sprint. This is nothing but the Burn Down Chart. It is updated
daily.

A burndown chart gives a quick overview of the project progress, this chart
contains information like the total amount of work in the project that must be
completed, amount of work completed during each sprint and so on.

Velocity history graph: The velocity history graph predicts the velocity of the
team reached in each sprint. It is a bar graph and represents how teams
output has changed over time.

The additional metrics that may be useful are schedule burn, budget burn,
theme percent complete, stories completed - stories remaining and so on.

Do you have any tips or experiences to share for Scrum Testing? Do leave a
comment below-

Automation Testing Framework for

Agile/Scrum Methodology
Agile Automation Testing
Agile Automation Testing in software development is an approach of using
test automation in agile methodologies. The purpose of agile automation
testing is to make the software development process more effective and
efficient while maintaining the quality and time as well as resource
consumption. Thus, the implementation of such a process requires a lot of
coordination and collaboration between teams.

In the last few years, ever since the agile methodology came on board with its
founders shouting and willing to do away with the mundane and laborious
realities of the traditional waterfall model, the impact of the same can be also
felt when it comes to Automation Testing.

Automation in Waterfall Vs Automation in Agile

 In the realm of the traditional process of software testing life cycle,
Automation Testing is normally feasible when the application is stable,
steady and the requirement is involving with a real considerable amount
of time and in most cases involving a set of very skillful automation expert
resources as well as a considerable amount of set-up costs. The basic
purpose of Automation Testing is to reduce costs over a long time and to
ensure no new defects have been introduced as a result of existing test cases

Automation testing by the very nature of the technology is not

exploratory in nature since the main role of Automation Testing is saving time
and reducing costs. Automation Testing is not meant to come up with new and
innovative defects. Automation Testing aims at mostly confirmation of the
already existing.

How to automate in Agile Methodology

Now by its very definition agile methodology talks about doing away with
laborious and tedious documentation so that new and innovative ideas could
be implemented and people could interact freely with each other so that more
of these innovative and explorative ideas could be implemented.

Thus we could see a contradiction between the basic fundamental

philosophies of agile methodologies and Automation Testing.
Fundamental Points for Agile Test Automation
So we need to consider certain fundamental points here when it comes to
evaluating the use of agile methodologies with respect to the Automation
Testing methods and techniques. Thus we need to consider some
fundamental points like time taken for design and coding, validation of the
designed scripts with the existing test data and the adoption of the same for
testing (whether the tests are of functional or regression purposes) So the real
fact of all these events is that in order to perform all these facts, we need to
ensure that a considerable amount of time is required for these tasks and in
an agile environment where an average sprint takes an average 1-2 weeks to
complete and thus it is obviously too difficult to contemplate affording so much
time for automating scripts in such a way.

Another significant factor remains here that the type of changes in

requirements which come into picture when the agile methodology is at play.
The agile methodology by its own very definition is a sort of technique which is
very helpful for responding to quick customer induced change requirements
and which thus lends itself well to frequent changes during the overall
development of the application.

In contrast, automation testing is very useful when it comes to the more stable
and less frequent types of requirements. Thus by definition automation testing
does not lend itself well to various types of frequent changes in requirements
which comes alongside the adoption of any agile methodologies.

Agile Automation Tools

The selection of relevant automation tool is also a potentially very important
factor when it comes to the adoption of automation testing within the scope of
an overall agile methodology. Licensed automation tools, for example, impose
strict security access criterion to different types and levels of users when it
comes accessing various important resources belonging to that particular
testing automation framework.
In contrast agile methodology emphasizes upon mostly open collaboration
and open-ended interaction between team members and thus restrictive
policies which directly affects how the users would have a negative impact on
the overall cohesion within the team and thus may be leading into results
which are neither very helpful nor very conducive to the overall success of the
project

Therefore the primary importance of the process should be to ensure that in

order to obtain the quality delivery of automation test scripts within a stipulated
time as afforded by agile methodology; we need to choose our prospective
test cases which would be automated in a more nuanced way such that these
automated test scripts lend themselves well for future re-use as well as
ensuring that they can be prepared within the proper duration of the allotted
time (as required during the agile methodology process).

After consideration of all the above factors we thus can realize that even while
adopting agile methodologies, we need to bring into picture the types of tests
like for example regression tests (since even during agile testing there is a
considerable amount of testing work which is required to put into the job of
agile methodologies for ensuring better quality of the overall product)  

Now let us look at the most basic situations whereby automation testing can
be used and how we can adopt the same towards the realm of agile testing.
SAFe Methodology Tutorial: What is
Scaled Agile Framework
What is a Scaled Agile Framework (SAFe)?
Scaled Agile Framework (SAFe) is a freely available online knowledge base
that allows you to apply lean-agile practices at the enterprise level. It provides
a simple and lightweight experience for software development. It is a set of
organizations and workflow patterns intended to guide enterprises for scaling
lean and agile practices. It is divided into three segments which are Team,
Program and portfolio

SAFe was first developed in the field and was elaborated in Dean
Leffingwell's books and blog. Version 1.0 is the first official release in 2011.
The latest version is 4.6, was released in October 2018. It provides guidance
to work at enterprise Portfolio, Value Stream, Program, and Team levels.

Why to use Agile Framework

It is simpler and lighter in weight, yet it expands to handle the needs of large
value streams and complex system development. By implementing an Agile
Framework, you will have following benefits,
  Productivity increased by 20 - 50%
 Quality increased more than 50%
 Time to Market is faster than 30 -75%
 Increased employee engagement and job satisfaction.

The detailed framework diagram is available on the website. It shows all of the

key roles, Activities, deliverables, and flows. It also serves as a navigational
aid to the rest of the site.

The below image explains how agile process works. Epics are a large body of
work, which is further broken down into a number of smaller stories or sub-
epics. These sub-epics are allocated to the team as a story. Each team then
work on these stories or software features accordingly.
When to Use Scaled Agile Framework

 When a team is interested to implement an agile approach consistently

across larger, multi-team programs and portfolios.
 When multiple teams are running their own way of Agile implementation
but regularly facing obstacles, delays, and failures.
 When teams want to work independently.
 When you want to scale Agile across the organization but not sure what
new roles may be needed or what existing roles (i.e., management)
need to change and how.
 When you have attempted to scale the Agile across your organization
but struggling in alignment to achieve uniform or consistent strategy
across business departments from portfolio to program and team levels.
 When an organization needs to improve its product development lead
time and want to know how other companies have succeeded in scaling
Agile with SAFe.

How different than other Agile practices

Let's see how Scaled Agile framework is different from other agile practices,

 It's publicly available and free to use.

 Available in a highly approachable and usable form.
 It's lightweight, practically proven results and specific to level.
  It constantly/regularly modifies/maintains most commonly used agile
practices.
 Offers useful extensions to common agile practices.
 Grounds agile practices to an enterprise context.
 Offers complete picture of software development.
 Visibility or transparency is more on all the levels.
 Continues or regular feedback on quality and improvement.

Foundations of Scaled Agile Framework

Scaled Agile Framework(SAFe): It stands on the foundations of its

1. Lean-Agile Principles
2. Core Values,
3. Lean-Agile Leadership
4. Lean-Agile Mind-set,
5. Communities of Practice(Group of people who are constantly working
on SAFe practices)
6. Implementing 1-2-3

1. SAFe Lean-Agile Principles

These basic principles and values for SAFe must be understood, exhibited
and continued in order to get the desired results.
  Take an economic view
 Apply systems thinking
 Assume variability; preserve options
 Build incrementally with fast, integrated learning cycles
 Base milestones on an objective evaluation of working systems
 Visualize and limit WIP, reduce batch sizes and manage queue lengths
 Apply cadence, synchronize with cross-domain planning
 Unlock the intrinsic motivation of knowledge workers
 Decentralize decision-making

2. SAFe Agile Core Values

The SAFe agile is based on these four values.

Alignment:

 SAFe supports alignment.

 Alignment starts at,
o Strategic Themes in Portfolio Backlog and
o Moves down to Vision and Roadmap of Program Backlogs and
then
o Moves to the Team Backlogs.

Built-in Quality:

 It ensures that every incremental delivery reflects the quality standards.

 Quality is not "added later" is built in.
 Built-in quality is a prerequisite of Lean and its mandatory

Transparency:

 Transparency is the enabler for trust.

 SAFe helps the enterprise to achieve transparency at all levels-
Executives, Portfolio Managers, and other stakeholders.
 Everyone can see into the portfolio backlog/Kanban, program
backlogs/Kanban, and Team Backlog/Kanban.
 Each level has a clear understanding of the PI goals.
 Train Programs have visibility into the team's backlogs, as well other
program backlogs
  Teams and programs have visibility into business and architecture
Epics. They can see what might be headed their way.

Program Execution:

 SAFe places great focus on working systems and resultant business

outcomes.
 SAFe is not useful if teams can't execute and continuously deliver value.

3. Lean Agile Leaders:

The Lean-Agile Leaders are lifelong learners and teachers. It helps teams to
build better systems through understanding and exhibiting the Lean-Agile
SAFe Principles.

As an enabler for the teams, the ultimate responsibility is adoption, success

and ongoing improvement of Lean-Agile developments. For the change and
continuous improvement, leaders must be trained.

Leaders need to adopt a new style of leadership. One that truly empowers
and engages individuals and teams to reach their highest potential.

Principles of these Lean-Agile Leaders

 Lead the Change

 Know the Way; Emphasize Lifelong Learning
 Develop People
 Inspire and Align with Mission; Minimize Constraints
 Decentralize Decision-Making
 Unlock the Intrinsic Motivation of Knowledge Workers

4. Lean Agile Mind-Set:

Lean-Agile mindset is represented in two things:

1. The SAFe House of Lean

2. Agile Manifesto

The SAFe House of Lean:

SAFe is derived from Lean manufacturing principles and practices. Based on
these factors SAFe presents the "SAFe House of Lean". It is inspired by
"house" of lean Toyota.

The Goal of lean is unbeatable: To deliver maximum customer value in the

shortest lead time with the highest possible quality to customer

Below figure explains the Goal, Pillars, and Foundation of "SAFe House of
Lean."
Agile Manifesto
We are uncovering better ways of developing software by doing it and helping
others do it. Through this work we have come to value:

That' why , while there is a value in the items on the right, we value the items
on the left more.

Agile Manifesto

1. The highest priority is to satisfy the customer through continuous and

early delivery of valuable software.
2. Embrace the changing requirements, even late in development. Agile
processes harness change for the customer's benefit.
3. Deliver working software frequently, from a couple of weeks to a couple
of months, with a preference to the shorter timescale.
4. Developers and business people must work together daily throughout
the project.
5. Build projects around motivated individuals. Give them support and the
environment they need, and trust them to get the job done.
6. The most efficient method for communication with a development team
is a face-to-face conversation.
7. Working software is the primary measure of progress.
8. Agile processes promote sustainable development. The sponsors,
developers, and users should be able to maintain a constant pace
indefinitely.
9. Continuous attention to technical excellence and good design enhances
agility.
10. Simplicity--the art of maximizing the amount of work not done--is
essential.
 11. The best architectures, requirements, and designs emerge from
self-organizing teams.
12. At regular intervals, the team reflects on how to become more
effective, then tunes and adjusts its behavior accordingly.

Different Levels in SAFE

There are two different types of SAFe implementation:

1. SAFe 4.0 implementation

2. SAFe 3.0 implementation

 In SAFe 4.0 implementation we have 4-Levels: Portfolio, Value

Stream, Program, and Team.
 In SAFe 3.0 implementation we have 3-Levels: Portfolio, Program,
and Team
 3-Level SAFe is for smaller implementations with 100 or fewer people.
Programs that do not require significant collaboration.
 4-Level SAFe is for solutions that typically require many hundreds of
practitioners to develop deploy and maintain software.

Team Level
Roles/Teams   Events   Artifacts

* Agile Team   * Sprint Planning   * Team Backlog

* Product Owner   * Backlog Grooming   * Non-Functional Requirements

* Scrum Master   * Daily Stand-Up   * Team PI Objectives

    * Execution   * Iterations

    * Sprint Demo   * Stories(Working Software)

    * Sprint Retrospective   * Sprint Goals

    * IP Sprints   * Built-In Quality

        * Spikes

        * Team Kanban

 All SAFe teams are part of one or other Agile Release Train (ART).
 SAFe teams are empowered, self-organizing, self-managing, cross-
functional teams
 Each team is equally responsible for defining, building and testing
stories from their Team Backlog in a fixed-length Iterations
 Teams plan and execute two-week time-boxed iterations in accordance
with agreed-to Iteration Goals.
 Teams will use ScrumXP/Team Kanban routine to deliver high-quality
systems to produce a System Demo on every two weeks.
 All different teams in the ART (Agile Release Trains) will create an
integrated and tested system. Stakeholders will evaluate and respond
with fast feedback
 They apply Built-in Quality practices.
 Each ScrumXP team will have 5-9 team members, which includes all
the roles necessary to build a quality incremental value in each Iteration.
 ScrumXP roles includes:
o Team(Dev+QA)
o Scrum Master
o Product Owner. Etc..
  SAFe divides the development timeline into a set of iterations within a PI
(Program Increment).
 PI duration is between 8 -12 weeks.
 The team will use stories to deliver the value. The Product Owner will
have content authority over their creation and acceptance of the stories.
 Stories contain Customer's requirements.
 Team Backlog includes user and enabler stories, which are identified
during PI planning. When the Product Management presents the
Roadmap, Vision, and Program Backlog.
 Identifying, elaborating, prioritizing, scheduling, implementing, testing,
and accepting the stories are the primary requirements of management
work in team level.
 Each iteration provides:
o A valuable increment of new functionality
o Accomplish via constantly repeating pattern
o Plan the iteration
o Commit to some functionality
o Execute the iteration by building and testing Stories
o Demo the new functionality
o Retrospective
o Repeat for the next iteration
 Teams also support the System Demo at the end of each Iteration.
which is the critical integration point for the ART.
 Larger Value Streams will have multiple ARTs.
 The Innovation and Planning (IP) Iterations leverage the teams with an
opportunity for innovation and exploration.

Program Level
Roles/Teams   Events   Artifacts

* DevOps   * PI(Program Increment)   * Vision

Planning

* System Team   * System Demos   * Roadmap

* Release Management   * Inspect and Adopt   * Metrics

 Workshop

* Product Management   * Architectural Runway   * Milestones

* UEX Architect   * Release Any Time   * Releases

* Release Train   * Agile Release Train   * Program Epics

Engineer(RTE)

* System   * Release   * Program Kanban

Architect/Engineer

* Business Owners       * Program Backlog

* Lean-Agile Leaders       * Non-Functional

Requirements

* Communities of Practice       * Weighted Shortest Job

First (WSJF)

* Shared Services       * Program PI Objectives

* Customer       * Feature

        * Enabler

        * Solution

        * Value Stream Coordination

 In Program level, Value of SAFe is delivered by long-lived Agile Release
Trains (ART). Iteration is for team and train is for the program.
 Agile Release Trains (ART) is the primary vehicle for value delivery at
the program level. It delivers a value stream to the organization.
 The Program Increments (PIs) duration is of 8 to 12 weeks.
 ART is of 5 - 12 Agile Teams (~50 – 125+ people) which includes all the
roles and infrastructure needed to deliver fully tested, working, system-
level software.
 Each PI is a multiple-Iteration time box. During which a significant,
valuable increment of the system is developed and delivered.
 In each PI a "demo" and "Inspect and adapt" sessions will happen, and
Planning begins for the next PSI.
 At the Program level, SAFe emphasis on the principle of alignment. This
is because multiple agile team efforts are integrated to create customer
value.
 SAFe artifact hierarchy is Epics->features->user stories.
 At a Program level, Product Manager/Program Manager has content
authority. He defines and prioritizes the program backlog.
 Program backlog is a prioritized list of features.
 At the program level, features can be originated, or they can derive from
epics defined at the portfolio level.
 Features decompose to user stories and flow into team-level backlogs.
 Product Manager or the Release Train Engineer role could be handled
by the Program Manager/Senior Project Manager
 System Architect role at the program level is to collaborate day to day
work with the teams. It ensures that non-functional requirements are
met. Also, they work with the enterprise architect at the portfolio level to
make sure there is sufficient architectural runway to support upcoming
user and business needs.
 Interface design, user experience guidelines and design elements for
the teams are provided by UX Designers.
 Chief-Scrum Master role is played by 'Release Train Engineer'.
 Various team (from marketing, development, quality, operations, and
deployment) forms 'Release Management Team'. They will approve
routine releases of quality solutions to customers.
 Deployment of software into customer environments and successful
delivery is taken care of by DevOps team.
Portfolio Level
Roles/Teams   Events   Artifacts

* Enterprise Architect   * Strategic Investment   * Strategic Themes

Planning

* Program Portfolio   * Kanban Portfolio(Epic)   * Enterprise

Mgmt Planning

* Epic Owners       * Portfolio Backlog

        * Portfolio Kanban

        * Non-Functional
Requirements

        * Epic and Enabler

        * Value Stream

        * Budgets(CapEx and
OpEx)

 Highest level of interest/ concern /involvement/ in SAFe is SAFe

Portfolio
 The portfolio provides the basic blocks for organizing the Lean-Agile
Enterprise flow of value via one or more Value Streams.
 The portfolio helps to develop systems and solutions which are
described in strategic themes (links a SAFe portfolio to the changing
business strategy of an enterprise).
 To meet strategic objectives, portfolio level encapsulates these
elements. It provides basic budgeting and other governance
 mechanisms. This way it assures that the investment in the value
streams provides the returns necessary for the enterprise.
 A portfolio is connected to business bi-directionally:
o In order to guide the Portfolio to the larger changing business
objectives, it provides strategic themes.
o Another direction indicates the constant flow of portfolio values.
 Program Portfolio Management acts as stakeholders, and they are
accountable to deliver the business results.
 SAFe Portfolio Level contains people, processes and necessary build
systems and solutions that an enterprise needs to meet its strategic
objectives.
 Value Streams are the primary objectives in Portfolio, with which
funding for the people and other resources required to build the
Solutions.
 Important key concepts used here are:
o Connection to the Enterprise,
o Program Portfolio Management,
o Managing the Flow of Portfolio Epics.

Value Stream Level

Roles/Teams   Events   Artifacts

* DevOps   * Pre and Post PI(Program   * Vision

Increment) Planning

* System Team   * Solution Demos   * Roadmap

* Release Management   * Inspect and Adopt   * Metrics

Workshop

* Solution Management   * Agile Release Train   * Milestones

* UEX Architect       * Releases

* Value Stream       *Value Stream Epics

Engineer(RTE)

* Solution       * Value Stream Kanban

Architect/Engineer

* Shared Services       * Value Stream Backlog

* Customer       * Non-Functional
Requirements

* Supplier       * Weighted Shortest Job

First (WSJF)

        * Value Stream PI
Objectives

        * Capability

        * Enabler

        * Solution Context

        * Value Stream
Coordination

        * Economic Framework

        * Solution Intent

        * MBSE
        * Set Based

        * Agile Architecture

 The Value Stream Level is optional in SAFe.

 Value Stream Level is new in SAFe 4.0.
 The Value Stream Level is intended/designed for Enterprises
/builders/organization who are:

1. Large in size
2. Independent
3. Have complex solutions
4. Their solutions typically require multiple ARTs
5. They have Suppliers contribution.
6. They face the largest systems challenges
7. For cyber-physical systems
8. For software, hardware, electrical and electronics, optics, mechanics,
fluidics and more.

 Building this kind of systems often takes hundreds, even thousands of

practitioners, external and internal suppliers.
 If the systems are mission crucial. Failure of the Solution, or even a
subsystem, has unacceptable economic and social consequences.
 If the Enterprises can be built with a few hundred practitioners, it may
not need the constructs of this level. In that case, they can use from the
'collapsed view' which is 3-level SAFe.
 Building value stream solutions in a Lean-Agile pattern requires
additional artifacts, coordination, and constructs. So this level contains
an Economic Framework to provide financial boundaries for Value
Stream
 It supports cadence and synchronization for multiple ARTs and
Suppliers. It includes Pre-and Post-PI Planning meetings and Solution
Demo.
 It gives additional roles which are: Value Stream Engineer, Solution
Architect/Engineering, and Solution Management.

Summary:
  SAFe is an industry-proven, value-focused method for scaling Agile at
the Enterprise level.
 It answers the questions like "How do we plan?", "How do we budget?",
and "How do we become cross-functional in architecture and DevOps?"
 SAFe helps large organization teams to meet an organization's strategic
goals, not just individual project goals.
 The framework offers the ability to maintain and create a centralized
strategy to deliver value.
 The SAFe model has three/four levels that centralize the strategic
themes of an organization.
 Centralized strategy, combined with the de-centralized agile
development execution.

Web Application Testing: 8 Step Guide to

Website Testing
Before we pen down more details on the type of web testing, lets quickly define Web Testing.

What is Web Testing?

WEB TESTING, or website testing is checking your web application or website for potential
bugs before its made live and is accessible to general public. Web Testing checks for
functionality, usability, security, compatibility, performance of the web application or website.

During this stage issues such as that of web application security, the functioning of the site, its
access to handicapped as well as regular users and its ability to handle traffic is checked.

How to test Web Application

In Software Engineering, the following testing types/technique may be performed depending on
your web testing requirements.

1. Functionality Testing of a Website

Functionality Testing of a Website is a process that includes several testing
parameters like user interface, APIs, database testing, security testing, client
and server testing and basic website functionalities. Functional testing is very
convenient and it allows users to perform both manual and automated testing.
It is performed to test the functionalities of each feature on the website.
Web based Testing Activities includes:

Test all links in your webpages are working correctly and make sure there
are no broken links. Links to be checked will include -

 Outgoing links
 Internal links
 Anchor Links
 MailTo Links

Test Forms are working as expected. This will include-

 Scripting checks on the form are working as expected. For example- if a

user does not fill a mandatory field in a form an error message is shown.
 Check default values are being populated
 Once submitted, the data in the forms is submitted to a live database or
is linked to a working email address
 Forms are optimally formatted for better readability

Test Cookies are working as expected. Cookies are small files used by

websites to primarily remember active user sessions so you do not need to
log in every time you visit a website. Cookie Testing will include

 Testing cookies (sessions) are deleted either when cache is cleared or

when they reach their expiry.
 Delete cookies (sessions) and test that login credentials are asked for
when you next visit the site.

Test HTML and CSS to ensure that search engines can crawl your site easily.
This will include

 Checking for Syntax Errors

 Readable Color Schemas
 Standard Compliance. Ensure standards such W3C, OASIS, IETF, ISO,
ECMA, or WS-I are followed.

Test business workflow- This will include

 Testing your end - to - end workflow/ business scenarios which takes

the user through a series of webpages to complete.
  Test negative scenarios as well, such that when a user executes an
unexpected step, appropriate error message or help is shown in your
web application.

2. Usability testing:
Usability Testing has now become a vital part of any web based project. It can
be carried out by testers like you or a small focus group similar to the
target audience of the web application.

Test the site Navigation:

 Menus, buttons or Links to different pages on your site should be easily

visible and consistent on all webpages

Test the Content:

 Content should be legible with no spelling or grammatical errors.

 Images if present should contain an "alt" text

3.Interface Testing:
Three areas to be tested here are - Application, Web and Database Server

 Application: Test requests are sent correctly to the Database and

output at the client side is displayed correctly. Errors if any must be
caught by the application and must be only shown to the administrator
and not the end user.
 Web Server: Test Web server is handling all application requests
without any service denial.
 Database Server: Make sure queries sent to the database give
expected results.

Test system response when connection between the three

layers (Application, Web and Database) cannot be established and
appropriate message is shown to the end user.

4. Database Testing:
Database is one critical component of your web application and stress must
be laid to test it thoroughly. Testing activities will include-
  Test if any errors are shown while executing queries
 Data Integrity is maintained while creating, updating or deleting data in
database.
 Check response time of queries and fine tune them if necessary.
 Test data retrieved from your database is shown accurately in your web
application

5. Compatibility testing.
Compatibility tests ensures that your web application displays correctly across
different devices. This would include-

Browser Compatibility Test: Same website in different browsers will display

differently. You need to test if your web application is being displayed correctly
across browsers, JavaScript, AJAX and authentication is working fine. You
may also check for Mobile Browser Compatibility.

The rendering of web elements like buttons, text fields etc. changes with
change in Operating System. Make sure your website works fine for various
combination of Operating systems such as Windows, Linux, Mac and
Browsers such as Firefox, Internet Explorer, Safari etc.

6. Performance Testing:
This will ensure your site works under all loads. Software Testing activities will
include but not limited to -

 Website application response times at different connection speeds

 Load test your web application to determine its behavior under normal
and peak loads
 Stress test your web site to determine its break point when pushed to
beyond normal loads at peak time.
 Test if a crash occurs due to peak load, how does the site recover from
such an event
 Make sure optimization techniques like gzip compression, browser and
server side cache enabled to reduce load times
7. Security testing:
Security Testing is vital for e-commerce website that store sensitive customer
information like credit cards. Testing Activities will include-

 Test unauthorized access to secure pages should not be permitted

 Restricted files should not be downloadable without appropriate access
 Check sessions are automatically killed after prolonged user inactivity
 On use of SSL certificates, website should re-direct to encrypted SSL
pages.

8. Crowd Testing:
You will select a large number of people (crowd) to execute tests which
otherwise would have been executed a select group of people in the
company. Crowdsourced testing is an interesting and upcoming concept and
helps unravel many a unnoticed defects.

This concludes the tutorial. It includes almost all testing types applicable to
your web application.

As a Web-tester its important to note that web testing is quite an arduous

process and you are bound to come across many obstacles. One of the major
problems you will face is of course deadline pressure. Everything is always
needed yesterday! The number of times the code will need changing is also
taxing. Make sure you plan your work and know clearly what is expected of
you. Its best define all the tasks involved in your web testing and then create
a work chart for accurate estimates and planning.

Web Application Testing Checklist:

Example Test Cases for Website
While testing the web applications, one should consider the below mentioned
template. The below mentioned checklist is almost applicable for all types of
web applications depending on the business requirements. Usability
Testing
What is Usability Testing?
  Usability testing is nothing but the User-friendliness check.
 In Usability testing, the application flow is tested so that a new user can
understand the application easily.
 Basically, system navigation is checked in Usability testing.

What is the purpose or Goal of Usability testing?

A Usability test establishes the ease of use and effectiveness of a product

using a standard Usability test practices.

Example Usability Test Cases

 Web page content should be correct without any spelling or grammatical

errors
 All fonts should be same as per the requirements.
 All the text should be properly aligned.
 All the error messages should be correct without any spelling or
grammatical errors and the error message should match with the field
label.
 Tool tip text should be there for every field.
 All the fields should be properly aligned.
 Enough space should be provided between field labels, columns, rows,
and error messages.
 All the buttons should be in a standard format and size.
 Home link should be there on every single page.
 Disabled fields should be grayed out.
 Check for broken links and images.
 Confirmation message should be displayed for any kind of update and
delete operation.
 Check the site on different resolutions (640 x 480, 600x800 etc.?)
 Check the end user can run the system without frustration.
 Check the tab should work properly.
 Scroll bar should appear only if required.
 If there is an error message on submit, the information filled by the user
should be there.
 Title should display on each web page
 All fields (Textbox, dropdown, radio button, etc) and buttons should be
accessible by keyboard shortcuts and the user should be able to
perform all operations by using keyboard.
  Check if the dropdown data is not truncated due to the field size. Also,
check whether the data is hardcoded or managed via administrator.

Functional Testing:
What is Functional Testing?

 Testing the features and operational behavior of a product to ensure

they correspond to its specifications.
 Testing that ignores the internal mechanism of a system or component
and focuses solely on the outputs generated in response to selected
inputs and execution conditions.

What is the purpose or Goal of Functional testing?

 The goal of Functional Testing is to verify whether your product meets

the intended functional specifications mentioned in your development
documentation.

Example Functional Test Scenarios:

 Test all the mandatory fields should be validated.

 Test the asterisk sign should display for all the mandatory fields.
 Test the system should not display the error message for optional fields.
 Test that leap years are validated correctly & do not cause
errors/miscalculations.
 Test the numeric fields should not accept the alphabets and proper error
message should display.
 Test for negative numbers if allowed for numeric fields.
 Test division by zero should be handled properly for calculations.
 Test the max length of every field to ensure the data is not truncated.
 Test the pop up message ("This field is limited to 500 characters")
should display if the data reaches the maximum size of the field.
 Test that a confirmation message should display for update and delete
operations.
 Test the amount values should display in currency format.
 Test all input fields for special characters.
 Test the timeout functionality.
 Test the Sorting functionality.
 Test the functionality of the buttons available
  Test the Privacy Policy & FAQ is clearly defined and should be available
for users.
 Test if any functionality fails the user gets redirected to the custom error
page.
 Test all the uploaded documents are opened properly.
 Test the user should be able to download the uploaded files.
 Test the email functionality of the system.
 Test the Java script is properly working in different browsers (IE,
Firefox, Chrome, safari and Opera).
 Test to see what happens if a user deletes cookies while in the site.
 Test to see what happens if a user deletes cookies after visiting a site.
 Test all the data inside combo/list box is arranged in chronological
order.

Compatibility Testing:
What is Compatibility testing?

 Compatibility testing is used to determine if your software is compatible

with other elements of a system with which it should operate, e.g.
Browsers, Operating Systems, or hardware.

What is the purpose or Goal of Compatibility testing?

 The purpose of Compatibility testing is to evaluate how well software

performs in a particular browser, Operating Systems, hardware or
software.

Sample Compatibility Test Scenarios:

 Test the website in different browsers (IE, Firefox, Chrome, Safari and
Opera) and ensure the website is displaying properly.
 Test the HTML version being used is compatible with appropriate
browser versions.
 Test the images display correctly in different browsers.
 Test the fonts are usable in different browsers.
 Test the java script code is usable in different browsers.
 Test the Animated GIF's across different browsers.
Database Testing:
What is Database Testing?

 In Database testing backend records are tested which have been

inserted through the web or desktop applications. The data which is
displaying in the web application should match with the data stored in
the Database.

To perform the Database testing, the tester should be aware of the below
mentioned points:

 The tester should understand the functional requirements, business

logic, application flow and database design thoroughly.
 The tester should figure out the tables, triggers, store procedures, views
and cursors used for the application.
 The tester should understand the logic of the triggers, store procedures,
views and cursors created.
 The tester should figure out the tables which get affected when insert
update and delete (DML) operations are performed through the web or
desktop applications.

With the help of the above mentioned points, the tester can easily write
the test scenarios for Database testing.

Example Test Cases for Database Testing:

 Verify the database name: The database name should match with the
specifications.
 Verify the Tables, columns, column types and defaults: All things should
match with the specifications.
 Verify whether the column allows a null or not.
 Verify the Primary and foreign key of each table.
 Verify the Stored Procedure:
 Test whether the Stored procedure is installed or not.
 Verify the Stored procedure name
 Verify the parameter names, types and number of parameters.
 Test the parameters if they are required or not.
 Test the stored procedure by deleting some parameters
 Test when the output is zero, the zero records should be affected.
  Test the stored procedure by writing simple SQL queries.
 Test whether the stored procedure returns the values
 Test the stored procedure with sample input data.
 Verify the behavior of each flag in the table.
 Verify the data gets properly saved into the database after each page
submission.
 Verify the data if the DML (Update, delete and insert) operations are
performed.
 Check the length of every field: The field length in the back end and
front end must be same.
 Verify the database names of QA, UAT and production. The names
should be unique.
 Verify the encrypted data in the database.
 Verify the database size. Also test the response time of each query
executed.
 Verify the data displayed on the front end and make sure it is same in
the back end.
 Verify the data validity by inserting the invalid data in the database.
 Verify the Triggers.

What is Security Testing?

Security Testing involves the test to identify any flaws and gaps from a
security point of view.

Sample Test Scenarios for Security Testing:

 Verify the web page which contains important data like password, credit
card numbers, secret answers for security question etc should be
submitted via HTTPS (SSL).
 Verify the important information like password, credit card numbers etc
should display in encrypted format.
 Verify password rules are implemented on all authentication pages like
Registration, forgot password, change password.
 Verify if the password is changed the user should not be able to login
with the old password.
 Verify the error messages should not display any important information.
 Verify if the user is logged out from the system or user session was
expired, the user should not be able to navigate the site.
  Verify to access the secured and non-secured web pages directly
without login.
 Verify the “View Source code” option is disabled and should not be
visible to the user.
 Verify the user account gets locked out if the user is entering the wrong
password several times.
 Verify the cookies should not store passwords.
 Verify if, any functionality is not working, the system should not display
any application, server, or database information. Instead, it should
display the custom error page.
 Verify the SQL injection attacks.
 Verify the user roles and their rights. For Example, the requestor should
not be able to access the admin page.
 Verify the important operations are written in log files, and that
information should be traceable.
 Verify the session values are in an encrypted format in the address bar.
 Verify the cookie information is stored in encrypted format.
 Verify the application for Brute Force Attacks

What is Performance Testing?

Performance Testing is conducted to evaluate the compliance of a system or
component with specified performance requirements.

General Test scenarios:

 To determine the performance, stability and scalability of an application

under different load conditions.
 To determine if the current architecture can support the application at
peak user levels.
 To determine which configuration sizing provides the best performance
level.
 To identify application and infrastructure bottlenecks.
 To determine if the new version of the software adversely had an impact
on response time.
 To evaluate product and/or hardware to determine if it can handle
projected load volumes.

How to do Performance testing? By Manual Testing or by Automation

Practically it is not possible to do the Performance Testing manually because
of some drawbacks like:

 More number of resources will be required.

 Simultaneous actions are not possible.
 Proper system monitoring is not available.
 Not easy to perform the repetitive task.

Hence to overcome the above problems we should use Performance Testing

tool. Below is the list of some popular testing tools.

 Apache JMeter
 Load Runner
 Borland Silk Performer.
 Rational Performance Tester
 WAPT
 NEO LOAD

Banking Domain Application Testing:

Sample Test Cases
he BFSI (Banking, Financial services and Insurance) sector is the biggest
consumer of IT services. Banking Applications directly deal with confidential
financial data. It is mandatory that all the activities performed by banking
software run smoothly and without any error. Banking software perform
various functions like transferring and depositing fund, balance inquiry,
transaction history, withdrawal and so on. Testing banking application assures
that these activities are not only executed well but also remain protected from
hackers.

What is Domain in Testing?

Domain in Testing is nothing but the industry for which the software testing
project is created. When we talk about software projects or development, this
term is often referred to. For example, Insurance domain, Banking domain,
Retail Domain, Telecom Domain, etc.
Usually, while developing any specific domain project, domain expert help is
sought out. Domain expert are master of the subject, and he may know the
inside-out of the product or application.

Why Domain Knowledge Matters?

Domain knowledge is quintessential for testing any software product, and it
has its own benefits like

Banking Domain Knowledge - Introduction

Banking domain concepts are huge, and basically it is sub-characterized into
two sectors

1. Traditional banking sector

2. Service based banking sector

Below is the table of the services these two sub-sectors of banking

encompass

Traditional banking sector  Core banking

 Corporate banking
 Retail banking
 Service based banking sector  Core
 Corporate
 Retail
 Loan
 Trade finance
 Private banking
 Consumer finance
 Islamic banking
 Customer delivery channels/Front end delivery

Based on the scope of your project you may need to test one or all of the
above service offerings. Before you begin testing, ensure you have enough
background on the service being tested.

Characteristics of a Banking Application

Before you begin testing, it's important to note the standard features expected
of any banking application. So that, you can gear your test efforts to achieve
these characteristics.

A standard banking application should meet all these characteristics as

mentioned below.

 It should support thousands of concurrent user sessions

 A banking application should integrate with other numerous applications
like trading accounts, Bill pay utility, credit cards, etc.
 It should process fast and secure transactions
 It should include massive storage system.
 To troubleshoot customer issues, it should have high auditing capability
 It should handle complex business workflows
 Need to support users on multiple platforms (Mac, Linux, Unix,
Windows)
 It should support users from multiple locations
 It should support multi-lingual users
 It should support users on various payment systems (VISA, AMEX,
MasterCard)
 It should support multiple service sectors (Loans, Retail banking etc.)
 Foolproof disaster management mechanism
Test Phases in Testing Banking Applications
For testing banking applications, different stages of testing include

 Requirement Analysis: It is done by business analyst; requirements

for a particular banking application are gathered and documented
 Requirement Review: Quality analysts, business analysts, and
development leads are involved in this task. The requirement gathering
document is reviewed at this stage, and cross-checked to ensure that it
does not affect the workflow
 Business Requirements Documentation: Business requirements
documents are prepared by quality analysts in which all reviewed
business requirements are covered
 Database Testing: It is the most important part of bank application
testing. This testing is done to ensure data integrity, data loading, data
migration, stored procedures, and functions validation, rules testing, etc.
 Integration Testing: Under Integration Testing all components that are
developed are integrated and validated
 Functional Testing: The usual software testing activities like Test
Case preparation, test case review and test case execution is done
during this phase
 Security Testing: It ensures that the software does not have any
security flaws. During test preparation, QA team needs to include both
negative as well as positive test scenarios so as to break into the
system and report it before any unauthorized individual access it. While
to prevent from hacking, the bank should also implement a multi-layer of
access validation like a one-time password. For Security Testing,
automation tools like IBM AppScan and HPWebInspect are used while
for Manual Testing tools like Proxy Sniffer, Paros proxy, HTTP watch,
etc. are used
 Usability Testing: It ensures that differently able people should be able
to use the system as normal user. For example, ATM with hearing and
Braille facility for disabled
 User Acceptance Testing: It is the final stage of testing done by the
end users to ensure the compliance of the application with the real
world scenario.
Sample Test Case for Net Banking Login Application
Security is prime for any banking application. Therefore, during test
preparation, QA team should include both negative and positive test scenarios
in order to sneak into the system and report for any vulnerabilities before any
unauthorized individual get access to it. It not only involves writing negative
test cases but may also include destructive testing.

Following are generic test cases to check any banking application

Sample test cases

For Admin  Verify Admin login with valid and Invalid data
 Verify admin login without data
 Verify all admin home links
 Verify admin change password with valid and invalid data
 Verify admin change password without data
 Verify admin change password with existing data
 Verify admin logout

For new Branch  Create a new branch with valid and invalid data
 Create a new branch without data
 Create a new branch with existing branch data
 Verify reset and cancel option
 Update branch with valid and invalid data
 Update branch without data
 Update branch with existing branch data
 Verify cancel option
 Verify branch deletion with and without dependencies
 Verify branch search option

For New Role  Create a new role with valid and invalid data
 Create a new role without data
 Verify new role with existing data
 verify role description and role types
 Verify cancel and reset option
 Verify role deletion with and without dependency
 verify links in role details page
For customer & Visitors  Verify all visitor or customer links
 Verify customers login with valid and invalid data
 Verify customers login without data
 Verify banker's login without data
 Verify banker's login with valid or invalid data

For New users  Create a new user with valid and invalid data
 Create a new user without data
 Create a new user with existing branch data
 Verify cancel and reset option
 Update user with valid and invalid data
 Update user with existing data
 Verify cancel option
 Verify deletion of the user

Challenges in testing Banking domain & their

Mitigation
Challenges tester might face during testing banking domain are

Challenge Mitigation

 Getting access to production data and  Ensure that test data meets regulatory
replicating it as test data, for testing is compliances requirements and guidelines
challenging  Maintain the data confidentiality by
following techniques like data masking,
synthetic test data, testing system
integration, etc.

 The biggest challenge in testing banking  Ensure Data Migration Testing is

system is during the migration of the complete
system from the old system to the new  Ensure Regression Test cases are
system like testing of all the routines, executed on old and new systems, and
procedures and plans. Also how the data the results match.
will be fetched, uploaded and transferred
to the new system after migration
  There may be the cases where  The test should participate in the project
requirements are not documented well right from Requirement Analysis phases
and may lead to functional gaps in test and should actively review the Business
plan Requirements
 Many non-functional requirements are
not fully documented, and testers do not
know whether to test it or not

 The most important point is to check  Compliance or Regulatory Policies testing

whether the said system follows the must be done
desired policies and procedures

 The scope and the timelines increases as  Ensure Time budget for Integration
banking application are integrated with Testing is accounted if your banking
other application like internet application has many external interfaces
or Mobile banking

Summary
Banking domain is the most vulnerable area for cyber-theft, and safeguarding
the software requires precise testing. This tutorial gives a clear idea of what it
takes for banking domain testing and how important it is. One must
understand that -

 Majority of banking software are developed on Mainframe and Unix

 Testing helps to lessen possible glitches encounter during software
development
 Proper testing and compliance to industry standards, save companies
from penalties
 Good practices help develop good results, reputation and more
business for companies
 Both manual and automated testing have respective merits and usability
eCommerce Testing: How to Test an E-
Commerce Website

What is Ecommerce Testing?

eCommerce testing is defined as testing of an eCommerce (online shopping)
application. It helps in the prevention of errors and adds value to the product
by ensuring conformity to client requirements.

The objective of testing is to ensure

 Software reliability
 Software quality
 System Assurance
 Optimum performance and capacity utilization

Setting up an E-commerce system is a complex process and subject to many

market-specific variables. To maintain the integrity of the E Commerce
system, testing becomes compulsory

Types of Testing for E-commerce System

A common type of testing included into e commerce system is

Sr.
Type of Testing Testing Process
#

 Lack of support for early browsers

 Browser specific extensions
1 Browser compatibility  Browser testing should cover the main platforms (Linux, Windows,
Mac etc.)

 Incorrect display of pages

 Runtime error messages
2 Page display  Poor page download time
 Dead hyperlink, plugin dependency, font sizing, etc.

 Session Expiration
3 Session Management  Session storage
Sr.
Type of Testing Testing Process
#

 Non-intuitive design
 Poor site navigation
4 Usability  Catalog navigation
 Lack of help-support

 Misleading, offensive and litigious content

 Royalty free images and copyright infringement
5 Content Analysis  Personalization functionality
 Availability 24/7

 Denial of service attacks

6 Availability  Unacceptable levels of unavailability

 Failure or fall over recovery

Back-up and Recovery  Backup failure
7
 Fault tolerance

 Transaction Integrity
Transactions  Throughput
8
 Auditing

 Shopping cart functionality

Shopping order processing and  Order processing
9 purchasing  Payment processing
 Order tracking

 Language support
 Language display
10 Internationalization  Cultural sensitivity
 Regional Accounting

 How well e-procedure copes

11 Operational business procedures  Observe for bottlenecks

 Data Interface format

 Interface frequency and activation
 Updates
12 System Integration
 Interface volume capacity
 Integrated performance

13 Performance  Performance bottlenecks

 Load handling
Sr.
Type of Testing Testing Process
#

 Scalability analysis

 Login capability
 Penetration and access control
 Insecure information transmission
14 Login and Security  Web attacks
 Computer viruses
 Digital signatures

Performance testing- a top priority in E-commerce

Just delay about 250 milliseconds of a page load time, is what keeps your
customer going to your competitor. Retail giant Walmart overhaul their site
speed and noticed an increase of 2% in visitor's conversion rate and revenue
by 1%.

Performance of your site depends on these factors

 Throughput
o Request per second
o Transactions per minute
o Executions per click
 Response Time
o Duration of a task
o Seconds per click
o Page Load
o DNS Lookup
o Length of time between click and seeing page

Useful Tools for Mapping E-commerce Site

 UsabilityHub: UsabilityHub's user testing platform and research panel
help you improve the UX of your apps and websites. Get feedback from
real people.
 HotJar: It shows the most clicked and unclicked zones of sites by
visitors
  FiveSecondTest: This tool ensures that your message is
communicated as effectively as possible, in just five seconds it tells
what a person recalls about your website design
 Feng-GUI: It simulates the human vision during the first five seconds
and predicts what a real human would most likely look at
 Optimizely: It enables you to test track, clicks, conversions or anything
else that matters to e-commerce business

Challenges of E-commerce Testing

 Compliance with security guidelines to safeguard customer data and
identity
 Compliance with accessibility standards to support multi-lingual markets
and business regions
 End to end testing and test management for large e-commerce
transformation programs
 Scalability and reliability of applications

Payment Gateway Testing Tutorial with

Example Test Cases

Payment Gateway Testing

Payment Gateway Testing is a testing of Payment Gateway in a system for
online purchases and transactions by the users. The purpose of payment
gateway testing is to ensure the security, reliability and performance of a
payment gateway by encrypting and securing the payment details between
user and merchant while providing a smooth payment experience.

A payment gateway system is an e-commerce application service that

approves credit card payment for online purchases. Payment gateways
safeguard the credit card details by encrypting sensitive information like credit
card numbers, account holder details and so on. This information is passed
safely between the customer and the merchant and vice versa.

Modern payment gateways also securely approve payments via debit cards,
electronic bank transfers, cash cards, reward points etc.
Types of Payment Gateway System
 Hosted Payment Gateway:

Hosted payment gateway system direct customer away from an e-

commerce site to gateway link during the payment process. Once the
payment is done, it will bring a customer back to an e-commerce site.
For such type of payment you don't need a merchant id, an example of
a hosted payment gateway are PayPal, Noche, and WorldPay.

 Shared Payment Gateway:

In a shared payment gateway, while processing payment customer is

directed to the payment page and stays on the e-commerce site. Once
the payment detail is filled, the payment process proceeds. Since it
does not leave the e-commerce site while processing payment, this
mode is easy and more preferably, an example of a shared payment
gateway is eWay, Stripe.

Testing Types for Payment Domain

Testing for Payment Gateway should include

Functional Testing: It is the act of testing the base functionality of the

payment gateway. It is to verify whether the application behaves in the same
way as it is supposed to be like handling orders, calculation, an addition of
VAT as per the country etc.

Integration: Test integration with your credit card service.

Performance: Identify various performance metrics like the highest possible

number of users coming through gateways during a specific day and
converting them to concurrent users

How to test Payment Gateway: Complete Checklist

Before you begin testing -

 Collect proper test data for the dummy credit card number for the
maestro, visa, master etc.
  Collect payment gateway information like Google Wallet, Paypal or
else
 Collect payment gateway document with error codes
 Understand the session and parameters passed through application
and payment gateway
 Understand and test the amount related information passed through
query string or variable or session
 Along with payment gateway language check the language of the
application
 Under the various settings of payment gateway like currency format,
subscriber data collected.

Example Test Cases for Payment Gateway Testing

Following are important Test Scenarios/Cases to check Payment Gateway

Sr# Test Cases

1 During the payment process try to change the payment gateway language

2 After successful payment, test all the necessary components, whether it is retrieved
or not

3 Check what happens if payment gateway stops responding during payment

4 During the payment process check what happens if the session ends

5 During the payment process check what happens in the backend

6 Check what happens if payment process fails

7 Check the Database entries whether they store credit card details or not

8 During the payment process check error pages and security pages
9 Check settings of pop-up blocker, and see what happens if a pop-up blocker is on and
off

10 Between payment gateway and application check buffer pages

11 Check on successful payment, a success code is sent to the application and a

confirmation page is shown to the user

12 Verify whether the transaction processes immediately or processing is hand to your

bank

13 After successful transaction check if the payment gateway returns to your application

14 Check all format and messages when successful payment process

15 Unless you don't have an authorization receipt from the payment gateway, good
should not be shipped

16 Inform the owner for any transaction processed through e-mail. Encrypt the content
of the mail

17 Check the amount format with currency format

18 Check if each of the payment options is selectable

19 Check if each listed payment option opens the respective payment option according
to specification

20 Verify whether the payment gateway defaults to the desired debit/credit card option
 21 Verify the default option for debit card shows card selection drop down menu

Things to consider before Buying Gateway Package

 If you have bought a shopping cart package, find out about its
compatibility
 If shopping gateway package is due, ask the payment gateway provider
for a list of supported applications
 The gateway must offer Address Verification System Protection
 Find out the types of transaction protection being offered
 Check what types of debit or credit cards are accepted by your chosen
payment gateway
 Check the transaction fees levied by a payment gateway
 Check whether the gateways collect the payment right on the form or
direct to another page to complete the purchase

Mainframe Testing - Complete Tutorial

What is a Mainframe?
The mainframe is a high performance and a high-speed computer system. It is
used for larger scale computing purposes that requires great availability and
security. It is mostly used in sectors like finance, insurance, retail and other
critical areas where huge data are processed multiple times.

Mainframe Testing
Mainframe Testing is a process of testing software applications and services
based on Mainframe Systems. The purpose of mainframe testing is to ensure
the performance, reliability and quality of software application or service by
verification and validation methods and check if it is ready to deploy.

While performing Mainframe testing, the tester only needs to know about the
navigations of the CICS screens. They are custom built for specific
applications. Any changes made to the code in COBOL, JCL, etc. tester does
not have to worry about the emulator set up on the machine. The changes
that work on one terminal emulator will work on others.
  The Mainframe application (otherwise called job batch) is tested against
the test cases developed using requirements
 Mainframe Testing is usually performed on the deployed code using
various data combinations set into the input file.
 Applications that run on the mainframe can be accessed through
terminal emulator. The emulator is the only software that needs to be
installed on the client machine.

Mainframe Attributes
1. Virtual Storage
1. It is a technique that lets a processor simulate main storage that is larger than the actual
amount of real storage.
2. It is a technique to use memory effectively to store and execute various sized tasks.
3. It uses disk storage as an extension of real storage.
2. Multiprogramming
1. The computer executes more than one program at the same time. But at any given
moment only one program can have control of CPU.
2. It is a facility provided to make efficient use of the CPU.
3. Batch Processing
1. It is a technique by which any task is accomplished in units known as jobs.
2. A job may cause one or more programs to execute in a sequence.
3. The Job scheduler makes a decision about the order in which the jobs should be
executed. To maximize the average throughput, jobs are scheduled as per their priority
and class.
4. The necessary information for batch processing is provided through JCL (JOB CONTROL
LANGUAGE). JCL describes the batch job – programs, data and resources needed.
4. Time Sharing
1. In a time-sharing system, each user has access to the system through the terminal
device. Instead of submitting jobs that are scheduled for later execution, the user enters
commands that are processed immediately.
2. Hence this is called "Interactive Processing". It enables the user to interact directly with
the computer.
3. Time-share processing is known as "Foreground Processing" and the batch job
processing is known as "Background Processing."
5. Spooling
1. SPOOLing stands for Simultaneous Peripheral Operations Online.
2. SPOOL device is used to store the output of program/application. The spooled output is
directed to output devices like a printer (if needed).
3. It is a facility exploiting the advantage of buffering to make efficient use of the output
devices.

Classification of Manual Testing in Mainframe

Mainframe Manual Testing can be classified into two types :
 1. Batch Job Testing –
o Testing process involves executions of batch jobs for the functionality implemented in
the current release.
o The test result extracted from the output files and the database are verified and
recorded.
2. Online Testing –
o Online Testing refers to testing of CICS screens which is similar to testing of the web
page.
o The functionality of the existing screens could be changed, or new screens could be
added.
o Various applications can have enquiry screens and update screens. The functionality of
these screens needs to be checked as part of the online testing.

How to do Mainframe Testing

1. The Business team prepares requirement documents. Which determines how a particular item
or process is going to be modified in the cycle of release.
2. The testing team and the development receive the requirement document. They will figure out
how many processes will be affected by the change. Usually, in a release, only 20-25% of the
application affected directly by the customized requirement. The other 75% of the release will
be for the out-box-functionalities like testing the applications and processes.
3. So, a Mainframe application has to be tested in two parts:
1. Testing Requirements – Testing the application for the functionality or the change
mentioned in the requirement document.
2. Testing Integration – Testing the whole process or other application which receive or
send data to the affected application. Regression Testing is the primary focus of this
testing activity.

Mainframe Automation Testing Tools

Below is the list of tools which can be used for mainframe Automation Testing.

 REXX
 Excel
 QTP

Methodology in Mainframe Testing

Let us consider an example: An XYZ insurance company has member enrollment module. It
takes data both from member enrollment screen and offline enrollment. As we discussed earlier,
it takes two approaches for Mainframe testing, online testing, and batch testing.

 Online testing is done on the member enrollment screen. Just like a web page the database is
validated with data entered through the screens.
  Offline enrollment can be paper enrollment or enrollment on a third party website. The Offline
data (also referred to as batch) will be entered into the company database through batch jobs.
An input flat file is prepared as per the prescribed data format and fed to the sequence of batch
jobs. So for mainframe application testing we can use the following approach.
o The first job in the line of batch jobs validates the data entered. Let say for example
special character, alphabets in number only fields, etc.
o The second job validates the consistency of data based on business conditions. For
example, a child enrollment should not contain dependent data, member zip code
(which is not available for service by the enrolled plan), etc.
o The third job modifies the data in the format that can be entered into the database. For
instance, deleting the plan name (database will store only plan ID, and insurance plan
name), appending date of entry, etc.
o The fourth job loads the data into the database.
 Batch job testing is done on this process in two phases –
o Each job is validated separately, and the
o Integration between the jobs is validated by providing input flat file to the first job and
validating the database. (Intermediary results have to be validated for extra caution)

The following is the method followed for Mainframe testing:

Step 1): Shakedown/Smoke Testing

The main focus in this stage is to validate whether the code deployed is in the right test
environment. It also ensures that there are no critical issues with the code.

Step 2): System Testing

Below are the types of testing done as part of System Testing.

1. Batch Testing – This testing will be done by validating the test results on output files and data
changes done by the batch jobs under testing scope and recording of them.
2. Online Testing – This testing will be done on the front end of the mainframe application. Here
the application is tested for correct entry field like an insurance plan, interest on the plan, etc.
3. Online-Batch Integration testing – This testing will be done on the systems having batch
processes and online application. The data flow and interaction between the online screens and
the batch jobs is validated.

(Example for this type of testing – Consider an update on Plan details like increase of
interest rate. The change of interest is done on an update screen, and the balance details
on the affected accounts will be modified only by a nightly batch job. Testing in this case
will be done by validating the Plan details screen and the batch job run for updating all
the accounts).

4. Database Testing – The databases where the data from the mainframe application (IMS, IDMS,
DB2, VSAM/ISAM, Sequential datasets, GDGs) are validated for their layout and the data
storage.
Step 3): System Integration Testing

The primary purpose of this testing is to validate the functionality of the systems which are
interacting with the system under test.

These systems are not directly affected by the requirements. However, they use data from the
system under test. It is important to test the interface and different types of messages (like Job
Successful, Job Failed, Database updated, etc. ) that can possible flow between the systems and
the resulting actions taken by the individual systems.

Types of testing done in this stage are

1. Batch Testing
2. Online Testing
3. Online – Batch Integration Testing

Step 4): Regression Testing

Regression Testing is a common phase in any type of testing project. This testing in Mainframes
ensures that batch jobs and the online screens which do not directly interact with the system
under test (or do not come in the scope of requirements) are not affected by the current project
release.

In order to have effective regression testing, a particular set of test cases should be shortlisted
depending on their complexity and a regression bed (Test cases repository) should be created.
This set should be updated whenever there is a new functionality rolled out into the release.

Step 5): Performance Testing

This testing is done to identify the bottlenecks in high hit areas like front end data, upgrading
online databases and to project the scalability of the application.

Step 6): Security Testing

This testing is done to evaluate how well the application is designed and developed to counter
anti-security attacks.

Two fold security testing should be done on the system – Mainframe security and Network
security.

The features which need to the tested are

1. Integrity
2. Confidentiality
3. Authorization
4. Authentication
 5. Availability

Steps involved in Batch testing

1. After the QA team receives the approved package (Package contains procedures, JCL, Control
Cards, Modules, etc.), the tester should preview and retrieve the contents into PDS as required.
2. Convert the production JCL or Development JCL into QA JCL otherwise called JOB SETUP.
3. Copying production file and preparing test files.
4. For every functionality, there will be a job sequence defined. (As explained in the example in
Methodology in Mainframe section).The jobs should be submitted using the SUB command with
the test data files.
5. Check the intermediate file in order to identify the reasons for missing or error-out data.
6. Check the final output file, database and the Spool to validate the test results.
7. If the job fails, the spool will have the reason for the job failure. Address the error and resubmit
the job.

Test Reporting – Defect should be logged if the actual result deviates from expected.

Steps involved in Online Testing

1. Select the Online screen in a test environment.
2. Test each field for the acceptable data.
3. Test the Test Scenario on the screen.
4. Verify the database for the data updates from the online screen.

Test Reporting – Defect should be logged if the actual result deviates from expected.

Steps involved in Online – Batch Integration testing

1. Run the job in a Test Environment and validate the data on the online screens.
2. Update the data on the online screens and validate if the batch job is properly run with the
updated data.

Commands used in Mainframe Testing

1. SUBMIT – Submit a background job.
2. CANCEL – Cancel background job.
3. ALLOCATE – Allocate a dataset
4. COPY – Copy a dataset
5. RENAME – Rename data set
6. DELETE – Delete Dataset
7. JOB SCAN –To bind the JCL with the program, libraries, file, etc. without executing it.

There are many other commands used when required, but they are not that frequent.
Pre-requisites to start mainframe testing
Basic details needed for mainframe testing are:

 Login ID and password for logging into the application.

 Brief knowledge on ISPF commands.
 Names of the files, file qualifier and their types.

Before starting mainframe testing, the below aspects should be verified.

1. Job
1. Do a job scan (Command – JOBSCAN) to check for errors before executing it.
2. CLASS parameter should be pointed to the test class.
3. Direct the job output into spool or a JHS or as required by using MSGCLASS parameter.
4. Reroute the email in the job to spool or a test mail ID.
5. Comment the FTP steps for initial testing and then point the job to a test server.
6. In case an IMR (Incident Management record) is generated in the job, just add comment
"TESTING PURPOSE" in the job or param card.
7. All the production libraries in the job should be changed and pointed to test libraries.
8. The job should not be left unattended.
9. To prevent the job to run in an infinite loop incase of any error, TIME parameter should
be added with specified time.
10. Save the output of the job including the spool. The spool can be saved using XDC.

2. File
1. Create test file of needed size only. Use GDGs(Generation Data Groups – Files with the
same name but with sequential version numbers–
MYLIB.LIB.TEST.G0001V00,MYLIB.LIB.TEST.G0002V00 so on ) when necessary to store
data into consecutive files with the same name.
2. The DISP (Disposition - describes the system to perform keep or delete the dataset after
normal or abnormal termination of the step or job) parameter for the files should be
coded correctly.
3. Ensure that all the files used for job execution are saved and closed properly to prevent
job to go into HOLD.
4. While testing using GDGs make sure that the right version is pointed at.
3. Database
1. While executing the job or online program, ensure that unintended data is not inserted
or updated or deleted.
2. Also, ensure that the correct DB2 region is used for testing.
4. Test cases
1. Always test for boundary conditions like – Empty file, First record processing, Last record
processing, etc.
2. Always include both positive and negative test conditions.
3. In case if standard procedures are used in the program like Check point restart, Abend
Modules, Control files, etc. include test cases to validate if the modules have been used
correctly.
5. Test Data
 1. Test data setup should be done before the beginning of the testing.
2. Never modify the data on the test region without notifying. There may be other teams
working with same data, and their test would fail.
3. In case the production files are needed during the execution, proper authorization
should be obtained before copying or using them.

Best Practices
1. Incase of a Batch Job run, MAX CC 0 is an indicator that the job has run successfully. It does not
mean that the functionality is working fine. The job will run successfully even when the output is
empty or not as per the expectation. So it is always expected to check all the outputs before
declaring the job successful.
2. It is always a good practice to do a dry run of the job under test. Dry run is done with empty
input files. This process should be followed for the jobs which are impacted by the changes
made for the test cycle.
3. Before the test cycle begins the test job set up should be done well in advance. This will help in
finding out any JCL error in advance hence saving time during execution.
4. While accessing DB2 tables through SPUFI (Option on the emulator to access DB2 tables), always
set auto commit as "NO" in order to avoid accidental updates.
5. Test Data availability is the primary challenge in batch testing. Required data should be created
well in advance of the test cycle and should be checked for completeness.
6. Some online transactions and batch jobs may write data into MQs (Message Queue) for
transmitting data to other applications. If the data is not valid, it may disable/stop MQs, this will
affect the whole testing process. It is a good practice to check that MQs are working fine after
testing.

Mainframe testing Challenges and Troubleshooting

Challenges Approach

Incomplete / Unclear Requirements Testers should be involved in the SDLC from the requirements
phase onwards. This will help to verify if the requirements are
There may be access to user manual/ training guide, testable.
but those are not same as documented requirements.

Data Setup/ Identification For data setup, homegrown tools can be used as per the need
For fetching existing data, queries should be built in advance.
There may be situations where existing data should be In case of any difficulty, a request can be placed to data
reused as per the requirement. It is sometimes difficult management team for creating or cloning required data.
to identify the required data from the existing data.

Job Setup Job setup tools should be used so as to overcome human

 errors made during setup.
Once the jobs are retrieved into PDS, the job needs to
be setup in the QA region. So that the jobs are not
submitted with production qualifier or path detail.

Ad-hoc Request Use of automation scripts, regression scripts, and skeleton

scripts could help in reducing the time and effort overhead.
There may be situations when end to end testing needs
to be supported due to a problem in upstream or
downstream application issues. These requests
increase the time and effort in execution cycle.

On-Time Releases for scope change Scope change management process and Impact analysis shou
be in place.
There may be a situation where the code impact may
completely change the look and feel of the system. This
may require a change to test cases, scripts, and data.

Common Abends encountered

1. S001 – An I/O error occurred.

Reason – Reading at the end of the file, file length error, attempt to write
into read-only file.

2. S002 – Invalid I/O record.

Reason – Attempt to write a record longer than record length.

1. S004 – Error occurred during OPEN.

Reason – Invalid DCB

2. S013 – Error opening a dataset.

Reason – PDS member does not exist, record length in the program
does not match the actual record length.

3. S0C1 – Operation Exception

 Reason –Unable to open file, missing DD card

4. S0C4 – Protection exception/ Storage violation

5. Reason – Trying access storage not available to the program.
6. SC07 – Program Check Exception – Data
7. Reason – Change in record layout or file layout.
8. Sx22 – Job has been canceled
9. S222 – Job canceled by the user without a dump.
10. S322 – Job or Step time exceeded the specified limit, or the
program is in a loop or insufficient time parameter.
11. S522 – TSO session timeout.
12. S806 –Unable to link or load.

Reason - Job id unable to find the specified load module.

13. S80A – Not enough virtual storage to satisfy GETMAIN or

FREEMAIN requests.
14. S913 – Trying to access the dataset which the user is not
authorized.
15. Sx37 – Unable to allocate enough storage to the dataset.

Error Assist – A very popular tool to get detailed information on various types
of abends.

Common issue faced during mainframe testing

 Job Abends – For successful completion of the job, you should check
the data, input file and the modules present at the specific location or
not. Abends can be faced due to multiple reasons, the most common
being – Invalid data, Incorrect input field, date mismatch, environmental
issues, etc.

 Output file empty–Though the job might run successfully (MaxCC 0),
the output might not be as expected. So before passing any test case,
the tester have to make sure that the output is cross verified. Only then
proceed further.
 Input file empty – In some applications, files will be received from the
upstream processes. Before using the received file for testing current
application, the data should be cross verified to avoid re-execution and
rework.
Testing Retail Point Of Sale(POS)
Systems: Example Test Cases
What is POS Testing?
POS Testing is defined as Testing of a Point of Sale Application. A POS or
Point Of Sale software is a vital solution for retail businesses to carry out retail
transactions effortlessly from anywhere. You must have seen Point of Sale
terminal while checking out at your favorite Mall.

The system is more complex than you think and is tightly integrated with other
software systems like Warehouse, Inventory, purchase order, supply chain,
marketing, merchandise planning etc. POS Domain Knowledge is important
for testing.

Test Architecture for POS Application

POS test architecture includes three components for testing - POS terminal,
store server, and enterprise server. Basically, it is classified into three levels
for testing of POS application.

Level 1- (POS Terminal ) Level 2- (Store Server) Level 3- (Enterprise Server

 Device and hardware testing (RFID, Scanner,  Security Testing  Security Testing
Printer, Barcode reader)  BI & Analytics  BI & Analytics
  Interoperability Testing Testing Testing
 BI and Analytics Testing  Disaster Recovery  Disaster Recovery
 Performance Testing Testing Testing
 Interface Testing  Interface Testing

Types of Testing for POS system

Testing of POS System can be broken down into two levels

1. Application Level
2. Enterprise Level

Testing Performed At Application Level Testing Performed At Enterprise Level

 Functionality Testing  Compliance Testing

 Compatibility Testing  Performance Testing
 Payment Gateway Testing  Interoperability Testing
 Report Testing  Data Migration
 Mobility

Sample Test Cases for POS used in Retail

To ensure quality of the POS system, proper POS software testing is
mandatory. The POS testing spans many things like

Test Scenario Test Cases

Cashier activity  Test the entry of items purchased by a customer is correct
 Test discounts are applied correctly
 Verify store value cards can be used
 Check petty cash management works as expected
 Check totals and closings match
 Check cash drawer loans are handled properly
 Test the POS system is compatible with peripherals like RFID Reader, Bar Code
Scanner etc.

Payment Gateway  Test the validity of CVV number of Credit Card

Processing  Test swiping of cards from both sides and chips
 Verify that the captured card details are properly encrypted and decrypted

Sales  Check for a regular sale process

 Check sales can be processed with debit/credit cards
 Check for loyalty membership purchase
 Check for correct prices are displayed for merchandise purchased
 Test for "0" or null transaction
 Tie UPC or barcodes to vendors
 Test for billing details or shipping details in payment manager
 Test for reference transaction
 Test the print format of the receipt generated
 Verify that the correct code is generated for approved, hold or declined
transactions

Return & Exchange  Make sure the in-house inventory is well integrated with other outlets or supply
scenarios chain
 Check for exchange or return of an item with cash
 Check whether system responds on exchange or return of an item with a credit
card
 Check system process the sale with receipt or without a receipt
 Verify that system should allow entering bar-code manually incase scanner don'
work
 Verify system display both the current amount as well as the discount amount o
an exchange of item if applicable
Performance  Check for speed or time taken to receive a response or send a request
 Check the transaction based rules are applicable (discounts/tax/ rebates etc.)
 Verify that the correct code is generated for approved, hold or declined
transactions

Negative Scenarios  Test system with expired card details

 Test with an invalid PIN for credit card
 Check the inventory by entering a wrong code for the item
 Check how a system responds while entering a wrong invoice number
 Test for a negative transaction
 Test the response of system while entering an invalid date for promotional offer
online items

Managing Promotions and  Test system for various discount like a veteran discount, seasonal discount,
Discounts undergage or overgage discount etc.
 Test system for various promotional offers on certain line items
 Test alert system that notifies end or beginning of seasonal offers
 Test whether receipt print the exact discount or offers that are leveraged
 Test system for allocating wrong offers or discount online item
 Test the order management process
 Verify product data obtained after scanning a barcode is accurate

Tracking customer's data  Test for system response with incorrect customer data input
 Test system for allowing authorized access to customer's confidential data
 Test the database for recording customer's buying history like (what they buy,
how frequent they buy, etc.)

Security & Regulatory  Verifying POS system as per regulatory compliances

Compliance  Test alert system that notifies security defenders
 Make sure you can void a payment before posting
 Test user profiles and access levels on the POS Software
 Test database consistency
 Verify specific information about each tender cash, coupon identifier, check
number and so on

Report testing  Testing of a trend analysis report

 Test information related to credit card transaction should be reflected in reports
 Test for the individual as well as consolidated reports of customers buying
 history
 Test for online report generation

Security Testing for Retail POS Systems

Some recent studies have Point of Sale Systems very high-security
vulnerabilities. Following measures will help with security of POS

 Security testing in compliance with PCI standard is very crucial to be

addressed as the part of enterprise testing
 Actively manage all software on the network so that only authorized
software can only execute and installed
 Conduct regular Penetration Testing to identify attack vectors and
vulnerabilities
 Include tests for the presence of unprotected system information and
artifacts that would be useful to hackers
 Use vulnerability testing tools
 Create a testbed that imitate a production environment for specific
penetration tests and attacks against elements that are not tested in
production

Challenges in POS testing

 Multiple Configurations
 Complex interfaces
 Peripheral issues
 Upgrades
 PCI compliance
 Test lab maintenance

Summary

 Retail POS demands a high level of testing keeping in mind that its
performance and correct functioning directly affect business revenues.
 To reduce the risk and chances of POS failure during the transaction
process, testing under the extreme condition is essential.
 Testing needs to perform at Application as well as Enterprise Level
 Your Testing should cover the following scenarios - Cashier activity,
Payment Gateway Processing, Sales, Return & Exchange scenarios,
 Performance, Negative Scenarios, Managing Promotions and
Discounts, Security & Regulatory Compliance.
 Multiple configuration settings, peripheral issues, upgrades are few
issues you will need to tide over while testing.

HealthCare Domain Testing with Sample

Test Cases

Before we begin testing, let's quickly study the basic healthcare domain
knowledge.

Basic knowledge of Health Care Domain

The entire health care system is weaved with each other by the single body
that is hospital or provider (doctor).

While the other entities include-

 Insurance company: Medicare, Medicaid, BCBS, etc.

 Patient/Consumers: Patient Enrolled
 Regulatory Authority: HIPAA, OASIS assessment, HCFA 1500 and
UB92, etc.
 Health-care and Life-Science solution Vendors
Basic Terminology of Health Care System

 Provider: A health care professional (doctor), medical group, clinic, lab,

hospital, etc. licensed by health care services
 Claim: A request to your health insurance company to pay a bill for
health care service
 Broker: An insurance professional, who negotiates, procures insurance
on behalf of insured or prospective insured
 Finance: Insurance bodies that pay for medical expenses, it could be
government (Medicare or Medicaid) or commercial (BCBS)
 Medicare: A federal health insurance program for senior citizen and
permanently disabled people
 Medicaid: A joint and state program that helps low-income families and
individuals pay for the cost associated with medical care
 CPT code: A current procedural terminology code is a medical code set
to describe medical, surgical and diagnostic services
 HIPAA: It is a set of rules and regulations which doctors, hospitals,
healthcare providers and health plan must follow in order to provide
their services
Healthcare Business Process
Most health-care organization have adapted software program to process the
smooth functioning of the system. This software system gives all the
information in a single document for each entity dealing with this.

Interconnecting this whole system to a single web application is a huge task

and making it work effectively is even a bigger task. Rigorous testing of this
health application is compulsory, and it has to go through various testing
phases.

In this tutorial, we will learn,

Testing of Providers system

Sample Test Scenarios and Test cases for providers (doctor/hospital)
system:

Test Scenario Test Cases

1. Access to  Provider system should let us enter, edit and save the provider's
providers system data
 2. Positive  It includes scenarios to enter different types of provider, change
flow System providers details, save and inquire them
Testing

3. Negative flow  Allows to save provider information with incomplete data,

System Testing contract's effective date, entering details about existing
providers in the system

4. System Integration  Validate the feed to members system, finance system, claim

Testing system, and provider portal. Also, validate if the changes from
provider portal are entered into the respective provider's
record

5. Positive flow  Login and view providers details, claim status, and member
providers portal details
testing  Make change request to change the name, address, phone
number, etc.

6. Negative flow  View the member details with an invalid ID

providers portal  Login with invalid credentials
testing

7. Positive flow  Login and view details about broker and commission payment
Broker portal  Make a request to change the name, address, phone number,
testing etc.

8. Negative flow  It should include scenarios to log in with invalid credentials

Broker portal
testing

Testing of Broker System

Sample Test Scenarios and Test cases for Broker System:
Sr Test Scenario Test Cases
#

1) Broker System  It should be capable of edit, enter and save broker data
 Broker commission calculation based on the premium payment details from the
member system

2) Positive Flow  Enter, save and edit brokers record for different types of broker
System Testing  For active brokers calculate the commission by creating a feed file with the
respective record for members with a different plan

3) Negative flow  Enter a broker record with incomplete data and save for different types of broker
System Testing  By creating the feed file with the respective record for members with different plan
calculate the commission for the terminated broker
 By creating the feed file with the respective record for members with different plan
calculate the commission for the invalid broker

4) System Testing  To downstream system such as finance system, broker portal and member system
validate the feeds
 Validate if the changes from broker portal are incorporated in the respective broke
record

Testing of Member System

Sample Test Scenarios and Test cases for Member (Patient) System:

Sr Test Scenario Test Cases

#

1) Member system  Enroll, reinstate and terminate a member

 Remove and add a dependent
 Generate premium bill
 Process premium payments

2) Positive Flow System  With the current, past, and future effective dates enroll different types of
Testing members
 Inquire and change members
  Produce premium bill for an active member for the following month
 Terminate an active member with past, current and future termination dates
greater than the effective date
 Re-enroll a terminated member with current, past and future effective dates
 Reinstate a terminated number

3) Negative flow System  With insufficient data enroll a member

Testing  For a terminated member produce a premium bill for the following month

4) System Integration  Validate the feed to downstream systems such as provider portal, broker portal
Testing finance system, and claim system
 Validate if the alterations from member portal are incorporated in the respectiv
member record
 Process the payment of premium bill generated with the feed from members
portal that has details of payment made

Testing of Claims System

Sample Test Scenarios and Test cases for Claims System:

# Test Scenarios Test Cases

1 Claim System  Claims in health-care should edit, enter and process claims for a member as wel
) as dependent
 For invalid claims, it should throw errors when incorrect data is entered

2 Positive Flow System  It should include the scenario to edit, enter and process claims for a member as
) Testing well as dependent

3 Negative Flow System  It should validate and enter a claim with invalid procedure code and diagnosis
) Testing code
 Validate and enter a claim with the inactive provider ID
 Validate and enter a claim with a terminated member

4 System Integration  It should include a scenario to validate the feed to downstream systems such as
)
 provider and finance portal

Testing of Finance System

Sample Test Scenarios and Test cases for Finance System

Sr Test Scenarios Test Cases

#

1) Finance System  Enroll, reinstate and terminate a member

2) Positive flow system  It should check whether correct account number or address is chosen for the
testing respective member, provider or broker for the payment

3) Negative flow  Verify whether payment is done for an invalid member, provider or broker ID by
system testing creating a respective record in the feed
 Verify whether payment is done for an invalid amount for the member, provider o
broker by creating respective records in the feed

Testing for regulatory compliance

Protecting patient sensitive data and health information is an utmost priority
for health regulatory bodies. The testing should be done in compliance of such
regulatory bodies.

Sample Test Scenarios and Test cases for Regulatory Compliance:

Sr Test Scenarios Test Cases

#

1) User's Authentication  Using verification method to ensure that correct users get a login and deny
to others

2) Information Disclosure  Authorizing access to information is based on the user's role and patient
 limitation

3) Data Transfer  At all transfer, points ensure that data is encrypted

4) Audit Trail  All transactions and all attempts to access data with a proper set of audit
trail information are recorded

5) Sanity Testing related to  Perform sanity testing and verify the encryption of the data is done in
regulatory body particular areas like EPHI ( Electronic Protected Health Information)

Performance testing of Healthcare Application

Before preparing test scenarios certain requirement of the system should be
considered. For example, health-care providers (Doctors/Hospitals) provide
care 24/7, so the patient check-in software needs to be available at all times.
Also, it needs to communicate with insurance companies to validate policy
information, send claims and receive remittances. Here, the architecture
should define the different components of the system, the protocol to
communicate with insurance companies, and how to deploy the system so
that it complies 24/7.

As a tester, you need to ensure that the healthcare software system meets the
desired load/performance benchmark.
Other Testing Types for Healthcare Application

 Functional Testing: Testing healthcare application against functional

capabilities
 Conformance Testing: Conformance test Healthcare security
requisites and industry frameworks
 Platform Testing: Testing of applications on Mobile platform and
applications testing for cross-browser compatibility
 Interoperability Testing: Testing conformance to interoperability
standards ( Eg; DICOM, HL7, CCD/CDA)

Testing Challenges in Healthcare Application

Testing challenges in testing healthcare application are no different than other
web application testing.

 Requires expertise in testing, and usually, it is high in cost

 Requires interoperability, compliance, regulatory, security, safety testing
besides regular testing techniques (Non-Functional, Functional and
Integration testing)
 Testing should be done keeping in mind the safety and regulatory
standards- as any error can cause a direct effect on patient's life
 Testing team needs to be well aware of the various functionalities,
clinical usage, and the environment the software will be used for
 A health-care product should comply with various standards like FDA,
ISO, and CMMI before it can be used
  Cross dependency of software- testers need to ensure that any changes
in one component or layer should not lead to side effect on the other.

Healthcare device Testing

While health-care device software is not the direct concern of patient, they
also require rigorous testing like another software testing. For example, X-ray
machines that are controlled by software programs should be tested well
because any testing error in software can lead to a serious effect on the
patient.

FDA (Food and Drug Administration) has guidelines for mobile and web
applications for medical devices. While testing medical devices the proper
functional Test Plan along with pass and fail criteria is also the part of FDA
guidelines. When a test plan is executed, the results are collected and
reported to FDA. This process ensures that the device meets the standard of
the regulatory bodies.

Useful tips for Healthcare Testing

While testing software, you can consider some important tips for the testing
healthcare system.

 Dates are important and need to be accurate

 While designing test cases consider various parameters like different
types of plan, brokers, members, commission, etc.
 Complete knowledge of the domain is required

Testing Insurance Domain Applications

with Sample Test Cases
Insurance Companies rely heavily on Software to run their business. Software
Systems helps them to deal with various insurance activities like developing
standard policy forms, handling billing process, managing customer's data,
rendering quality services to the customer, coordinating between branches
and so on.

Though this software is designed to meet the customer's expectations, its

durability and consistency needs to be tested before its actual deployment.
Software testing assures the quality of the insurance software by identifying
bugs before go-live.

What is Domain in Testing?

Domain is nothing but the industry for which the software testing project is
created. When we talk about software project or development, this term is
often referred. For example, Insurance domain, Banking domain, Retail
Domain, Health Care Domain, etc.
Usually, while developing any specific domain project, domain expert help is
sought out. Domain expert are master of the subject, and he may know the
inside-out of the product or application.

Why Insurance Domain Knowledge Matters?

Domain knowledge is quintessential for testing any software product, and it
has its own benefits like
What is Insurance? Type of Insurance
Insurance is defined as the equitable transfer of the risk of a loss from one
entity to another in exchange for payment. Insurance Company, which sells
the policy is referred as INSURER while the person or company who avails
the policy is called the INSURED.

Insurance policies are usually classified into two categories, and insurer buy
these policies as per their requirement and budget.

However, there are other types of insurance that falls under these categories

 Unemployment insurance
 Social Security
 Workers Compensation
What is Premium? How is Premium calculated?
Premium is defined as the amount to be charged for a certain amount of
insurance coverage or policy the insured has bought.

Premium for the insurance is determined by on the basis of two factors

 The frequency of claims

 The Severity of claims (Cost of each claim)

For example, we will see how insurance system works,

Suppose an insurance company provides insurance to all houses in a

village

Home Insurance Amount

Total number of house in village = 1000

Value of each house = $ 800

Contribution of each house owner as premium =$8

Total Premium Collected = $8000

Statistically, it has calculated that in case of fire a maximum of 10 houses are

burnt which it need to compensate.

So incase, of fire, it will have to pay 10 house $800 which comes $8000 equal
to the premium it collected.

The risk of 10 house owners is spread over 1000 house owner in the village
hence reducing the burden on any one of the owner.

In case of no fire in a particular year, the entire sum goes to its profit while if
more than 10 houses burn the insurer will incur a loss.
Testing required in different process area of
Insurance
Testing can mitigate the risk of business disruption during and after
deployment of software. There are many branches of an insurance company
that requires testing.

Sponsored by IBM Security

Manage your defenses against growing threats

Let's put security everywhere, so you can thrive in the face of uncertainty.
Discover how.
See More

 Policy Administration Systems

 Claim Management Systems
 Distribution Management Systems
 Investment Management Systems
 Third party Administration Systems
 Risk Management Solutions
 Regulatory and Compliance
 Actuarial Systems (Valuation & Pricing)
What to Test in Insurance?
The insurance sector is a network of small units that deals directly or indirectly
with processing claims. For smooth functioning of an insurance company, it is
necessary that each of this unit is tested rigorously before it is sync together
to deliver the desired outcome. The testing includes

 Call Center  IVR Integration Testing

 Call routing and assignment
 Security and access
 Reflexive Questions

 Policy Serving  Policy life cycle testing

 Financial and Non-financial policy changes
 Policy lapse and Re-instatement
 Policy aging-run cycles
 Premium due alerts
 Valuation of NPV/NAV

 Claims  Claims triage and assignment

 Testing claims life cycle
 Claims accounting/reserving
 Third party EDI/messaging

 Direct channel  Mobile access

 Cross browser/cross platform accessibility
 Application performance
 Usability of application

 Reports/BI  Behaving to regulatory requirements

 Generate quality data for reporting
 Create bulk data for roll-up reports
 Testing formula based fields in reports

 Underwriting  Underwriting quality

 Manual and Straight through processing
 Complex business rules
  Rating efficiency
 Requirements Management (Vendor Interfacing)

 Integration  Data integration

 Complex interface integration
 Source/Destination formats
 Production like interface
 Web service pull/push efficiency

 New Business  Validate rates-factor combinations

 Batch job schedules and runs
 Commissioning calculations settlements
 Quick and detailed quote
 Benefit illustration
 Benefit summary validation
 Quick and detailed quote

Sample Test Case for Insurance Application Testing

Sr# Test Cases for Insurance Application

1 Validate claims rule

2 Ensure that claim can occur to the maximum and minimum payment

3 Verify data is transferred accurately to all sub-systems including accounts and reporting.

4 Check that the claims can be processed via all channels example web, mobile, calls, etc

5 Test for 100% coverage and accuracy in calculations determining premium rates

6 Make sure formula for calculating dividend and paid up values gives correct value

7 Verify surrender values are calculated as per the policy requirement

8 Verify fiduciary details and bookkeeping requirements

9 Test complex scenarios for policy lapse and revivals

10 Test various conditions for non-forfeiture value

11 Test scenarios for policy termination

12 Verify general ledger account behave same as to reconcile with subsidiary ledger

13 Test calculation of net liability for valuation

14 Test conditions for extended term insurance

15 Verify policy for a non-forfeiture option

16 Check different insurance product term behaves as expected

17 Verify premium value as per product plan

18 Test automatic messaging system to inform customer about new products

19 Validate all the data entered by users as it progresses through the workflow to trigger
warnings, compliance, notification and other workflow events

20 Verify insurance document template supports the document format like MS-Word

21 Test system for generating invoice automatically and send it to customer through e-mail
Summary
Timely process of the insurance policy and managing client's data is a
foremost priority for any insurance company. Their complete dependency on a
software solution for handling claims, as well as customers, requires software
solution to be precise and accurate. Considering all the key aspects of
insurance company's requirement some of the testing strategy and scenarios
are represented in this tutorial.

Testing Telecom Domain with Sample

OSS/BSS Test cases
What is Telecom Testing?
Telecom Testing is defined as the testing of Telecommunication software.
Since the shift of the telecom sector to digital and computer networks,
telecommunication industry uses software indispensable. Telecom sector
depends on the various types of software components to deliver many
services like routing and switching, VoIP broadband access, etc. Hence,
telecom software testing is inevitable.

What is Domain in Testing?

A domain is nothing but the industry for which the software testing project is
created. When we talk about software project or development, this term is
often referred. For example, the Insurance domain, Banking domain, Retail
Domain, Telecom Domain, etc.

Usually while developing any specific domain project, domain expert help is
sought out. Domain expert is master of the subject and he may know the
inside-out of the product or application.

Why Testing Domain Knowledge Matters?

Domain knowledge is quintessential for testing any software product, and it
has its own benefits like
Business Processes in the Telecom Industry
For telecom testing end-to-end service verification is important. To ensure
efficient testing a good understanding of the different Business process is a
must.

You need to understand each stage of service deliverability before drafting the
test cases.

Telecom services are either based on a business support system that

includes IVR's, Call Centers, generating invoices, etc. or an operation support
system that includes routers, switches, cell towers, etc.

The following table shows what activities are performed at different levels

Telecom Telecom Activities

Department

Pre-sales  It handles all the sales information like discounts, services, promos, etc.
 Ordering  Applying for a new connection or disconnecting a connection

Provisioning  This division deals with the physical connection between customers and TSP ( Telecom
Service Provider)

Billing  Under this division, all billing work is done

Service Assurance  In case of any failure, this division corrects the problem

Inventory Systems  It is the repository of all information

Tracking  This division tracks the ordering system and the status of an order

Typical Telecom Business Process

Following is a typical business process in the Telecom Industry.
Types of Protocols used in Telecom Industry
Here the popular protocols used in the Telecom industry

 VoIP technologies: VoIP, IMS, MPLS, ISDN, PSTN

 Signaling and Protocols: SIP, ISDN, Codecs, H.323
 Wireless technologies: GPRS, CDMA, GSM, UMTS
 Network Management: SNMP
 Layer 2 Protocols: ARP, STP, L2TP, PPP
  Layer 3 protocols/routing: ICMP, BGP, ISIS, MPLS
 Infrastructure/Security: ATM, TCP/IP, LAN/VLAN, SSH

Testing LifeCycle in the Telecom Industry

The Test Lifecycle in the telecom industry is similar to that of any other
industry but with a stress on details. Here is how the test lifecycle looks like
along with the test artifacts.

Telecom Testing Stage Test artifacts

 Business View  Requirement based test artifacts

 Feasibility based artifacts
 Standard and policy identification based test artifacts
 Operation and maintenance considerations related test artifacts

 System/ Architecture  System test artifacts (Security, Installation)

 Test artifacts for virtual prototype
 Special System Testing artifacts ( interoperability, disaster recovery)

 Implementation  Unit test artifacts

 Integration test artifacts
 Quality and performance artifacts
 Regression, load testing, sanity, etc.

 Deployment  Acceptance test artifacts

 Integration test artifacts
 Quality and performance artifacts
 Functional test artifacts
 Alpha/Beta test artifacts

Types of Testing Performed on Telecom Software

 Interconnection Testing
 Conformance Testing
 IVR Testing
 Performance Testing
  Security Testing
 Interoperability Testing
 Protocol Testing
 Functional Testing
 Automation Testing

Sample TestCases for Telecom Testing

In Telecom Testing, one must consider the testing following

Various Telecom Testing Testing activities in Telecom

Billing System  Verify, the telephone number of the customer is registered

under telecom operator
 Verify whether the number is still working
 Verify the number entered is valid, and it is 10 digit number
 Verify the number is not blocked due to some reasons
 Verify if the number has any outstanding bills, if exist, display
it on screen
 Verify the number has all previous accounts or bills cleared
 Verify the system enables statement generation as per
customer requirement
 Verify the system has recorded a number of calls accurately
 Verify the plan chosen by the customer displays on the billing
system
 Verify the total amount billed is accurate and mapped to the
service offered

Application Testing  Protocols, signaling, field testing for IOT

 Usage and Functional Testing for core Mobile handset
applications like a call, SMS, transfer/hold, etc.
 Testing of various applications like finance, sports and
location-based services, etc. OSS-BSS testing

OSS-BSS Testing  Billing, customer case, interconnect billing, order and fraud
management, revenue assurance
 Network management, mediation, provisioning, etc.
 EAI, CRM & ERP, data warehousing, etc.
 Conformance Testing  Electrical interface compatibility
 Conformance of protocol
 Conformance of transport layers

IVR Testing  Interactive test scenarios

 Detection of voice energy
 Broadband audio tones
 Extensive conditional branching sequences
 DTMF Entries

Summary

The telecom service is a very broad field consists of a various component

including cables, networks, signals, protocols, etc. and their testing requires a
broad range of testing techniques, so the choice of testing techniques and
strategy highly depends on what component of telecom is tested.

The test requirement, scope, test scenarios, testing techniques, testing tools,
etc. varies with the type of testing involves, it can be protocol testing for VoIP
or wireless device testing for CDMA. The tutorial gives a basic but complete
overview of how telecom testing can be performed and discuss various
prospects that are crucial for telecom testing.

Business Intelligence (BI) Testing:

Sample Test Cases

What is BI Testing?
BUSINESS INTELLIGENCE (BI) is the process of gathering, cleansing,
analyzing, integrating and sharing data to derive actional insights that drive
business growth. Business Intelligence Testing or BI testing verifies the
staging data, ETL process, BI reports and ensures the implementation is
correct. BI Testing ensures data credibility and accuracy of insights derives
from the BI process.
Sample Test Cases for BI
Following are generic test cases that need to be validated for any BI Testing
Project

Test Scenarios Test Cases

ETL verification  Verify data is mapped correctly from source to target system

  Verify all tables and their fields are copied from source to target

  Verify keys configured to be auto-generated are created properly in target system

  Verify that null fields are not populated

  Verify data is neither garbled nor truncated

  Verify data type and format in target system is as expected

  Verify there is no duplicity of data in the target system

  Verify transformations are applied correctly

  Verify that the precision of data in numeric fields is accurate

  Verify exception handling is robust

Staging data  Reconciliation check- record count between the STG (staging) tables and target tables are
same after applying filter rules

  Insert a record which is not loaded into target table for given key combination
   Copy records, sending same records that are already loaded into target tables-should not be
loaded

  Update a record for a key when value columns changed on day_02 loads

  Delete the records logically in the target tables

  Values loaded by process tables

  Values loaded by reference tables

Data Loading in  Check if the target and source data base are connected well and there are no access issues.
BI

  For a full load, check the truncate option and ensure its working fine.

  While loading the data, check for the performance of the session

  Check for non-fatal errors.

  Verify you can fail the calling parent task if the child task fails.

  Verify that the logs are updated

  Verify mapping and workflow parameters are configured accurately

  Verify the number of tables in source and target systems is the same

  Compare the attributes from stage tables to that of the target tables. They should be
matched.

BI Reports  Display date and time

   Decimal precision for key figures

  In a given page display the number of rows and columns

  Free characteristics in the report

  How are blank values/data displayed for both characteristics and key figures in the report

  Whether search for characteristics is based on key or key&text as applicable

  Does search option on text is case sensitive- Upper, Lower or both

What is WHITE Box Testing?

Techniques, Example, Types & Tools
White Box Testing
White Box Testing is software testing technique in which internal structure,
design and coding of software are tested to verify flow of input-output and to
improve design, usability and security. In white box testing, code is visible to
testers so it is also called Clear box testing, Open box testing, Transparent
box testing, Code-based testing and Glass box testing.

It is one of two parts of the Box Testing approach to software testing. Its
counterpart, Blackbox testing, involves testing from an external or end-user
type perspective. On the other hand, Whitebox testing is based on the inner
workings of an application and revolves around internal testing.

The term "WhiteBox" was used because of the see-through box concept. The
clear box or WhiteBox name symbolizes the ability to see through the
software's outer shell (or "box") into its inner workings. Likewise, the "black
box" in "Black Box Testing" symbolizes not being able to see the inner
workings of the software so that only the end-user experience can be tested

What do you verify in White Box Testing?

White box testing involves the testing of the software code for the following:

 Internal security holes

 Broken or poorly structured paths in the coding processes
 The flow of specific inputs through the code
 Expected output
 The functionality of conditional loops
 Testing of each statement, object, and function on an individual basis

The testing can be done at system, integration and unit levels of software
development. One of the basic goals of whitebox testing is to verify a working
flow for an application. It involves testing a series of predefined inputs against
expected or desired outputs so that when a specific input does not result in
the expected output, you have encountered a bug.

How do you perform White Box Testing?

To give you a simplified explanation of white box testing, we have divided it
into two basic steps. This is what testers do when testing an application
using the white box testing technique:

STEP 1) UNDERSTAND THE SOURCE CODE

The first thing a tester will often do is learn and understand the source code of
the application. Since white box testing involves the testing of the inner
workings of an application, the tester must be very knowledgeable in the
programming languages used in the applications they are testing. Also, the
testing person must be highly aware of secure coding practices. Security is
often one of the primary objectives of testing software. The tester should be
able to find security issues and prevent attacks from hackers and naive users
who might inject malicious code into the application either knowingly or
unknowingly.

Step 2) CREATE TEST CASES AND EXECUTE

The second basic step to white box testing involves testing the application's
source code for proper flow and structure. One way is by writing more code to
test the application's source code. The tester will develop little tests for each
process or series of processes in the application. This method requires that
the tester must have intimate knowledge of the code and is often done by the
developer. Other methods include Manual Testing, trial, and error testing and
the use of testing tools as we will explain further on in this article.

WhiteBox Testing Example

Consider the following piece of code

Printme (int a, int b) { ------------ Printme is a function

int result = a+ b;
If (result> 0)
Print ("Positive", result)
Else
Print ("Negative", result)
} ----------- End of the source code

The goal of WhiteBox testing in software engineering is to verify all the

decision branches, loops, statements in the code.

To exercise the statements in the above code, WhiteBox test cases would be

 A = 1, B = 1
 A = -1, B = -3
White Box Testing Techniques
A major White box testing technique is Code Coverage analysis. Code
Coverage analysis eliminates gaps in a Test Case suite. It identifies areas of a
program that are not exercised by a set of test cases. Once gaps are
identified, you create test cases to verify untested parts of the code, thereby
increasing the quality of the software product

There are automated tools available to perform Code coverage analysis.

Below are a few coverage analysis techniques a box tester can use:

Statement Coverage:- This technique requires every possible statement in

the code to be tested at least once during the testing process of software
engineering.

Branch Coverage - This technique checks every possible path (if-else and
other conditional loops) of a software application.

Apart from above, there are numerous coverage types such as Condition
Coverage, Multiple Condition Coverage, Path Coverage, Function Coverage
etc. Each technique has its own merits and attempts to test (cover) all parts of
software code. Using Statement and Branch coverage you generally
attain 80-90% code coverage which is sufficient.

Following are important WhiteBox Testing Techniques:

 Statement Coverage
 Decision Coverage
 Branch Coverage
 Condition Coverage
 Multiple Condition Coverage
 Finite State Machine Coverage
 Path Coverage
 Control flow testing
 Data flow testing

Types of White Box Testing

White box testing encompasses several testing types used to evaluate the
usability of an application, block of code or specific software package. There
are listed below --
  Unit Testing: It is often the first type of testing done on an
application. Unit Testing is performed on each unit or block of code as it
is developed. Unit Testing is essentially done by the programmer. As a
software developer, you develop a few lines of code, a single function or
an object and test it to make sure it works before continuing Unit Testing
helps identify a majority of bugs, early in the software development
lifecycle. Bugs identified in this stage are cheaper and easy to fix.

 Testing for Memory Leaks: Memory leaks are leading causes of

slower running applications. A QA specialist who is experienced at
detecting memory leaks is essential in cases where you have a slow
running software application.

Apart from above, a few testing types are part of both black box and white box
testing. They are listed as below

 White Box Penetration Testing: In this testing, the tester/developer

has full information of the application's source code, detailed network
information, IP addresses involved and all server information the
application runs on.  The aim is to attack the code from several angles
to expose security threats
 White Box Mutation Testing: Mutation testing is often used to discover
the best coding techniques to use for expanding a software solution.

White Box Testing Tools

Below is a list of top white box testing tools.

 Parasoft Jtest
 EclEmma
 NUnit
 PyUnit
 HTMLUnit
 CppUnit

Advantages of White Box Testing

 Code optimization by finding hidden errors.
 White box tests cases can be easily automated.
 Testing is more thorough as all code paths are usually covered.
  Testing can start early in SDLC even if GUI is not available.

Disadvantages of WhiteBox Testing

 White box testing can be quite complex and expensive.
 Developers who usually execute white box test cases detest it. The
white box testing by developers is not detailed can lead to production
errors.
 White box testing requires professional resources, with a detailed
understanding of programming and implementation.
 White-box testing is time-consuming, bigger programming applications
take the time to test fully.

Ending Notes:

 White box testing can be quite complex. The complexity involved has a
lot to do with the application being tested. A small application that
performs a single simple operation could be white box tested in few
minutes, while larger programming applications take days, weeks and
even longer to fully test.
 White box testing in software testing should be done on a software
application as it is being developed after it is written and again after
each modification

What is Static Testing? What is a Testing

Review?
What is Static Testing?
Static Testing is a software testing technique which is used to check defects
in software application without executing the code. Static testing is done to
avoid errors at an early stage of development as it is easier to identify the
errors and solve the errors. It also helps finding errors that may not be found
by Dynamic Testing.

Its counterpart is Dynamic Testing which checks an application when the code
is run. Refer to this tutorial for a detailed difference between static and
dynamic testing.
The two main types of static testing techniques are

 Manual examinations: Manual examinations include analysis of code

done manually, also known as REVIEWS.
 Automated analysis using tools: Automated analysis are basically
static analysis which is done using tools.

What is Testing Review?

A review in a Static Testing is a process or meeting conducted to find the
potential defects in the design of any program. Another significance of review
is that all the team members get to know about the progress of the project and
sometimes the diversity of thoughts may result in excellent suggestions.
Documents are directly examined by people and discrepancies are sorted out.

Reviews can further be classified into four parts:

 Informal reviews
 Walkthroughs
 Technical review
 Inspections

During the Review process four types of participants that take part in testing
are:

 Moderator: Performs entry check, follow up on rework, coaching team

member, schedule the meeting.
 Author: Takes responsibility for fixing the defect found and improves
the quality of the document
 Scribe: It does the logging of the defect during a review and attends the
review meeting
 Reviewer: Check material for defects and inspects
 Manager: Decide on the execution of reviews and ensures the review
process objectives are met.

Types of defects which can be easier to find during static testing are:

 Deviations from standards

 Non-maintainable code
 Design defects
 Missing requirements
  Inconsistent interface specifications

Usually, the defect discovered during static testing are due to security
vulnerabilities, undeclared variables, boundary violations, syntax violations,
inconsistent interface, etc

Why Static Testing?

Static testing is performed due to the following reasons

 Early defect detection and correction

 Reduced development timescales
 Reduced testing cost and time
 For improvement of development productivity
 To get fewer defect at a later stage of testing

What is Tested in Static Testing

In Static Testing, following things are tested

 Unit Test Cases

 Business Requirements Document (BRD)
 Use Cases
 System/Functional Requirements
 Prototype
 Prototype Specification Document
 DB Fields Dictionary Spreadsheet
 Test Data
 Traceability Matrix Document
 User Manual/Training Guides/Documentation
 Test Plan Strategy Document/Test Cases
 Automation/Performance Test Scripts

How Static Testing is Performed

To perform Static Testing, it is done in the following ways,

 Carry out the inspection process to completely inspect the design of the
application
  Use a checklist for each document under review to ensure all reviews
are covered completely

The various activities for performing Static Testing are:

1. Use Cases Requirements Validation: It validates that all the end-user

actions are identified, as well as any input and output associated with
them. The more detailed and thorough the use cases are, the more
accurate and comprehensive the test cases can be.
2. Functional Requirements Validation: It ensures that the Functional
Requirements identify all necessary elements. It also looks at the
database functionality, interface listings, and hardware, software, and
network requirements.
3. Architecture Review: All business level process like server locations,
network diagrams, protocol definitions, load balancing, database
accessibility, test equipment, etc.
4. Prototype/Screen Mockup Validation: This stage includes validation
of requirements and use cases.
5. Field Dictionary Validation: Every field in the UI is defined well enough
to create field level validation test cases. Fields are check for min/max
length, list values, error messages, etc.

Static Testing Techniques

 Informal Reviews
 Walkthroughs
 Technical Reviews
 Inspections
 Static Analysis
o Data Flow
o Control Flow
Tools used for Static Testing
Various tools used for Static Testing are as follow,

 Checkstyle
 Soot
 SourceMeter

Tips for Successful Static Testing Process

Some useful tips to perform a static testing process in Software Engineering.
  Focus only on things that really count
 Explicitly plan and track review activities. A software walkthrough and
inspection are generally composite into peer's reviews
 Train participants with Examples
 Resolve people issues
 Keep process formal as the project culture
 Continuous Improvement – Process and Tools
 By removing the major delays in test execution, testing cost and time
can be reduced

Summary:

 Static testing is to find defects as early as possible.

 Static testing not a substitute for dynamic testing, both find a different
type of defects
 Reviews are an effective technique for Static Testing
 Reviews not only help to find defects but also understand missing
requirements, design defects, non-maintainable code.

Mccabe's Cyclomatic Complexity:

Calculate with Flow Graph (Example)
To understand Cyclomatic Complexity, lets first understand -

What is Software Metric?

Measurement is nothing but quantitative indication of size / dimension /
capacity of an attribute of a product / process. Software metric is defined as a
quantitative measure of an attribute a software system possesses with respect
to Cost, Quality, Size and Schedule.

Example-

Measure - No. of Errors

Metrics - No. of Errors found per person

Cyclomatic Complexity in Software Testing

Cyclomatic Complexity in Software Testing is a testing metric used for
measuring the complexity of a software program. It is a quantitative measure
of independent paths in the source code of a software program. Cyclomatic
complexity can be calculated by using control flow graphs or with respect to
functions, modules, methods or classes within a software program.

Independent path is defined as a path that has at least one edge which has
not been traversed before in any other paths.

This metric was developed by Thomas J. McCabe in 1976 and it is based on a

control flow representation of the program. Control flow depicts a program as
a graph which consists of Nodes and Edges.

In the graph, Nodes represent processing tasks while edges represent control
flow between the nodes.

Flow graph notation for a program:

Flow Graph notation for a program defines several nodes connected through
the edges. Below are Flow diagrams for statements like if-else, While, until
and normal sequence of flow.
How to Calculate Cyclomatic Complexity
Mathematical representation:

Mathematically, it is set of independent paths through the graph diagram. The

Code complexity of the program can be defined using the formula -

V(G) = E - N + 2

Where,

E - Number of edges

N - Number of Nodes

V (G) = P + 1

Where P = Number of predicate nodes (node that contains condition)

Example -

i = 0;
n=4; //N-Number of nodes present in the graph
while (i<n-1) do
j = i + 1;

while (j<n) do

if A[i]<A[j] then
swap(A[i], A[j]);

end do;
i=i+1;

end do;

Flow graph for this program will be

Computing mathematically,

 V(G) = 9 - 7 + 2 = 4
 V(G) = 3 + 1 = 4 (Condition nodes are 1,2 and 3 nodes)
 Basis Set - A set of possible execution path of a program
 1, 7
 1, 2, 6, 1, 7
 1, 2, 3, 4, 5, 2, 6, 1, 7
 1, 2, 3, 5, 2, 6, 1, 7

Properties of Cyclomatic complexity:

Following are the properties of Cyclomatic complexity:
 1. V (G) is the maximum number of independent paths in the graph
2. V (G) >=1
3. G will have one path if V (G) = 1
4. Minimize complexity to 10

How this metric is useful for software testing?

Basis Path testing is one of White box technique and it guarantees to execute
atleast one statement during testing. It checks each linearly independent path
through the program, which means number test cases, will be equivalent
to the cyclomatic complexity of the program.

This metric is useful because of properties of Cyclomatic complexity (M) -

1. M can be number of test cases to achieve branch coverage (Upper

Bound)
2. M can be number of paths through the graphs. (Lower Bound)

Consider this example -

If (Condition 1)
Statement 1

Else
Statement 2

If (Condition 2)
Statement 3

Else
Statement 4

Cyclomatic Complexity for this program will be 8-7+2=3.

As complexity has calculated as 3, three test cases are necessary to the

complete path coverage for the above example.

Steps to be followed:
The following steps should be followed for computing Cyclomatic complexity
and test cases design.
 Step 1 - Construction of graph with nodes and edges from the code

Step 2 - Identification of independent paths

Step 3 - Cyclomatic Complexity Calculation

Step 4 - Design of Test Cases

Once the basic set is formed, TEST CASES should be written to execute all
the paths.

More on V (G):
Cyclomatic complexity can be calculated manually if the program is small.
Automated tools need to be used if the program is very complex as this
involves more flow graphs. Based on complexity number, team can conclude
on the actions that need to be taken for measure.

Following table gives overview on the complexity number and corresponding

meaning of v (G):

Complexity Number Meaning

1-10 Structured and well written code

High Testability

Cost and Effort is less

10-20 Complex Code

Medium Testability

Cost and effort is Medium

20-40 Very complex Code

Low Testability
 Cost and Effort are high

>40 Not at all testable

Very high Cost and Effort

Tools for Cyclomatic Complexity calculation:

Many tools are available for determining the complexity of the application.
Some complexity calculation tools are used for specific technologies.
Complexity can be found by the number of decision points in a program. The
decision points are if, for, for-each, while, do, catch, case statements in a
source code.

Examples of tools are

 OCLint - Static code analyzer for C and Related Languages

 Reflector Add In - Code metrics for .NET assemblies
 GMetrics - Find metrics in Java related applications

Uses of Cyclomatic Complexity:

Cyclomatic Complexity can prove to be very helpful in

 Helps developers and testers to determine independent path executions

 Developers can assure that all the paths have been tested atleast once
 Helps us to focus more on the uncovered paths
 Improve code coverage in Software Engineering
 Evaluate the risk associated with the application or program
 Using these metrics early in the cycle reduces more risk of the program

Conclusion:
Cyclomatic Complexity is software metric useful for structured or White Box
Testing. It is mainly used to evaluate complexity of a program. If the decision
points are more, then complexity of the program is more. If program has high
complexity number, then probability of error is high with increased time for
maintenance and trouble shoot.
Code Coverage Tutorial: Branch,
Statement, Decision, FSM

What is Code coverage?

Code coverage is a measure which describes the degree of which the source
code of the program has been tested. It is one form of white box testing which
finds the areas of the program not exercised by a set of test cases. It also
creates some test cases to increase coverage and determining a quantitative
measure of code coverage.

In most cases, code coverage system gathers information about the running
program. It also combines that with source code information to generate a
report about the test suite's code coverage.

Why use Code Coverage?

Here, are some prime reasons for using code coverage:

 It helps you to measure the efficiency of test implementation

 It offers a quantitative measurement.
 It defines the degree to which the source code has been tested.

Code Coverage Methods

Following are major code coverage methods

 Statement Coverage
 Decision Coverage
 Branch Coverage
 Toggle Coverage
 FSM Coverage

Statement Coverage
Statement Coverage is a white box testing technique in which all the
executable statements in the source code are executed at least once. It is
used for calculation of the number of statements in source code which have
been executed. The main purpose of Statement Coverage is to cover all the
possible paths, lines and statements in source code.

Statement coverage is used to derive scenario based upon the structure of

the code under test.

In White Box Testing, the tester is concentrating on how the software works.
In other words, the tester will be concentrating on the internal working of
source code concerning control flow graphs or flow charts.

Generally in any software, if we look at the source code, there will be a wide
variety of elements like operators, functions, looping, exceptional handlers,
etc. Based on the input to the program, some of the code statements may not
be executed. The goal of Statement coverage is to cover all the possible
path's, line, and statement in the code.

Let's understand this with an example, how to calculate statement coverage.

Scenario to calculate Statement Coverage for given source code. Here we are
taking two different scenarios to check the percentage of statement coverage
for each scenario.

Source Code:

Prints (int a, int b) { ------------ Printsum is a function

int result = a+ b;
If (result> 0)
Print ("Positive", result)
Else
Print ("Negative", result)
} ----------- End of the source code

Scenario 1:

If A = 3, B = 9
The statements marked in yellow color are those which are executed as per
the scenario

Number of executed statements = 5, Total number of statements = 7

Statement Coverage: 5/7 = 71%

Likewise we will see scenario 2,

Scenario 2:

If A = -3, B = -9

The statements marked in yellow color are those which are executed as per
the scenario.

Number of executed statements = 6

Total number of statements = 7

Statement Coverage: 6/7 = 85%

But overall if you see, all the statements are being covered by 2nd scenario's
considered. So we can conclude that overall statement coverage is 100%.

What is covered by Statement Coverage?

1. Unused Statements
2. Dead Code
3. Unused Branches
4. Missing Statements

Decision Coverage
Decision Coverage is a white box testing technique which reports the true or
false outcomes of each boolean expression of the source code. The goal of
decision coverage testing is to cover and validate all the accessible source
code by checking and ensuring that each branch of every possible decision
point is executed at least once.

In this coverage, expressions can sometimes get complicated. Therefore, it is

very hard to achieve 100% coverage. That's why there are many different
methods of reporting this metric. All these methods focus on covering the
most important combinations. It is very much similar to decision coverage, but
it offers better sensitivity to control flow.
Example of decision coverage
Consider the following code-

Demo(int a) {
If (a> 5)
a=a*3
Print (a)
}

Scenario 1:

Value of a is 2

The code highlighted in yellow will be executed. Here the "No" outcome of the
decision If (a>5) is checked.

Decision Coverage = 50%

Scenario 2:

Value of a is 6

The code highlighted in yellow will be executed. Here the "Yes" outcome of
the decision If (a>5) is checked.
Decision Coverage = 50%

Test Case Value of A Output Decision Coverage

1 2 2 50%

2 6 18 50%

Branch Coverage
Branch Coverage is a white box testing method in which every outcome from
a code module(statement or loop) is tested. The purpose of branch coverage
is to ensure that each decision condition from every branch is executed at
least once. It helps to measure fractions of independent code segments and
to find out sections having no branches.

For example, if the outcomes are binary, you need to test both True and False
outcomes.

The formula to calculate Branch Coverage:

Example of Branch Coverage

To learn branch coverage, let's consider the same example used earlier

Consider the following code

Demo(int a) {
If (a> 5)
a=a*3
Print (a)
}
Branch Coverage will consider unconditional branch as well

Test Case Value of A Output Decision Coverage Branch Coverage

1 2 2 50% 33%

2 6 18 50% 67%

Advantages of Branch coverage:

Branch coverage Testing offers the following advantages:

 Allows you to validate-all the branches in the code

 Helps you to ensure that no branched lead to any abnormality of the
program's operation
 Branch coverage method removes issues which happen because of
statement coverage testing
 Allows you to find those areas which are not tested by other testing
methods
 It allows you to find a quantitative measure of code coverage
 Branch coverage ignores branches inside the Boolean expressions

Condition Coverage
Condition Coverage or expression coverage is a testing method used to test
and evaluate the variables or sub-expressions in the conditional statement.
The goal of condition coverage is to check individual outcomes for each
logical condition. Condition coverage offers better sensitivity to the control flow
than decision coverage. In this coverage, expressions with logical operands
are only considered.

For example, if an expression has Boolean operations like AND, OR, XOR,
which indicates total possibilities.

Condition coverage does not give a guarantee about full decision coverage.

The formula to calculate Condition Coverage:

Example:

For the above expression, we have 4 possible combinations

 TT
 FF
 TF
 FT

Consider the following input

X=3 (x<y) TRUE Condition Coverage is ¼ = 25%

Y=4

A=3 (a>b) FALSE

B=4

Finite State Machine Coverage

Finite state machine coverage is certainly the most complex type of code
coverage method. This is because it works on the behavior of the design. In
this coverage method, you need to look for how many time-specific states are
visited, transited. It also checks how many sequences are included in a finite
state machine.

Which Type of Code Coverage to Choose

This is certainly the most difficult answer to give. In order to select a coverage
method, the tester needs to check that the

 code under test has single or multiple undiscovered defects

 cost of the potential penalty
 cost of lost reputation
 cost of lost sale, etc.

The higher the probability that defects will cause costly production failures, the
more severe the level of coverage you need to choose.

Code Coverage vs. Functional Coverage

Code Coverage Functional Coverage

Code coverage tells you how well the Functional coverage measures how well the
source code has been exercised by your functionality of the design has been covered
test bench. by your test bench.

Never use a design specification Use design specification

Done by developers Done by Testers

Code Coverage Tools

Here, is a list of Important code coverage Tools:

Tool Name Description

Coco It is an Cross-platform and cross-compiler

code coverage analysis for C, C++,
 SystemC, C#, Tcl and QML code.
Automated measurement of test coverage
of statements, branches and conditions. No
changes to the application are necessary
Learn more about coco

Cobertura It is an open source code coverage tool. It

measures test coverage by instrumenting a
code base and analyze which lines of code
are executing and which are not executed
when the test suite runs.

Clover Clover also reduces testng time by only

running the tests which cover the applicatio
code which was modified since the previous
build.

DevPartner DevPartner enables developers to analyze

Java code for Code Quality and Complexity

Emma EMMA supports class, method, line, and

base block coverage, aggregated source
file, class, and method levels.

Kalistick Kalistick is a third party application which

analyzes the codes with different
perspectives.

CoView and CoAnt Coding Software is a code coverage tool fo

metrics, mock object creation, code
testability, path & branch coverage, etc.

Bullseye for C++ BulseyeCoverage is a code coverage tool

for C++and C.
Sonar Sonar is an open code coverage tool which
helps you to manage code quality.

Advantages of Using Code Coverage

 Helpful to evaluate a quantitative measure of code coverage
 It allows you to create extra test cases to increase coverage
 It allows you to find the areas of a program which is not exercised by a
set of test cases

Disadvantages of Using Code Coverage

 Even when any specific feature is not implemented in design, code
coverage still report 100% coverage.
 It is not possible to determine whether we tested all possible values of a
feature with the help of code coverage
 Code coverage is also not telling how much and how well you have
covered your logic
 In the case when the specified function hasn't implemented, or a not
included from the specification, then structure-based techniques cannot
find that issue.

Summary
 Code coverage is a measure which describes the degree of which the
source code of the program has been tested
 It helps you to measure the efficiency of test implementation
 Five Code Coverage methods are 1.) Statement Coverage 2.) Condition
Coverage 3) Branch Coverage 4) Toggle Coverage 5) FSM Coverage
 Statement coverage involves execution of all the executable statements
in the source code at least once
 Decision coverage reports the true or false outcomes of each Boolean
expression
 In the branch coverage, every outcome from a code module is tested
 Conditional will reveal how the variables or subexpressions in the
conditional statement are evaluated
 Finite state machine coverage is certainly the most complex type of
code coverage method
  In order to select a coverage method, the tester needs to check the cost
of the potential penalty, lost reputation, lost sale, etc.
 Code coverage tells you how well the source code has been exercised
by your test bench while Functional coverage measures how well the
functionality of the design has been covered
 Cobertura, JTest, Clover, Emma, Kalistick are few important code
coverage tools
 Code Coverage allows you to create extra test cases to increase
coverage
 Code Coverage does not help you to determine whether we tested all
possible values of a feature

Path Testing & Basis Path Testing with

EXAMPLES

What is Path Testing?

Path testing is a structural testing method that involves using the source code
of a program in order to find every possible executable path. It helps to
determine all faults lying within a piece of code. This method is designed to
execute all or selected path through a computer program.

Any software program includes, multiple entry and exit points. Testing each of
these points is a challenging as well as time-consuming. In order to reduce
the redundant tests and to achieve maximum test coverage, basis path testing
is used.

Basis Path Testing in Software Engineering

Basis Path Testing in software engineering is a White Box Testing method in
which test cases are defined based on flows or logical paths that can be taken
through the program. The objective of basis path testing is to define the
number of independent paths, so the number of test cases needed can be
defined explicitly to maximize test coverage.

In software engineering, Basis path testing involves execution of all possible

blocks in a program and achieves maximum path coverage with the least
number of test cases. It is a hybrid method of branch testing and path testing
methods.

Here we will take a simple example, to get a better idea what is basis path
testing include

In the above example, we can see there are few conditional statements that is
executed depending on what condition it suffice. Here there are 3 paths or
condition that need to be tested to get the output,

 Path 1: 1,2,3,5,6, 7
 Path 2: 1,2,4,5,6, 7
 Path 3: 1, 6, 7

Steps for Basis Path testing

The basic steps involved in basis path testing include

 Draw a control graph (to determine different program paths)

 Calculate Cyclomatic complexity (metrics to determine the number of
independent paths)
 Find a basis set of paths
 Generate test cases to exercise each path
Advantages of Basic Path Testing
 It helps to reduce the redundant tests
 It focuses attention on program logic
 It helps facilitates analytical versus arbitrary case design
 Test cases which exercise basis set will execute every statement in a
program at least once

Performance Testing Tutorial: What is,

Types, Metrics & Example
Performance Testing
Performance Testing is a software testing process used for testing the
speed, response time, stability, reliability, scalability and resource usage of a
software application under particular workload. The main purpose of
performance testing is to identify and eliminate the performance bottlenecks in
the software application. It is a subset of performance engineering and also
known as “Perf Testing”.

The focus of Performance Testing is checking a software program's

 Speed - Determines whether the application responds quickly

 Scalability - Determines maximum user load the software application
can handle.
 Stability - Determines if the application is stable under varying loads

Why do Performance Testing?

Features and Functionality supported by a software system is not the only

concern. A software application's performance like its response time,
reliability, resource usage and scalability do matter. The goal of Performance
Testing is not to find bugs but to eliminate performance bottlenecks.

Performance Testing is done to provide stakeholders with information about

their application regarding speed, stability, and scalability. More importantly,
Performance Testing uncovers what needs to be improved before the product
goes to market. Without Performance Testing, software is likely to suffer from
issues such as: running slow while several users use it simultaneously,
inconsistencies across different operating systems and poor usability.

Performance testing will determine whether their software meets speed,

scalability and stability requirements under expected workloads. Applications
sent to market with poor performance metrics due to nonexistent or poor
performance testing are likely to gain a bad reputation and fail to meet
expected sales goals.

Also, mission-critical applications like space launch programs or life-saving

medical equipment should be performance tested to ensure that they run for a
long period without deviations.

According to Dunn & Bradstreet, 59% of Fortune 500 companies experience

an estimated 1.6 hours of downtime every week. Considering the average
Fortune 500 company with a minimum of 10,000 employees is paying $56 per
hour, the labor part of downtime costs for such an organization would be
$896,000 weekly, translating into more than $46 million per year.

Only a 5-minute downtime of Google.com (19-Aug-13) is estimated to cost the

search giant as much as $545,000.

It's estimated that companies lost sales worth $1100 per second due to a
recent Amazon Web Service Outage.

Hence, performance testing is important.

Types of Performance Testing

 Load testing - checks the application's ability to perform under
anticipated user loads. The objective is to identify performance
bottlenecks before the software application goes live.
  Stress testing - involves testing an application under extreme
workloads to see how it handles high traffic or data processing. The
objective is to identify the breaking point of an application.
 Endurance testing - is done to make sure the software can handle the
expected load over a long period of time.
 Spike testing - tests the software's reaction to sudden large spikes in
the load generated by users.
 Volume testing - Under Volume Testing large no. of. Data is populated
in a database and the overall software system's behavior is monitored.
The objective is to check software application's performance under
varying database volumes.
 Scalability testing - The objective of scalability testing is to determine
the software application's effectiveness in "scaling up" to support an
increase in user load. It helps plan capacity addition to your software
system.

Common Performance Problems

Most performance problems revolve around speed, response time, load time
and poor scalability. Speed is often one of the most important attributes of an
application. A slow running application will lose potential users. Performance
testing is done to make sure an app runs fast enough to keep a user's
attention and interest. Take a look at the following list of common performance
problems and notice how speed is a common factor in many of them:

 Long Load time - Load time is normally the initial time it takes an
application to start. This should generally be kept to a minimum. While
some applications are impossible to make load in under a minute, Load
time should be kept under a few seconds if possible.
 Poor response time - Response time is the time it takes from when a
user inputs data into the application until the application outputs a
response to that input. Generally, this should be very quick. Again if a
user has to wait too long, they lose interest.
 Poor scalability - A software product suffers from poor scalability when
it cannot handle the expected number of users or when it does not
accommodate a wide enough range of users. Load Testing should be
done to be certain the application can handle the anticipated number of
users.
 Bottlenecking - Bottlenecks are obstructions in a system which
degrade overall system performance. Bottlenecking is when either
 coding errors or hardware issues cause a decrease of throughput under
certain loads. Bottlenecking is often caused by one faulty section of
code. The key to fixing a bottlenecking issue is to find the section of
code that is causing the slowdown and try to fix it there. Bottlenecking is
generally fixed by either fixing poor running processes or adding
additional Hardware. Some common performance bottlenecks are
o CPU utilization
o Memory utilization
o Network utilization
o Operating System limitations
o Disk usage

Performance Testing Process

The methodology adopted for performance testing can vary widely but the
objective for performance tests remain the same. It can help demonstrate that
your software system meets certain pre-defined performance criteria. Or it can
help compare the performance of two software systems. It can also help
identify parts of your software system which degrade its performance.

Below is a generic process on how to perform performance testing

1. Identify your testing environment - Know your physical test

environment, production environment and what testing tools are
available. Understand details of the hardware, software and network
configurations used during testing before you begin the testing process.
It will help testers create more efficient tests.  It will also help identify
possible challenges that testers may encounter during the performance
testing procedures.
2. Identify the performance acceptance criteria - This includes goals
and constraints for throughput, response times and resource allocation. 
It is also necessary to identify project success criteria outside of these
goals and constraints. Testers should be empowered to set
performance criteria and goals because often the project specifications
will not include a wide enough variety of performance benchmarks.
Sometimes there may be none at all. When possible finding a similar
application to compare to is a good way to set performance goals.
 3. Plan & design performance tests - Determine how usage is likely to
vary amongst end users and identify key scenarios to test for all
possible use cases. It is necessary to simulate a variety of end users,
plan performance test data and outline what metrics will be gathered.
4. Configuring the test environment - Prepare the testing environment
before execution. Also, arrange tools and other resources.
5. Implement test design - Create the performance tests according to
your test design.
6. Run the tests - Execute and monitor the tests.
7. Analyze, tune and retest - Consolidate, analyze and share test results.
Then fine tune and test again to see if there is an improvement or
decrease in performance. Since improvements generally grow smaller
with each retest, stop when bottlenecking is caused by the CPU. Then
you may have the consider option of increasing CPU power.

Performance Testing Metrics: Parameters Monitored

The basic parameters monitored during performance testing include:

 Processor Usage - an amount of time processor spends executing

non-idle threads.
 Memory use - amount of physical memory available to processes on a
computer.
 Disk time - amount of time disk is busy executing a read or write
request.
 Bandwidth - shows the bits per second used by a network interface.
 Private bytes - number of bytes a process has allocated that can't be
shared amongst other processes. These are used to measure memory
leaks and usage.
 Committed memory - amount of virtual memory used.
 Memory pages/second - number of pages written to or read from the
disk in order to resolve hard page faults. Hard page faults are when
code not from the current working set is called up from elsewhere and
retrieved from a disk.
 Page faults/second - the overall rate in which fault pages are
processed by the processor. This again occurs when a process requires
code from outside its working set.
 CPU interrupts per second - is the avg. number of hardware interrupts
a processor is receiving and processing each second.
 Disk queue length - is the avg. no. of read and write requests queued
for the selected disk during a sample interval.
 Network output queue length - length of the output packet queue in
packets. Anything more than two means a delay and bottlenecking
needs to be stopped.
 Network bytes total per second - rate which bytes are sent and
received on the interface including framing characters.
 Response time - time from when a user enters a request until the first
character of the response is received.
 Throughput - rate a computer or network receives requests per
second.
 Amount of connection pooling - the number of user requests that are
met by pooled connections. The more requests met by connections in
the pool, the better the performance will be.
 Maximum active sessions - the maximum number of sessions that can
be active at once.
 Hit ratios - This has to do with the number of SQL statements that are
handled by cached data instead of expensive I/O operations. This is a
good place to start for solving bottlenecking issues.
 Hits per second - the no. of hits on a web server during each second of
a load test.
 Rollback segment - the amount of data that can rollback at any point in
time.
 Database locks - locking of tables and databases needs to be
monitored and carefully tuned.
 Top waits - are monitored to determine what wait times can be cut
down when dealing with the how fast data is retrieved from memory
  Thread counts - An applications health can be measured by the no. of
threads that are running and currently active.
 Garbage collection - It has to do with returning unused memory back
to the system. Garbage collection needs to be monitored for efficiency.

Example Performance Test Cases

 Verify response time is not more than 4 secs when 1000 users access
the website simultaneously.
 Verify response time of the Application Under Load is within an
acceptable range when the network connectivity is slow
 Check the maximum number of users that the application can handle
before it crashes.
 Check database execution time when 500 records are read/written
simultaneously.
 Check CPU and memory usage of the application and the database
server under peak load conditions
 Verify response time of the application under low, normal, moderate and
heavy load conditions.

During the actual performance test execution, vague terms like acceptable
range, heavy load, etc. are replaced by concrete numbers. Performance
engineers set these numbers as per business requirements, and the technical
landscape of the application.

Performance Test Tools

There are a wide variety of performance testing tools available in the market.
The tool you choose for testing will depend on many factors such as types of
the protocol supported, license cost, hardware requirements, platform support
etc. Below is a list of popularly used testing tools.

 LoadNinja – is revolutionizing the way we load test. This cloud-based

load testing tool empowers teams to record & instantly playback
comprehensive load tests, without complex dynamic correlation & run
these load tests in real browsers at scale. Teams are able to increase
test coverage. & cut load testing time by over 60%.
 NeoLoad - is the performance testing platform designed for DevOps that
seamlessly integrates into your existing Continuous Delivery pipeline.
With NeoLoad, teams test 10x faster than with traditional tools to meet
 the new level of requirements across the full Agile software
development lifecycle - from component to full system-wide load tests.
 HP LoadRunner - is the most popular performance testing tools on the
market today. This tool is capable of simulating hundreds of thousands
of users, putting applications under real-life loads to determine their
behavior under expected loads. Loadrunner features a virtual user
generator which simulates the actions of live human users.
 Jmeter - one of the leading tools used for load testing of web and
application servers.

FAQ
Which Applications should we Performance Test?
Performance Testing is always done for client-server based systems only.
This means, any application which is not a client-server based architecture,
must not require Performance Testing.

For example, Microsoft Calculator is neither client-server based nor it runs

multiple users; hence it is not a candidate for Performance Testing.

What is the difference between Performance Testing & Performance

Engineering
It is of significance to understand the difference between Performance Testing
and Performance Engineering. An understanding is shared below:

Performance Testing is a discipline concerned with testing and

reporting the current performance of a software application under various
parameters.
Performance engineering is the process by which software is tested and
tuned with the intent of realizing the required performance. This process aims
to optimize the most important application performance trait i.e. user
experience.

Historically, testing and tuning have been distinctly separate and often
competing realms. In the last few years, however, several pockets of testers
and developers have collaborated independently to create tuning teams.
Because these teams have met with significant success, the concept of
coupling performance testing with performance tuning has caught on, and
now we call it performance engineering.

Load Testing Tutorial: What is? How to?

(with Examples)
Load Testing
Load Testing is a non-functional software testing process in which the
performance of software application is tested under a specific expected load.
It determines how the software application behaves while being accessed by
multiple users simultaneously. The goal of Load Testing is to improve
performance bottlenecks and to ensure stability and smooth functioning of
software application before deployment

This testing usually identifies -

 The maximum operating capacity of an application

 Determine whether the current infrastructure is sufficient to run the
application
 Sustainability of application with respect to peak user load
 Number of concurrent users that an application can support, and
scalability to allow more users to access it.

It is a type of non-functional testing. In Software Engineering, Load testing is

commonly used for the Client/Server, Web-based applications - both Intranet
and Internet.
Need of Load Testing:
Some extremely popular sites have suffered serious downtimes when they get
massive traffic volumes. E-commerce websites invest heavily in advertising
campaigns, but not in Load Testing to ensure optimal system performance,
when that marketing brings in traffic.

Consider the following examples

 Popular toy store Toysrus.com, could not handle the increased traffic
generated by their advertising campaign resulting in loss of both
marketing dollars, and potential toy sales.
 An Airline website was not able to handle 10000+ users during a festival
offer.
 Encyclopedia Britannica declared free access to their online database
as a promotional offer. They were not able to keep up with the
onslaught of traffic for weeks.

Many sites suffer delayed load times when they encounter heavy traffic. Few
Facts -

 Most users click away after 8 seconds' delay in loading a page

 $ 4.4 Billion Lost annually due to poor performance

Why Load Testing?

 Load testing gives confidence in the system & its reliability and
performance.
 Load Testing helps identify the bottlenecks in the system under heavy
user stress scenarios before they happen in a production environment.
 Load testing gives excellent protection against poor performance and
accommodates complementary strategies for performance management
and monitoring of a production environment.

Goals of Load Testing:

Loading testing identifies the following problems before moving the application
to market or Production:

 Response time for each transaction

  Performance of System components under various loads
 Performance of Database components under different loads
 Network delay between the client and the server
 Software design issues
 Server configuration issues like a Web server, application server,
database server etc.
 Hardware limitation issues like CPU maximization, memory limitations,
network bottleneck, etc.

Load testing will determine whether the system needs to be fine-tuned or

modification of hardware and software is required to improve performance.

Prerequisites of load testing:

The chief metric for load testing is response time. Before you begin load
testing, you must determine -

 Whether the response time is already measured and compared -

Quantitative
 Whether the response time is applicable to the business process -
Relevant
 Whether the response time is justifiable - Realistic
 Whether the response time is achievable - Achievable
 Whether the response time is measurable using a tool or stopwatch -
Measurable

An environment needs to be set up before starting the load

testing:
Hardware Platform Software Configuration

 Server Machines  Operating System

 Processors  Server Software
 Memory
 Disk Storage
 Load Machines configuration
 Network configuration
Strategies of load Testing:
There are many numbers of ways to perform load testing. Following are a few
load testing strategies-

 Manual Load Testing: This is one of the strategies to execute load

testing, but it does not produce repeatable results, cannot provide
measurable levels of stress on an application and is an impossible
process to coordinate.
 In house developed load testing tools: An organization, which
realizes the importance of load testing, may build their own tools to
execute load tests.
 Open source load testing tools: There are several load testing tools
available as open source that are free of charge. They may not be as
sophisticated as their paid counterparts, but if you are on a budget, they
are the best choice.
 Enterprise-class load testing tools: They usually come with
capture/playback facility. They support a large number of protocols.
They can simulate an exceptionally large number of users.

How to do Load Testing

The load testing process can be briefly described as below -

1. Create a dedicated Test Environment for load testing

2. Determine the following
3. Load Test Scenarios
4. Determine load testing transactions for an application
o Prepare Data for each transaction
o Number of Users accessing the system need to be predicted
o Determine connection speeds. Some users may be connected via
leased lines while others may use dial-up
 o Determine different browsers and operating systems used by the
users 
o A configuration of all the servers like web, application and DB
Servers
5. Test Scenario execution and monitoring. Collecting various metrics
6. Analyze the results. Make recommendations
7. Fine-tune the System
8. Re-test

Guidelines for load testing

1. Load testing should be planned once the application becomes

functionally stable.
2. A large number of unique data should be ready in the data pool
3. Number of users should be decided for each scenario or scripts
4. Avoid creation of detailed logs to conserve the disk IO space
5. Try to avoid downloading of images in the site
6. The consistency of response time over the elapsed period should be
logged & the same should be compared with various test runs

Difference between Load and Stress testing:

Load Testing Stress Testing

Load testing identifies the bottlenecks in the Stress Testing determines the breaking
system under various workloads and checks point of the system to reveal the maximum
how the system reacts when the load is point after which it breaks.
gradually increased
 Difference between Functional and Load Testing:
Functional Testing Load Testing

Results of functional tests are easily Results of load tests are unpredictable
predictable as we have proper steps and
preconditions defined

Results of functional tests vary slightly Load test results vary drastically

Frequency of executing Functional A frequency of executing load testing will be

Testing will be high low

Results of functional tests are dependent on Load testing depends on the number of
the test data users.

Load Testing Tools:

LoadNinja
LoadNinja – is revolutionizing the way we load test. This cloud-based load
testing tool empowers teams to record & instantly playback comprehensive
load tests, without complex dynamic correlation & run these load tests in real
browsers at scale. Teams are able to increase test coverage. & cut load
testing time by over 60%.

NeoLoad:
NeoLoad is the enterprise-grade load testing platform designed for Agile and
DevOps. NeoLoad integrates with your continuous delivery pipeline to support
performance testing across the complete software lifecycle - from component
to full system-wide load tests.

Load Runner:
Load runner is HP tool used to test the applications under normal and peak
load conditions. Load runner generates load by creating virtual users that
emulate network traffic. It simulates real time usage like a production
environment and gives graphical results.
Advantages and disadvantages of Load testing:
Following are the advantages of Load testing:

 Performance bottlenecks identification before production

 Improves the scalability of the system
 Minimize risk related to system downtime
 Reduced costs of failure
 Increase customer satisfaction

Disadvantages of Load testing:

 Need programming knowledge to use load testing tools.

 Tools can be expensive as pricing depends on the number of virtual
users supported.

What is STRESS Testing in Software

Testing? Tools, Types, Examples
Stress Testing
Stress Testing is a type of software testing that verifies stability & reliability of
software application. The goal of Stress testing is measuring software on its
robustness and error handling capabilities under extremely heavy load
conditions and ensuring that software doesn't crash under crunch situations. It
even tests beyond normal operating points and evaluates how software works
under extreme conditions.

In Software Engineering, Stress Testing is also known as Endurance Testing.

Under Stress Testing, AUT is be stressed for a short period of time to know its
withstanding capacity. A most prominent use of stress testing is to
determine the limit, at which the system or software or hardware breaks.
It also checks whether the system demonstrates effective error management
under extreme conditions.

The application under testing will be stressed when 5GB data is copied from
the website and pasted in notepad. Notepad is under stress and gives 'Not
Responded' error message.

Need for Stress Testing

Consider the following scenarios - 

 During festival time, an online shopping site may witness a spike in

traffic, or when it announces a sale.
 When a blog is mentioned in a leading newspaper, it experiences a
sudden surge in traffic.

It is imperative to perform Stress Testing to accommodate such abnormal

traffic spikes. Failure to accommodate this sudden traffic may result in loss of
revenue and repute.

Stress testing is also extremely valuable for the following reasons:

 To check whether the system works under abnormal conditions.

 Displaying appropriate error message when the system is under stress.
 System failure under extreme conditions could result in enormous
revenue loss
 It is better to be prepared for extreme conditions by executing Stress
Testing.
Goals of Stress Testing
The goal of stress testing is to analyze the behavior of the system after a
failure. For stress testing to be successful, a system should display an
appropriate error message while it is under extreme conditions.

To conduct Stress Testing, sometimes, massive data sets may be used which
may get lost during Stress Testing. Testers should not lose this security-
related data while doing stress testing.

The main purpose of stress testing is to make sure that the system recovers
after failure which is called as recoverability.

Load Testing Vs Stress Testing

 

Load Testing Stress Testing

 Load Testing is to test the system behavior Stress testing is to test the system behavior
under normal workload conditions, and it is under extreme conditions and is carried out
just testing or simulating with the actual till the system failure.
workload

Load testing does not break the system stress testing tries to break the system by
testing with overwhelming data or
resources.

Types of Stress Testing:

Following are the types of stress testing and are explained as follows:

Distributed Stress Testing:

In distributed client-server systems, testing is done across all clients from the
server. The role of stress server is to distribute a set of stress tests to all
stress clients and track on the status of the client. After the client contacts the
server, the server adds the name of the client and starts sending data for
testing.

Meanwhile, client machines send signal or heartbeat that it is connected with

the server. If the server does not receive any signals from the client machine,
it needs to be investigated further for debugging. From the figure, a server can
connect with the 2 clients (Client1 and Client2), but it cannot send or receive a
signal from Client 3 & 4.

Night run is the best option to run these stress testing scenarios. Large server
farms need a more efficient method for determining which computers have
had stress failures that need to be investigated.

Application Stress Testing:

This testing concentrate on finding defects related to data locking and
blocking, network issues and performance bottlenecks in an application.

Transactional Stress Testing:

It does stress testing on one or more transactions between two or more
applications. It is used for fine-tuning & optimizing the system.

Systemic Stress Testing:

This is integrated stress testing which can be tested across multiple systems
running on the same server. It is used to find defects where one application
data blocks another application.

Exploratory Stress Testing:

This is one of the types of stress testing which is used to test the system with
unusual parameters or conditions that are unlikely to occur in a real scenario.
It is used to find defects in unexpected scenarios like

1. A large number of users logged at the same time

2. If a virus scanner started in all machines simultaneously
3. If Database has gone offline when it is accessed from a website,
4. When a large volume of data is inserted to the database simultaneously
How to do Stress Testing?
Stress Testing process can be done in 5 major steps:

1. Planning the Stress Test. Here you gather the system data, analyze the
system, define the stress test goals
2. Create Automation Scripts: In this phase, you create the Stress testing
automation scripts, generate the test data for the stress scenarios.
3. Script Execution: In this stage, you run the Stress testing automation
scripts and store the stress results.
4. Results Analysis: In this stage, you analyze the Stress Test results and
identify bottlenecks.
5. Tweaking and Optimization: In this stage, you fine-tune the system,
change configurations, optimize the code with goal meet the desired
benchmark.

Lastly, you again run the entire cycle to determine that the tweaks have
produced the desired results. For example, it's not unusual to have to 3 to 4
cycles of the Stress Testing process to achieve the performance goals

Tools recommended for Stress Testing:

LoadRunner
LoadRunner from HP is a widely-used Load Testing tool. Load Test Results
shaped by Loadrunner are considered as a benchmark.

Jmeter
Jmeter is an Open Source testing tool. It is a pure Java application for stress
and Performance Testing. Jmeter is intended to cover types of tests like load,
functional, stress, etc. It needs JDK 5 or higher to function.

Stress Tester
This tool provides extensive analysis of the web application performance,
provides results in graphical format, and it is extremely easy to use. No high-
level scripting is required and gives a good return on investment.
Neo load
This is a popular tool available in the market to test the web
and Mobile applications. This tool can simulate thousands of users in order to
evaluate the application performance under load and analyze the response
times. It also supports Cloud-integrated - performance, load and stress
testing. It is easy to use, cost-effective and provides good scalability.

Metrics for Stress Testing

Metrics help in evaluating a System's performance and generally studied at
the end of Stress Test. Commonly used metrics are -

Measuring Scalability & Performance

 Pages per Second: Measures how many pages have been requested /
Second
 Throughput: Basic Metric - Response data size/Second
 Rounds: Number of times test scenarios have been planned Versus
Number of times a client has executed

Application Response

 Hit time: Average time to retrieve an image or a page

 Time to the first byte: Time is taken to return the first byte of data or
information
 Page Time: Time is taken to retrieve all the information in a page

Failures

 Failed Connections: Number of failed connections refused by the client

(Weak Signal)
 Failed Rounds: Number of rounds it gets failed
 Failed Hits: Number of failed attempts done by the system (Broken links
or unseen images)

What is Volume Testing? Learn with

Examples
What is Volume Testing?
VOLUME TESTING is a type of Software Testing, where the software is
subjected to a huge volume of data. It is also referred to as flood
testing. Volume testing is done to analyze the system performance by
increasing the volume of data in the database.

With the help of Volume testing, the impact on response time and system
behavior can be studied when exposed to a high volume of data.

For example, testing the music site behavior when there are millions of user to
download the song.

Benefits of Volume Testing

 By identifying load issues, a lot of money can be saved which otherwise
will be spent on application maintenance.
 It helps in a quicker start for scalability plans
 Early identification of bottlenecks
 It assures your system is now capable of real-world usage

Why to do Volume Testing

The objective of performing the volume testing is to

 Check system performance with increasing volumes of data in the

database
 To identify the problem that are likely to occur with large amount of data
 To figure out the point at which the stability of the system degrades
 Volume Testing will help to identify the capacity of the system or
application - normal and heavy volume

How to do Volume Testing

In volume testing, following things need to be tested

 Test to check if there is any data loss

 Check the system's response time
 Check if the data is stored correctly or not
 Verify if the data is overwritten without any notification
  Check for warning and error messages, whether it comes at all for
volume problems
 Check whether high volume data affects the speed of processing
 Does system have the necessary memory resources
 Does volume test executed on the whole system
 Is there any risk if data volume is greater than specified
 Is there any guarantee that no larger date volume will occur than
specified

Best practices for high volume testing

The best practices for high-volume testing are

 Stop all servers and check all logs

 Before the load test manually execute the application scenario
 For most useful results stagger the number of users
 To overcome license constraints, balance think time
 Be cautious with the new build
 Analyze the use case for improvement once a baseline has been
established
 A repetition of particular parts of volume testing becomes inevitable in
case there is a performance bottleneck

Volume Testing Vs Load Testing

Volume Testing Load Testing

 Volume testing is testing of an  While in load testing, application is

application with large number of data subjected to certain level of load to
in database is possible analyze the behavior of the application

 Volume testing verifies if the system  Load testing checks the performance
responds as expected for a certain of the system when the user load is
volume of data. It may include increased. It may include increasing
increasing size of the file number of the file
Challenges in Volume Testing
 Fragmentation of memory difficult to generate
 Dynamic generation of keys
 Relational Integrity of generated data

Summary:

 In Software Engineering, Volume testing is a way to check system

performance for high data loads
 It is a non-functional testing
 Fragmentation of memory difficult to generate and relational Integrity of
generated data

are few challenges of Volume Testing

What is Scalability Testing? Learn with

Example
Scalability Testing
Scalability Testing is a non functional testing method that measures
performance of a system or network when the number of user requests are
scaled up or down. The purpose of Scalability testing is to ensure that the
system can handle projected increase in user traffic, data volume, transaction
counts frequency, etc. It tests system ability to meet the growing needs.

It is also referred to as performance testing, as such, it is focused on the

behavior of the application when deployed to a larger system or tested under
excess load. In Software Engineering, Scalability Testing is to measure at
what point the application stops scaling and identify the reason behind it.

Why do Scalability Testing

 Scalability testing lets you determine how your application scales with
increasing workload.
 Determine the user limit for the Web application.
 Determine client-side degradation and end user experience under load.
 Determine server-side robustness and degradation.
What to test in Scalability Testing
Here are few Scalability Testing Attributes:

 Response Time
 Screen transition
 Throughput
 Time (Session time, reboot time, printing time, transaction time, task
execution time)
 Performance measurement with a number of users
 Request per seconds, Transaction per seconds, Hits per second
 Performance measurement with a number of users
 Network Usage
 CPU / Memory Usage
 Web Server ( request and response per seconds)
 Performance measurement under load

Test Strategy for Scalability testing

Test Strategy for Scalability Testing differ in terms of the type of application is
being tested. If an application accesses a database, testing parameters will be
testing the size of the database in relation to the number of users and so on.
Prerequisites for Scalability Testing
 Load Distribution Capability- Check whether the load test tool
enables the load to be generated from multiple machines and controlled
from a central point.
 Operating System- Check what operating systems do the load
generation agents and load test master run under
 Processor- Check what type of CPU is required for the virtual user
agent and load test master
 Memory- Check how much memory would be enough for the virtual
user agent and load test master

How to do Scalability Testing

1. Define a process that is repeatable for executing scalability tests
throughout the application life-cycle
2. Determine the criteria for scalability
3. Shortlist the software tools required to run the load test
4. Set the testing environment and configure the hardware required to
execute scalability tests
5. Plan the test scenarios as well as Scalability Tests
6. Create and verify visual script
7. Create and verify the load test scenarios
8. Execute the tests
9. Evaluate the results
10. Generate required reports

Scalability Test Plan

Before you actually create the tests, develop a detailed test plan. It is an
important step to ensure that the test conforms as per the application
requirement.

Following are the attributes for creating a well-defined Test Plan for Scalability

Testing.

 Steps for Scripts: The test script should have a detailed step that
determines the exact actions a user would perform.
 Run-Time Data: The test plan should determine any run-time data that
is required to interact with the application
  Data Driven Tests: If the scripts need varying data at run-time, you
need to have an understanding of all the fields that require this data.

Scalability testing Vs Load testing

Scalability Testing Load Testing

 It focusses on the performance of your  Load testing focusses on testing an

websites, software, hardware, and application under heavy loads, to
application when changes are done in determine at what point the system
the size or volume of the system to meet response time fails.
a growing need.

What is Soak Testing? Definition,

Meaning, Examples
Soak Testing
Soak Testing is a type of non functional testing that is used to measure
performance of a software application under a huge volume of load for an
extended period of time. The goal of Soak testing is to ensure whether the
software application sustains high volume of usage and to check what would
happen outside its design expectations.

The image below depicts a testing cycle that shows at which stage the Soak
Testing (Type of Performance Test) is performed on an application.
In this type of testing, what basically monitored is the memory utilization by an
application in a system. It is testing at a system level, to find whether the
system will stand up to a very high volume of usage and to see what would
happen outside its design expectations

Why do Soak Testing?

A system may behave normally when used for 2 hours, but when the same
system is used continuously for 10 hours or more than that then it may fail or
behave abnormally/randomly/it may crash. To predict such failure Soak
Testing is performed.
When to do Soak Testing?
Soak Testing should be done in the following scenarios: -

1. Before the built is deployed to the client i.e. prior to the release of any
application on a specific platform, it needs to go through a successful
series of load tests at high or equivalent traffic levels. After that soak
testing is performed. It helps us to determine how to run any particular
application for an extended period. If issues like memory leaks/memory
corruption are found during the period i.e. when it is on Soak, then it
should be immediately reported.
2. The best time to do a soak testing is over the weekends as an
application need to be in a running state for as long as over a day or
night. It totally depends on the limitations of the testing situation. Soak
tests are one of the most important compliance requirements which
need to be very strictly followed by every company.

Soak Testing Strategy

Long Session Soak Testing is a strategy where a system is under load
for a longer time.

A simple example is where the user stays logged into a system for many
hours executing a number of business transactions. In this way, a lot of data
gets created. There can be lots of load on the system/database server which
can result in stalling/crashing of the system/database server.

Under Long Session Soak Testing, multiple day's (say 30 days) activities are
performed in a restrained time frame (say 2 days). The number of transaction
in this restrained timeframe should match or surpass multiple days' worth of
transactions. The focus should be on the number of transactions processed.
The most important part of Soak Testing is to check the available memory in
the CPU and the amount of the memory that will be in usage. We need to
record the memory usage at the start and the end of a soak test. If necessary,
then the memory usage of facilities such as Java Virtual Machines are also
important and needs to be monitored.

Below are few more checks that need to be done by any user/tester before
they begin with Soak Testing:

a) Monitor the database resource consumption.

b) Monitor the server resource consumption (ex- CPU usage).

c) Soak test should run with realistic user concurrency.

Characteristics of Soak Testing

A standard Soak Testing Method should have the following characteristics: -

 The duration of most Soak Test is often determined by the time

available.
 Any application must run without any interruption if it requires an
extended period of time.
 It should cover all the scenarios that are agreed upon by the
stakeholders.
 Mostly every system has a regular maintenance window time period and
the time between such window periods is a key driver for determining
the scope of a Soak Test.

EXAMPLES of Soak Testing

 In the case of banking domain when there is a large amount of data
from merchants, the tester will put the system under load continuously
for 70hrs to 150 hrs to check how the application behaves during this
loading period.
 Suppose there are 33,000 logins, which needs to be put through the
system, it represents seven and a half days of activity. In this case, a
60- 70 hours of Soak Test can be started by Friday evening around 6
pm which can be completed by Monday morning at 6 am. Only with
such a test, it will be possible to observe any degradation of
performance under the controlled conditions.
 In the case of Video games, Mobile applications, etc. involve leaving the
game or application in a running state for a prolonged time period, in
various modes of operation- such as idling, paused at the title screen
and so on to find out whether an application can handle the
continuously expected load.

Common issues observed during Soak Testing

1. Memory allocation (memory leaks that would eventually result in a
memory crisis or rounding errors that manifest only over time).
 2. Database resource utilization (Failure to close database cursors under
some conditions which would eventually result in the entire system
stalling).
3. It can also lead to performance degradation, i.e. to ensure that the
response time after a long period of sustained activity is as good as it
was at the beginning of the test.
4. Failure to close connections between tiers of a multi-tiered system
under some circumstances which could stall some or all modules of the
system.
5. The gradual degradation of a response time of some functions as
internal data-structures become less efficient during a long test.

Summary
 In Software Engineering, Soak testing is done to determine if the
application under test can sustain the continuous load.
 It is a type of performance test.
 It helps the system to determine whether it will stand up to a very high
volume of usage
 In this type of testing, what basically monitored is the memory utilization
by an application in a system
 Checks that need to be done by any user/tester before they begin with
Soak Testing include
o Monitor the database resource consumption.
o Monitor the server resource consumption (ex- CPU usage).
o Soak test should run with realistic user concurrency.

Stability Testing in Software Testing

Stability Testing
Stability Testing is a type of non functional software testing performed to
measure efficiency and ability of a software application to continuously
function over a long period of time. The purpose of Stability testing is checking
if the software application crashes or fails over normal use at any point of time
by exercising its full range of use.

Stability Testing is done to check the efficiency of a developed product

beyond normal operational capacity, often to a breakpoint. There is greater
significance is on error handling, software reliability, robustness and scalability
of a product under heavy load rather than checking the system behavior under
normal circumstances.

Stability testing assesses stability problems. This testing is primarily intended

to stress the software component to the maximum. It is a Non-Functional
Technique.

Stability testing is also referred to as a Load or endurance testing

Problems if a System under test has not undergone

Stability Test
For an application under test where a large number of users are introduced
and applications that have to run for months without restarting, a number of
problems are likely to occur:

The possible error can be faced,

 the system slows down

 the system encounters functionality problems
 the system shows wired behavior
 the system crashes altogether

In Software Engineering, Stability Testing typically involves exercising the

system with heavy users (virtual) and measuring the performance parameters
to verify whether the system can support the anticipated load.

Why do Stability Testing

This kind of testing helps users to understand the ways the system will work in
real-life situations.
Hence, Stability Testing allows you to check,

 Provide confidence in the stability of your system under test.

 Ensure that your system can handle large programs.
 Monitor the effectiveness of your system.
 Test system stability under stress.

It plays an important role in product development as it is used to determine

the limitations of a software product under test before it is released or the
areas of more improvement before the product goes live or at Production.

A very common example of Stability testing technique is

Online Shopping Portals: Stability testing will check how the website will

behave when -

 High Amount of data entered at peak time

 Number of hits at a certain specific time
 Page load issue at the same time
 Behavior of system
 Responsiveness of the system and many more come under Stability
Testing

Another example

A CPU test is a popular form of stability test under Performance

Testing technique. This test checks for processor stability and also monitors
its performance as the processor's workload is increased.

How to do Stability Testing

 To determine the scope and objective of the testing, we must ensure
that the Application Server(s) do not crash during the Load Test
executions.
 To determine the Business issues, verify the system performance and
load as per end user perspective.
 To assign the different Responsibilities and Roles like -Creating Test
plan, Test Case design, Test case review, Test execution, etc.
 To ensure the Test deliverables within the specified time
 To ensure proper Load Testing tools and experience team is present for
the same.
  To measure the risk and cost involves in the testing. This will determine
the cost of each execution in terms of CPU utilization and memory.
 Determine the Defect tracking and reporting and there proper mapping
with the requirements.

Test Case for Stability Testing for CPU Performance

 To verify the Upper limit of the system.
 How system crashes or recovers.
 A total number of transactions completed per request.
 Whether or not transaction response stays steady or increases over
time.
 How the system behaves under heavy load.
 Its response and behavior under heavy load.

Test Reports for Stability Testing

Several statistics are gathered and measured during test executions; these
numbers are analyzed in order to generate a report and to identify possible
performance problems.

Examples of statistics collected under test are:

 Transaction Response Times: The average time is taken to perform

transactions during the test. This statistic will evaluate whether the
performance of the server is within the acceptable minimum and
maximum transaction performance time periods defined for the system.

This information will evaluate the time taken in processing the request
by the web server and sent to the application server, which in most of
the cases will make a request to a database server.

 Hits Per Second: The number of hits made on the server by users.

This statistics benefits to determine the number of load users generate,
with respect to a number of hits.
 Throughput: The amount of throughput on the Web server during the
test which is measured in bytes. Throughput means the amount of data
that the users received from the server at any given time. This statistic
helps to evaluate the amount of load that users generate.
  Transaction per second: These are the total number of completed
transactions (both successful and failed) performed during a test. This
statistic helps to check the actual transaction load on the system.
 CPU: CPU percentage utilization spent during a test.
 Memory: Memory usage during a test.
 Disk: utilization of disk spaces spent during a test.

By the time, you can now easily identify that –

Stability Testing comes under Performance Testing –a technique that is

performed to check some of the quality attributes of Software like stability,
reliability, and availability.

This testing is used to determine how fast a system or sub-system performs

under a particular workload.

Performance Testing has many types and stability testing is one of them.

 Stress testing: It is a testing type which checks the robustness of the

system beyond the system capacity.
 Spike testing: It is used to check the behavior of a system by
increasing a load of a system instantly. The objective is to check at
which point the system will be having performance issues, or it will pass.
 Scalability testing: It is used to check the capabilities of a system. How
effective the system will going to behave in growing needs, change in
size and change in volume.
 Volume testing: It is a non-functional testing technique where software
under test is subjected to a huge volume of data, and the behavior of a
system is checked and verified accordingly.
 Load or Stability testing: (already discussed above)

Some of the tools for Performance Testing are as –

 WebLOAD
 LoadRunner
 Apache JMeter
 NeoLoad
 CloudTest
 Loadstorm
 LoadUI
 WAPT
  LoadImpact
 Loadster
 Httperf
 OpenSTA
 Performance Tester
 Testing Anywhere

What is Spike Testing? Learn With

Example
Spike Testing
Spike Testing is a performance testing type used to test software
applications with extreme increments and decrements in load. The main
purpose of spike testing is to evaluate the behavior of software applications
under sudden increment or decrement in user load and determine recovery
time after a spike of user load. It is performed to estimate weaknesses of
software applications.
Goal of Spike Testing
The goal of Spike testing is to see how the system responds to unexpected
rise and fall of the user load. In Software Engineering Spike testing helps
determine system performance will deterioration when there is a sudden high
load.

Another goal of Spike Testing is to determine the recovery time. Between two
successive spikes of user load, the system needs some time to stabilize. This
recovery time should be as low as possible.

How to do Spike Testing?

Here are six simple steps to perform spike testing

Step 1) Determine the Max User Load capacity of your software application

Step 2) Prepare the Testing Environment and configure it to record

performance parameters.

Step 3) Apply expected maximum load to your Software Application using a

Performance Tool of your choice

Step 4) Rapidly increase in load, to the system for a set period.

Step 5) Gradually reduce the load back to its original level.

Step 6) Analyze the performance graphs. Metric to be considered are

Failures, Time Taken, Virtual Users, etc.

Example Spike Testing Scenarios:

 When an eCommerce store is launching special deals with great
discounts such as on Black Friday.
 When a web application is live streaming a favorite TV program.
 When a flash sale is going on a daily deal site.
 When the certain content of a site goes viral over the Internet.
 A new system is released for production, and multiple users want to
access the system.
  A power outage may cause all users to lose access to a system. After
the outage issue resolved all users then log back onto the system
simultaneously.

Recovery Scenarios on Spike Loads

Three main recovery scenarios can be configured to guard against spikes

1. Use cloud platforms like AWS, Azure to dynamically increase server

capacity in tandem with the user load
2. Do not allow the application-access to some users, so that system does
not face heavy load. This stops people above the maximum designed
load from entering into the system. Thus protects the system from the
threat of an excessive load.
3. The site admin allows users to join the system. However with warning
that they may face slow response because of heavy load. This may
result in the adverse effect on the system performance. However, the
user will be able to work with the system.

Advantages and Disadvantages of Spike testing

Advantages Disadvantages

 The performance of the software must  The only disadvantage of Spike

be maintained at any costs. However, Testing is that it is an expensive
when there is an extreme increase in testing process. So, it needed setting
the load of any system, there are high up special test conditions. However,
chances of the issues. Spike Testing over the longer duration, it will surely
helps test such scenario. give a positive ROI.

 In the standard testing method, the  

bad to worst-case scenarios may not
be addressed. However, ignoring
them does not mean that they will
never occur. Therefore, every
software should ready for such
possibilities. One such worst case
scenario is loading which can be
judged and minimized with the help of
 spike testing.

Tools for Spike Testing:

1) Jmeter

The Apache JMeter is a java open source spike testing tool. It is specially
designed to load functional test behavior and measure performance. This
performance testing tool can be used to analyze and measure the
performance of web application or a variety of services. Today, it is widely
used for the functional test, database server test.

2) Loadrunner

Loadrunner is a load testing tool for Windows and Linux, which allows spike
testing of web and other apps. It helps to determine the performance and
result of the application even under the heavy load.

Load Testing vs Stress Testing vs

Performance Testing: Difference
Discussed
hat is Performance testing?
Performance testing is a type of testing for determining the speed of a
computer, network or device. It checks the performance of the components of
a system by passing different parameters in different load scenarios.

What is Load testing?

Load testing is the process that simulates actual user load on any application
or website. It checks how the application behaves during normal and high
loads. This type of testing is applied when a development project nears to its
completion.
What is Stress testing?
Stress testing is a type of testing that determines the stability and robustness
of the system. It is a non-functional testing technique. This testing technique
uses auto-generated simulation model that checks all the hypothetical
scenarios.

Performance Testing vs. Load Testing vs. Stress

Testing
Performance testing Load testing Stress tes
Superset of load and stress testing. A subset of performance testing. A subset o
Helps to set the benchmark and To recognize the upper limit of the To check o
standards for the application. system, set SLA of the app and check behaves u
how the system can handle a heavy how it reco
load.
The aim of performance testing is to get Generating increased load on a web Stress test
an indication of how an application application is the main aim of load under a su
behaves under regular parameters. testing. considerab
don't crash
Resource usage, availability, and The attributes which are checked in a This kind o
reliability of the product are validated load test are peak performance, response t
under this testing. server quantity and response time.
In performance testing, Load limit is In load testing load limit is a threshold In stress te
both below and above the threshold of a of a break. the thresho
break.
Example of performance testing are : Example of Load testing are Test of a
Checking concurrent users, HTTP word processor by make change in
connections or checking Suitable the large volume of data,
response time.
test a printer by transferring heavy
data.

Check mail server with thousands of

concurrent users.
When do you use Performance Testing?
Performance testing is done to check the performance of website servers,
database, and network. If you are adopting waterfall methodology, then it is
important for you to check each time release of a new version. However, if
you are using the agile software development approach, then you need to test
application continuously.

When do you use Load testing?

Load testing is performed to determine how many users the system can
handle. You can also check out different scenarios that let you focus on
different parts of your system. Like the home page or checkout webpage on
your website for web load testing. It also helps you to determine how the load
builds and sustains in the system.

When do you Use Stress Testing?

Website and app stress tests are significant before big events such as ticket
selling for a famous concert with high demand of the people. So it is very
important to test regularly to the capacity of the system endurance. This also
helps you to prepare for unexpected traffic spikes and gives more time and
resources to fix any bottlenecks.

Conclusion

What is Response Time Testing?

What is Response Time Testing?
RESPONSE TIME TESTING measures the time taken for one system node to
respond to the request of another. It is the time a system takes to reach a
specific input until the process is over. For example, you have API, and you
want to know exactly how much time it takes to execute it and return data in
JSON. Response Time measures the server response of every single
transaction or query.

Response time starts when a user sends a request and ends at the time that
the application states that the request has completed.
How to Measure Response Time?
To Measure Response Time, we can use test tools by surrounding an
important business process with Start and End transactions. A business
process may be an action or a set of actions which users perform in
application to complete a business task like login to the application or to
purchase a book on Amazon.com.

For the same process, the response will vary slightly from tool to tool. Here is
why

 Method of calculating metrics gathered by each tool

 Tools Simulate the load and capture speed which can make a difference
in the response time
 Extra items recorded when monitoring user loads
 Computing metrics gathered by each tool increases the response time
due to high resource consumption.
 The architecture of the two tools may be different

Types of Response Metrics:

Response Time Metrics Explanation

Average Response Time The Average Response Time is the averag

time taken for every round trip request. The
 Average Response Time includes the
loading time of HTML, CSS, XML, images,
JavaScript files, etc. Therefore, the average
is affected when slow components are
present in the system.

Peak Response Time The Peak Response Time helps us to find

potentially problematic components. It helps
us to find all the irregularity in the website o
system where a certain request is not
handled correctly. For example, there may
be a large database query executed which
can affect the response time. This query
does not permit the page to load at the
desired time.

Error Rate The Error Rate is a mathematical calculatio

which displays the percentage of problem
requests against all requests. This
percentage counts all HTTP status codes
displaying an error on the server. It also
counts requests which are time out.

Three important response time values:

Response time testing has two most essential characteristic:

 Average response time

 Maximum response time.

It shows how long a user needs to wait for the server to the response its
request.

Following are key response time values

Response Time Significance

 0.1 Second It is most preferred response time. If the
response time is 0.1, users always feel that
the application or system is responding
instantly, and do not feel any interruption.

1.0 Second It is the defined as the maximum limit of

acceptable response time. Users are
unlikely to feel any interruption, though they
may experience some delay. The response
time of more than 1-second may interrupt
user experience.

10 Seconds It is a maximum limit after which response

time goes beyond the acceptable limit.
However, in today's time, if response time
exceeds 6 seconds, the user will leave that
site or quit the application.

Generally, response time should be fast as possible in the interval of 0.1 - 1

second. However, people can adapt a slower response times, but they will
never be happy with a response time greater than 2 seconds. Lesser the
response times, better is client satisfaction, lesser costs, higher customer
satisfaction.

Response Time testing Tools:

There are many Response Time testing tools are available in the market.
Three most prominent names are:

1) JMeter:

Jmeter can be used for load and performance testing on the target
application.

Download link: http://jmeter.apache.org/download_jmeter.cgi

2) Load Runner:
Load Runner is a load testing product developed by Microfocus. LoadRunner
response testing tool works on the principle of simulating Virtual Users on the
subject application.

Download link: https://software.microfocus.com/de-de/products/loadrunner-
load-testing/free-trial

3) AEM:

Adobe Experience manage which is shortly known as AEM is another

effective tool for testing response time. It allows checking problematic queries,
request and error message.

Download link: https://helpx.adobe.com/in/experience-manager/6-
3/sites/developing/using/aem-eclipse.html

What is Benchmark Testing? Test Plan,

Tools, Example
Before we learn Benchmark Testing, let's understand-

Benchmark in Performance Testing

A Benchmark in Performance Testing is a metric or a point of reference
against which software products or services can be compared to assess the
quality measures. In other words, Benchmark means a set standard that helps
to determine the quality of software product or service. We can benchmark a
software product or service to assess its quality.

For example, a football team has a benchmark for a player to qualify as

a premier league player. The skill set for the benchmark could be a number
of goals scored in the last five seasons, or ball possession during the first half
time, or minimum games played in the local tournament, etc.

What is Benchmark Testing?

BENCHMARK TESTING measures a repeatable set of quantifiable results
that serves as a point of reference against which products/services can be
compared. The purpose of benchmark testing results is to compare the
present and future software releases with their respective benchmarks.
A benchmark must be repeatable. For instance, with every iteration of load a
test, if the response times varies too much, system performance be
benchmarked. Response time needs to be stable amongst different load
conditions.

A benchmark must be quantifiable. For example, the user experience cannot

be quantified in numbers, but time a user spends on a webpage due to good
UI can be quantified.

Benchmark Testing is not a term related to just software testing, but it

also deals with Hardware Testing and considered as one of the most
important tasks in the business world.

In this tutorial, we will discuss benchmark for a client-server application or in

other word web application.

The benchmark testing can be done for following services,

 Browser compatibility
 Broken Links
 HTML compliance
 Load Time
 Accessibility
 Link popularity
Why Benchmark Testing is important
At the business level, benchmark testing can be helpful in determining

 How well a web-based application is performing with respect to the

competitors
 How different types of customers experience the response time and
availability of a site
 It ensures that websites complies with standards and best practices
 It enables to evaluate third- party service providers prior to making a
contracting decision
 Allows to figure out the mistakes to be avoided

Creating a Benchmark Test Plan

Test plan is the most crucial element of the benchmark testing process.
Benchmark Test plan follows steps as given below;

1. Scaling and invoking workload

2. Collect and store measures for benchmark testing
3. Define the time span required and the terminal point of a test process
4. Prepare a backup plan to overcome any new test case failure
5. Decide the authority to call termination of the end process

Phases of Benchmark Testing

There are 4 phases involved in Benchmark Testing:

 1. Planning Phase
o Identifying and prioritize standards and requirements
o Decide benchmark criteria
o Define benchmark test process
2. Analysis Phase
o Identify root cause of error to improve quality
o Setting goals for test process
3. Integration Phase
o Share outcomes with concerned person and get approval
o Establish functional goals
4. Action Phase
o Develop test plan and documentation
o Implement actions specified in previous phases and monitor
progress
o Run the process continuously

Benchmark Testing Frameworks

Benchmark Testing Frameworks help to perform some fundamental tasks for
performance check.

These fundamental tasks include;

1. Database Access
2. Server-side composition
 3. JSON Serialization
4. Configuration

Benchmark Framework 2.0 & TechEmpower are mostly used Benchmark

Testing frameworks

Let's have a look towards TechEmpower. We will have a quick glance towards
its salient features

 It is an open-source framework to perform benchmark tasks

 It needs correct configuration of benchmarking environment
 There are mainly two modes such as Benchmark Mode for comparing
results and Verify Mode for thosewho are not experts
 There are several files required which are unique and may vary system
to system accordingly
 It includes 'Benchmark Config File' that defines set of test instructions
and metadata for a benchmark program
 It is based on several languages such as Java, Python, Ruby, PHP,
JavaScript, Perl, C, Groovy, Haskell, Scala, etc.
 Object – Relational Mapper (ORM) is used to perform tests on
databases, JSON Serialization, etc.

Things to consider while Benchmark Testing

 Consistency and control are important measures to perform
benchmark testing
 Understand the system architecture to design test criteria and test data
 Examine initial static data and update according to a number of users
 Check for 'Reset' functionality wherever required and define per second
transaction rate (this specifically deals with users transaction against a
database)
 Split the system elements according to is functionalities
 Every system has different architecture and design, which needs to be
taken into consideration while performing Benchmark Testing

Components of Benchmark Testing

Different systems have a varying degree of complexity and requires different
techniques for testing application.
There are 3 major components of Benchmark Testing. They are

1. Workload Specifications: Determining type and frequency of request

that is to be submitted to the system under test.
2. Specifications of Metrics: Determine which element to be measured
e.g.; Download speed
3. Specification of measurement: Determining how to measure the
specified elements to find appropriate values

For running successful benchmark testing following facts needs to be

considered.

1. Make sure all software components are in working condition

2. Operating system and supporting drivers should work accurately
3. Remove prefetch and temporary files from the system before running a
benchmark
4. Close all processes and applications running in the background
5. Check for operating systems updates and real- world configurations

Benchmark Testing Tools

Benchmark Testing Tools are being categorized accordingly. We will enlist
one by one.

Tools for Testing Windows PCs:

1. Prime95
2. Novabench
3. 3DMark
4. SiSoftware Sandra

Tools for Testing CPU performance:

1. Cinebench
2. Geekbench

Tools for Testing System's Speed and Mobile Device batteries:

1. Phoronix (Linux)
2. CPU-M (CPU Benchmark test)
3. Vellamo (Web browsing performance)
Tools for Performing Comparative Benchmark Test between different
machines:

Everest Ultimate Edition

Benchmark Testing can be performed with Android phone and tablets too,
some of the tools are listed above for the same. Among all Passmark is well-
known and widely used Benchmark Testing tool.

What is Endurance Testing in Software

Testing? (with Example)

Endurance Testing
Endurance Testing is non-functional type of software testing where a
software is tested with high load extended over a significant amount of time to
evaluate the behavior of software application under sustained use. The main
purpose of endurance testing is to ensure that the application is capable
enough to handle extended load without any deterioration of response time.

This type of testing is performed at the last stage of the performance run
cycle. Endurance testing is a long process and sometimes lasts for even up to
a year. This may include applying external loads such as Internet traffic or
user actions. This makes endurance testing differ from Load Testing, which
usually ends in a couple of hours or so.

Goals of Endurance Testing

 Primary goal of Endurance testing is to check for memory leaks.
 To discover how the system performs under sustained usage.
 To ensure that after a long period, the system response time will remain
the same or better than the start of the test.
 To determine the number of users and/or transactions a given system
will support and meet performance goals.
 To manage the future loads, we need to understand how many
additional resources (like processor capacity, disk capacity, memory
usage, or network bandwidth) are necessary to support usage in future.
  Endurance testing is generally done by either overloading the system or
by reducing certain system resources and evaluating the
consequences.
 It is performed to ensure that defects or memory leaks do not occur
after what is considered to be a relatively "normal" usage period.

What to monitor in Endurance Testing

In Endurance Testing following things are tested.

 Test memory leakage- Checks are done to verify if there is any

memory leakage in the application, which can cause crashing of the
system or O.S.
 Test connection closure between the layer of the system – If the
connection between the layers of the system is not closed successfully,
it may stall some or all modules of the system.
 Test database connection close successfully- If the database
connection is not closed successfully, may result in system crash
 Test response time – System is tested for the response time of the
system as the application becomes less efficient as a result of the
prolonged use of the system.

How to perform Endurance Testing

Below is the basic testing approach for Endurance Test
  Testing Environment – Identify the hardware, software, operating
system require for the endurance testing, assigning roles and
responsibilities within the team, etc. The environment should be ready
before the test execution. You also need to estimate common database
production size and yearly growth. This is required as such you need to
test how your application will respond after a year, two or five.
 Creating the Test Plan, Scenarios - Based on nature of testing -
manual or automation or combination of both, Test Case design,
reviews, and execution should be planned. Testing to stress the system,
break point testing, etc. should also be part of the test plan. Testing to
stress the system determines the break point in the application.
 Test Estimation – Provide the estimation of how long it will take to
complete the testing phase. It should be analyzed on the basis of a
number of testers involved and the number of test cycles required.
 Risk Analysis - Analyzing the risk and taking an appropriate action for
the prevention. Prioritization of test cases as per the Risk factor and
identify the below risk and issues tester may phase during the
endurance test.
o Will performance remain consistent over time?
o Are there other minor issues that have not yet been detected?
o Is there external interference that was not addressed?
 Test Schedule - Determine the budget, deliverables within the time
frames. As Endurance Testing applies a huge but natural load
arrangement of transactions to the system/application for a continuous
period of time.

Endurance Testing Example

While Stress testing takes the tested system to its limits, Endurance
testing takes the application to its limit over time.

For Example, the most complex issues – memory leaks, database server
utilization, and unresponsive system – happen when software runs for an
extended period of time. If you skip the endurance tests, your chances of
detecting such defects prior to deployment are quite low.

Endurance Testing Tools

 WebLOAD
 LoadComplete
  Apache JMeter
 LoadRunner
 Appvance
 LoadUI
 OpenSTA
 Rational Performance Tester

Advantages of Endurance Testing

 It helps in determining how workload can the System Under Load
handle.
 Provides accurate data that customer can use to validate or enhance
their infrastructure needs.
 Identifies performance problems that may occur after a system has
been running at a high level for longer period of time
 Typical issues are identified in smaller targeted performance tests,
which means it ensures application remain available even when there is
huge load in a very short span time.
 The endurance test is also used to check whether there is any
performance degradation after a long period of execution

Disadvantages of Endurance Testing

 It is often hard to define how much stress is worth applying.
 Endurance Testing could cause application and/or network failures that
may result in significant disruption if Test Environment are not isolated.
 Permanent Data loss or corruption can occur by over-stressing the
system.
 Resource utilization remains very high after the stress is removed.
 Some application components fail to respond.
 Unhandled exceptions are observed by the end user.

Reliability Testing Tutorial: What is,

Methods, Tools, Example
What is Reliability Testing?
RELIABILITY TESTING is a software testing type, that checks whether the
software can perform a failure-free operation for a specified period of time in a
particular environment.

Reliability means "yielding the same," in other terms, the word "reliable"
means something is dependable and that it will give the same outcome every
time. The same is true for Reliability testing. Reliability testing in software
assures that the product is fault free and is reliable for its intended purpose.

Reliability Testing Example

The probability that a PC in a store is up and running for eight hours without
crashing is 99%; this is referred as reliability.

Reliability Testing can be categorized into three segments,

 Modeling
 Measurement
 Improvement

The following formula is for calculating the probability of failure.

Probability = Number of failing cases/ Total number of cases under consideration

Factors Influencing Software Reliability

1. The number of faults presents in the software
2. The way users operate the system

 Reliability Testing is one of the key to better software quality. This

testing helps discover many problems in the software design and
functionality.
 The main purpose of reliability testing is to check whether the software
meets the requirement of customer's reliability.
 Reliability testing will be performed at several levels. Complex systems
will be tested at unit,assembly,subsystem and system levels.
Why to do Reliability Testing
Reliability testing is done to test the software performance under the given
conditions.

The objective behind performing reliability testing are,

1. To find the structure of repeating failures.

2. To find the number of failures occurring is the specified amount of time.
3. To discover the main cause of failure
4. To conduct Performance Testing of various modules of software
application after fixing defect

After the release of the product too,we can minimize the possibility of
occurrence of defects and thereby improve the software reliability. Some of
the tools useful for this are- Trend Analysis,Orthogonal Defect Classification
and formal methods, etc..

Types of reliability Testing

Software reliability testing includes Feature Testing, Load
Testing and Regression Testing

Feature Testing:-

Featured Testing check the feature provided by the software and is conducted
in the following steps:-

 Each operation in the software is executed at least once.

 Interaction between the two operations is reduced.
 Each operation have to be checked for its proper execution.

Load Testing:-

Usually, the software will perform better at the beginning of the process and
after that, it will start degrading. Load Testing is conducted to check the
performance of the software under maximum work load.

Regression Test:-
Regression testing is mainly used to check whether any new bugs have been
introduced because of the fixing of previous bugs. Regression Testing is
conducted after every change or updation of the software features and their
functionalities.

How to do Reliability Testing

Reliability Testing is costly compared to other types of testing. So Proper
planning and management is required while doing reliability testing. This
includes testing process to be implemented, data for test environment, test
schedule, test points, etc.

To begin with reliability testing, tester has to keep following things,

 Establish reliability goals

 Develop operational profile
 Plan and execute tests
 Use test results to drive decisions

As we discussed earlier, there are three categories in which we can perform

the Reliability Testing,-Modeling, Measurement and Improvement.

The key parameters involved in Reliability Testing are:-

 Probability of failure-free operation

 Length of time of failure-free operation
 The environment in which it is executed

Step 1) Modeling
Software Modeling Technique can be divided into two subcategories:

1. Prediction Modeling

2. Estimation Modeling

 Meaningful results can be obtained by applying suitable models.

 Assumptions and abstractions can be made to simplify the problems
and no single model will suitable for all the situations.

The major differences of two models are:-

Issues Prediction Models Estimation Models
It uses current data from the
Data Reference It uses historical data
software development.
It will be usually created before It will be usually used at the later
When used in
the development or testing stage of Software Development Lif
Development Cycle
phases. Cycle.
It will predict the reliability either fo
It will predict the reliability in the
Time Frame the present time or in the future
future.
time.

Step 2) Measurement
Software reliability cannot be measured directly and hence, other related
factors are considered in order to estimate the software reliability. The current
practices of Software Reliability Measurement are divided into four
categories:-

1. Product Metrics:-

Product metrics are the combination of 4 types of metrics:

 Software size: - Line of Code (LOC) is an intuitive initial approach for

measuring the size of the software. Only the source code is counted in
this metric, and the comments and other non-executable statements will
not be counted.
 Function point Metric:- Function Pont Metric is the method for
measuring the functionality of the Software Development. It will consider
the count of inputs, outputs, master files, etc. It measures the
functionality delivered to the user and is independent of the
programming language.
 Complexity:- It is directly related to software reliability, so representing
complexity is important. Complexity-oriented metric is a method of
determining the complexity of a program's control structure, by
simplifying the code into a graphical representation.
 Test Coverage Metrics:- It is a way of estimating fault and reliability by
performing the complete test of software products. Software reliability
means it is the function of determining that the system has been
completely verified and tested.
2. Project Management Metrics

 Researchers have realized that good management can result in the

better products.
 A good management can achieve higher reliability by using better
development process,risk management process,configuration
management process, etc.

3. Process Metrics

The quality of the product is directly related to the process. The process
metrics can be used to estimate, monitor and improve the reliability and
quality of software.

4. Fault and Failure Metrics

Fault and Failure Metrics are mainly used to check whether the system is
completely failure-free. Both the types of faults found out during the testing
process (i.e. before delivery) as well as the failure reported by users after
delivery are collected, summarized and analyzed to achieve this goal.

Software reliability is measured in terms of mean time between failures

(MTBF). MTBF consists of

 Mean to failure (MTTF): It is the difference of time between two

consecutive failures
 Mean time to repair (MTTR): It is the time required to fix the failure.

MTBF = MTTF + MTTR

Reliability for good software is a number between 0 and 1.

Reliability increases when errors or bugs from the program are removed.

Step 3) Improvement
Improvement completely depends upon the problems occurred in the
application or system, or else the characteristics of the software. According to
the complexity of the software module,the way of improvement will also differ.
Two main constraints time and budget, which will limit the efforts are put into
the software reliability improvement.
Example Methods for Reliability Testing
Testing for reliability is about exercising an application so that failures are
discovered and removed before the system is deployed.

There are mainly three approaches used for Reliability Testing

 Test-Retest Reliability
 Parallel Forms Reliability
 Decision Consistency

Below we tried to explain all these with an example.

Test-Retest Reliability

To estimate test-retest reliability, a single group of examinees will perform

testing process only a few days or weeks apart. The time should be short
enough so that the examinees skills in the area can be assessed. The
relationship between the examinee's scores from two different administrations
is estimated, through statistical correlation. This type of reliability
demonstrates the extent to which a test is able to produce stable, consistent
scores across time.
Parallel Forms Reliability

Many exams have multiple formats of question papers, this parallel forms of
exam provide Security. Parallel forms reliability is estimated by administrating
both forms of the exam to the same group of examinees. The examinees
scores on the two test forms are correlated in order to determine how similarly
the two test forms functions. This reliability estimate is a measure of how
consistent examinees scores can be expected to across test forms.

Decision Consistency
After doing Test-Retest Reliability and Parallel Form Reliability, we will get a
result of examinees either pass or fail. It is the reliability of this classification
decision that is estimated in decision consistency reliability

Importance of Reliability Testing

A thorough assessment of reliability is required to improve the performance of

software product and process. Testing software reliability will help the
software managers and practitioners to a great extent.

To check the reliability of the software via testing:-

1. A large number of test cases should be executed for an extended period

of time to find out how long the software will execute without failure.
2. The test cases distribution should match the actual or planned
operational profile of the software. The more often a function of the
 software is executed, the greater the percentage of test cases that
should be allocated to that function or subset.

Reliability Testing Tools

Some of the Tools used for Software Reliability are

1. WEIBULL++:- Reliability Life Data Analysis

2. RGA:- Reliability Growth Analysis

3. RCM:-Reliability Centered Maintenance

Summary:

Reliability Testing is the important part of a reliability engineering program.

More correctly, it is the soul of reliability engineering program.

Furthermore, reliability tests are mainly designed to uncover particular failure

modes and other problems during software testing.

In Software Engineering, Reliability Testing can be categorized into three

segments,

 Modeling
 Measurement
 Improvement

Factors Influencing Software Reliability

 The number of faults presents in the software

 The way users operate the system

10 BEST Performance Testing Tools |

Load Testing Tools (2020)
Performance Testing Tools help in the process of determining the speed,
effectiveness, reliability, scalability and interoperability of the system,
computer, network, program or application.
Below is a hand-picked list of Best Load Testing Tools. The list contains both
open-source(free) and commercial load testing tools(paid) with latest features
and download links

est Load Testing Tools: Open Source (Free) and Paid

Name Features Link
• Flexible test scenario creation
WebLOAD • Easy-to-reach customer support Learn More
• Automatic bottleneck detection
• On-premise or cloud testing supported
StresStimulus • Works as a stand-alone tool Learn More
• Supports three scripting languages
• Real browser load test execution at scale
LoadNinja • VU Debugger – debug tests in real time Learn More
• Manage virtual user activity in real time
• Integration with CI servers
NeoLoad • Monitors all major servers Learn More
• Efficient and infinitely scalable
• Cloud-based load testing in real browsers
LoadView • Supports Rich Internet Applications Learn More
• Identify bottlenecks and ensure scalability

1) WebLOAD

WebLOAD is an enterprise-scale load testing solution capable of generating

reliable, real life load scenarios, even for the most complex systems, with
smart analytics providing performance insights.

WebLOAD is based on a flexible platform with built in support for hundreds of

technologies and integration with many tools from CI/CD pipelines to
monitoring.

WebLOAD Features:
  WebLOAD is offered on the Cloud (SAAS) or as an On-Prem
deployment
 Flexible test scenario creation
 Smart Scripting based native JavaScript with Powerful correlation,
parametrization and validation engines
 Supports every major web technology and easily extendible
 Automatic bottleneck detection
 Easy-to-reach customer support

2) StresStimulus

StresStimulus targets application scenarios that are difficult to test with other

tools. It automatically fixes playback errors thanks to its proprietary
autocorrelation. StresStimulus records user actions and replays them to
emulate variable usage patterns. It also monitors load impact on application
responsiveness and server infrastructure. The result analyzer finds hidden
concurrency errors left unexposed by functional testing and includes detailed
application performance metrics.

Features:

 End-to-end Test Wizard covers all steps from recording to test results,
which reduces the learning curve
 On-premise or cloud testing supported with multiple load generators
 Works as a stand-alone tool or Fiddler add-on
 Exports script to Visual Studio test format to help with scenarios that fail
when recorded in Visual Studio directly
 Manual scripting is not required but available; supports three scripting
languages
3) LoadNinja

LoadNinja by SmartBear allows you to quickly create scriptless sophisticated

load tests, reduce testing time by 50%, replace load emulators with real
browsers, and get actionable, brower-based metrics, all at ninja speed. You
can easily capture client-side interactions, debug in real time, and identify
performance problems immediately. LoadNinja empowers teams to increase
their test coverage without sacrificing quality by removing the tedious efforts of
dynamic correlation, script translation, and script scrubbing. With LoadNinja,
engineers, testers and product teams can focus more on building apps that
scale and less on building load testing scripts.

Features:

 Scriptless load test creation & playback with InstaPlay recorder

 Real browser load test execution at scale
 VU Debugger – debug tests in real time
 VU Inspector – manage virtual user activity in real time.
 Hosted on the cloud, no server machine & upkeep required
 Sophisticated browser-based metrics with analytics and reporting
features

Protocols:

 HTTP, HTTPS, SAP GUI Web, WebSocket, Java based protocol,

Google Web Toolkit, Oracle forms, & more
4) NeoLoad

NeoLoad is an innovative performance testing platform designed to automate

test design, maintenance, and analysis for Agile and DevOps teams. NeoLoad
integrates with continuous delivery pipelines to support performance testing
across the life cycle - from component to the full system-wide load tests.

Features:

 Automated test design enabling 10x faster test creation and update than
traditional solutions
 Integration with CI servers for automated test runtime
 Collaboration: shared test scripts and reports, in real -time and after test
completion through an on-premise or SaaS web interface
 Hybrid on-premise and cloud load generation from over 70 global
localizations

5) LoadView

With LoadView by Dotcom-Monitor, you can show actual performance of your

applications under load — just as your users experience it. LoadView utilizes
real browser-based load testing for websites, web applications, and APIs.
Easily create multi-step scripts that simulate users interacting with your
website or application using the EveryStep Web Recorder, or even manually
edit the script using your own C# code.
  Cloud-based load testing in real browsers
 Supports Rich Internet Applications (RIA), such as Flash, Silverlight,
Java, HTML5, PHP, Ruby, and more
 Quickly and easily build test scripts without touching a line of code
 Test compatibility on 40+ desktop/mobile browsers and devices
 13+ world-wide cloud locations using Amazon Web Services and
Google Cloud Platform
 Identify bottlenecks and ensure scalability
 Performance metrics and reports that can be shared with various
internal stakeholders for capacity planning

6) Apache Jmeter

JMeter is an open source tool that can be used for performance and load
testing for analyzing and measuring the performance of a variety of services.
It is one of the best load testing tools that is mainly used as website load
testing tools for web service applications.

Features:

 This free tool doesn't demand state of the art infrastructure for load
testing and supports multiple load injectors managed by a single
controller
 It is one of the performance test tools which is highly portable and
supports 100% all the Java based apps
 Less scripting efforts as compared to other api performance testing
tools because of its user-friendly GUI
 Simple charts and graphs sufficient for analyzing key load related
statistics and resource usage monitors.
 Supports Integrated real-time, Tomcat collectors for Monitoring

Protocols
Web : HTTP, HTTPS, WebServices : XML, SOAP, etc., Java based protocols,
FTP

Download JMeter

7) HP Performance Tester (LoadRunner)

This is an enterprise performance testing version of Loadrunner and a

platform enabled both global standardization and formation Performance CoE.

Features:

 Lower the cost of distributed load testing

 Scale from single projects to a full-scale testing Center of Excellence
(CoEs) that consolidates hardware, standardizes best practices, and
leverages global testing resources
 Reduce the risk of deploying systems that do not meet performance
requirements through the use of effective enterprise load tester tool
 It is one of the best performance testing tools that lowers hardware and
software costs by accurately predicting system capacity
 Pinpoint the root cause of application performance problems quickly and
accurately
 It is one of the Effective performance test tools for utilization tracking
 Browser based access to global test resources and optimal usage of
load generator farm.

Protocols:

All protocols are supported by Load Runner load test tool

Download Loadrunner
8) Silk Performer
Silk Performer tool is an enterprise class load and stress testing tool and has
the ability to test multiple application environments with the thousands of
concurrent users. It is one of the best load testing tools which supports the
widest range of protocols.

Features:

There are many good load and stress test features in Silkperformer and those
are listed as follows:

 Components -> Workbench, True Log Explorer and Performance

Explorer
 Requires minimum hardware resources for virtual user simulation
 Simulates modifiable virtual users
 This load test tool Supports integrated server monitoring
 Customer friendly licensing
 Correlation and Parameterization is user friendly
 No License requirement for Controllers or Individual Protocols
 Handles Load Test in Project approach
 Generates reports with tables and graphs and allows customization.
 This stress testing tool supports six models of workloads.
 This load tester tool provides Agent Health Control
 Stress test website with server side diagnostics
 Resource Management
 It is one of the best website load testing tools that provides Version
Control Management

Protocols

HTTP/HTML, HTTPS/HTML, HTTP/HTTPS, Flash, Email (SMTP/ POP), FTP,

TCP/IP, LDAP, XML/SOAP, .NET and many more
Download Silk Performer

9) Rational Performance Tester

Rational Performance Tester(RPT) is a performance and load testing tool

developed by IBM Corporation. It is performance test creation, execution, and
analysis tool that helps development team to validate the scalability and
reliability of web based applications before deployment into production.

Features:

There are good features which gives confidence to the users that all
performance related bottlenecks are resolved. Following are some of the
features of this tool:

 No coding involved
 It is one of the best performance testing tools that provides scheduled
and event based testing
 Real time reporting for immediate performance problem identification
 It is one of the best website load testing tools that can run with large
multi users tests
 Accurate user profile workloads
 This load test software has automated test data variation
 Automatic identification of dynamic server responses
 Performs the website load test with rendered HTML view of web pages
visited during test recordings
 Environment and Platform support
 Entrust security protocol support
 Java code insertion for customization

Protocols:
Citrix, Socket Recording, Web HTTP, SOA, SAP, XML, Websphere, Weblogic

Download Rational Functional Tester

10) SmartMeter.io

SmartMeter.io is an alternative to JMeter and aims to fix its drawbacks. It is

one of the best load testing tools that allows for easy scriptless test scenario
creation using the so-called Recorder, yet still lets you make advanced edits
of the test. It also excels in test reporting and makes use of functions such as
automatic test criteria evaluation, test runs comparison and trend analysis. It
fully supports CI/CD integration. Available for Windows, Mac OS, and Linux.

Features:

 This load test software provides scriptless test scenario creation

 Comprehensive reporting with automatic evaluation and test runs
comparison
 GUI test run with real time results
 State of the art response body extractor (Boundary body extractor)
 CI/CD ready

Protocols:

 HTTP, JDBC, LDAP, SOAP, JMS, and FTP

Download SmartMeter.io

FAQ
⚡ What is Performance Testing?
Performance Testing checks the speed, response time, reliability, resource
usage, scalability of a software program under their expected workload. The
purpose of Performance Testing is not to find functional defects but to
eliminate performance bottlenecks in the software or device
✔️What is Load Testing?
Load Testing determines a system's performance under a specific expected
load. This purpose of Load Test is to determine how the application behaves
when multiple users access it simultaneously.

❓ Which factors should you consider while selecting

a Load/Performance Testing Tool?
You should consider the following factors before selecting a tool

 Desired Protocol Support: Web, Desktop, Citrix, SAP, etc

 License Cost if applicable
 In the case of an outsourcing project, you need to factor customer/Client
preference of the automation tool
 The cost involved in training employees on the tool
 Hardware/Software requirements of the automation tool
 Support and Update policy of the automation tool vendor.