MSBTE STE Chapter 3

Download as pdf or txt
Download as pdf or txt
You are on page 1of 89

1

22518
Chapter 3
Test Management
1.Test Planning : Preparing a test plan , Deciding Test Approach,
Setting up criteria for testing , Identifying Responsibilities,
Staffing, Resource Requirements, Test Deliverables, Testing
Tasks
2.Test Management: Test Infrastructure Management, Test
People Management.
3.3 Test Process: Baselining a Test Plan, Test Case
Specification.
3.4 Test Executing Test Cases,Preparing
Reporting: Test
Summary Report.
What is a Test Plan?
► A TEST PLAN is a detailed document that describes the test
strategy, objectives, schedule, estimation and deliverables
and resources required for testing.
► Test Plan helps us determine the effort needed to validate
the quality of the application under test.
► The test plan serves as a blueprint to conduct software
testing activities as a defined process which is minutely
monitored and controlled by the test manager.
Importance of Test Plan
► It serves as a roadmap to the testing process to ensure your testing
project is successful and helps you control risk.
► It clearly defines the roles and responsibilities of every team
member, outlines the resource requirements which are essential to
carry out the testing process
► Planning and a test plan encourages better communication with
other project team members, testers, peers, managers, and other
stakeholders
► Help people outside the test team such as developers, business
managers, customers understand the details of testing.
► Important aspects like test estimation, test scope, Test Strategy are
documented in Test Plan, so it can be reviewed by Management
Team and re-used for other projects.
How to write a Test Plan
How to write a Test Plan

1. Analyze the product


2. Design the Test Strategy
3. Define the Test Objectives
4. Define Test Criteria
5. Resource Planning
6. Plan Test Environment
7. Schedule & Estimation
8. Determine Test Deliverables
1. Analyze the product

► It’s not possible to test a product without any information


about it.
► So, the first step towards creating a test plan is to analyze
the product, its features, and functionalities.
► Need to have a proper understanding of user
requirements and expectations.
2. Develop Test Strategy

► Test Strategy is a critical step in making a Test Plan. A Test


Strategy document, is a high-level document, which is
usually developed by Test Manager. This document
defines:
► The project’s testing objectives and the means to achieve
them
► Determines testing effort and costs
2.1) Define Scope of
Testing
► The componentsof the system to be tested
(hardware, software, middleware, etc.) are defined as "in
scope“.
► The components of the system that will not be tested also
need to be clearly defined as being "out of scope."
► Defining the scope of your testing project is very
important for all stakeholders.
2.1) Define Scope of
Testing
► A precise scope helps :
► Give everyone a confidence & accurate information of the
testing you are doing
► All project members will have a clear understanding about
what is tested and what is not
► How do you determine scope your project?
► Precise customer requirement
► Project Budget
► Product Specification
► Skills & talent of your test team
2.2) Identify Testing Type
► A Testing Type is a standard test procedure that gives an
expected test outcome.
► Each testing type is formulated to identify a specific type
of product bugs. But, all Testing Types are aimed at
achieving one common goal “Early detection of all the
defects before releasing the product to the customer”
► Unit test
► Integration test
► System test
► Acceptance test
2.3) Document Risk & Issues
► Risk is future’s uncertain event with a probability
of occurrence and a potential for loss.
► When the risk actually happens, it becomes the ‘issue’.
2.4) Create Test Logistics
The Test Manager should answer the following questions:
► Who will test?
► When will the test occur?
3.Define Test Objective

► To release a high-quality bug-free software product


to market on time.
► So, as a part of a test plan, need to clearly
definethe testing scope and its boundaries.
► You could just follow these two simple steps to define test
objectives:
► List all the software features which may need testing
► Define thetarget of testing, based on features listed
previously
4. Define Test Criteria

► Test Criteria is a standard or rule on which a


procedure or test judgment can be based test
► There Are 2 types of test criteria as following
Suspension Criteria
► If the suspension criteria are met during testing, the active test cycle
will be suspended until the criteria are resolved.
Exit Criteria
► It specifies the criteria that denote a successful completion of a test
phase. The exit criteria are the targeted results of the test and are
necessary before proceeding to the next phase of development.
5.Resource Planning
► The resource plan is basically a detailed summary
of all types of resources required to complete the project
task.
► The resources here could be human, equipment
and materials needed to complete a project.
► It is an important step which helps you determine
the number of resources to be used for the project.
6.Plan Test Environment

► A testing environment is a setup of software and hardware


on which the testing team is going to execute test cases.
► The test environment consists of real business and user
environment, as well as physical environments, such as
server, front end running environment.
7. Schedule & Estimation
► Creating a schedule helps you control the progress of the
testing process.
► While drawing up a schedule, you should consider factors
like:
► Project estimate
► Project risk estimate
► Resource estimate
► Employee roles & responsibilities
► Test activity deadlines
8.Test Deliverables
► Test Deliverables is a list of all the documents, tools and
other components that has to be developed and
maintained in support of the testing effort.
► There are different test deliverables at every phase of the
software development lifecycle.
Test Plan Template
Parameter Description
Test Plan Identifier Uniquely identifies the test plan & may include the version
number
Introduction Sets objectives, scope, goals, resource & budget constraints
Test Items Lists the systems and subsystems which are to be tested
Features to be Tested All the features & functionalities to be tested are listed here
Features not to be Lists the features of the software product that need not be
Tested tested

Approach Has sources of test data, inputs and outputs, testing priorities
Item pass/fail Describes a success criteria for evaluating the test results
Test Plan Template
Parameter Description
Suspension Criteria Has criteria that may result in suspending testing activities
Test Deliverables Includes test cases, sample data, test report, issue log
Testing Tasks Describes dependencies between tasks & resources needed
Environmental Needs Lists software, hardware or other testing requirements
Responsibilities Lists roles and responsibilities assigned to the testing team
Staffing Needs Describes the additional training needs for the staff

Schedule Details on when the testing activities will take place are listed
Risks Lists overall risk of the project as it pertains to testing
Approvals Contains signature of approval from stakeholders
Test Plan Guidelines

► Make the plan concise.


► Avoid redundancy and unnecessary data.
► Be specific.
► Make use of lists and tables wherever possible.
► Avoid lengthy paragraphs.
► Have the test plan reviewed a number of times
prior to baselining it or sending it for approval.
► Update the plan as and when necessary.
Deciding Test Approach/Strategy
What needs to be tested, to enable estimation of size, effort,
and schedule. This includes identifying :
1. What type of testingwould you use for testingthe
functionality?
2. What aretheconfigurations or scenarios for testingthe
features?
3. What integration testingwould you do to ensure
these
features work together?
4.What localization validations would be needed?
5.What “non-functional” tests would you need to do?
Setting up Criteria for Testing
► Entry and exit criteria : The entry criteria for a test specify
threshold criteria for each phase or type of test. The
completion/exit criteria specify when a test cycle or a
testing activity can be deemed complete.

► Pass or Fail criteria- Specify the criteria that


will use to determine whether a test item has passed
or failed testing
Setting up Criteria for Testing
► Suspended criteria - Specify criteria to be used to suspend
the testing activity.

► Resume criteria- Specified testing activity which must be


redone when testing is resumed.
IdentifyingResponsibilities, Staffing, and Training
Needs
► A testing project requires different people to play different
roles. The different role definitions should
► Ensure there is clear accountability for a given task, so that
each person knows what he or she has to do;
► Clearly list the responsibilities for various functions to
various people, so that everyone knows how his or her work
fits into the entire project;
► Complement each other, ensuring no one steps on an
others’ toes; and
► Supplement each other, so that no task is left unassigned.
Staffing
► Staffing is done based on estimation of effort involved and
the availability of time for release.
► People are assigned to tasks that achieve the best possible
fit between the requirements of the job and skills and
experience levels needed to perform that function.
► In case there are gaps between the requirements and
availability of skills, they should be addressed with
appropriate training programs.
Identifying Resource Requirements
The following factors need to be considered:
► Machine configuration needed to run the product under test.
► Overheads required by the test automation tool, if any.
► Supporting tools such as compilers, test data
generators,
configuration management tools, and so on.
► The different configurations of thesupporting
software that must be present.
► Special requirements for running machine-intensive tests such
as load tests and performance tests.
► Appropriate number of licenses of all the software.
Identifying Test Deliverables
► The deliverables include the following, all
reviewed and approved by the appropriate people.
► The test plan itself (master test plan, and various other test
plans for the project)
► Test case design specifications
► Test cases, including any automation that is specified in the
plan
► Test logs produced by running the tests
► Test summary reports
Testing Tasks: Size and Effort Estimation
► Size estimate quantifies the actual amount of testing that
needs to be done.
► LOC
► FP
► Effort estimate is given in person days, person
months, or person years.
► The effort estimate is then translated to a schedule estimate.
Test Management
► An important part of software quality is the
process of testing and validating the software.

► Test management is the practice of organizing and


controlling the process and artifacts required for the
testing effort.
Test Management
► Traditional tools used for test management include
► Pen and paper
► Word processors
► Spreadsheets
► Larger testing efforts usually built on spreadsheets or
databases, or commercial test management applications.
► Generally test management allows different teams to plan,
develop, execute, and assess all testing activities.
Choice of Standards
► Standards comprise an important part of planning in any
organization.

► Standards are of two types—external standards


and internal standards.
Internal standards
► Naming and storage conventions for test artifacts.
► Document standards
► Test coding standards
► Test reporting standards.
Naming and storage conventions for test artifacts

► Every test artifact (test specification, test case, test results


and so on) have to be named appropriately and
meaningfully.
► It enables
► Easy identification of the product functionality.
► Reverse mapping to identify the functionality corresponding
to a given set of tests.
► E.g. modules shall be M01, M02. Files types can be .sh, .SQL.
Documentation standards:

► Appropriate header level comments at the beginning of a


file that outlines the functions to be served by the test.
► Sufficient inline comments, spread throughout the file
► Up-to-Date change history information, reading all
the changes made to the test file.
Test coding standards:

► Enforce right type of initialization


► Stipulate ways of naming variables.
► Encourage reusability of test artifacts.
► Provide standard interfaces to external entities like
operating system, hardware and so on.
Test reporting standard:

► All the stakeholders must get a consistent and timely view


of the progress of tests.
► It provides guidelines on the level of details that should be
present in the test report, their standard formats and
contents.
External Standards:
► These are the standards made by an entity external to an
organization.

► These standards are standards that a product should


comply with, are externally visible and are usually
stipulated by external parties.
External Standards:
The three types of external standards are:
► Customer standard: refer to something defined by the
customer as per his/her business requirement for the
given product.
► National Standard: refer to something defined by the
regulatory entities of the country where the supplier
/customer resides.
► International Standard: are defined at international level
and these are applicable to all customers across the globe.
Test Infrastructure Management
► The testing infrastructure is used to support
automated and manual, software testing
► It consists of the testing activities,events, tasks
processes.
and
► More stability, continuity and reliability is provided to the
automated testing process using stronger
infrastructure
and its good management.
Test Infrastructure Management
► Testing requires a robust infrastructure to be
planned upfront.
► This infrastructure is made up of three essential elements.
► A test case database (TCDB)
► A defect repository
► Configuration management repository and tool
A test case database captures all the relevant information about the
test cases in an organization.
Entity Purpose Attributes
Test case Records all the “static” Test case ID
information about the tests Test case name (file name)
Test case owner
Associated files for the test
case
Test case - Product Provides a mapping between the Test case ID Module ID
cross- reference tests & the corresponding product
features; enables identification of
tests for a given feature.
Test case run Gives the history of when a test Test case ID Run date Time
history was run and what was the result; taken Run status
provides inputs on selection of (success/failure)
tests for regression runs
Test case—Defect Gives details of test cases Test case ID
cross-reference introduced to test certain specific Defect reference (points to
defects detected in the product; a record in the defect
provides inputs on the selection of repository)
A defect repository captures all the relevant details of defects
reported for a product.
Entity Purpose Attributes
Defect details Records all the “static” Defect ID
information about the Defect priority/severity Defect
tests description
Affected product(s)
Any relevant version information
Environmental information (for
example, OS version)
Customers who encountered the
problem
Date and time of defect occurrence
Defect test Provides details of test Defect ID
details cases for a given defect. Test case ID
Cross-references the
TCDB
A defect repository captures all the relevant details of defects
reported for a product.
The defect repository is an important vehicle of communication that
influences the work flow within a software organization.
Entity Purpose Attributes
Fix details Provides details of fixes for a given Defect ID
defect; cross-references the Fix details
configuration management (file changed, fix
repository release information)
Communication Captures all the details of the Test case ID
communication that transpired for Defect reference #
this defect among the various Details of
stakeholders. These could include communication
communication between the
testing team and development
team, customer communication,
and so on. Provides insights into
effectiveness of communication
Software configuration management (SCM) repository

► An SCM repository also known as (CM repository)


► Software Configuration Management is defined as a
process to systematically manage, organize, and control
the changes in the documents, codes, and other entities
during the Software Development Life Cycle.
► keeps track of change control and version control of all the
files/entities that make up a software product.
► A particular case of the files/entities is test files.
Software configuration management (SCM) repository

Change control ensures that


► Changes to test files are made in a controlled fashion and
only with proper approvals.
► Change are made by one test engineer are not accidently
lost or overwritten by other changes.
► Each change produces distinct version of the file
that is re-creatable at any point of time.
► Everyone gets access to only the most recent
version of the test files.
Relationship
Relationship
► TCDB, defectrepository, and SCM repository
complement each should and work
integrated
other fashion. together in an
► For example, the defect repository links the defects, fixes,
and tests.
► The files for all these will be in the SCM.
► The meta data about the modified test files will be in the
TCDB.
Test People Management
► An application or product success is high just because of
use of efficient and effective testing techniques .
► Bugs/defects are found based on the skills and knowledge
of a tester and the dedication of the test teams.
► A test team is formed of people with varying
skill and experience levels.
Test People Management
► Team members come up with different expertise levels,
► Different attitudes
► Different expectations
► Interests' levels.
► Maximizequality all these factors has to be
integrated correctly ,
► Need to work together,
► Follow the applied test processes,
► work within specified schedule.
TEST PROCESS
Baselining a Test Plan
► The test plan is reviewed by a designated set of competent
people in the organization.
► Approved by a competent authority, who is
independent of the project manager directly responsible for
testing.
► After this, thetest plan is baselined into the
configuration management repository.
► From then on, the baselined test plan becomes the basis for
running the testing project.
► Baseline is a formal document which acts as a base document
for future work
TEST PROCESS
Test Case Specification
The basis for preparing individual test cases, Test case
specification should clearly identify
► The purpose of the test: This lists what feature or part the
test is intended for. The test case should follow the
naming conventions that are consistent with the module
being tested.

► Items being tested, along with their version/release


numbers as appropriate.
TEST PROCESS
► Environment that needs to be set up for running the test
case: This can include the hardware environment setup,
supporting software environment setup (for example,
setup of the operating system, database, and so on), setup
of the product under test.

► Input data to be used for the test case: The choice of


input data will be dependent on the test case itself and
the technique followed in the test case.
TEST PROCESS
► Steps to be followed to execute the test: If automated
testing is used, then, these steps are translated to the
scripting language of the tool.

► The expected results that are considered to be “correct


results.” these expected result can be what the user may
see in the form of a GUI, report and so on..
TEST PROCESS

► A step to compare the actual results produced with the


expected results: This step should do an “intelligent”
comparison of the expected and actual results to highlight
any discrepancies.

► Any relationship between this test and other tests: This


can be in the form of dependencies among the tests or the
possibility of reuse across the tests.
TEST PROCESS
Update of Traceability Matrix
► Matrix which is associates the requirements to
its work product and test cases.
► Every test case should associate to a requirement
and every requirement should associate with two
or more test cases.
► Traceability Matrix is a tools to validate that every
requirement is tested or not.
Update of Traceability Matrix
► The traceability matrix is created during the
requirements gathering phase itself by filling up
the unique identifier for each requirement.
Developing and Baselining Test Cases
► The development of test cases entails translating the test
specifications to a form from which the tests can be
executed.
► If a test case is a candidate for automation, then, this step
requires writing test scripts in the automation language.
► If the test case is a manual test case, then test case
writing maps to writing detailed step-by-step instructions
for executing the test and validating the results.
Executing Test Cases and Keeping
Traceability Matrix Current

► During test design and execution, the


traceability matrix should be kept current.
► As and when tests get designed and executed successfully,
the traceability matrix should be updated..
Collecting and Analyzing Metrics
► When tests are executed, information about test
execution gets collected in test logs and other files.
Preparing Test Summary Report
► At the completion of a test cycle, a test summary report is
produced.
► This report gives insights to the senior management about
the fitness of the product for release.
Recommending Product Release Criteria
The job of the testing team is to articulate to
the senior management and the product release team
► What defects the product has;
► What is the impact/severity of each of the defects; and
► What would be the risks of releasing the product with the
existing defects.

The senior management can then take a meaningful business


decision on whether to release a given version or not.
TEST REPORTING
► Test reporting is a means of achieving this communication.
There are two types of reports or communication that are
required: test incident reports and test summary reports
(also called test completion reports).
Test incident report
► A test incident report is a communication that happens
through the testing cycle as and when defects are
encountered.

► A test incident report is nothing but an entry made in the


defect repository. Each defect has a unique ID and this is
used to identify the incident. The high impact test
incidents (defects) are highlighted in the test summary
report.
Test summary report
► A report that summarizes the results of a test cycle is the
test summary report.
► There are two types of test summary reports.
► Phase-wise test summary, which is produced at the end of
every phase
► Final test summary reports (which has all the details of all
testing done by all phases and teams, also called as
“release test report”)
Test summary report
A Summary report should content:
1. Test Summary report Identifier
2. Description : Identify the test items being reported in this
report with test id
3. Variances: Mention any deviation from test plans, test procedures, if
any.
4. Summary of results: All the results are mentioned here with the
resolved incidents and their solutions.
5. Comprehensive assessment and recommendation for release
should include Fit for release assessment and recommendation of
release
Preparing Test summary report
Step #1: Purpose of the document
<Short description about the objective of
preparing the document>

Example: This document explains the various activities


performed as part of the Testing of ‘ABCD transport
system’ application.
Preparing Test summary report
Step #2: Application Overview
<Brief description of the application
tested> Example:
‘ABCD transport system’ is a web-based Bus ticket
booking application. Tickets for various buses can
be booked using the online facilities.
There are several modules like Registration, Booking,
Payment and Reports which are integrated to fulfill
the purpose.
Preparing Test summary report
Step #3: Testing Scope
► In Scope
► Out of Scope
► Items not tested

<This section explains about the functions/modules


in scope & out of scope for testing; Any items
which are not tested due to any
constraints/dependencies/restrictions>
Preparing Test summary report
a) In Scope : Functional Testing for the following
modules are in Scope of Testing
Registration
Booking
Payment

b) Out of Scope : Performance Testing was not done


for this application.
c) Items not tested
Verification of connectivity with the third party
system ‘Central repository system’ was not tested,
as the connectivity could not be established due to
Preparing Test summary report
Step #4: Metrics
<Metrics will help to understand the test
execution results, the status of test cases &
defects etc.
Required Metrics can be added as necessary.
Example: Defect Summary-Severity wise;
Defect
Distribution-Function/Module wise; Charts/Graphs
can be attached for better visual representation>
Preparing Test summary report
Preparing Test summary report
Preparing Test summary report
Preparing Test summary report
Step #5: Types of testing performed
► Smoke Testing
► System Integration Testing
► Regression Testing
<Describe the various types of Testing performed for
the Project.
This will make sure the application is being tested
properly through testing types agreed as per
Test Strategy.
Preparing Test summary report
Example:
a) Smoke Testing
This testing was done whenever a Build is
received (deployed into Test environment) for
Testing to make sure the major functionality is
working fine, Build can be accepted and
Testing can start.
Preparing Test summary report
b) System Integration Testing
This is the Testing performed on the Application under
test, to verify the entire application works as per
the requirements.
Critical Business scenarios were tested to make sure
important functionality in the application works
as intended without any errors.
Preparing Test summary report
c) Regression Testing
Regression testing was performed each time a new
build is deployed for testing which contains defect
fixes and new enhancements if any.
Regression Testing is being done on the entire
application and not just the new functionality
and Defect fixes.
Preparing Test summary report
Step #6: Test Environment & Tools
<Provide details on Test Environment in which the
Testing is carried out. Server, Database,
Application URL etc. If any Tools were used like
Quality Center (now HP ALM) for logging defects>
Preparing Test summary report
Step #7: Lessons Learned
<This section is used to describe the critical issues
faced and their solutions (how they were
solved during the Testing).
Lessons learned will help to make proactive
decisions during the next Testing engagement, by
avoiding these mistakes or finding a suitable
workaround>
Example:
Preparing Test summary report
Step #8: Recommendations
<Any workaround or suggestions can be
mentioned here>
Step #9: Best Practices
<There will be a lot of activities done by the
Testing team during the project.
Some of them could have saved time, some proved to
be a good & efficient way to work, etc. These can
be documented as a ‘Value Add’ to showcase to the
Stakeholders>
Preparing Test summary report
Step #10: Exit Criteria
<Exit Criteria is defined as a Completion of Testing by
fulfilling certain conditions like
(i) All planned test cases are executed;
(iI) All Critical defects are Closed etc.>

Example:
a) All test cases should be executed – Yes
b)All defects in Critical, Major, Medium severity
should be verified and closed – Yes.
Preparing Test summary report
Step #11: Conclusion/Sign Off
In this section exit criteria for an application is set
and checked to release it.
Testing team verifies the application against set
criteria.
If it is not met the decision about its release is
taken by management team.
Preparing Test summary report
Step #12: Definitions, Acronyms, and Abbreviations
<This section mentions the meanings of
Abbreviated terms used in this document and
any other new definitions>
Few Points To Note While Preparing The
Test Summary Report
As part of Test Execution, collect all required information on the Testing
performed. This will help to prepare a sound Test summary report.

Lessons learned can be explained in detail, which will convey the


Responsibility which was taken to solve these issues. Also, this will be a
reference for upcoming projects to avoid these.

Mentioning the Best Practices will describe the efforts taken by the team
apart from regular testing, which will also be treated as a “Value Addition”.
Few Points To Note While Preparing The
Test Summary Report
Mentioning the Metrics in graphics form (Charts, Graphs) will be a good way
to visually represent the status & data.

Remember, the Test summary report shall mention and explain the activities
performed as part of the Testing, to the recipients to understand better.

A few more appropriate sections can be added if required.


Test plan template

https://docs.google.com/document/d/16WfzJGEZxiNFFKn-B
wtv7EtsRn1Ej0rsAJoTayUpTIw/edit
Test Plan
ABC Bank Web Application

https://docs.google.com/document/d/1USvzAh8vN1YERuM-I
OcML4-TlyBbejn_jHKol4TqO-M/edit

You might also like