International Journal of Computer Applications (0975 – 8887)

Volume 116 – No. 16, April 2015

A Review on Various Techniques for Regression Testing

and Test Case Prioritization
Jaspreet Singh Rajal Shivani Sharma
Assistant Professor Research Scholar, Master of Engineering
Department Of Computer Science & Engineering Department of Computer Science & Engineering
Chandigarh University, Gharuan, Punjab, India Chandigarh University, Gharuan, Punjab, India

ABSTRACT changes, rather than changes in the original copy).It is a

In the field of software engineering, different applications critical and exceptionally testing assignment for the product
have been developed. An application requires changes due to analyzers to test the whole programming inside the restricted
changes in the customer requirements. Regression testing has time and assets. Every business in the United States now
to be performed for the validation of data modification. depends on software for the development, production,
Various test cases have to be developed to perform the distribution and after sales support of products and
regression testing. In this paper, various test case prioritization services[3].A study conducted by NIST in 2002 reports that
techniques have been discussed for the generation of priority software bugs cost U.S. economy $59.5(Approx) billion
of test suites and regression testing approaches provide annually[3].More than a third of this cost could be avoided if
information about which strategies have to be followed or not. better software testing was performed [3].In case of complex
software development, running the entire test cases require
Keywords many weeks. Test engineers may want to prioritize and
Test Case, Prioritization, FEP, Test Suite, Regression Testing. schedule those test cases in the order that those test cases with
higher priority are executed first.
1. INTRODUCTION Test case prioritization methods and process are required,
1.1 Software Engineering because[4]:
Software is instructions (computer programs) that when (a) the regression testing phase consumes a lot of time and
executed provide desired functionality and performance[1]. A cost to run
good software must provide the required functionality,
performance to the user and should be maintainable, reliable (b) there is not enough time or resources to run the entire test
and usable. Software may be developed for a particular suite
customer or may be developed for a general market. (c) there is a need to decide which test cases to run first.
Computer science focuses on theory and fundamentals but
software engineering is concerned with the practicalities of When the time required to execute all test cases in a test suite
developing and delivering useful software. Software is short, test case prioritization may not be cost effective - it
Engineering is the application of a systematic, disciplined, may be most expedient simply to schedule test cases in any
quantifiable approach to the development, operation, and order [4].When the time required to run all test cases in the
maintenance of software;[1]. While all softwares have to be test suite is sufficiently long, the benefits offered by test case
professionally managed and developed, different techniques prioritization methods become more significant.
are appropriate for different types of system. For example,
games should always be developed using a series of
prototypes whereas safety critical control systems require a
complete and analyzable specification to be developed.
Software testing is a comprehensive set of activities
conducted with the intent of finding errors in the software.
The changed programming must be tried completely with the
purpose so that the changed bit of code does not influence
different parts of the code. Regression testing involves
running test sets that have successfully executed after changes
have been made to the system. The regression test checks that
these changes have not introduced new bugs into the system
and that the new code interacts as expected with the existing
code.[2]. Common methods of regression testing include
rerunning previously completed tests and checking whether
program behavior has changed and whether previously fixed
faults have re-emerged. Regression testing can be performed
to test a system efficiently by systematically selecting the Fig 1: Software Engineering
appropriate minimum set of tests needed to adequately cover a
particular change. 1.2 Regression Testing
Regression testing is important because changes and error Regression testing is frequently applied but it is expensive
corrections tend to be much more error prone than the original maintenance process that aims to (re)verify modified
program code (in much the same way that most typographical software. It is full or partial selection of already executed test
errors in newspapers are the result of last-minute editorial cases which are re-executed to ensure existing functionalities

8
 International Journal of Computer Applications (0975 – 8887)
Volume 116 – No. 16, April 2015

work fine. Regression testing does not belong to either unit test case prioritization technique is the need for present
test, integration test or system testing. Instead, it is a separate software industries. Test case prioritization for the entire
dimension to these three forms of testing[5]. If regression program consumes more time and the selection of test case for
testing exposes problems, then test engineers check whether entire software is also affecting the test performance. In order
there are problems in the previous increment that the new to resolve it, author proposed modular based test case
increment has exposed or whether these are due to the added prioritization for regression testing. In this method the
increment of functionality. program is divided into multiple modules. The test cases
corresponding to each module is prioritized first. In the
For example, there are three modules in the project named second stage, the individual modular based prioritized test
Admin Module, Personal Information and Employment suites are combined together and further prioritized for the
Module and suppose bug occurs in the Admin Module like on whole program. This method is verified for fault coverage and
Admin Module existing user is not able to login with valid compared with overall program test case prioritization
login credentials so this is the bug. Now Testing team sends method. The proposed method is assessed using three
the above - mentioned bug to the development team to fix it standard applications namely University Students Monitoring
and when development team fixes the bug and hand over to System, Hospital Management System, and Industrial Process
testing team. Testing team checks that fixed bug does not Operation System. The empirical studies conducted by the
affect the remaining functionality of the other modules author showed that the proposed algorithm performed
(Admin, PI, Employment) and also the functionality of the significantly well.
same module (Admin) so this is known as the process of
regression testing done by software testers. Thus, regression S.Raju,G.V.Uma[8]"Factors Oriented TestCase Prioritization
testing assures programs not adversely affected by unintended Technique in Regression Testing using Genetic Algorithm",
modifications by running them over existing regression test In this paper the regression testing based test suite
suites. The information collected in the previous executions of prioritization technique is illustrated. A new prioritization
the test cases (e.g., the fault history, the change history, and technique is proposed for the requirement based system level
execution profiles) on the previous versions can be used to test cases to improve the rate of fault detection of severe
optimize the current round of regression testing. Basic faults.
techniques for regression testing incorporate rerunning
already finished tests and checking whether program conduct Chen L., Wang Z. et. al[9] "Test Case Prioritization for Web
has changed and whether beforehand altered shortcomings Service Regression Testing" proposed a dependence analysis
have re-developed. Regression testing can be performed to based test case prioritization technique for Web Service
test a framework productively by deliberately selecting the regression testing. First, they analyzed the dependence
fitting least set of tests expected to sufficiently cover a relationship using control and data flow information in an
specific change. orchestration language: WS-BPEL. Then they construct a
weighted graph. After that, they prioritize test cases according
1.2.1 Need Of Regression to covering more modification-affected elements with the
highest weight. Finally authors conduct a case study to
Regression Testing is required when there is a
illustrate the applicability of method.
 Change in requirements and code is modified according
Gaurav Duggal, Bharti Suri[10] "Understanding Regression
to the requirement
Testing Techniques" described regression testing is done in
 New feature is added to the software
the maintenance phase of the software development life cycle
 Defect fixing to retest the software for the modifications it has undergone.
 Performance issue fixing Approximately 50% of the software cost is involved in the
1.2.2 Types Of Regression maintenance phase. Thus, researchers are working hard to
come up with best results by developing new regression
Regression Testing can be classified as [6].
testing techniques so that the expenditure made in this phase
 Local - Changes introduce new bugs.
can be reduced to some extent. This paper discussed
 Unmasked - Changes unmask previously existing bugs. techniques of regression testing ,test case prioritization,
importance of regression testing and its scope.
 Remote - Changing one part breaks another part of the
program. For example, Module A writes to a database. Shweta A. Joshi et al[11] "Literature Review of Model Based
Module B reads from the database. If changes to what Test case Prioritization" described algorithms for prioritizing
Module A writes to the database break Module B, it is test cases for testing the component-based software systems.
remote regression. The studies show that testing of component-based software
systems is expensive. The importance is given to component
 Corrective interactions because maximum defect occur when components
are going to interact with each other. This approach is
It is applied when specifications are unmodified and test
applicable to test the component composition in case of
cases can be reused. It is triggered by corrections made to
component based software maintenance.
the previous version.
S.Roongruangsuwan et.al[12] "Test-Case Prioritization
 Progressive
techniques" researchers propose many methods to prioritize
It is applied when specifications are modified and new and reduce the effort, time and cost in the software testing
test cases must be designed. It is triggered by new phase, such as test case prioritization methods, regression
features added to the previous version. selection techniques and test case reduction approaches. This

2. LITERATURE REVIEW paper concentrates on test case prioritization techniques

N.Prakash[7] "Modular Based Multiple Test Case researched between 1998 and 2008.
Prioritization " concluded that cost and time effective reliable

9
 International Journal of Computer Applications (0975 – 8887)
Volume 116 – No. 16, April 2015

This paper reveals that there are many research challenges and 3. REGRESSION TESTING
gaps in the test case prioritization area.
TECHNQUIES
Animesh Caturvedi et al [13] “A Tool Supported Approach Software maintenance is an activity which includes
to Perform Efficient Regression Testing of Web Services” enhancements, error corrections, optimization and deletion of
authors presented a tool for perform regression testing of web existing features. These modifications may cause the system
services. In this paper, functional and non-functional testing is to work incorrectly. Therefore, Regression Testing becomes
performed by using WSDL and Regression testing is necessary.
performed by identifying the changes made. Authors identify,
categorize and capture the web service regression testing
needs into three different categories, namely, changes in
WSDL, changes in code and selective re-testing of web
service operations. This paper proposed tool that can used to
find out the exactly which operations are changed and how to
use only those changed operations to minimize regression
testing.
Lijun Mei,Zhenyu Zhang et al[14] "Test Case Prioritization
for Regression Testing of Service-Oriented Business
Applications", researchers have examined the important
impact of heterogeneous artifacts on test case prioritization in
the regression testing of service-oriented business applications
and demonstrated the shortcomings of traditional test case
prioritization techniques in this aspect. The proposed test case
prioritization techniques take into account the coverage data Fig 2: Regression Testing Techniques [18]
of test cases at three levels i.e. workflow, XPath, and WSDL. Regression Testing can be carried out using following
Sebastian Elbaum, Alexey G. Malishevsky[15] "Prioritizing techniques [18].
Test Cases for Regression Testing", empirically examined the Retest All
abilities of several test case prioritization techniques to This is one of the methods for regression testing in which all
improve the rate of fault detection of test suites. It focus on the tests in the existing test bucket or suite should be
version-specific test case prioritization, in which test cases are reexecuted.
prioritized and rate of fault detection is measured, relative to
 This is very expensive as it requires huge time and
specific modified versions of a program.
resources due to execution of unnecessary tests.
Chu-Ti Lin et al [16] "History-Based Test Case Prioritization  When the change to a system is minor, this strategy
with Software Version Awareness" described that numerous would be rather wasteful.
existing prioritization techniques are code-based, in which Regression Test Selection
the testing of every product form is considered as an  Instead of re-executing the entire test suite, it is better to
autonomous procedure. The test after effects of the former select a part of test suite to be run.
programming renditions may be valuable for booking the  Test cases selected can be categorized as
experiments of the later programming forms. The paper a) Reusable Test Cases b) Obsolete Test Cases.
described history-based ways to address this issue. History a) Re-usable Test cases can be used in succeeding
based prioritization is assigned to test case based upon the regression cycles.
software versions and experimentation results of similar type b) Obsolete Test Cases cannot be used in succeeding
of test cases. cycles.
Prioritization of Test Cases
Md. Hossain et al[17] "Regression Testing for Web
 Prioritize the test cases depending on business impact,
Applications Using Reusable Constraint Values" described
critical & frequently used functionalities.
that companies often encounter various security attacks and
frequent feature update demands from users so they need to  Selection of test cases based on priority will greatly
perform frequent regression testing. Web applications require reduce the regression test suite.
regression testing processes that require minimal test effort
because they have already been deployed and used in the
4. TEST CASE
Test case is the triplet [I,S,O], where I is the data input to the
field. In this paper, researchers presented a technique that
system, S is the state of the system at which the data is input,
identifies reusable constraint values for regression test cases
and O is the expected output of the system. Test suite is the
by analyzing definitions and uses of the variables for two
set of all test cases with which a given software product is to
consecutive versions. A large number of constraint input
be tested[5]. Test Cases represent a set of conditions or
values may be reused from the previous version’s test cases.
variables in which a tester will determine whether a system
The constraints and actual values for variables may be
under test satisfies requirements or works correctly. Test
reusable across several versions as long as the definition and
cases should be written by a team member who understands
use relationships of the variables hold across versions. This
the function or technology being tested and each test case
will create greater savings as the applications evolve over
should be submitted for peer review. Test cases act like a
time. It may also reduce testing effort as well as improve
starting point of the test execution. Test case can be designed
testing effectiveness when the major releases are tested
using only the functional specification of the software or
because new test cases and other associated artifacts
thorough knowledge about the internal structure of software.
accumulate over time. The proposed approach is evaluated
using widely used open source web applications with three
versions.

10
 International Journal of Computer Applications (0975 – 8887)
Volume 116 – No. 16, April 2015

4.1 Types of Test Cases c) Mutation Faults

There are two types of test case: Test cases are prioritized by FEP (Fault-Exposing-Potential)
Technique. This technique is achieved by the ability to expose
A. Formal test cases
faults and mutation analysis is used to determine this value.
Formal test case is designed by a known input and expected Each application of a mutation operator will create a mutant
output. The known input should test a precondition and the of the source code, which makes a single valid syntactic
expected output should test a post condition. If the output is as change. The mutation score represents the ratio of mutants
expected then the code is working correctly otherwise it needs that a test-suite can distinguish from the source code. The
modification. In order to fully test that all the requirements of mutation score can be calculated for each test case
an application are met, there must be at least two test cases for separately[15].
each requirement: one positive test and one negative test. The FEP is calculated as
link between the requirement and the test is frequently done
using a traceability matrix. Positive test will represent those | mutants ( s j ) killed by ti |
FEP(ti , s j ) 
conditions in which the software is working properly & | mutants ( s j ) |
negative test will specify those conditions in which the
software is not working as expected or fails. FEP(ti )   j FEP(ti , s j )
B. Informal test cases
Test cases are not written in the formal way but the activities Given program P and test suite T, for each test case tT,
and results are reported after the tests have been run. for each statement s in P, determine the mutation score of t on
Generally, hypothetical stories are used to help the tester think s i.e. the ratio of mutants of s exposed by t to total mutants of
through a complex problem or system. These scenarios are s. A mutant is killed by a test case if the original and the
usually not written down in any detail. They can be as simple mutated program give different outputs when the test case is
as a diagram for a testing environment or a description written executed.
in prose. Informal test case can be used for applications or d) Customer Requirement-Based Prioritization
systems when there is no formal requirements, test cases can Customer requirement-based techniques are the methods to
be written based on the accepted normal operation of prioritize test cases based on the requirement documents.
programs of a similar class. Many weight factors have been used in these techniques,
including custom-priority, requirement complexity and
5. TEST CASE PRIORITIZATION requirement volatility[20]. Prioritization techniques use
TECHNIQUES several factors to weight (or rank) the test cases. Those factors
Test case prioritization techniques provide a way to schedule may be customer-assigned priority (CP), requirements
and run test cases which have the highest priority in order to complexity (RC) and requirements volatility (RV). The
provide earlier detection of faults. Each test case is assigned a scaling factor can be assigned value (1 to 10) for each factor
priority. Priority is set according to some criterion and test for the measurement. Highest factor values indicate a need for
cases with highest priority are scheduled first. prioritization of test case related to that requirement. If
For example criterion may be that the test case which has requirements have high complexity then it leads to maximum
faster code coverage gets the highest priority. There are number of faults.
various prioritization techniques. Fault Severity
Some common test-case prioritization Techniques are:
a) Fault Severity
Test case prioritization depends upon the severity of faults. Code Coverage
The order of execution of test case depends on the size of the
test suite and how long each test case takes time to run. Thus,
through the use of an effective prioritization technique, testers Test Case Mutation Faults
can re-order the test cases to obtain an increased rate of fault Prioritization
Techniques
detection.
The rate of fault detection is good if Customer Requirement
 It reveal faults earlier that have high risk
 It reveal faults related to critical code sections earlier
History
 Provides confidence in the system’s reliability earlier
b) Code Coverage Technique
Test coverage analysis is a measure used in software testing Cost Effective
known as code coverage analysis for practitioners. It describes
the quantity of source code of a program that has been Fig 3: Test Case Prioritization Techniques
exercised during testing[19]. The following lists a process of
e) History based Technique
coverage-based techniques:
History-based techniques are the methods to prioritize test
 Finding areas of a program not exercised by a set of test
cases based on test execution history. It can enhance the
cases
effectiveness of regression testing. For a test case, using
 Creating additional test cases to increase coverage historical information of each prioritization, increase or
 Determining a quantitative measure of code coverage, decrease its likelihood in the current test session. A
which is an indirect measure of quality probability value is given to the history and the value of the
 Identifying redundant test cases that do not increase current session is calculated based on the number of the fault
coverage. detection (or coverage).

11
 International Journal of Computer Applications (0975 – 8887)
Volume 116 – No. 16, April 2015

Finally, the calculated value of a test case is equal to the sum Following are most important tools used for both functional
of the current number multiplied by a probability and the and regression testing[18]:
historical value multiplied by another probability[21]. Quick Test Professional (QTP):HP Quick Test Professional is
f) Cost Effective-based Technique automated software designed to automate functional and
Cost effective-based techniques are methods to prioritize test regression test cases. It uses VbScript language for
cases based on costs, such as cost of analysis and cost of automation. It is a Data driven , Keyword based tool.
prioritization[22]. The cost of a test case is related to the Rational Functional Tester (RFT):IBM's rational functional
resources required to execute and validate it. Cost-cognizant tester is a java tool used to automate the test cases of software
prioritization requires an estimate of the severity of each fault applications. This is primarily used for automating regression
that can be revealed by a test case. To minimize the costs test cases and it also integrates with Rational Test Manager.
associated with testing and with software failures, a goal of Selenium: This is an open source tool used for automating
testing must be to uncover as many defects as possible with as web applications. Selenium can be used for browser based
little testing as possible i.e. write test cases that have a high regression testing.
likelihood of uncovering the faults that are the most likely to
be observed as a failure in normal use. The cost–benefits 8. CONCLUSION
should be considered first before they are applied to large In this paper, regression testing and test case prioritization
programs. methodologies are presented. There are various factors on the
Table 1. Cost Factors basis of which test case prioritize can be decided. It includes
customer requirements, history based, cost analysis, fault
Factor Description exposing, code coverage etc. Test prioritization can strengthen
Execution cost Total cost of running a set of test regression testing for finding more severe fault in earlier
cases. stages. Test case prioritization varies from project to project.
Cost of It includes the cost of source code With earlier prioritization of test cases we can reduce cost,
analysis analysis, analysis of changes time, effort and maximize customer satisfaction. A system can
between old and new versions and be developed using MATLAB to analyze the test case
collection of execution traces. prioritization after performing regression testing on the
Cost of data A total cost of preparing all input developed software. In future, if we have a large test suite
preparation values for test cases. then we can implement the clustering to categorize the faults
Cost of A total cost for validating the and then perform the cluster based prioritization approach.
validation expected result and actual result. Moreover for these techniques, soft computing approaches
like Genetic Algorithms, Fuzzy Logic, Artificial Neural
6. SELECTING TEST CASES FOR Network etc. may be used for experimentation and validation
REGRESSION TESTING purpose.
It was found from the industry data that good number of the
defects reported by customers were due to last minute bug
9. REFERENCES
fixes creating side effects and hence selecting the test case for [1] Pressman, Roger S., "Software engineering: A
regression testing is an art and not that easy. It requires practitioner's approach", McGraw-Hill Companies, 5th
knowledge on the bug fixes and how it affect the system. edition, 2005.
Effective Regression Tests can be done by selecting following [2] Sommerville, Ian. "Software Engineering", Addison
test cases Wesley, 9th edition ,2011.
 Test cases which have frequent defects
 Functionalities which are more visible to the users [3] Hema Srikanth, Laurie Williams and Jason Osborne,
 Test cases which verify core features of the product “System Test Case Prioritization of New and Regression
Test Cases”, Proceedings of the 4th International
which are mandatory requirements of the customer
Symposium on Empirical Software Engineering (ISESE),
 Test cases of functionalities which has undergone more
pp.62–71, IEEE Computer Society, 2005.
and recent changes
 All Integration Test Cases [4] Dennis Jeffrey and Neelam Gupta, “Test Case
 All Complex Test Cases Prioritization Using Relevant Slices”, Proceedings of the
 Boundary value test cases 30th Annual International Computer Software and
 Sample of Successful test cases Applications Conference, Volume 01, 2006, pp.411-420,
 Sample of Failure test cases 2006.
Selection of test cases for regression testing depends more on [5] Mall, Rajib, "Fundamentals of Software engineering",
the criticality of bug fixes than the criticality of the defect PHI Learning Pvt. Ltd., 2014.
itself. A minor defect can result in major side effect and a bug [6] http://www.onestoptesting.com/regression-
fix for an extreme defect can have no or a just a minor side testing/types.asp
effect. So the test engineer needs to balance these aspects for
[7] Prakash, N., & Rangaswamy, T. R., "Modular based
selecting the test cases for regression testing. multiple test case prioritization", Computational
Intelligence & Computing Research (ICCIC), IEEE
7. REGRESSION TESTING TOOLS International Conference (pp. 1-7),2012
When a software undergoes frequent changes, regression
testing costs will escalate. In such cases, manual execution of [8] Raju, S., and G. V. Uma. "Factors oriented test case
test cases increases test execution time as well as costs. prioritization technique in regression testing using
Automation of regression test cases is required in such cases. genetic algorithm." European Journal of Scientific
Extent of automation depends on the number of test cases that Research ,pp. 389-402,2012.
remain re-usable for successive regression cycles.

12
 International Journal of Computer Applications (0975 – 8887)
Volume 116 – No. 16, April 2015

[9] Chen, L., Wang, Z., Xu, L., Lu, H., & Xu, B. "Test case [16] Lin, C. T., Chen, C. D., Tsai, C. S., & Kapfhammer, G.
prioritization for web service regression testing",Service M.,"History-based test case prioritization with software
Oriented System Engineering (SOSE), Fifth IEEE version awareness", 18th International Conference on
International Symposium (pp. 173-178). 2010. Engineering of Complex Computer Systems (ICECCS),
pp. 171-172, 2013.
[10] Duggal, Gaurav, and Bharati Suri. "Understanding
regression testing techniques." Proceedings of the 2nd [17] Md. Hossain, Hyunsook Do, Ravi Eda," Regression
National Conference on Challenges and Opportunities, Testing for Web Applications Using Reusable Constraint
Mandi Gobindgarh, India, March. Vol. 29. 2008. Values", IEEE International Conference on Software
Testing, Verification, and Validation Workshops,pp-312-
[11] Joshi, Shweta A., and B. S. Tiple. "Literature Review of 321 ,DOI 10.1109/ICSTW.2014.35,2014
Model Based Test case Prioritization." International
Journal of Computer Science & Information [18] http://www.guru99.com/regression-testing.html
Technologies,Volume 5, no. 5 ,2014.
[19] Hyunsook Do and Gregg Rothermel, “A Controlled
[12] Roongruangsuwan, Siripong. Jirapun Daengdej,"Test Experiment Assessing Test Case Prioritization
Case prioritization techniques." Journal of theoretical and Techniques via Mutation Faults”, Proceedings of the
applied informational technology,2005. IEEE International Conference on Software
Maintenance, pages 411-420, 2005.
[13] Chaturvedi, Animesh, and Atul Gupta. "A tool supported
approach to perform efficient regression testing of web [20] G. Rothermel, R. Untch, C. Chu, and M. Harrold,“Test
services." Maintenance and Evolution of Service- Case Prioritization”, IEEE Transactions on Software
Oriented and Cloud-Based Systems (MESOCA), IEEE Engineering, vol. 27, pp. 929-948, 2001.
7th International Symposium, pp. 50-55, 2013.
[21] Jung-Min Kim and A. Porter.,"A history-based test
[14] Mei, L., Zhang, Z., Chan, W. K., & Tse, T. H. ,"Test case prioritization technique for regression testing in resource
prioritization for regression testing of service-oriented constrained environments", ICSE ’02: Proceedings of the
business applications", Proceedings of the 18th 24th International Conference on Software Engineering,
international conference on World wide web, pp. 901- pages 119–129, New York, NY, USA, 2002. ACM Press.
910. ACM,2009.
[22] Alexey G. Malishevsky, Gregg Rothermel and Sebastian
[15] Elbaum, S., Malishevsky, A. G., & Rothermel, Elbaum, “Modeling the Cost-Benefits Tradeoffs for
G.,"Prioritizing test cases for regression Regression Testing Techniques", Proceedings of the
testing",ACM,Vol. 25, No. 5, pp. 102-112,2000 International Conference on Software Maintenance
(ICSM’02), 2002.

IJCATM : www.ijcaonline.org 13