Received January 18, 2022, accepted February 11, 2022, date of publication March 22, 2022, date of current

version March 30, 2022.

Digital Object Identifier 10.1109/ACCESS.2022.3161522

A Systematic Literature Review on the

Characteristics and Effectiveness of Web
Application Vulnerability Scanners
SULIMAN ALAZMI 1,2 , (Member, IEEE), AND DANIEL CONTE DE LEON 1, (Member, IEEE)
1 Department of Computer Science, University of Idaho, Moscow, ID 83844, USA
2 Department of Computer Science, Majmaah University, Riyadh 11362, Saudi Arabia

Corresponding author: Suliman Alazmi ([email protected])

The work of Suliman Alazmi was supported by Majmaah University, Saudi Arabia. The work of Daniel Conte de Leon was supported by
the University of Idaho, USA.

ABSTRACT Web applications have been a significant target for successful security breaches in the last few
years. They are currently secured, as a primary method, by searching for their vulnerabilities with specialized
tools referred to as Web Application Vulnerability Scanners (WVS’s). Although, these dynamic approaches
of testing have some advantages, there is still a scarcity of studies that explore their features and detection
capabilities in a systematic way. This article reports findings from a Systematic Literature Review (SLR)
to look into the characteristics and effectiveness of the most frequently used WVS’s. A total of 90 research
papers were carefully evaluated. Thirty (30) WVS’s were collected and reported, with only 12 having at
least one quantitative assessment of effectiveness. These 12 WVS’s were evaluated by 15 original evaluation
studies. We found that these evaluations tested mostly only two of the Open Web Application Security
Project (OWASP) Top Ten vulnerability types: SQL injection (SQLi) (13/15) and Cross-Site Scripting (XSS)
(8/15). Additionally, only one work evaluated six of the OWASP Top Ten vulnerability types and for only one
scanner. We also found that the reported detection rates were highly dissimilar between these 15 evaluations.
Based on these surprising results we suggest avenues for future directions.

INDEX TERMS Web applications, black-box testing, web vulnerability scanner, effectiveness and
performance, OWASP top ten, detection rate.

I. INTRODUCTION user-supplied input, and capable of thinking outside the box.

Web application vulnerabilities such as SQL injection Black-box testing is the most common approach used for
(SQLi), Cross-Site Scripting (XSS), and Cross-Site Request identifying vulnerabilities by testing them dynamically [6].
Forgery (CSRF) are becoming common and widely reported Tools that use a dynamic or Black-box testing approach
nowadays. These vulnerabilities give attackers unauthorized are usually called Web Vulnerability Scanners (WVS’s).
access to sensitive information, such as credit card data, Software developers and cyber security experts use these
accounts, and medical information. Approaches for identi- scanners to find vulnerabilities in web applications. Theses
fying vulnerabilities in web applications are classified as scanners have the capability of automatically evaluating
either Black-box testing or White-box. They are also known the security of web applications with minimal human
as Dynamic Application Security Testing (DAST) or Static intervention and are usually marketed as ‘point-and-click
Application Security Testing (SAST). White-box testing is pentesting’ tools [25]. Many WVS’s, both commercial and
used by security consultants who are well versed in different free/open source, are available to help developers and
programming languages, creating algorithms, and skilled at security analysts discover vulnerabilities in web applications.
inspecting application code [25]. On the other hand, Black- However, these scanners vary in their technical features
box testing is used by cyber security professionals who are and vulnerability detection performance. Therefore, the
experts in the different technologies, skilled in analyzing selection of WVS should be based on various factors such
as scanner characteristics, availability of documentations
The associate editor coordinating the review of this manuscript and and the capability of the scanner to detect vulnerabilities.
approving it for publication was Dongxiao Yu . To the best of our knowledge and date, the present article is

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
33200 VOLUME 10, 2022
S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

the most comprehensive systematic literature review on the Adam et al. [1] defined four phases for a Black-box
effectiveness and characteristics of WVS’s. The contributions vulnerability test:
of this study are the following: 1) Planning phase: The rules and objectives for the test
1) This survey performed a systematic literature review of can be set in this phase.
published studies about WVS’s. It includes four search 2) Discovery phase: This phase is divided into two stages.
engines and four key phrases for a total of 16 searches. The first stage includes the initiation of the test and the
2) The returned publications were then analyzed and collection of information. The second stage performs
classified based on their type of contributions - vulnerability analysis, which occurs after the attack
methodology, approach, evaluation, or survey. phase.
3) Each of the acquired web vulnerability scanners (total 3) Attack phase: This phase examines the various
30) were classified based on seven different charac- vulnerabilities in the target application, which is also
teristics: number of citations, license type, last update known as ‘‘the heart of the test’’.
date, scanner technology, run platform, user interface 4) Reporting phase: This phase provides documentation
type, documentation availability, and capability of with a combination of other phases.
detecting the OWASP Top Ten vulnerabilities. An assessment plan is developed in the planning phase,
4) Data was then collected and tallied regarding the whereas the discovery and attack phases involve recording
reported effectiveness and vulnerability type detection and periodical reporting of events to the director. Finally,
rate of web vulnerability scanners as reported by the a report is presented which explains known vulnerabilities,
obtained (total 15) evaluation studies. the ranking of risks, and tips for the improvement of the
5) All details of the researchers’ method, data, and recognized weaknesses [16].
findings were presented in tables and graphs within this
article to make it a complete, self-contained, and state- C. BLACK-BOX WEB APPLICATION SECURITY TESTING
of-the-art account on web vulnerability scanners. ADVANTAGES
The rest of this article is organized as follows: Section II: • Consistent: The Black-box testing is capable of showing
Background introduces web application vulnerability types the consistencies or inconsistencies of the system’s
and web application scanner types and approaches; requirements specifications [87].
Section III Related Work, describes related but less compre- • Simple: The tester does not have to deal with the
hensive surveys; Section IV: Research Questions; Section V: tested system’s internal structure or code, so the
Research Methods, describes our systematic approach to this tester does not face many difficulties in performing
literature search; Section VI Results and Findings, describes the Black-box testing. It merely involves examining
in detail the findings and results; Section VII: Discussion, the inputs and outputs of the tested system, so it
analyzes the results collected by the study. is not imperative to have in-depth internal knowl-
edge of the system. Moreover, the source code also
does not need to be accessible for conducting the
II. BACKGROUND test [87].
A. WEB APPLICATION VULNERABILITIES • Rapid: Black-box tests do not require a long time for
The vulnerability can be described as the security flaw its preparation because the tester is not required to have
which silently exists in a software program or application. full knowledge of the system in question. These tests
Attackers or malicious entities try to exploit the vulnerability follow the user paths, which are limited in relatively
to gain illegal access to data owned by the application. small systems [87].
The whole exploitation activity includes three actors: the • Impartial: Black-box tests show the result whether or not
application itself (victim), the implementation of the attack the system works. Rather than the ‘‘developer’’ point
to compromise the application (attack), and the entity that of view, the tests are taken from a user point of view,
carries the attack (attacker) [63]. OWASP Top 10 2010, creating independence for each party [87].
OWASP Top 10 2013, OWASP Top 10 2017 and OWASP Top
10 2021 are detailed in Table 1.
D. BLACK-BOX WEB APPLICATION SECURITY TESTING
DISADVANTAGES
B. BLACK-BOX WEB APPLICATION SECURITY TESTING • It is hard to make precise test cases without exact
For many years, the most commonly used approach for specifications [87].
testing a web application is Black-box security testing. • It is not easy to distinguish potential and practicable
This strategy serves as a technique to test the running web inputs in constrained testing time.
applications to discover security vulnerabilities and loopholes • It is possible that the already executed tests may be
without priory knowledge of the application’s internal re-performed by the coder [87].
coding. Typically, the testing team would be considered users • In this testing, several areas of the program may remain
of the application as they are provided with valid access to the untested [87].
user account, where the tester acts like an attacker to find out • It is difficult to insure covering all functionality of web
vulnerabilities and flaws in tested web applications [1]. application [87].

VOLUME 10, 2022 33201

 S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

TABLE 1. Merging of the OWASP Top 10 Vulnerability Types for 2010, 2013, 2017 and 2021. Columns labeled 10, 13, and 17 correspond to the OWASP Top
Ten Vulnerability Types for 2010, 2013, 2017 and 2021 respectively.

33202 VOLUME 10, 2022

S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

TABLE 1. (Continued.) Merging of the OWASP Top 10 Vulnerability Types for 2010, 2013, 2017 and 2021. Columns labeled 10, 13, and 17 correspond to the
OWASP Top Ten Vulnerability Types for 2010, 2013, 2017 and 2021 respectively.

VOLUME 10, 2022 33203

 S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

E. WEB APPLICATION VULNERABILITY were identified as the most cited ones. It also identified
SCANNERS(WVS’S) AND THEIR ARCHITECTURE frameworks, methodologies, and security, testing models.
Black-box WVS’s are the automated testing tools used for Additionally, it analyzed the relationship between scanners
examining and detecting vulnerabilities in web applications. and models besides some challenges of penetration testing.
Several WVS’s test the prevalent vulnerabilities in web The researchers further identified process efficiency and
applications and web servers. These scanners are either effectiveness as critical challenges besides the vulnerability
academic research projects or open-source tools developed assessment process. Also, they noted that challenges in the
by academic members and researchers who are interested in analysis model and security scanners influence the security
studying and improving web application vulnerability tools or of scanners. Our research method is more extensive than
commercial products that are owned by software companies. the above study. We analyzed 320 studies in our work,
The commercialized scanners usually provide more effective and a total of 30 scanners were collected and identified
results than open-source scanners; however, they can cost in the paper. All of the returned scanners ’ characteristics
from just under 100 to over 6000 US dollar [81]. were provided based on what was indicated in the research
The design of a WVS includes three core components papers and information available on the scanners’ websites.
as per the usage scenario. First, the crawler module grabs Similarly, a survey study by Mirjalili et al. [65] explored
the content of the web pages. Second, the attacker module the applications of web penetration testing and its models
is designated for launching the attacks. Third, the analysis and highlighted the comparison between web vulnerability
module highlights vulnerabilities. scanners. The survey reviewed previous literature on pen test
• The Crawling Module is the most crucial component methods and scanners and divided it into three categories.
of a WVS and is performed by utilizing a ‘‘crawler’’ The first category examined and compared various methods
component. It investigates the web application to and scanners. The second one proposed a new method or
identify and recover web pages and the related input scanner for detecting vulnerabilities in web applications.
vectors like input fields in HTML forms, and request The third category involved proposing a proper testing envi-
parameters GET and POST, and cookies. Moreover, the ronment for executing web penetration testing. Moreover,
crawler generates an indexed list of all the crawled web the paper observed a correlation between 13 open-source
pages. The detection of web vulnerabilities determines and seven commercial scanners. It also noted that there are
the quality of the crawler. If the scanner’s attack engine two core factors to judge the effectiveness and efficiency
is subpar, a vulnerability may be missed [48]. of the scanners. First is the ‘‘Structural Design’’ which
• The Fuzzing Module is used to investigate the URLs deals with the GUI (Graphical User Interface), user ease,
of the pages and input vectors. After the crawling, customization, and performance. The other key decision
the attack patterns recognized in the previous step are factor is the ‘‘Supported Features and Functionalities’’,
sent by the crawler to the entry points. It produces a which incorporate crawling techniques (automatic/manual),
vulnerable value that triggers a type of vulnerability analysis techniques, auditing, and logging along with the
for each entry tested using the WVS. For example, the generation of user reports. The researchers found that some
fuzzer tries to detect XSS vulnerabilities by injecting of the reviewed scanners had technical problems such as
malicious JavaScript code or SQL injection vulnerabil- the inability to detect some types of attacks, such as stored
ities by using SQL strings with specific meanings, such SQLi and stored XSS attacks. Also, some scanners did not
as ticks and SQL operators [48]. support new technologies and were incapable of detecting
• The Analysis Module examines the pages that the vulnerabilities attributed to application logic flows. In our
WVS returns due to the attack that the attacker module survey, we identified 21 free/open source and 9 commercial
launched to detect potential vulnerabilities and provide scanners. In addition, our research covers additional aspects
feedback to other modules. For instance, input testing of such as the developer that created the scanner, the technology
SQL injection will return a page that contains a database utilized to design it, and the scanner’s operating platform
error message; then, the analysis module may deduce the (e.g. Windows, Mac OS X or Linux). We also looked at
presence of an SQL injection vulnerability [48]. the scanner’s user interface, whether it was GUI or CLI.
Furthermore, we included the availability of documents, such
III. RELATED WORK as the user manual and installation guide. Another study
Black-box testing has been the focus of many recent studies conducted by Kyriakos et al. [55] reviewed existing literature
aimed at improving security in data, systems, and networks. on web vulnerability scanners. The researchers delved deep
However, only a few surveys and overviews on Black-box into fundamental open-source scanners and databases. They
web vulnerability scanners were returned by this research. examined the web vulnerability of fundamental open-source
Bertoglio and Zorzo [18] systemically reviewed 54 pri- scanners and databases by comparing them based on config-
mary studies using quality criteria to selected papers to uration, functionality, and support. The study also examined
determine reliability and credibility. The criteria grouped the scanners by comparing their accuracy of identifying
papers as ’Good’, ’ Very good ’, and ’Excellent’. The vulnerability, errors in a web application, and their frequency.
study identified scanners used for penetration testing and Moreover, it evaluated the functionality of the scanners based
their characteristics. Based on their analysis, 13 scanners on categorization, vulnerability coverage, risk assessment

33204 VOLUME 10, 2022

S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

inference, and counter-measuring. Besides, the researchers application security scanner remained unknown. It only
determined configuration using architecture, operation sys- showed that the number of testbeds used to benchmark
tem support, level of usage, required resources, modularity, web vulnerability scanners ranged from zero to thousands.
and access control mode. The researchers concluded that Besides, the researchers did not specify measurement metrics
complete benchmarking of vulnerabilities, scanning strategy used in describing the test coverage of web application
and workflow is essential to support the execution of the vulnerability scanners, attack coverage, and vulnerability
scanners. In comparison to this study, our analysis is more detection rate.
complete because it covers the most prevalent commercial
and open-source scanners, whereas this study solely focused IV. RESEARCH QUESTIONS
on open-source scanners. Additionally, we examined and dis- This paper investigates the WVS’s to address three main
cussed common web vulnerability scanner features, whereas Research Questions (RQs):
this study only addressed three: settings, functionality, and RQ.1 What are the most cited web application vulnerability
support. Furthermore, the performance of the researched scanners?
scanners in finding vulnerabilities in web applications was Ans. Table 3 reports the most cited web vulnerability
not included in this study. However, we reviewed in-depth the scanners by other researchers’ studies.
findings of evaluation studies undertaken on these scanners, RQ.2 What are the general characteristics of the reported
as well as the knowledge gap in this domain. scanners?
Kumar and Sheth [56] conducted a review on the Zero-day Ans. Table 4 and Table 5 are built to list all the characteris-
vulnerabilities and the web application scanners that are tics of the scanners to satisfy this question.
used to detect these vulnerabilities in web services. The RQ.3 What are the most common OWASP Top Ten vulner-
study explained different techniques used to detect and abilities tested by the reported scanners?
prevent zero-day vulnerability based on statistical-based Ans. To answer this question and respond, Table 6 and
methods, behaviour and signature-based methods, and hybrid Table 7 contain the evaluation results of studies
techniques. The primary objective of each technique is to conducted by other researchers.
recognize the exploits’ existence, eliminate them in real-
time, and minimize the damage induced by the attack. One V. RESEARCH METHODS
significant challenge is to ensure that the victim’s machine Systematic Literature Review (SLR) refers to the type
threshold delay for analysis and quarantine is not exceeded. of literature review that assists researchers in finding,
However, in some cases, this can cause undermining of the classifying and investigating the existing literature for any
affected system. The researchers concluded that Zero-day particular research query. As its main objective, SLR assesses
attacks could misuse obscure vulnerabilities due to the the already present literature in accordance with the research
absence or lack of antivirus, patches, and intrusion-detection question and finds the gap in it. This SLR is following the
signatures. To combat zero-day attacks, updating the system guidelines provided by PRISMA [91].
can disclose patches for most of the unknown vulnerabilities
that were not detected during the system’s development. A. SEARCH STRATEGY
In addition to that, the researchers suggested a robust PRISMA refers to the minimum set of evidence-based items
framework designed to help the penetration tester detect and used to detail meta-analyses and systematic reviews [91].
prevent zero-day vulnerabilities and remote code execution. As its primary focus, it makes sure that the systematic
Our research is thorough, and it includes information on reviews are reported transparently and completely and also
all vulnerabilities identified by the Open Web Application details information flow through the various phases, such as
Security Project (OWASP), including the OWASP Top identification, screening, eligibility and included as given in
Ten - 2010, the OWASP Top Ten - 2013, and the OWASP Top Figure 1:
Ten - 2017. Furthermore, we looked into both the commercial
and open- source scanners for detecting these security laws. 1) IDENTIFICATION
Seng et al. [78] conducted another survey on the available Researchers used chains of related words to get relevant
methodologies used to assess web vulnerability scanners papers in order to meet the objective of the study. Some of the
regarding test coverage, attack coverage, and vulnerability keywords used in this search included Black-box, penetration
detection rate. It also highlighted the OWASP Top Ten testing, scanner, and tool. The researchers further developed
vulnerabilities in web applications and the popular test-beds an adequate set of search phrases by studying relevant
used to evaluate the web vulnerability scanners. In this study, literature. The selected search phrases include ‘‘vulnerability
the authors investigated some popular web vulnerability scanner’’, ‘‘web application vulnerability scanner’’, ‘‘pene-
scanners, including Acunetix Web Vulnerability Scanner, tration testing tool’’, and ‘‘injection tool’’. However, entering
BurpSuite, N-Sparker, Wapiti, W3af, Vega, Arachni, and ‘‘benchmarking vulnerability scanner’’ and ‘‘benchmarking
Owasp Zap. Nevertheless, the paper could not answer web application vulnerability scanner’’ as key words did not
some of the research questions aimed at quantifying the give any result. The authors surveyed journal papers and
quality of web application security scanners. For instance, international conference proceedings from databases such as
the suitable number of testbeds used to benchmark a web Google Scholar, ACM Digital Library, SpringerLink, and

VOLUME 10, 2022 33205

 S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

FIGURE 1. PRISMA flow diagram for studies selection.

IEEE xplore to acquire relevant research papers. Overall, 3) ELIGIBILITY

320 manuscripts were retrieved with keywords stated above. The researchers assessed the full texts of the 179 articles and
2) SCREENING only collected studies that introduced, compared, evaluated,
The researchers merged the results from all searches, thus or reviewed web vulnerability scanners. As a result, 89 stud-
eliminating duplicate entries. After removing the duplicated ies did not meet the research objectives and were excluded,
articles, the authors obtained a total of 233 papers from the hence.
resources outlined above. They also read the title and the
abstract of each paper for screening purposes and they found 4) INCLUDED
only 179 articles related to web application vulnerability The authors considered ninety (90) studies relevant and
scanners. therefore included them in this study as they fulfilled the

33206 VOLUME 10, 2022

S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

FIGURE 2. Number of papers returned from each source.

FIGURE 3. Distribution of paper types elicited from the research.
objectives of the study and answered the present research
questions. conference proceedings (34 papers), and workshop papers
(5 papers). A majority of these papers (total of 57%),
B. INCLUSION AND EXCLUSION as shown in the figure, are journal articles; the conference
A set of inclusion and exclusion criteria was used to filter all papers represent 38% of the papers, and the rest of the
research papers after their discovery. The following criteria studies compose workshop papers (5%). This implies that
were used to determine papers’ inclusion: the workshop papers have a smaller impact than the other
• Only peer-reviewed articles must be considered. types of literature. Conclusively, the majority (95%) of
• The article should cover Black-box web vulnerability the aforementioned sources constitute journal articles and
scanners. conference proceedings.
• Choosing the most complete version of the study for
inclusion if it has been published in more than one B. STUDIES CLASSIFICATION
journal. The returned studies are classified based on their contribution
However, the exclusion criteria were as follows: to the research. They are categorized into methodology,
• Duplicate studies. approach, evaluation or survey.
• Papers that are unrelated to Black-box web vulnerability Table 2, summarizes the distribution of returned papers,
scanners were omitted. based on their contribution’s classification. This table is
• Inaccessible articles: To receive a private copy of them, divided into three columns as follows:
an email was written to their writers. The articles were • Reference: The number of the study, as cited in this
discarded if no response was received. article, and the name of the first author.
• Articles that are written in a language other than English • Publication Year.
• Very brief publications (e.g., posters) that make only a • Paper Contribution: For example, the contribution of
minor contribution the selected paper may be developing methodology
for detecting vulnerabilities in web applications or
VI. RESULTS AND FINDINGS introducing a new approach which includes all stud-
This section presents the results of the data collection, as well ies that developed a new tool, model, framework,
as how each question was answered. Please note that we only or algorithm to detect the flaws and loopholes in a
report on what was discovered in the reviewed papers. We do web application. Evaluation studies include those which
not personalize the information gathered. introduced comparative evaluation for analyzing the
effectiveness of the web vulnerability scanners, and all
A. PAPERS DISTRIBUTION studies that produced analysis and assessment of the web
The 90 papers that were returned are distributed on the basis application flaws and loopholes. Finally, survey studies
of the search engines they were obtained from, as depicted in that have been conducted on WVS’s or web application
figure 2 below. vulnerabilities are also returned by this research.
There are clear differences in the divisions which are As shown in Figure 4 most of the papers returned from the
apparent in the graph. For example, Google Scholar elicited searches described and presented a new approach with a
the most relevant papers, while SpringerLink produced the total of 33 papers which represent 37% of all the returned
lowest amount. This demonstrates the different outcomes papers. This includes all the studies which have proposed
from these resources when using the designated search terms. revolutionary algorithms or frameworks or developed new
Furthermore, Figure 3 shows that the returned papers fall tool for the purpose of detecting web vulnerabilities. Further-
into three distinct categories: journal articles (51 papers), more, the number of studies that have carried out comparative

VOLUME 10, 2022 33207

 S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

TABLE 2. Classification of papers based on their contributions.

evaluations of the scanners was 28, representing 31% of all Moreover, 19 studies include methodologies for detecting
the returned papers. Interestingly, only 15 of these evaluate vulnerabilities in web applications, which represent 21%
the WVS’s in detecting the OWASP Top 10 vulnerabilities. of all returned papers. Finally, out of the studies collected

33208 VOLUME 10, 2022

S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

• Citation No: Number of citations that mention the given

scanner.
• Studies Mentioning Scanner: List of studies that men-
tioned the scanner, referred to by reference number.
• Number of Evaluation Studies: The number of studies
that evaluated or compared a given scanner.
• Studies Evaluating this Scanner: List of studies that
evaluate a given scanner, referred to by reference
number.

D. GENERAL CHARACTERISTICS OF THE REPORTED

VULNERABILITY SCANNERS
FIGURE 4. Studies’ classification. Table 4 and Table 5 provide some answers to the research
question R2 by illustrating the most important characteristics
of the scanners returned by this study, including information
such as the scanner name, company, license, technology, run
platform, usage, documentation, and OWASP Top 10 Vulner-
ability detection. These characteristics are detailed as below:
• Scanner: A Black-box testing scanner, including both
commercial and open-source.
• Author or Organization: the scanner inventor or pro-
ducer.
• License: The scanner’s license whether it is commercial
or open-source.
• Technology: The programming language that a given
FIGURE 5. Frequency of WVS’s in the returned papers. scanner is implemented in.
• Run Platform: The platform that can be used to run the
through this review, ten surveys returned by this study given scanner.
represent 11% of all the returned papers; however, only five • Usage: The use of the given scanner can be Graphical
surveys were conducted on the Black-box web vulnerability User Interface (GUI) as in Windows or Command
scanners. Line (CLI) as in Linux.
• Documentation: The types of documentation available
C. FREQUENCY OF WEB VULNERABILITY SCANNERS IN for installing and using the scanner.
THE RETURNED PAPERS • OWASP Top 10 vulnerabilities detection: It includes the
Table 3 provides some answers to research question R1, list of scanners that can scan and detect the OWASP Top
where it shows a total of thirty WVS’s were retrieved through 10 vulnerabilities.
the systematic literature review. The table lists all of the A pertinent point that should be highlighted is that most
scanners that were reported by the selected studies and of the studies do not include the scanner’s features and
also displays the number of citations for each. As evident characteristics required for this purpose. To get the more
from Figure 5, Acunetix was the most reported scanner detailed information regarding the scanner’s features and
amongst the selected papers, where a total of 39 returned usage, the respective websites were explored.
studies mentioned this scanner. This was followed by IBM
AppScan as the next highest, which was reported by a total E. COMPARATIVE EVALUATION
of 33 studies. On the other hand, many of the scanners were In consultation with the previous research, it was found
discussed in very few of the selected studies. To name a that numerous comparative studies have been carried out
few of these scanners: SQLInjectionGen, SecuriFly, SQL to improve the accuracy of Black-box web vulnerability
Guard, WebSSARI, VinJect, SQL check, SQLDOM, Havij scanners in detecting web application vulnerabilities. While
and JSPChecker, were all only reported on by one study each. some of the studies focused primarily on comparing scanners
Furthermore, it can also be noted that whilst most of the to report the most effective one, others focused on comparing
scanners are mentioned in numerous publications, very few the effectiveness of a new detection approach to the effec-
studies conducted any critical evaluation with regards to the tiveness of the existing scanners. Based on this research, only
respective scanners. These evaluative studies would look to fifteen (15) evaluation studies were conducted to compare the
compare the scanners based on their capability in detecting capability of twelve (12) WVS’s in detecting vulnerabilities
vulnerabilities outlined by the OWASP Top 10 Vulnerabili- found in web applications. The number of studies evaluating
ties.The columns of the table are described as below: the scanners vary among the scanners. As evident from
• Scanner: A Black-box testing scanner, including both Figure 6, the most evaluated scanners was Acunetix WVS and
commercial and open source. it was evaluated by nine different studies, whereas the lowest

VOLUME 10, 2022 33209

 S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

TABLE 3. The studies reporting on and evaluating WVS’s.

evaluated scanner is QualysGuard as it was evaluated by only Table 6 and Table 7 present the results of all studies
one study. evaluated the scanners and provide some answers to the
33210 VOLUME 10, 2022
S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

TABLE 4. Characteristics of the WAVSs outlined in the literature Part [1].

research question R3. The two tables summarize the detec- rate was used to calculate the ‘group score’. It is clear from
tion rate of the evaluated scanners against the OWASP the two tables (6 and 7) that SQL Injection and Cross-Site
Top 10 Vulnerability types. An empty cell indicates that Scripting are the most frequently detected vulnerabilities
the given scanner has not detected the given vulnerability. from the OWASP Top 10 list as evaluated by prior research.
Some scanners have been placed into ‘groups’ in cases where It can also be observed that the vulnerability detection rates
the source paper did not disclose the specific scanner that of the evaluated WVS’s, generally fall between 0% and 100%
produced the result. In such cases, the scanners performance for detecting SQLi, and they fall between 6% and 100%, for
hit rate was retrieved, and the mathematical mean of the hit detecting XSS. Another interesting observation was made,

VOLUME 10, 2022 33211

 S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

TABLE 5. Characteristics of the WAVSs outlined in the literature Part [2].

33212 VOLUME 10, 2022

S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

used to identify the security loopholes in the available web

services. The researchers decided not to mention the versions
and brands of the evaluated scanners. Thus, they referred
to the scanners as VS1.1, VS1.2, VS2, and VS3, where
VS1.1 and VS1.2 refer to different versions of the same brand.
The four evaluated scanners reported different performances
in detecting SQLi. The detection rate of the fours scanners,
VS1.1, VS1.2, VS2, and VS3 was 17.5%, 16.8%, 20%
and 31.4%, respectively. Therefore, the mean value of the
detection rate for all scanners is 21 %.
Further ahead, Makino and Klyuev [61] evaluated and
compared OWASP ZAP and Skipfish in detecting (SQLi)
and (XSS) in web applications. Two benchmarks were
FIGURE 6. Number of studies evaluated each WVS.
used for evaluating the effectiveness of the compared
scanners, WAVSEP (Web Application Vulnerability Scanner
in one of the works, OWASP ZAP was able to show a 100% Evaluation Project) and DVWA (Damn Vulnerable Web
detection rate of SQLi, while in another work, its detection Application). This evaluation study is used to characterize the
rate was 0%. Vega observed the lowest detection rate for XSS, distinctive features and the detailed analysis of each scanner’s
with that of only 6%; whereas, OWASP ZAP showed the reports and features for the vulnerability analysis. After
highest rate, 100% the very same sort of vulnerability. the detailed analysis, it was concluded that OWASP ZAP
To begin with, Mburano and Si [63] evaluated two performed better than Skipfish in detecting vulnerabilities,
available open-source vulnerability scanners, Arachni and raising fewer false positives.
OWASP ZAP. Two benchmarks were used in this study, Moreover, Antunes and Vieira [7] compared the effective-
namely OWASP and Web Application Vulnerability Security ness of penetration testing and static code analysis techniques
Evaluation Project (WAVSEP) benchmarks. By combining on the detection of SQLi in web services code. They used
the performances of the two evaluated scanners in both three popular commercial WVS’s to detect vulnerabilities
benchmarks, the researchers concluded that OWASP ZAP in a set of vulnerable services. The used scanners include
performed better than Arachni in detecting SQLi whereas, HP WebInspect, IBM Rational AppScan and Acunetix Web
Arachni performed the best in detecting XSS. Furthermore, Vulnerability Scanner. The brands of the scanners were
Alsaleh et al. [6] presented a comparative evaluation of three not mentioned to assure neutrality. Thus, the scanners
open-source WVS’s. This evaluation includes comparing were referred to in this study as VS1, VS2, VS3 (without
the security features of the evaluated scanners as well as any order in particular). The performances of the three
measuring their capabilities in detecting the common web scanners were 50.8%, 36.1% and 9.8% for VS1, VS2, VS3
vulnerabilities. The evaluated scanners include Arachni, respectively. In this evaluation analysis, the mean value was
Wapiti and Skipfish. While, the results of the conducted taken to be 32.2%. Moving further, Šuteva et al. [85] tested
experiment showed disagreements between the generated and assessed six open-source or free WVS’s (principally
reports by the different scanners, the comparative evaluation aimed at false-negative rates) by using the famous and
study did not show significant differences among the vulnerable web application, ‘WackoPicko’. The rates of false
evaluated scanners. The researcher concluded that the latest negatives of all the scanners were very high, ranging from
version of Arachni performed the best among the evaluated 68.8 for IronWasp to 100 for W3af. NetSparker showed
scanners in detecting SQLi and XSS vulnerabilities. a high rate in finding all possible XSS vulnerabilities.
Moving forward, Shelly [81] analyzed the flaws and Also, Aliero et al. [5] conducted an analytical evaluation
limitations of several WVS’s. The evaluated scanners include to compare the effectiveness of their proposed approach-
W3af, Acunetix WVS, Burp Suite Pro, HP WebInspect,IBM SQLIV- with the effectiveness of existing academic scanners
Security AppScan and Netsparker. The researcher developed (Acunetix WVS, IBM Security AppScan, OWAZP ZAP,
a custom vulnerable web application as a testbed used to Wapiti, Vega and W3af). The results showed that the two
test the selected scanners. It had two versions: a secure commercial scanners Acunetix WVS and IBM AppScan
version for detecting false-positive results and an insecure as well as the open-source scanner W3af achieved a high
version for detecting false-negative results. In this evaluation performance of 80 % in detecting SQLi vulnerabilities. Fur-
study, the researcher referred to the evaluated scanners as thermore, Antunes and Vieira [12] proposed a new approach
Scanner A, Scanner B, Scanner C, Scanner D, Scanner E, to designing a vulnerability testing scanner for web services.
and Scanner F without considering the order. The mean value The researchers executed a case study to demonstrate their
of these scanners’ detection rate in detecting SQLi and XSS scanner’s effectiveness in detecting SQLi vulnerabilities in
vulnerabilities was % 96 and % 43 respectively. web services. In this experiment, three commercial scanners
Additionally, Vieira et al. [88] conducted an experimental representing the state-of-the-art vulnerability testing for web
study to evaluate SQLi vulnerability in different web services. applications and web services were used. They include
In this study, three well-known vulnerability scanners were IBM Rational AppScan, HP WebInspect, and Acunetix Web

VOLUME 10, 2022 33213

 S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

Vulnerability Scanner. They referred to them as VS1, VS2, Lastly, Shah [79] conducted an evaluation study to measure
and VS3 without any particular order. The coverage of the the Burp Suite capability in detecting vulnerabilities in web
tools, VS1, VS2, and VS3, stood at 51%, 38%, and 3% applications. The researcher used the OWASP Benchmark to
respectively. Also, the mean value was calculated to be 31%. evaluate the scanner’s detection rate and crawling coverage.
Going further, Martirosyan [62] evaluated the effectiveness of The total number of vulnerabilities detected by the scanner
Acunetix WVS in detecting OWASP Top ten vulnerabilities. is 26, representing 50 % of the SQLi in the tested web
The researcher used the MusicStore web application as a application and produced 0.0 false-positive results. The time
testbed for this study. The evaluation result showed that used to complete the scanning process was six hours and
the scanner detected Insecure Direct Object References twenty minutes. The two benchmarks used for evaluating the
vulnerability with a perfect detection rate of 100 %. However, effectiveness of the compared scanners were Web Applica-
it performed poorly in detecting Insecure Cryptographic tion Vulnerability Scanner Evaluation Project (WAVSEP) and
Storage with detection rate was only 28%. Moreover, Antunes Damn Vulnerable Web Application (DVWA).
and Vieira [10] evaluated three commercial scanners anony-
mously to compare their effectiveness with the effectiveness VII. DISCUSSION
of their approach (SignWS) in detecting SQLi vulnerabilities. The content of this section revolves around examining and
The three commercial scanners include Acunetix, IBM discussing the results of the preceding section. Based on
Rational AppScan, and HP WebInspect. The commercial the results derived from this research, it was found that
scanners were named VS1, VS2, and VS3 without any only a very small number of surveys and overviews have
consideration for the order. The detection rates of VS1, VS2, been conducted on Black-box web vulnerability scanners;
and VS3 were 32.28%, 24.05%, and 1.90%, respectively. a majority of them revolve around merely summarizing the
Additionally, the mean value of the detection rate for the three concepts of the approaches without targeting their charac-
scanners stood at 19%. teristics and effectiveness [18], [55], [65], [78]. However,
Further, Garn et al. [40] provided a methodology for a the present study contains a systematic literature review
better detection process of XSS in web applications. They on the most cited web vulnerability scanners, summarizing
used Burp Suite Pro and OWASP Zed Attack Proxy (ZAP) their characteristics and discussing the results of different
to test their methodology. Mutillidae II version 2.6.3. was evaluation studies conducted to compare their effectiveness
used as a testbed for running this experiment. The result in detecting the common web applications vulnerabilities.
showed that Burp Suite Pro performed better in finding XSS Based on the data collected from the reviewed studies, thirty
vulnerabilities with a detection rate of 88.9%, whereas the (30) scanners were identified and it was found that their
detection rate for ZAP was only 80%. Moving forward, frequencies in the reviewed studies varied from scanner to
DURIĆ [27] ran an evaluation experiment to compare the scanner. For example, it was found that Acunetix WVS was
performance of his approach for detecting SQLi with the the most cited scanner as it was cited by 39 papers; however,
performance of some well-known WVS’s. The selected scan- some scanners including JSPChecker, Havij, SQLDOM,
ners were four open-source scanners: W3af, Nikto, Wapiti, SQL check, Vinject, WebSSARI, SQL Guard, SecuriFly and
Vega, and ZAP, and one commercial scanner, Acunetix. SQLInjectionGen were only reported by one paper each.
The author employed six experienced master students to We also found that there was no major difference between
develop the testing environment for this experiment. The frequency of the commercial and open-source scanners in
result showed that Acunetix achieved the best performance the reviewed papers, which may indicate that open-source
with a detection rate of 50%. Acunetix detected eight scanners have similar importance for the researchers as the
vulnerabilities out of 16, Wapiti detected six, W3af detected commercial scanners. Interestingly, among all returned stud-
five, and Vega detected only one vulnerability. Interestingly, ies, we found that the technical features and characteristics
ZAP did not detect any vulnerabilities in any of the three of the web vulnerability scanners were only discussed by a
applications. Additionally, Antunes and Vieira [8] ran an small number of studies. Consequently, we investigated the
evaluation experiment to compare their approach (VS.WS) scanners’ official websites, and we documented their main
with four commercial vulnerability scanners (two of them characteristics, including the technology utilized to design the
were different versions of the same vendor). The goal scanner, and the scanner’s operating platform (e.g., Windows,
of this study was to identify SQLi vulnerabilities in web Mac OS X or Linux). We also looked at the scanner ’s user
services. The evaluated scanners included HP WebInspect, interface, whether it was GUI or CLI. Moreover, we included
IBM Rational AppScan and Acunetix. Also, to maintain the availability of documentations, such as the user manual
anonymity and equality, the specific scanner applications and installation guide. As a result, we found Java to be
names and their corresponding versions were not mentioned the most frequently used language for designing the tools.
by the researchers. They referred to the four scanners in their We also found that all the identified commercial scanners
study as VS1.1, VS1.2, VS2, and VS3, with VS 1.1 and including Acunetix WVS, HP Webinspect, IBM Security
2.2 being two different versions of the same vendor. The AppScan, Burp Suite pro, NetSparker and QualysGuard were
detection rate of SQL vulnerabilities by VS1.1, VS1.2, VS2, provided with a Graphical User Interface (GUI), while some
and VS3 was 84%, 84%, 30%, and 38% respectively. Also, of the open-source scanners such as Wapiti, Skipfish were
their mean value was calculated to be 59%. implemented with a command-line interface (CLI). Users can

33214 VOLUME 10, 2022

S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

TABLE 6. WVS detection rate results per OWASP Top Ten vulnerability type [1st part].

interact easily with the scanners that use GUI to perform Injection and Cross-Site Scripting. This might be because
the scanning process; however, using scanners with CLI SQL Injection and Cross-Site Scripting are the most popular
mode requires more technical knowledge from the users. web application vulnerabilities and because they are the
Furthermore, even though the present study identified thirty most exploited web application vulnerabilities that yield
(30) web vulnerability scanners, only twelve (12) of them effective results. Furthermore,SQLi attacks enable attackers
were evaluated by prior research. The evaluation studies to access the back-end database of web applications and to
focused on measuring the capabilities of the scanners in exfiltrate, destroy, and modify confidential information. XSS
detecting the OWASP Top 10 vulnerabilities. Based on the attacks may also result in major negative implications. For
data collected from the evaluative studies, it was found example, with XSS, attackers might hijack accounts, and
that most of the OWASP Top 10 vulnerabilities tested by steal credentials and/or other sensitive data. It can also be
the previous studies were SQL Injection and Cross-Site found that the detection rates of the evaluated WVS’s fall
Scripting [6], [7], [10], [12], [27], [40], [79], [81], [85], [88]. between 0% and 100% for SQLi, whereas those for XSS
Only one work evaluated Acunetix WAS scanner against six fall between 6% and 100%. Interestingly, the evaluations
vulnerability types from the OWASP Top 10 list, including conducted in these studies show inconsistencies in the results
Broken Authentication and Session Management, Insecure reported by the different scanners. Moreover, these scanners
Direct Object References, Insecure Cryptographic Storage, significantly vary in the detected vulnerability types, and
and Insufficient Transport Layer Protection, besides SQL the detection rates. In turn,this may drastically decrease the

VOLUME 10, 2022 33215

 S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

TABLE 7. WVS detection rate results per OWASP Top Ten vulnerability type [2nd part].

level of trustworthiness that may be attributed to WVS’s reported detection rates and accuracy of these scanners
and subsequently increase the demand for further research to detect OWASP Top Ten vulnerability types. We have
that quantitatively evaluates the quality and accuracy of web achieved this by examining the published research field of
application vulnerability scanners. web vulnerability scanners in three ways:
VIII. CONCLUSION AND KNOWLEDGE GAPS 1) By examining articles that proposed a new, revolution-
In this article, we have systematically surveyed, collected, ary method, algorithm, or scanner for detecting web
organized, and evaluated most of the available knowledge vulnerabilities.
on web vulnerability scanners. We identified the most 2) By examining the articles that themselves analyzed
frequently used scanners and we investigated their features and compared the existing scanners for detecting web
and characteristics. We also collected and analyzed the vulnerabilities.

33216 VOLUME 10, 2022

S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

3) By drawing insights from the existing surveys and They would also like to thank Dr. Dilshani Sarathchandra,
literature reviews. Dr. Jia Song, and Dr. Stu Steiner for their time and
When we analyzed the relatively few (15) published constructive comments. They would also like to thank the
evaluations of the performance of web vulnerability scanners, reviewers and editors of IEEE Access for their comments and
we discovered two unexpected and, we believe, three very help improving this article.
important findings:
1) SQLi and XSS vulnerability types were the most REFERENCES
common tested types among the OWASP Top Ten [1] A. Doupé, L. Cavedon, C. Kruegel, and G. Vigna, ‘‘Enemy of the state: A
state-awareblack-box web vulnerability scanner,’’ in Presented at the 21st
vulnerability types. The other types of vulnerabilities
USENIX Secur. Symp. (USENIX Secur.), Aug. 2012.
in the OWASP Top Ten list were almost not tested. [2] M. Agarwal and A. Singh, Metasploit Penetration Testing Cookbook.
Only one evaluation was found that reported evaluating Birmingham, U.K.: Packt, 2013.
four (4) other OWASP Top Ten vulnerabilities and this [3] A. Akbulut, ‘‘VinJect: Toolkit for penetration testing and vulnerability
scanning,’’ Düzce Üniversitesi Bilim ve Teknoloji Dergisi, vol. 6, no. 4,
study evaluated only one commercial web vulnerability pp. 779–790, Apr. 2018. [Online]. Available: https://dergipark.org.tr/en/
scanner. A total of 13 studies evaluated SQLi and download/article-file/517130
8 studies evaluated XSS performance for several [4] M. S. Aliero and I. Ghani, ‘‘A component based SQL injection
vulnerability detection tool,’’ in Proc. 9th Malaysian Softw. Eng. Conf.
scanners; However, most studies only evaluated one or (MySEC), Dec. 2015, pp. 224–229, doi: 10.1109/MySEC.2015.7475225.
two scanners against only one or two non-standard, and [5] M. S. Aliero, I. Ghani, K. N. Qureshi, and M. F. Rohani, ‘‘An algorithm for
hence difficult to replicate, web applications. detecting SQL injection vulnerability using black-box testing,’’ J. Ambient
2) After analyzing and collating the efficacy results as Intell. Humanized Comput., vol. 11, no. 1, pp. 249–266, Jan. 2020, doi:
10.1007/s12652-019-01235-z.
published in the 15 evaluations, we found disparate and [6] M. Alsaleh, N. Alomar, M. Alshreef, A. Alarifi, and A. Al-Salman,
inconsistent efficacy reports as detailed in Table 6 and ‘‘Performance-based comparative assessment of open source web vul-
Table 7. nerability scanners,’’ Secur. Commun. Netw., vol. 2017, May 2017,
Art. no. 6158107.
3) We found no published evaluations assessing the [7] N. Antunes and M. Vieira, ‘‘Comparing the effectiveness of pen-
usability or quality of use of web vulnerability etration testing and static code analysis on the detection of SQL
scanners. injection vulnerabilities in web services,’’ in Proc. 15th IEEE Pacific
Rim Int. Symp. Dependable Comput., Nov. 2009, pp. 301–306, doi:
Based on this findings, we would like to make the
10.1109/PRDC.2009.54.
following recommendations for future directions: [8] N. Antunes and M. Vieira, ‘‘Detecting SQL injection vulnerabilities in
1) The effectiveness evaluation of all web vulnerability web services,’’ in Proc. 4th Latin-American Symp. Dependable Comput.,
Sep. 2009, pp. 17–24, doi: 10.1109/LADC.2009.21.
scanners should be carried out using a set of ‘‘bench- [9] N. Antunes and M. Vieira, ‘‘Benchmarking vulnerability detection tools
mark’’ web applications and for all OWASP Top for web services,’’ in Proc. IEEE Int. Conf. Web Services, Jul. 2010,
10 types of vulnerabilities; Such benchmark web appli- pp. 203–210, doi: 10.1109/ICWS.2010.76.
[10] N. Antunes and M. Vieira, ‘‘Enhancing penetration testing with attack
cations currently do not exist. Therefore, new standard signatures and interface monitoring for the detection of injection
and representative benchmark web applications should vulnerabilities in web services,’’ in Proc. IEEE Int. Conf. Services Comput.,
be created. These benchmarks should cover all specific Jul. 2011, pp. 104–111, doi: 10.1109/SCC.2011.67.
domains of web applications. This will help ensure [11] N. Antunes and M. Vieira, ‘‘Defending against web application vul-
nerabilities,’’ Computer, vol. 45, no. 2, pp. 66–72, Feb. 2012, doi:
web vulnerability scanner results are complete and 10.1109/MC.2011.259.
comparable. [12] N. Antunes and M. Vieira, ‘‘Designing vulnerability testing tools for web
2) Evaluations of web vulnerability scanners should be services: Approach, components, and tools,’’ Int. J. Inf. Secur., vol. 16,
no. 4, pp. 435–457, Jun. 2016, doi: 10.1007/s10207-016-0334-0.
based on the OWASP Top Ten vulnerability types or [13] L. Auronen, ‘‘Tool-based approach to assessing web application security,’’
other common nomenclature for web vulnerabilities. Helsinki Univ. Technol., Espoo, Finland, Tech. Rep. T-110.501, 2002,
The lack of standardization in this aspect makes it vol. 11, pp. 12–13. [Online]. Available: http://citeseerx.ist.psu.edu/
viewdoc/download? doi=10.1.1.104.893&rep=rep1&type=pdf
nearly impossible to adequately measure and compare [14] N. F. Awang and A. A. Manaf, ‘‘Detecting vulnerabilities in web
the efficacy of different scanners. applications using automated black box and manual penetration testing,’’
3) Evaluations of web vulnerability scanners should in Proc. Int. Conf. Secur. Inf. Commun. Netw. Cairo, Egypt: Springer,
include disclosures of affiliations or lack-of-thereof Sep. 2013, pp. 230–239, doi: 10.1007/978-3-642-40597-6_20.
[15] T. Basso, P. C. S. Fernandes, M. Jino, and R. Moraes, ‘‘Analysis of the
with commercial sponsors that may be potential biases effect of Java software faults on security vulnerabilities and their detection
for the evaluation. by commercial web vulnerability scanner tool,’’ in Proc. Int. Conf.
4) Evaluations of web vulnerability scanners from a Dependable Syst. Netw. Workshops (DSN-W), Jun. 2010, pp. 150–155, doi:
10.1109/DSNW.2010.5542602.
usability or quality-of-use perspective should also be [16] J. Bau, E. Bursztein, D. Gupta, and J. Mitchell, ‘‘State of the art: Automated
performed. black-box web application vulnerability testing,’’ in Proc. IEEE Symp.
Secur. Privacy, May 2010, pp. 332–345, doi: 10.1145/1135777.1135817.
[17] E. Bazzoli, C. Criscione, F. Maggi, and S. Zanero, ‘‘XSS PEEKER:
ACKNOWLEDGMENT Dissecting the XSS exploitation techniques and fuzzing mechanisms of
The authors would like to thank the faculty and staff blackbox web application scanners,’’ in Proc. 31st IFIP Int. Inf. Secur.
of the Department of Computer Science, the Center for Privacy Conf. (SEC). Ghent, Belgium: Springer, May 2016, pp. 243–258,
Secure and Dependable Systems, and the College of Engi- doi: 10.1007/978-3-319-33630-5_17.
[18] D. Dalalana Bertoglio and A. F. Zorzo, ‘‘Overview and open issues on
neering, at the University of Idaho, and Dr. Mohammad penetration test,’’ J. Brazilian Comput. Soc., vol. 23, no. 1, Dec. 2017, doi:
Alshehri of Majmaah University, for supporting this research. 10.1186/s13173-017-0051-1.

VOLUME 10, 2022 33217

 S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

[19] J. Bozic, D. E. Simos, and F. Wotawa, ‘‘Attack pattern-based combinatorial [40] B. Garn, I. Kapsalis, D. E. Simos, and S. Winkler, ‘‘On the applicability of
testing,’’ in Proc. 9th Int. Workshop Automat. Softw. Test, May 2014, combinatorial testing to web application security testing: A case study,’’ in
pp. 1–7, doi: 10.1145/2593501.2593502. Proc. Workshop Joining AcadeMiA Ind. Contrib. Test Autom. Model-Based
[20] M. Ceccato, C. D. Nguyen, D. Appelt, and L. C. Briand, ‘‘SOFIA: Test. (JAMAICA), 2014, pp. 16–21, doi: 10.1145/2631890.2631894.
An automated security Oracle for black-box testing of SQL-injection [41] J. N. Goel, M. H. Asghar, V. Kumar, and S. K. Pandey, ‘‘Ensemble based
vulnerabilities,’’ in Proc. 31st IEEE/ACM Int. Conf. Automated Softw. Eng., approach to increase vulnerability assessment and penetration testing
Aug. 2016, pp. 167–177, doi: 10.1145/2970276.2970343. accuracy,’’ in Proc. Int. Conf. Innov. Challenges Cyber Secur. (ICICCS-
[21] M. Ceccato and R. Scandariato, ‘‘Static analysis and penetration testing INBUSH), Feb. 2016, pp. 330–335, doi: 10.1109/ICICCS.2016.7542303.
from the perspective of maintenance teams,’’ in Proc. 10th ACM/IEEE [42] H. Gohel and P. Sharma, ‘‘Intelligent web security testing with threat
Int. Symp. Empirical Softw. Eng. Meas., Sep. 2016, pp. 1–6, doi: assessment and client server penetration,’’ in Proc. Int. Conf. ICT Sustain.
10.1145/2961111.2962611. Develop. Singapore: Springer, Feb. 2016, pp. 555–568, doi: 10.1007/978-
[22] S. Chen, ‘‘The web application vulnerability scanners benchmark,’’ Denim 981-10-0135-2_54.
[43] S. Gupta, Web Vulnerability Scanner. New York, NY, USA: Apress,
Group, Israel, Tech. Rep. 201108, 2014.
Nov. 2019, doi: 10.1007/978-1-4842-4341-1_11.
[23] A. S. Choudhary and M. L. Dhore, ‘‘CIDT: Detection of malicious code [44] M. J. Haber and B. Hibbert, Asset Attack Vectors: Building Effective
injection attacks on web application,’’ Int. J. Comput. Appl., vol. 52, no. 2, Vulnerability Management Strategies to Protect Organizations. New York,
pp. 19–26, Aug. 2012, doi: 10.5120/8174-1493. NY, USA: Apress, 2018.
[24] S. Cook. (Feb. 2003). A Web Developers Guide to Cross-Site [45] W. G. J. Halfond and A. Orso, ‘‘Preventing SQL injection attacks using
Scripting. [Online]. Available: https://www.sans.org/reading-room/ AMNESIA,’’ in Proc. 28th Int. Conf. Softw. Eng., New York, NY, USA,
whitepapers/securecode/paper/988 May 2006, pp. 795–798, doi: 10.1145/1134285.1134416.
[25] M. Curphey and R. Arawo, ‘‘Web application security assessment [46] K. P. Haubris and J. J. Pauli, ‘‘Improving the efficiency and effectiveness
tools,’’ IEEE Security Privacy, vol. 4, no. 4, pp. 32–41, Jul. 2006, doi: of penetration test automation,’’ in Proc. 10th Int. Conf. Inf. Technology:
10.1109/MSP.2006.108. New Generat., Apr. 2013, pp. 387–391, doi: 10.1109/ITNG.2013.135.
[26] A. Dessiatnikoff, R. Akrout, E. Alata, M. Kaaniche, and V. Nicomette, [47] H.-C. Huang, Z.-K. Zhang, H.-W. Cheng, and S. W. Shieh, ‘‘Web
‘‘A clustering approach for web vulnerabilities detection,’’ in Proc. IEEE application security: Threats, countermeasures, and pitfalls,’’ Computer,
17th Pacific Rim Int. Symp. Dependable Comput., Dec. 2011, pp. 194–203, vol. 50, no. 6, pp. 81–85, Jun. 2017, doi: 10.1109/MC.2017.183.
doi: 10.1109/PRDC.2011.31. [48] S. Idrissi, N. Berbiche, F. Guerouate, and M. Shibi, ‘‘Performance
[27] Z. C. Dhuri, ‘‘WAPTT-Web application penetration testing tool,’’ Adv. evaluation of web application security scanners for prevention and
Elect. Comput. Eng., vol. 14, no. 1, pp. 93–102, Nov. 2014, doi: protection against vulnerabilities,’’ Int. J. Appl. Eng. Res., vol. 12,
10.4316/AECE.2014.01015. no. 21, pp. 11068–11076, Jun. 2017. [Online]. Available: https://www.
[28] Z. Djuric, ‘‘A black-box testing tool for detecting SQL injection semanticscholar.org/paper/Performance-Evaluation-of-Web-Application-
vulnerabilities,’’ in Proc. 2nd Int. Conf. Informat. Appl. (ICIA), Sep. 2013, Security-Idrissi-Berbiche/5c56cf1cd211c2810f8217585123c4b99ba5b15a
[49] S. Jose, K. Priyadarshini, and K. Abirami, ‘‘An analysis of black-box web
pp. 216–221, doi: 10.1109/ICoIA.2013.6650259.
application vulnerability scanners in SQLi detection,’’ in Proc. Int. Conf.
[29] A. Doupé, L. Cavedon, C. Kruegel, and G. Vigna, ‘‘Enemy of the state: A
Soft Comput. Syst. New Delhi, India: Springer, Dec. 2016, pp. 177–185,
state-aware black-box web vulnerability scanner,’’ in Presented at the 21st
doi: 10.1007/978-81-322-2674-1_18.
USENIX Secur. Symp. (USENIX Secur.), Aug. 2012. [50] C. Joshi and U. Kumar, ‘‘Security testing and assessment of vulner-
[30] A. Doupé, M. Cova, and G. Vigna, ‘‘Why Johnny can’t pentest: An analysis ability scanners in quest of current information security landscape,’’
of black-box web vulnerability scanners,’’ in Proc. 7th Int. Conf. Detection Int. J. Comput. Appl., vol. 145, no. 2, pp. 1–7, Jul. 2016, doi:
Intrusions Malware, Vulnerability Assessment. Bonn, Germany: Springer, 10.5120/ijca2016910563.
2010, pp. 111–131, doi: 10.1007/978-3-642-14215-4_7. [51] K. Kemalis and T. Tzouramanis, ‘‘SQL-IDS: A specification-based
[31] D. M. Duchesne, ‘‘Using CABECTPortal as a case study to extend approach for SQL-injection detection,’’ in Proc. ACM Symp. Appl.
the capabilities of penetration testing tools,’’ in Proc. 46th ACM Tech. Comput. (SAC), New York, NY, USA, 2008, pp. 2153–2158, doi:
Symp. Comput. Sci. Educ., New York, NY, USA, Feb. 2015, p. 715, doi: 10.1145/1363686.1364201.
10.1145/2676723.2693629. [52] C. Khairnar, ‘‘Detection and automatic prevention against SQL injection
[32] N. Duhan and B. Saneja, ‘‘A two tier defense against SQL injection,’’ in attack and XSS attacks perform on web application,’’ Maharashtra
Proc. Int. Conf. Signal Propag. Comput. Technol. (ICSPCT), Jul. 2014, India, vol. 5, no. 11, Nov. 2015. [Online]. Available: https://www.
pp. 415–420, doi: 10.1109/ICSPCT.2014.6884906. semanticscholar.org/paper/Detection-and-Automatic-Prevention-against-
[33] J. Eckroth, K. Chen, H. Gatewood, and B. Belna, ‘‘Alpaca: Build- SQL-and-Khairnar/b2f6bee4c95ed56f2ba6df5efdc0f7f441875840
ing dynamic cyber ranges with procedurally-generated vulnerability [53] N. Khoury, P. Zavarsky, D. Lindskog, and R. Ruhl, ‘‘Testing and assessing
lattices,’’ in Proc. ACM Southeast Conf., Apr. 2019, pp. 78–85, doi: web vulnerability scanners for persistent SQL injection attacks,’’ in Proc.
10.1145/3299815.3314438. 1st Int. Workshop Secur. Privacy Preserving e-Societies (SeceS), 2011,
[34] B. Eshete, A. Villafiorita, K. Weldemariam, and M. Zulkernine, ‘‘Con- pp. 12–18, doi: 10.1145/2107581.2107584.
feagle: Automated analysis of configuration vulnerabilities in web [54] D. Kilaru, ‘‘Improving techniques for SQL injection defenses,’’ M.S. the-
applications,’’ in Proc. IEEE 7th Int. Conf. Softw. Secur. Rel., Jun. 2013, sis, Dept. Comput. Sci., University of Colorado Colorado Springs, Boulder,
pp. 188–197, doi: 10.1109/SERE.2013.30. CO, USA, 2017.
[55] K. Kritikos, K. Magoutis, M. Papoutsakis, and S. Ioannidis, ‘‘A survey on
[35] Y. Fang, X. Long, L. Liu, and C. Huang, ‘‘DarkHunter: A fingerprint
vulnerability assessment tools and databases for cloud-based web applica-
recognition model for web automated scanners based on CNN,’’ in Proc.
tions,’’ Array, vols. 3–4, Sep. 2019, Art. no. 100011. [Online]. Available:
2nd Int. Conf. Cryptogr., Secur. Privacy, Mar. 2018, pp. 10–15, doi:
https://www.sciencedirect.com/science/article/pii/S2590005619300116
10.1145/3199478.3199504. [56] P. Kumar and R. K. Sheth, ‘‘A review on 0-day vulnerability testing in web
[36] J. Fonseca and F. Matarese, ‘‘Using vulnerability injection to improve web application,’’ in Proc. 2nd Int. Conf. Inf. Commun. Technol. Competitive
security,’’ in Innovative Technologies for Dependable OTS-Based Critical Strategies (ICTCS), 2016, pp. 1–4, doi: 10.1145/2905055.2905357.
Systems. Milan, Italy: Springer, Jan. 2013, pp. 145–157, doi: 10.1007/978- [57] R. Kumar and K. Tlhagadikgora, ‘‘Internal network penetration test-
88-470-2772-5_11. ing using free/open source tools: Network and system administration
[37] J. Fonseca, M. Vieira, and H. Madeira, ‘‘Testing and comparing web approach,’’ in Proc. Int. Conf. Adv. Inform. Comput. Res. Shimla, India:
vulnerability scanning tools for SQL injection and XSS attacks,’’ in Proc. Springer, Jul. 2018, pp. 257–269, doi: 10.1007/978-981-13-3143-5_22.
13th Pacific Rim Int. Symp. Dependable Comput. (PRDC), Dec. 2007, [58] N. M. Z. Lee, S. Y. Ooi, and Y. H. Pang, ‘‘Vulnerability reports
pp. 365–372, doi: 10.1109/PRDC.2007.55. consolidation for network scanners,’’ in Proc. Int. Conf. Comput. Sci.
[38] J. Fonseca, M. Vieira, and H. Madeira, ‘‘Vulnerability & attack injection Technol. Kuala Lumpur, Malaysia: Springer, Feb. 2017, pp. 11–20, doi:
for web applications,’’ in Proc. IEEE/IFIP Int. Conf. Dependable Syst. 10.1007/978-981-10-8276-4_2.
Netw., Jun./Jul. 2009, pp. 93–102. [59] T. Lee, G. Won, S. Cho, N. Park, and D. Won, ‘‘Experimentation
[39] J. Fonseca, M. Vieira, and H. Madeira, ‘‘Evaluation of web security and validation of web application’s vulnerability using security testing
mechanisms using vulnerability & attack injection,’’ IEEE Trans. Depend- method,’’ in Computer Science and Its Applications. Dordrecht, The
able Secure Comput., vol. 11, no. 5, pp. 440–453, Sep./Oct. 2013, doi: Netherlands: Springer, Oct. 2012, pp. 723–731, doi: 10.1007/978-94-007-
10.1109/TDSC.2013.45. 5699-1_74.

33218 VOLUME 10, 2022

S. Alazmi, D. C. De Leon: SLR on Characteristics and Effectiveness of Web Application Vulnerability Scanners

[60] X. Li and Y. Xue, ‘‘A survey on server-side approaches to securing web [82] A. Kumar Singh and S. Roy, ‘‘A network based vulnerability scanner
applications,’’ ACM Comput. Surv., vol. 46, no. 4, pp. 1–29, Apr. 2014, for detecting SQLI attacks in web applications,’’ in Proc. 1st Int.
doi: 10.1145/2541315. Conf. Recent Adv. Inf. Technol. (RAIT), Mar. 2012, pp. 585–590, doi:
[61] Y. Makino and V. Klyuev, ‘‘Evaluation of web vulnerability scan- 10.1109/RAIT.2012.6194594.
ners,’’ in Proc. IEEE 8th Int. Conf. Intell. Data Acquisition Adv. [83] S. Singh and K. Singh, ‘‘Performance analysis of vulnerability detection
Comput. Syst., Technol. Appl. (IDAACS), Sep. 2015, pp. 399–402, doi: scanners for web systems,’’ in Cyber Security. Singapore: Springer,
10.1109/IDAACS.2015.7340766. Apr. 2018, pp. 387–399, doi: 10.1007/978-981-10-8536-9_37.
[62] Y. Martirosyan, ‘‘Security evaluation of web application vulnerability [84] A. Steinhauser and F. Gauthier, ‘‘JSPChecker: Static detection of context-
scanners strengths and limitations using custom web application,’’ Ph.D. sensitive cross-site scripting flaws in legacy web applications,’’ in Proc.
dissertation, Dept. Comput. Sci., California State Univ.-East Bay, Hay- ACM Workshop Program. Lang. Anal. Secur., Oct. 2016, pp. 57–68, doi:
ward, CA, USA, Oct. 2012. [Online]. Available: http://citeseerx.ist.psu. 10.1145/2993600.2993606.
edu/viewdoc/download?doi=10.1.1.722.5637&rep=rep1&type=pdf [85] N. Šuteva, D. Zlatkovski, and A. Mileva, ‘‘Evaluation and testing of
[63] B. Mburano and W. Si, ‘‘Evaluation of web vulnerability scanners based several free/open source web vulnerability scanners,’’ in Proc. 10th Conf.
on OWASP benchmark,’’ in Proc. 26th Int. Conf. Syst. Eng. (ICSEng), for Informat. Inf. Technol., Apr. 2013, pp. 221–224. [Online]. Available:
Dec. 2018, pp. 1–6, doi: 10.1109/ICSENG.2018.8638176. http://eprints.ugd.edu.mk/id/eprint/9096
[64] S. McAllister, E. Kirda, and C. Kruegel, ‘‘Leveraging user interactions for [86] A. Tajpour and S. Ibrahim, ‘‘SQL injection detection and prevention tools
in-depth testing of web applications,’’ in Proc. Int. Workshop Recent Adv. assessment,’’ in Proc. Int. Conf. Comput. Sci. Inf. Technol., vol. 9, Jul. 2010,
Intrusion Detection, Sep. 2008, pp. 191–210. pp. 518–522, doi: 10.1109/ICCSIT.2010.5563777.
[65] M. Mirjalili, A. Nowroozi, and M. Alidoosti, ‘‘A survey on web penetration [87] F. van der Loo, ‘‘Comparison of penetration testing tools for web
test,’’ Adv. Comput. Sci., Int. J., vol. 3, no. 6, pp. 107–121, 2014. [Online]. applications,’’ Ph.D. dissertation, Dept. Comput. Sci., Univ. Radboud,
Available: http://www.acsij.org/documents/v3i6/acsij-2014-3-6-604.pdf Nijmegen, The Netherlands, 2011.
[66] I. Mukhopadhyay, S. Goswami, and E. Mandal, ‘‘Web penetration testing [88] M. Vieira, N. Antunes, and H. Madeira, ‘‘Using web security scan-
using nessus and metasploit tool,’’ IOSR J. Comput. Eng., vol. 16, no. 3, ners to detect vulnerabilities in web services,’’ in Proc. IEEE/IFIP
pp. 126–129, 2014, doi: 10.9790/0661-1634126129. Int. Conf. Dependable Syst. Netw., Jun. 2009, pp. 566–571, doi:
[67] F. Román Muñoz, I. I. Sabido Cortes, and L. J. García Villalba, ‘‘Enlarge- 10.1109/DSN.2009.5270294.
ment of vulnerable web applications for testing,’’ J. Supercomput., vol. 74, [89] S. Wagner, ‘‘The use of application scanners in software product quality
no. 12, pp. 6598–6617, Dec. 2018, doi: 10.1007/s11227-017-1981-2. assessment,’’ in Proc. 8th Int. Workshop Softw. Qual., Sep. 2011,
[68] F. Román Muñoz and L. J. García Villalba, ‘‘An algorithm to find pp. 42–49, doi: 10.1145/2024587.2024597.
relationships between web vulnerabilities,’’ J. Supercomput., vol. 74, no. 3, [90] X. Wang, L. Wang, G. Wei, D. Zhang, and Y. Yang, ‘‘Hidden web
pp. 1061–1089, Mar. 2018, doi: 10.1007/s11227-016-1770-3. crawling for SQL injection detection,’’ in Proc. 3rd IEEE Int. Conf.
[69] B. Nagpal, N. Singh, N. Chauhan, and A. Panesar, ‘‘Tool based Broadband Netw. Multimedia Technol. (IC-BNMT), Oct. 2010, pp. 14–18,
implementation of SQL injection for penetration testing,’’ in Proc. doi: 10.1109/ICBNMT.2010.5704860.
Int. Conf. Comput., Commun. Autom., May 2015, pp. 746–749, doi: [91] A. Liberati, D. G. Altman, J. Tetzlaff, C. Mulrow, P. C. Gøtzsche,
10.1109/CCAA.2015.7148509. J. P. A. Ioannidis, M. Clarke, P. J. Devereaux, J. Kleijnen, and D. Moher,
[70] S. Nagpure and S. Kurkure, ‘‘Vulnerability assessment and pene- ‘‘The PRISMA statement for reporting systematic reviews and meta-
tration testing of web application,’’ in Proc. Int. Conf. Comput., analyses of studies that evaluate health care interventions: Explanation and
Commun., Control Autom. (ICCUBEA), Aug. 2017, pp. 1–6, doi: elaboration,’’ J. Clin. Epidemiol., vol. 62, no. 10, pp. e1–e34, Oct. 2009,
10.1109/ICCUBEA.2017.8463920. doi: 10.1016/j.jclinepi.2009.06.006.
[71] G. K. Pannu, ‘‘A survey on web application attacks,’’ [92] (2021). OWASP Top 10. Accessed: Mar. 11, 2022. [Online]. Available:
Int. J. Comput. Sci. Inf. Technol., vol. 5, no. 3, https://owasp.org/Top10/
pp. 4162–4166, 2014. [Online]. Available: https://pdfs.
semanticscholar.org/0672/d34966bf24aae5fd875111e8c9444c3132d6.pdf
[72] M. Parvez, P. Zavarsky, and N. Khoury, ‘‘Analysis of effectiveness
SULIMAN ALAZMI (Member, IEEE) received the bachelor’s degree in
of black-box web application scanners in detection of stored SQL
injection and stored XSS vulnerabilities,’’ in Proc. 10th Int. Conf. for computer science from King Saud University, Saudi Arabia, in 2004, and
Internet Technol. Secured Trans. (ICITST), Dec. 2015, pp. 186–191, doi: the master’s degree in computer science with concentration of computer and
10.1109/ICITST.2015.7412085. network security from the University of Wollongong, Australia, in 2009.
[73] W. Qianqian and L. Xiangjun, ‘‘Research and design on web application He is currently pursuing the Ph.D. degree with the Department of Computer
vulnerability scanning service,’’ in Proc. IEEE 5th Int. Conf. Softw. Eng. Science, College of Engineering, University of Idaho, USA. His research
Service Sci., Jun. 2014, pp. 671–674, doi: 10.1109/ICSESS.2014.6933657. interests include network security, information assurance, web vulnerability
[74] H. Radwan and K. Prole, ‘‘Code pulse: Real-time code coverage for analysis and assessment, and digital forensics.
penetration testing activities,’’ in Proc. IEEE Int. Symp. Technol. Homeland
Secur. (HST), Apr. 2015, pp. 1–6, doi: 10.1109/THS.2015.7225269.
[75] A. Sagala and E. Manurung, ‘‘Testing and comparing result scanning using
web vulnerability scanner,’’ Adv. Sci. Lett., vol. 21, no. 11, pp. 3458–3462,
Nov. 2015, doi: 10.1166/asl.2015.6598.
[76] A. Z. M. Saleh, N. A. Rozali, A. G. Buja, K. A. Jalil, F. H. M. Ali, and
T. F. A. Rahman, ‘‘A method for web application vulnerabilities detection
by using Boyer-Moore string matching algorithm,’’ Proc. Comput. Sci.,
vol. 72, pp. 112–121, Jan. 2015, doi: 10.1016/j.procs.2015.12.111. DANIEL CONTE DE LEON (Member, IEEE)
[77] S. Sandhya, S. Purkayastha, E. Joshua, and A. Deep, ‘‘Assessment of received the Ph.D. degree in computer science
website security by penetration testing using wireshark,’’ in Proc. 4th Int. with a focus on the security and safety of critical
Conf. Adv. Comput. Commun. Syst. (ICACCS), Jan. 2017, pp. 1–4, doi: systems from the University of Idaho, Moscow,
10.1109/ICACCS.2017.8014711. ID, USA, in 2006. He is currently a Cybersecurity
[78] L. K. Seng, N. Ithnin, and S. Z. M. Said, ‘‘The approaches to quantify web Researcher and an Educator at the University of
application security scanners quality: A review,’’ Int. J. Adv. Comput. Res., Idaho. He also works on developing innovative
vol. 8, no. 38, pp. 285–312, Sep. 2018. and hands-on methods and tools for cybersecurity
[79] M. P. Shah, ‘‘Comparative analysis of the automated penetration testing
learning. His teaching experience comes from
tools,’’ Nat. College Ireland, Ireland, Tech. Rep. X18139469, 2019.
[80] S. Shah and B. M. Mehtre, ‘‘An overview of vulnerability assessment and many years of teaching across the computer
penetration testing techniques,’’ J. Comput. Virol. Hacking Techn., vol. 11, science and cybersecurity curriculums. His current research interests
no. 1, pp. 27–49, Feb. 2015, doi: 10.1007/s11416-014-0231-x. include development of methods and tools for the design, development,
[81] D. A. Shelly, ‘‘Using a web server test bed to analyze the limitations of implementation, configuration, operation, and maintenance of safe and
web application vulnerability scanners,’’ Ph.D. dissertation, Dept. Comput. secure critical infrastructure systems.
Eng., Virginia Tech, Blacksburg, VA, USA, Jul. 2010.

VOLUME 10, 2022 33219