See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/348756994

A Process to Facilitate Automated Automotive Cybersecurity Testing

Preprint · January 2021

CITATIONS READS

0 97

9 authors, including:

Stefan Marksteiner Nadja Marko

AVL LIST GMBH Virtual Vehicle
36 PUBLICATIONS   127 CITATIONS    18 PUBLICATIONS   198 CITATIONS   

SEE PROFILE SEE PROFILE

Andre Smulders Stylianos Karagiannis

TNO Ionian University
6 PUBLICATIONS   45 CITATIONS    20 PUBLICATIONS   36 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

IoT4CPS - Trustworthy IoT for CPS View project

Preparatory study on Smart Appliances View project

All content following this page was uploaded by Stefan Kraxberger on 09 June 2021.

The user has requested enhancement of the downloaded file.

 A Process to Facilitate Automated Automotive
Cybersecurity Testing
Stefan Marksteiner Nadja Marko Andre Smulders
AVL List Virtual Vehicle Resarch TNO
Email: [email protected] Email: [email protected] Email: [email protected]

Stelios Karagiannis Florian Stahl Hayk Hamazaryan

Beyond Vision AVL Software & Functions ZF Friedrichshafen
Email: [email protected] Email: [email protected] Email: [email protected]

Rupert Schlick Stefan Kraxberger Alexandr Vasenev

Austrian Institute of Technology Seclnto Joint Innovation Centre ESI (TNO)
Email: [email protected] Email: [email protected] Email: [email protected]

Abstract-Modem vehicles become increasingly digital­ effort is ISO/SAE DIS 21434 [5]. Also, regulators be­
ized with advanced information technology-based solutions gin to take cybersecurity considerations into account;
like advanced driving assistance systems and vehicle-to-x the most prominent example being a recent regulation
communications. These systems are complex and intercon­
nected. Rising complexity and increasing outside exposure by the United Nations [6]. The rising incidents and
has created a steadily rising demand for more cyber-secure the regulators' requirements demand a substantially
systems. Thus, also standardization bodies and regulators higher amount of cybersecurity engineering and testing
issued standards and regulations to prescribe more secure of automotive systems. This also creates the need for
development processes. This security, however, also has to higher efficiency which makes a standardized method
be validated and verified. In order to keep pace with the need
for more thorough, quicker and comparable testing, today's of automotive cybersecurity testing necessary. Currently,
generally manual testing processes have to be structured automotive cybersecurity testing is mostly not holistic,
and optimized. Based on existing and emerging standards unstructured, non-reproducible and more art than crafts.
for cybersecurity engineering, this paper therefore outlines An approach to giving a standardized and industrial­
a structured testing process for verifying and validating grade testing process is therefore necessary to cope
automotive cybersecurity, for which there is no standardized
method so far. Despite presenting a commonly structured with these upcoming challenges and will also be a
framework, the process is flexible in order to allow imple­ prerequisite to automate steps of testing in this domain.
menters to utilize their own, accustomed toolsets. This paper therefore presents an approach to such a
Index Terms-Security, Cybersecurity, Testing, Automotive, standardized testing process.
Validation, Verification, Process This paper structures as follows: Section II contains
related work, Section III definitions, while Section IV
1. INT RODUCT ION describes some process-intemal relations. Section V de­
The rising complexity of modern automotive systems scribes the proposed automotive security testing process
make it increasingly difficult to assure their cyberse­ and, finally, Section VI concludes this paper.
curity. This is especially true due to the utilization of
advanced driving assistance systems (ADAS) and au­ II. REL ATED WORK
tonomous driving (AD) and the exposure to the outside The importance of creating generic testing frameworks
by to vehicle-to-x (V2X) functions. This usage of new or security testbeds to conduct automated security tests
technology is likely to accelerate even more; also market­ in automotive has already been highlighted in the past.
leading manufacturers are beginning to equip their most­ More specifically, fuzz testing methods in accordance
selling model with V2X off-the-shelf [1]. These devel­ to industry-specific technologies such as the Controller
opments facilitate cybersecurity incidents (e.g. [2], [3]). Area Network (CAN bus) and the vehicle's electronic
Furthermore, an exponential rise of events and an accel­ control unit (ECU) [7], [8], [9], [10] have been created.
eratingly adverse ratio between criminal activities versus Similarly, there are frameworks that address system­
benevolent security research results can be observed atic methods of security testing for automotive Blue­
[4]. This has also been recognized by standardization tooth, Vehicular Ad Hoc Networks (VANETS) in Intelli­
bodies - currently, the most important standardization gent Transportation Systems (ITS) and road services [11],

© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in
any current or future media, including reprinting/republishing this material for advertising or promotional
purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted
component of this work in other works.
[12], [13]. Finally there are integrated security testing a scenario. These means concretized with test scripts
frameworks that improve the standardized methods [9], (attack steps - e.g. ./cansend canO 7df#02010d) that even­
[14], [15], [16], [17]. All of these works, however, do tually form test cases (concretizations of test scenarios -
not encompass a defined process for automated security see Sections V-E and V-F). The difference is that scenarios
testing of complete automotive systems in a holistic and pattems only generically describe what to do, while
manner. This work therefore complements the standards the cases and scripts are concrete elaborations how to do
with a structured testing approach and underpins the it. The test cases therefore augment scenarios during the
technical testing solutions with a structured workflow tcg with information (e.g. CAN messages) from a specific
method. As the upcoming ISO/SAE 21434 [5] is regarded SUT in order to test it. The purpose of the test cases
to become the most important guideline, the process is to allow for automated testing using an appropriate
aligns to it (see V). There is a supplement to the ISO framework. Figure 1 illustrates the coherence between
standard regarding testing (ISO/WO PAS 5112), which, the test scenarios with their pattems and the cases with
however, is in a larval state. their scripts: scenarios consist of abstract pattems and
cases consist of concrete scripts.
III. DEFINITIONS
Testing in the context of this paper means verification
and validation in the sense of ISO/SAE DIS 21434 (Sec­
tions 10.4.2 and 11) [5]. An Item, according to the ISO
standard mentioned above, is a system or combination
thereof to implement a function at the vehicle level. In
the sense of this process, an item is understood as a
technical concept that defines such a system. A security
goal is a desired, security-related property of an item,
which is analyzed for threats and risks that lead to security
requirements that are collected in a security concept. A
System-under-test (SUT) is a concrete technology unit,
e.g. a vehicle, a single Electronic Control Unit (ECU)
or a software, that concretely instantiates an item that
is an examination subject in this process. Therefore, an
item can address multiple SUTs(e.g. different cars of the
same type). A fest system is the active unit that carries Figure 1. Relation between generic test scenarios/patterns and con­
crete test cases/scripts
out the test on an SUT. The proprietor of an SUT or
item, respectively, is an item owner. The user of a test
system is a fest operator. The compound of SUTs and V. AUTOMOTIVE SECURITY PROCESS
test system(s) including interfaces and surroundings (e.g.
This section outlines the security testing process which
an automotive testbed) constitute a fest environment. Test
consists of the following activities:
scenarios in the context of this paper are abstract test
descriptions that define what to test by which means, 1) Define Item;
consisting of fest patterns as their atomic elements that 2) Perform Risk and Threat Analysis;
describe single scenario stages. They are derived from 3) Define Security Concept (testing requirements);
a security analysis and requirements definition of the 4) Plan Test and Develop Scenarios;
respective item. Test cases are the concretization of sce­ a) Define Penetration Test Scenarios;
narios for a specific SUT, consisting of fest scripts, which b) Define Functional and Interface Test Scenarios;
are executable tests that run on a test system and target c) Define Fuzz Testing Scenarios;
an item. d) Define Vulnerability Scanning Scenarios;
5) Select Test Scripts;
IV. RELATION BETWEEN TEST SCENARIOS AND TEST
a) Develop Test Scripts;
CASES
b) Validate Test Scripts;
The reason to provide abstract test descriptions, that 6) Generate Test Cases;
preempt some test case generation (tcg) operations, is 7) Perform Test;
portability to other items. Although test scenarios derive
from threat assessments, they are described generic, con­ a) Prepare Test Environment;
veniently in a domain specific language (DSL - e.g. [18]). b) Execute Test Cases
Test patterns are, consequently, generic means to test a 8) Generate Test Reports.
part of the system - e.g. sending a CAN message with a The process is based on the security testing sections
break signal (e.g. SEND CAN_MSG()) - that constitute of ISO/SAE (DIS) 21434 [5]. The first three activities
 Fe ature s,
pre limin ary
1. Define item architecture o f 2. Perform Risk and
Start SUT Threat Analysis
threats, Risk
me tric

threats, Kn own
ite m de finitio n, se curity risk metric T h reats
go als

3. Define security
concept
Vu lnera bilitie s

security
req uirem ents

4. Plan test and develop scenarios

4a. Define 4b. Define functional 4c. Define fuzz 4d. Define
Penetration test and interface test testing scenarios vulnerability
scenarios scenarios scanning scenarios Test
sce nario s

test
sce nario s
fou nd
test
vul nerab ilitie s
sce nario s
SUT-spe cific informatio n 5. Select test scripts
SUT Da tabase

Test scripts
5a. Develop test
scripts
Test scripts
test
scripts
val idatio n
Test result
Test scripts
scripts
6. Generate test 5b. Validate test
Test cases cases scripts
exe cuta ble te st
cases int erface dete rmin ation ,
Test cal ibrati on
cases
Test cases 7. Perform Test en vironm ent
de script ion

Test results 7b.Execute Test 7a. Prepare test

Cases environment

Test Results (RAW)

test
results

Test results

8. Generate test
Test reports
report

Test
rep orts

End
requirements are the input for the test planning, as they the derived models (or constituting components, respec­
should be verified with regard to the consistency with tively) determines the used test patterns and, thus, the
the security goals and the item's functionality [5]. The test plan, including the test methods.
following steps derive such requirements [25] 1) Define Penetration Test Scenarios: Penetration testing
1) Collect the results from the threat analysis; is the legal and authorized process of exploiting systems
2) Define threat countermeasures; in order to retrieve information which is important
3) Map the resulting threats to countermeasures. for enhancing security of the system. Penetration tests
This method allows for using different attack mitigation focus on specific aspects of security and are deployed
techniques as building blocks that can later be referred manually or semi-automatic. To extend the capabilities,
by multiple requirements. The security requirements de­ global-based adversarial activities must be deployed to
rive the relevant scenarios for the tests and give direction maintain a holistic view of the system and deploy se­
on what needs to be tested. curity tests from the adversary's perspective. The above
methods are called red team assessments which usually
D. Plan Test and Develop Scenarios include penetration tests; however, such methods extend
A security test plan should organize the security test- the whole process [28]. A successful penetration testing
ing process and contain the following elements [26]: methodology will discover functional weaknesses, de­
sign flaws and provide recommendations for security
• Purpose/Objectives;
improvement [29]. To deploy penetration test scenarios,
• SuT overview and Test scope;
the scope and the context for deployment of appropriate
• Risk analysis;
attack strategies with respect to the system's potential
• Test strategy and requirements;
weaknesses must be defined. In penetration testing, it is
• Test environment (Hardware- or Software-in-the­
possible to attack vehicles without in-depth knowledge
Loop, actual vehicle, etc.);
(black box) or from the inside (white box - meaning that
• Test case specifications;
some or full information is available to the red team).
• Test execution and termination criteria.
The process suggests cyber kill chain [30] and attack trees
This also corresponds to a verification and a validation [31], where the latter approach allows for automated
specification according to ISO/SAE DIS 21434 (10.5 and decision making for generating attack vectors.
11.5, respectively) [5]. The output of this activity is a set 2) Define Functional and Interface Security Testing Sce­
of defined test scenarios which, dependent on the risk narios: Functional tests assess the system's adherence to
level and attack feasibility, apply different techniques. its functional requirements (correctness)and take place
Test scenarios are between system requirements and test throughout the whole process and at different levels of
cases [27] and are abstract test descriptions (consisting abstraction. Testing security functions focuses on test­
of test patterns) that define which vulnerabilities and ing the security requirements. Typical security require­
requirements are specifically tested with which meth­ ments may include specific elements of confidentiality,
ods. Testing methods, based on the identified risks and integrity, authentication, availability, authorization and
threats, are [5]: non-repudiation. There are two possibilities of formulat­
• Functional testing ing security requirements: 1. positive requirements and
• Interface testing 2. negative requirements. Positive formulated require­
• Static code analysis; ments describe how a security function should work.
• Penetration testing; Negative requirements state behaviour that the soft­
• Vulnerability scanning; ware should not exhibit. The mapping of requirements
• Fuzz testing. to specific software artifacts could be problematic for
A Test pattern is the generic description of a single such requirements, since this kind of requirement is
step inside a test (normally an action during an attack) not implemented in a specific place[26]. When negative
including potentially used tools, but not specific to an requirements are tested, security testers look for common
SuT. These methods should be made concrete with test mistakes and test assumed weaknesses in the applica­
scripts (containing e.g. an exploit) that eventually form tion. The emphasis is on finding vulnerabilities, often
test cases (see Section IV for the coherence between test by executing misuse tests. To derive the test cases, the
scenarios and test cases). Scenarios are derived by re­ following steps need to be carried out:
quirements analysis, equivalences classes, boundary val­ 1) Identify functions expected to perform.
ues analysis and error guessing [5].Furthermore, attack 2) Create test cases based on the function methods.
patterns are derived from generalizing existing attacks 3) Determine the output based on the function speci­
that may be derived from open databases (for well­ fications.
known attacks) or intrusion detection signatures as well 3) Define Fuzzing Scenarios: Fuzzing is a technique
as actual attack analyses. For an automated test system to use random input in order to put an SUT into a
implementing the process, the availability of attacks for non-intended state to uncover errors, which could be
more efficient than structured testing [32]. However, is optional and carried out if no appropriate test script is
randomness of fuzz testing does not have to be complete present beforehand. The scripts are concrete implemen­
but adapted to an SUT using passive listening [33]. A tations of test patterns, making use of the tools outlined
fuzzer consists of a generator (combining valid and ran­ in the scenario description targeting towards a specific
dom parts), a delivery mechanism, a monitoring system SUT. A test script is an executable script that contains:
and a test oracle [34]. The oracle, that determines the • The testing tool(s) to be used (parameters, interfaces,
test result (i. e. pass/fail), is obtained by monitoring oracle may be derived from the test scenario;
communications or using specific protocols (like XCP) • Needed parameters and information specific to the
as well [35]. Using fuzzing techniques, it is possible SUT;
to attack automobiles without any in-depth knowledge • Specifics of the test system (e.g. using Linux, avail-
[36]. In principle, any component that shows an external ability of a certain compiler/interpreter, etc.).
interface can be fuzzed. Similar to test scenarios, attack scripts are derived from
Fuzz testing can [37]: open sources by observing actual attacks. Test scripts are
• be used to reverse engineer vehicle messages; created by analysing an SUT or they are derived from
• be used to disrupt vehicle' s communication net­ various structured approaches like attack trees [39]. To
work; ensure that the case generation step (Section V-F) can re­
• be a form of cyber attack; use scripts from the database, the current step should:
• lead to vehicle component damage. 1) Either match an existing script;
For a significantly large test space, fuzzing should be 2) Or develop a test script to the specifications of the
combined with combinatorics to select test cases and be test scenario.
run in parallel as long as a test series runs and the space In the latter case, extensive technical knowledge about
is not covered. the SUT or further specifics might be needed (e.g., an
4) Vulnerability Scanning Scenarios: Vulnerability scan­ interpretation file for particular CAN messages).
ning uses tools, called vulnerability scanners, that com­ 2) Validate Test Scripts: This optional activity applies
pare a vulnerability database with the information ob­ to newly developed test scripts to validate them before
tained from a network scan to find possible vulnerabili­ actual tests. New test scripts are tried out on simulated
ties in the network [38]. or actual SUTs in a simulated or actual test environment.
A scanner typically enumerates known software vul­ Expected outcomes (derived from the test oracle) are
nerabilities and provide a comprehensive baseline of compared to actually acquired results. SUTs should be
existing vulnerabilities. To perform effective vulnerabil­ chosen in a way that both positive and negative results
ity scanning, the tools should be selected based on the can be obtained in specific well-defined conditions. In
scanning scope. This scope is needed to define and create order to validate a test script, the environment should
the vulnerability scanning scenarios. A typical scenario fulfill any prerequisite set in the test script (similar to an
for using vulnerability scanning is: actual test). Similar to test cases, the validation of the test
• Define which system to scan (i.e. the SUT or com- script should contain different SUTs or configurations
ponents thereof); thereof that include:
• Define the tool that should be used for the scanning; 1) an SUT configuration with a successful attack (pos­
• Perform the scan; itive validation);
• Analyze the resulting report (i.e. identify relevant 2) an SUT configuration with an unsuccessful attack
vulnerabilities); (negative validation);
• Specify further analysis/testing tasks. 3) several edge cases.
For automation, the results (in machine readable form) The validation test coverage should be comparable to
serve as an input for other scenarios. coverages in actual tests (see Section V-F).
E. Select Test Scripts F. Generate Test Cases
This section concerns the transition of generic test A test case includes multiple items from this non­
descriptions (from the test scenarios) addressing vulner­ exhaustive list derived and extended from [40], [41], [42]:
abilities (found in the threat assessment) into concrete • Test purpose and objectives;
test scripts to be executed onto a specific SUT. Test scripts • SUT /Function description (including
are selected from a database, if available, or otherwise software/hardware/firmware configurations);
developed. • Environmental needs including dependencies;
1) Develop Test Scripts: The purpose of this activity, in • Procedural requirements, test setup and condition;
general, is to populate a test script database with relevant • Test activities and input data;
tests, particularly attacks. The scripts correspond to the • Expected results, completion, stopping and resump­
plan and implement defined test patterns. This activity tion criteria (Pass/Fail criteria including metrics);
 • Traceability to related requirements and threats; H. Generate Test Report
• Variability and quality attributes. A test report is a presentation of the combined results
In the context of this process, the test case generation of the process, it should contain:
(tcg) is the fusion of a generic test scenario (Section • A management summary;
V-D) and the test scripts (Section V-E) that are specific to • An SUT description;
a distinct SUT. Augmenting the scenarios with specific • start time and duration;
information from an SUT database translates the test • An aggregated overview (dashboard);
scenarios into executable test cases. With both parts • The approach/method used;
available in machine-readable form, this activity is easy • Findings (passed and failed tests);
to automate. Combinatorial testing [43] allows for an For the executed tests, pass and fail information (and in
efficient coverage/effort ratio. The tcg can re-use the case of failed tests: sufficient information to understand
threat modelling outcomes in conjunction with a test the problem) must be given. In both cases, reference links
script database, giving the opportunity of automating to goals, requirements, used tools, the raw data, the test
the process using a framework (e.g. [44]) If a clear model results (including risk levels and severity categorization
is lacking completely, test coverage is most important. and conflicts with regulations, policies or best practices)
and information about aspects that were not tested (not
planned, technical problems, lack of time, funds or tools,
G. Perform Testing etc.) need to be included. The testing report should
also correspond to a verification and a validation report
To execute the test, a test environment shall be estab­ according to ISO/SAE DIS 21434 (10.5 and 11.5) [5].
lished using an description from the scenarios. The envi­
VI. CONCL US ION AND ÜUT LOOK
ronment description contains all required prerequisites,
while the test cases contain the performed operations This paper outlined a process for testing the cyber­
security of (particularly automotive) systems to fill the
1) Prepare Test Environment: Two inputs are needed to
gap between existing standards for automotive security
prepare a test environment: (a) an environment config­
engineering and their hands-on, actual-system testing.
uration and (b) interface descriptions. The resulting test
The process provides a comprehensive, automatable ap­
environment template is then used to execute tests. A
proach for system testing based on ISO/SAE DIS 21343.
configuration consists primarily of the system under test
Due to rising complexity and regulators' requirements
(SUT) and applicable test categories, including system
this is necessary as it facilitates a conceivable need
and service preconditions. Interface descriptions contain
for industrializing automotive cybersecurity testing. The
their stimulation and provisions, as well as verification
process itself is arranged generically in order to allow
procedures for their claimed properties. For automation,
for using already existing procedures (e.g. a present
they are organized in an object-oriented, serializable
risk assessment process) not mandating any specific
manner. The resulting combination of the environment
method. Future work will therefore involve a reference
and the interface description form a test environment
implementation on both processual and technical level.
template to applied with different test cases from diverse
categories, ideally in a microservices-based, container­ ACKNOWLEDGEMENT
ized style. This activity also includes saving a pre-attack This work was supported by the H2020-ECSEL pro­
state of the SUT and a clean-up procedure after the gram of the European Commission; grant no. 783119, SE­
conducted test (e.g. if a test involves flashing ECUs). CREDAS project. Special thanks to Rosita Jupri, Behrooz
2) Execute Test Cases: Each test case consists of a Sangcholie and Rauli Kaksonen for their help.
sequence of test scripts (as minimum verifiable actions
REFERENCES
- MVAs) that can be combined to form more complex
sequences. Resulting sequences can be combined again. [1] Vokswagen, "Eighth-generation Volkswagen Golf GTI makes
global debut at the Geneva Motor Show," 2020, [Online
The combinations can also contain permutations and re­ press release; retrieved 2020-07-16]. [Online]. Available:
organizations of scripts. For automation, final commands https://www.media.vw.com/releases/ 1261
take a shell-executable form. Test cases create specific [2] K. Koscher, A. Czeskis, F. Roesner, S. Patel, T. Kohno, S. Check­
oway, D. McCoy, B. Kantor, D. Anderson, H. Shacham, and S. Sav­
outputs on defined interfaces. This output is consumed age, "Experimental security analysis of a modern automobile," in
by an interface module that transforms the output into 2010 IEEE Symposium an Security and Privacy, 2010, pp. 447-462.
a correct call for the associated physical interface and [3] C. Miller and C. Valasek, "Remote exploitation of an unaltered
passenger vehicle," Black Hat USA, 2015.
the retumed response. A completed test case output is [4] Upstream Security, "Upstream Security Global Automotive Cy­
subsequently converted into a standardized test result. bersecurity Report," Upstream Security, Tech. Rep., 2020.
Test results are stored and used as input for other test [5] International Organization for Standardization and Society of
Automotive Engineers, "Road Vehicles - Cybersecurity Engi­
cases, further analysis, and reporting (see V-H), includ­ neering," International Standard, International Organization for
ing relevant meta data in a standardized format. Standardization, ISO/IEC Standard "21434", 2019.
 [6] United Nations Economic and Social Council - Economic Com­ in 2015 Design, Automation Test in Europe Conference Exhibition
mission for Europe, "UN Regulation on uniform provisions con­ (DATE), 2015, pp. 621-624.
cerning the approval of vehicles with regard to cyber security [25] S. Marksteiner, H. Vallant, and K. Nahrgang, "Cyber security
and of their cybersecurity management systems," UNECE, UN requirements engineering for low-voltage distribution smart grid
Regulation ECE/TRANS/WP.29/2020/79, 2020. architectures using threat modeling," Journal of Information Secu­
[7] D. S. Fowler, M. Cheah, S. A. Shaikh, and J. Bryans, "Towards a rity and Applications, vol. 49, p. 102389, 2019.
testbed for automotive cybersecurity," in 2017 IEEE International [26] C. Michael, K. van Wyk, and W. Radosevich, "Risk­
Conference on Software Testing, Verification and Validation (ICST), Based and Functional Security Testing," Cybersecurity
2017, pp. 540-541. and Infrastructure Security Agency (CISA), Tech. Rep.,
[8] R. Kurachi and T. Fujikura, "Proposal of hils-based in-vehicle net­ 2005, retrieved at April 17, 2020. [Online]. Available:
work security verification environment," in WCX World Congress https://www.us-cert.gov/bsi/articles/best-practices/security­
Experience. SAE International, apr 2018. testing/risk-based-and-functional-security-testing
[9] E. dos Santos, A. Simpson, and D. Schoop, "A formal [27] Wei-Tek Tsai, Xiaoying Bai, R. Paul, and Lian Yu, "Scenario­
model to facilitate security testing in modern automotive based functional regression testing," in 25th Annual International
systems," Electronic Proceedings in Theoretical Computer Science, Computer Software and Applications Conference, 2001, pp. 496-501.
vol. 271, pp. 95-104, May 2018. [Online]. Available: [28] B. J. Wood and R. A. Duggan, "Red teaming of advanced in­
http://dx.doi.org/10.4204/EPTCS.271.7 formation assurance concepts," in Proceedings DARPA Information
[10] T. Huang, J. Zhou, and A. Bytes, "Atg: An attack traffic generation Survivability Conference and Exposition, vol. 2, 2000, pp. 112-118.
tool for security testing of in-vehicle can bus," in Proceedings of the [29] R. R. Linde, "Operating system penetration," in Proceedings of the
13th International Conference on Availability, Reliability and Security, May 19-22, 1975, National Computer Conference and Exposition, ser.
ser. ARES 2018. New York, NY, USA: ACM, 2018. AFIPS '75. New York, NY, USA: ACM, 1975, pp. 361-368.
[11] M. Cheah, S. A. Shaikh, 0. Haas, and A. Ruddle, "Towards [30] 1. Tarnowski, "How to use cyber kill chain model to build cyber­
a systematic security evaluation of the automotive bluetooth security?" European Journal of Higher Education IT, 2017. [Online].
interface," Vehicular Communications, vol. 9, pp. 8 - 18, 2017. Available: http:/ /www. eunis. org/ download/TNC2017 /TNC17-
[12] M. R. Friesen and R. D. McLeod, "Bluetooth in intelligent trans­ IreneuszTarnowski-cybersecurity.pdf
portation systems: a survey," International Journal of Intelligent [31] S. Mauw and M. Oostdijk, "Foundations of attack trees," in
Transportation Systems Research, vol. 13, no. 3, pp. 143-153, 2015. Information Security and Cryptology - ICISC 2005, D. H. Won and
S. Kirn, Eds. Berlin, Heidelberg: Springer, 2006, pp. 186-198.
[13] M. Cheah, S. A. Shaikh, J. Bryans, and H. N. Nguyen, "Combining [32] J. W. Duran and S. Ntafos, "A report on random testing," in Pro­
third party components securely in automotive systems," in IFIP ceedings of the 5th International Conference on Software Engineering,
International Conference on Information Security Theory and Practice.
ser. ICSE '81. IEEE Press, 1981, pp. 179-183.
Springer, 2016, pp. 262-269. [33] D. S. Fowler, J. Bryans, M. Cheah, P. Wooderson, and S. A. Shaikh,
[14] J.-P. Monteuuis, A. Boudguiga, J. Zhang, H. Labiod, A. Servel, "A method for constructing automotive cybersecurity tests, a can
and P. Urien, "Sara: Security automotive risk analysis method," fuzz testing example," in 9th International Conference on Software
in Proceedings of the 4th ACM Workshop on Cyber-Physical System Quality, Reliability and Security Companion, 2019, pp. 1-8.
Security. New York, NY, USA: ACM, 2018, pp. 3-14. [34] R. McNally, K. Yiu, D. Grove, and D. Gerhardy, "Fuzzing: The
[15] L. Ming, G. Zhao, M. Huang, X. Kuang, J. Zhang, H. Cao, state of the art," Defence Science and Technology Organisation
and F. Xu, "A general testing framework based on veins for Edinburgh (Australia), DSTO-TN "1043", 2012.
securing vanet applications," in 2018 IEEE SmartWorld, Ubiquitous [35] P. Lapczynski, H. Heinemann, T. Schöneberger, and E. Metzker,
Intelligence Computing, Advanced Trusted Computing, Scalable Com­ "Automatically generating fuzz tests from automotive communi­
puting Communications, Cloud Big Data Computing, Internet of People cation databases," in 5th escar USA, Detroit, isits AG, 2017.
and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBD­ [36] H. Lee, K. Choi, K. Chung, J. Kirn, and K. Yim, "Fuzzing can
Com/IOP/SCI), 2018, pp. 2068--2073. packets into automobiles," in 2015 IEEE 29th International Confer­
[16] H. Srinivasan and K. Sarac, "A sip security testing framework," ence on Advanced Information Networking and Applications, 2015, pp.
in 2009 6th IEEE Consumer Communications and Networking Confer­ 817--821.
ence, 2009, pp. 1-5. [37] D. S. Fowler, J. Bryans, S. A. Shaikh, and P. Wooderson, "Fuzz
[17] S. Hagerman, A. Andrews, and S. Oakes, "Security testing of an testing for automotive cyber-security," in 2018 48th Annual IEEE/I­
unmanned aerial vehicle (uav)," in 2016 Cybersecurity Symposium FIP International Conference on Dependable Systems and Networks
(CYBERSEC), 2016, pp. 26-31. Workshops (DSN-W), 2018, pp. 239-246.
[18] C. Michel and L. Me, "Adele: An attack description language [38] H. Holm, T. Sommestad, J. Almroth, and M. Persson, "A quantita­
for knowledge-based intrusion detection," in Trusted Information, tive evaluation of vulnerability scanning," Information Management
M. Dupuy and P. Paradinas, Eds. Springer US, 2001, pp. 353-368. & Computer Security, 2011.
[19] D. R. Crow, S. R. Graham, and B. J. Borghetti, "Fingerprinting [39] W. Nichols, Z. Hili, P. Hawrylak, J. Haie, and M. Papa, "Automatie
vehicles with can bus data samples," in ICCWS 2020 15th In­ generation of attack scripts from attack graphs," in International
ternational Conference on Cyber Warfare and Security. Academic Conference on Data Intelligence and Security, 2018, pp. 267-274.
Conferences and publishing limited, 2020, p. 110. [40] P. M. Kamde, V. D. Nandavadekar, and R. G. Pawar, "Value of test
[20] B. K. Aichernig, R. Bloem, M. Ebrahimi, M. Tappler, and J. Win­ cases in software testing," in 2006 IEEE International Conference on
ter, "Automata learning for symbolic execution," in 2018 Formal Management of Innovation and Technology, vol. 2, 2006, pp. 668--672.
Methods in Computer Aided Design (FMCAD), 2018, pp. -9. [41] M. Nahas and R. Bautista-Quintero, "Applying the scheduler
[21] D. Ward, 1. Ibarra, and A. Ruddle, "Threat analysis and risk as­ test case technique to verify scheduler implementations in multi­
sessment in automotive cyber security," SAE International Journal processor time-triggered embedded systems," American Journal of
of Passenger Cars-Electronic and Electrical Systems, vol. 6, no. 2013- Engineering Research (AJER), vol. 5, no. 9, pp. 93-104, 2016.
01-1415, pp. 507-513, 2013. [42] K. Heussen, C. Steinbrink, 1. F. Abdulhadi, V. H. Nguyen, M. Z.
[22] J. Barnat, L. Brim, V. Havel, J. Havlfcek, J. Kriho, M. Lenco, Degefa, J. Merino, T. V. Jensen, H. Guo, 0. Gehrke, D. E. M.
P. Rockai, V. Still, and J. Weiser, "Divine 3.0 - an explicit-state Bondy et al., "Erigrid holistic test description for validating cyber­
model checker for multithreaded c & c++ programs," in Computer physical energy systems," Energies, vol. 12, no. 14, p. 2722, 2019.
Aided Verification, N. Sharygina and H. Veith, Eds. Berlin, [43] D. R. Kuhn, R. N. Kacker, and Y. Lei, "Practical combinatorial
Heidelberg: Springer, 2013, pp. 863--868. testing," NIST Special Publication, National Institute of Standards
[23] M. Islam, C. Sandberg, A. Bokesand, T. Olovsson, H. Broberg, and Technology, SP 800-142, 2010.
P. Kleberger, A. Lautenbach, A. Hansson, A. Söderberg-Rivkin, [44] S. Marksteiner and Z. Ma, "Approaching the automation of cyber
and S. P. Kadhirvelan, "Security Models," HEAVENS Project, security testing of connected vehicles," in Proceedings of the Central
HEAVENS Project Deliverable D2, 2014. European Cybersecurity Conference 2019, ser. CECC 2019. New
[24] G. Macher, H. Sporer, R. Bedach, E. Armengaud, and C. Kreiner, York, NY, USA: ACM, 2019.
"Sahara: A security-aware hazard and risk analysis method,"

View publication stats