SWE.

6 – Software Qualification Test

ADAS QMS
Internal
Understanding at the end of this session

› Understanding Level 1 requirements/ Base Practices for SWE.6 Software Qualification

Test
› Understanding your responsibility for Level 2-Generic Practices
› Understanding your responsibility for Level 3-Generic Practices
› Preparation for an ASPICE Assessment or any Process Compliance Check (PCC)

ADAS QMS 13 September 2019

Internal SPQE , © Continental AG 2
SWE.6 – Software Qualification Test

ADAS QMS 13 September 2019

Internal Thomas Brandstetterr, © Continental AG 3
ASPICE Basics
Compliance Level
› Level 0
› The Process is not implemented, or fails to achieve its process purpose.
› Level 1
› The implemented process achieves its process purpose
› A standard process or process description is not required.
› Level 2
› The previously described performed process is now implemented in a managed way (planned,
monitored and adjusted) and its work products are appropriately established, controlled and
maintained.
› Project Management, quality assurance and work product management are established. Work
product specific review checklists are used.
› Level 3
› The previously described managed process is now implemented using a defined process that is
capable of achieving its process outcomes.
› A standard process is documented in Stages. Standard templates are used. A Tailoring guideline is
used.

ADAS QMS 13 September 2019

Internal Thomas Brandstetterr, © Continental AG 4
ASPICE Basics
Compliance Level

Process Activity/
Level 3 in Stages Guideline

Work
Review Product
Level 2 Protocoll History in
Plan Template
MKS

Work
Level 1 Product

ADAS QMS 13 September 2019

Internal Thomas Brandstetterr, © Continental AG 5
Understanding Level 1
requirements for
Software Qualification
Test
Expected Outcome
➢ As a result of successful implementation of this process:
1. A software qualification test strategy including regression test strategy consistent with the
project plan and release plan is developed to test the integrated software;
2. A specification for software qualification test of the integrated software according to the
software qualification test strategy is developed that is suitable to provide evidence for
compliance with the software requirements;
3. Test cases included in the software qualification test specification are selected according
to the software qualification test strategy and the release plan;
4. the integrated software is tested using the selected test cases and the results of software
qualification test are recorded;

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 7
Expected Outcome

5. Consistency and bidirectional traceability are established between software requirements

and software qualification test specification including test cases and between test cases
and test results;
6. Results of the software qualification test are summarized and communicated to all
affected parties;

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 8
Base Practices SWE.6
ASPICE Level 1 Topics
ID Topic
Topic
MAN.3.BP1 Develop
SWE.6.BP1 Define the scope
software of work. test strategy including regression test strategy
qualification
MAN.3.BP2 Develop
SWE.6.BP2 Define project lifefor
specification cycle.
software qualification test.
MAN.3.BP3
SWE.6.BP3 Evaluate
Select feasibility of the project.
test cases.
MAN.3.BP4 Define, monitor and adjust project activities.
SWE.6.BP4 Test integrated software.
MAN.3.BP5 Determine, monitor und adjust project estimates and resources.
SWE.6.BP5 Establish bidirectional traceability.
MAN.3.BP6 Ensure required skills, knowledge, and experience.
SWE.6.BP6 Ensure consistency
Identify, monitor and adjust project interfaces and agreed
MAN.3.BP7
SWE.6.BP7 Summarize and communicate results.
commitments.

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 9
SWE.6 Software Qualification Test – Level 1 Base Practices

SWE.6.BP1: Develop software qualification test strategy including

regression test strategy

Develop a strategy for software qualification testing consistent with the project plan and the
release plan. This includes a regression test strategy for re-testing the integrated software if
a software item is changed. [OUTCOME 1]

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 10
SWE.6.BP1: Develop software qualification test strategy
including regression test strategy
› Responsiblility: SW/Algo Validation Engineer › Process + Activities:

› Precondition : › Update test plan for the qualification test strategy using
defined latest template (Scope, goals, schedule etc..) available
› SW/Algo project plan and Schedule review to be completed with in stages(Strategy (19-00))
relevant stakeholders as per SW/Algo Status sheet
› SW/Algo test scope must be defined for each release (in
line with project schedule)
› Test plan should be in lined with project plan and release scope
› Regression test strategy has to be defined
› Test engineer should be a part of requirement review to define
test plan › Test Entry and Test End criteria should be defined

› Reference documents like project plan, schedule , ADAS

stages,Risk managment etc.. Should be mentioned in test plan

› Check-in the updated SW/Algo Test plan in IMS

› Test Plan Review with mandatory participants

› Guidelines/Methods/Trainings:
› Work products (templates):
› Test Method(SW) › SW/Algo Test Plan –Test Plan (08-52)
› Review Method › SW/Algo Test plan Review ID Freezed – Review Record (13-19)

› Test Plan › SW/Algo Status Sheet(for Test plan Review ID)

Link to Stages

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 11
SWE.6 Software Qualification Test – Level 1 Base Practices
SWE.6.BP2: Develop specification for software qualification test.

Develop the specification for software qualification test including test cases based on the
verification criteria, according to the software test strategy. The test specification shall be
suitable to provide evidence for compliance of the integrated software with the software
requirements. [OUTCOME 2]

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 12
SWE.6.BP2:Develop specification for software qualification test.
› Responsiblility: SW/Algo Validation Engineer › Process + Activities:

› Precondition : › Define test cases for the attribute _Test equal to SW Req tests
and Algo Req Test for requirements defined in DOORS
› All the requirements for the next MLC to be in agreed status
› All the mandatory attributes to be filled as per RE Method
and reviewed
› Test cases and Test case description to have Test steps,
› Requirements to have satisfying links Expected results defined

› Requirements should have proper _Test defined as “ SW Req › All SW / Algo requirements (relevant for the sample loop /
test/ Algo Req test “ in DOORs sprint) should be covered

› Requirements should have proper verification criteria defined › SW/Algo Test specification Review as per review method,

› Update the Test specification review ID for the attribute

_ReviewID in DOORs

› Baseline test Specification with proper naming convention

› Guidelines/Methods/Trainings:

› RE Method( for mandatory attributes) › Work products (templates):

› Test Method(SW) › SW/Algo Test Plan -–Test Plan (08-52)

› Review Method
› SW/Algo Requirement Test Specifications on DOORs(baselined) –
Traceability Record (13-22)

› Test Specification Review ID - Review Record (13-19)

Link to Stages

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 13
SWE.6 Software Qualification Test – Level 1 Base Practices
SWE.6.BP3:Select test cases.

Select test cases from the software test specification. The selection of test cases shall have
sufficient coverage according to the software test strategy and the release plan. [OUTCOME
3]

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 14
SWE.6.BP3: Select test cases.
› Responsiblility: SW/Algo Validation Engineer › Process + Activities:

› Precondition › Test cases to be traced to the SW/Algo Requirements to ensure

sufficient coverage
› SW/Algo Project plan to have Release plan updated ( section
2.1.2) and reviewed › 100% of the planned scope should be ensured,

› SW/Algo Test plan updated for Test Strategy and reviewed › Regression test cases to be specified as per Regression
test strategy (when full validation is not planned)

› Test cases to be defined according to software test strategy

defined in SW Test plan and the release plan updated in
SW/Algo Project Plan

› Guidelines/Methods/Trainings:

› Test Method(SW)- section 3.1.3 for test coverage metrics

› Work products (templates):
› SW/Algo Test Plan
› SW/Algo Test specifications updated and baselined in DOORS-
› SW/Algo Project Plan Test Specification(08-50)

› Test Specification to Requirement Traceability by verifying links-

Requirement Traceability- Traceability Record(13-22)

Link to Stages

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 15
SWE.6 Software Qualification Test – Level 1 Base Practices
SWE.6.BP4:Test integrated software.

Test the integrated software using the selected test cases. Record the software test results
and logs. [OUTCOME 4] NOTE 1: See SUP.9 for handling of non-conformances.

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 16
SWE.6.BP4:Test integrated software.
› Responsiblility: SW/Algo Validation Engineer › Process + Activities:

› Precondition
› SW / Algo Requirements Validation for the integrated Checkpoint

› SW/Algo Requirements Test Report Updated(as per the latest template)

› Test Specification baseline on DOORS and Review checked –in in IMS

› Checkpoint creation from integrator label according to the › Test Statistics, Release Recommendation, Release Intention
naming convention described in ADAS CM Method)
› Risk Assessment for failed AND Not Executed test cases. With
Issue ID (FR) updated in DOORs(for failed test cases only)

› SW/Algo Requirements tests recorded and baselined in DOORS

› _ TestResult, _TestStatus_Project, _Test SW version, Tester,

_Review ID

› SW/Algo Test Reports available for each sample loop / sprint

› Work products (templates):

› Guidelines/Methods/Trainqaings:
› SW/Algo Test Report- Test Result(13-25), Verification Results(13-50)
› Test Method(SW)
› Risk Assessment Report for failed test cases- Analysis Report(15-01)
› RM Method
› SW/Algo Requirement Test Results baselined on DOORs-Test Result(13-
25), Verification Results(13-50)
› Review Method
› SW/Algo Status Sheet
› SW/Algo Test Plan

Link to Stages

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 17
SWE.6 Software Qualification Test – Level 1 Base Practices

SWE.6.BP5: Establish bidirectional traceability.

Establish bidirectional traceability between software requirements and test cases included in
the software qualification test specification. Establish bidirectional traceability between test
cases included in the software qualification test specification and software qualification test
results. [OUTCOME 5] NOTE 2: Bidirectional traceability supports coverage, consistency and
impact analysis.

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 18
SWE.6.BP5: Establish bidirectional traceability
› Responsiblility: SW/Algo Validation Engineer › Process + Activities:

› Precondition: › Verifying links from SW/Algo Test Specifications established to the

SW/Algo (L3) Requirements in DOORs
› SW/Algo Requirement Specification baselined on DOORs

› SW/Algo Requirement Validation completed

› Test Specification review , with review ID frozen in Review tool to ensure
the traceability with respect to the requirements
› SW/Algo Requirement test report updated
› Review Participants as per Status Sheet
› SW/Algo Requirement Test results updated on DOORs

› SW/Algo Requirement Test Results with review ID frozen in Review tool to

ensure the traceability with respect to the Test Specifications

› Review Participants as per Status Sheet

› Guidelines/Methods/Trainings:
› Test Method(SW) › Work products (templates):
› RM Method › Update SW/Algo test results on DOORs-Test Result(13-25), Verification
Results(13-50)
› Review Method
› SW/Algo Requirement Specification Review ID freezed-Review Record
(13-19)

› SW/Algo Test Results Review ID freezed-Review Record (13-19)

Link to Stages

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 19
SWE.6 Software Qualification Test – Level 1 Base Practices

SWE.6.BP6: Ensure consistency.

Ensure consistency between software requirements and test cases included in the software
qualification test specification. [OUTCOME 5] NOTE 3: Consistency is supported by
bidirectional traceability and can be demonstrated by review records.

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 20
SWE.6.BP6: Ensure consistency.
› Responsiblility: SW/Algo Validation Engineer › Process + Activities:

› Precondition: › Ensure verifying links from SW/Algo Requirements Test specifications

to SW/Algo Requirement Specification
› SW/Algo Test Plan updated for the scope of MLC(MLC 50 to
MLC 70)
› Ensure the scope of the SW/Algo Requirement Test is met as per test
scope defined in SW/algo Test plan

› SW/Algo Requirements Reviewed and baselined › Ensure the SW/Algo Requiement tests are covering the linked
Requirements
› SW/Algo Requirement test specifications reviewed and
baselined › Regression test been conducted according to SW / Algo Test
Plan, chap. 4.1.1

› Guidelines/Methods/Trainings:
› Work products (templates):
› Test Method(SW)
› SW/Algo Test Plan (review for test scope)-Review Record (13-19)
› SW/Algo Test plan
› SW/Algo Test Specification Review -Review Record (13-19)
› Review Method
› SW/Algo Test Report (review to ensure links)-Review Record (13-19)

› SW/Algo Test Results on DOORs (links) -Test Result(13-25),

Verification Results(13-50)

Link to Stages

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 21
SWE.6 Software Qualification Test – Level 1 Base Practices
SWE.6.BP7:Summarize and communicate results.

Summarize the software qualification test results and communicate them to all affected
parties. [OUTCOME 6] NOTE 4: Providing all necessary information from the test case
execution in a summary enables other parties to judge the consequences.

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 22
SWE.6.BP7:Summarize and communicate results.
› Responsiblility: SW/Algo Validation Engineer › Process + Activities:

› Precondition: › Summarize the software qualification test results and

communicate them to all affected parties through email
› SW/Algo Test execution

› SW/Algo Test Reports Review Freezed And Reports

checked-in in IMS

› SW/Algo Test Results updated and baselined on DOORs

› Work products (templates):

› Guidelines/Methods/Trainings:
› Test Reports in IMS-Verification Results(13-50)
› Test Method(SW)
› Review ID for SW/Algo Test report- - Review Record (13-19)

› FR- Issue created in IMS

› Risk assessment checked-in in IMS-Communication Record(13-

04)

Link to Stages

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 23
Understanding your
responsibility for
Level 2
Generic Practices - PA 2.1 Performance management
process attribute
The performance management process attribute is a measure of the extent to which the performance of
the process is managed.
› 2.1.1 Identify the objectives for the performance of the process › Review of the updated documents and review ID frozen

› SW/Algo test plan for an MLC to be defined

› 2.1.5 Define responsibilities and authorities for performing the
› SW/Algo project Schedule to be updated in parallel to the release plan process

› 2.1.2 Plan the performance of the process to fulfill the identified › Organization diagram in the SW/Algo Project plan to be updated for
objectives. performing Link to Stages

› Responsibilities for performing SW/Algo validation must be planned › RASI chart is defined in stages

› SW/Algo Test Plan, SW/Algo Test specification, SW/Algo Test › 2.1.6 Identify, prepare, and make available resources to perform
Results
the process according to plan
› Estimated effort must be documented in Project schedule and RO
› Training plan for the SW/Algo Validation engineers must be
defined
› 2.1.3 Monitor the performance of the process against the plans.
› SW/Algo project schedule to be updated based considering
› SW/Algo Validation activities must be performed as planned in
non-working time(holdays, vacations etc)
SW/Algo Project Schedule (as per WBS)

› Reviews for the work products conducted and frozen

› SW/Algo project plan to have all the tools required for
Performing SW/Algo validation activites
› 2.1.4 Adjust the performance of the process
› 2.1.7 Manage the interfaces between involved parties
› Update the SW/Algo project plan for any changes in the test scope and
timeline
› Team meetings to discuss the status of the SW/Algo Validation
activities
› Update the SW/Algo test scope incase of any changes
› Minutes of meeting
› Update the SW/Algo Project schedule for any change in the estimation/
› Action Item tracker
Link to Stages
when the objectives are not met

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 25
Generic Practices – PA 2.2 Work product management
process attribute
The work product management process attribute is a measure of the extent to which the work
products produced by the process are appropriately managed.
› 2.2.1 Define the requirements for the work products › 2.2.3 Identify, document and control the work products
SW/Algo validation
› The work products are documented and controlled in

› Quality criteria for the work products of the SW/Algo accordance with requirements

Validation must be defined

› The work products are made available through access

› Review Method, Review Frequency mechanisms

› 2.2.2 Define the requirements for documentation and 2.2.4 Review and adjust work products to meet the defined

control of the work products of the SW/Algo validation requirements

› All the required review of the work product to be performed
› Naming conventions to be defined adequately

› Access rights to be defined › Ensure correct and mandatory participants (as Per SWSS)

› Traceability to the work products to be defined › No overdue review findings

› Owner of the document…… › All “Not ok” questions to have RO’s assigned

› All Not relevant review questions to have relevant

comments
Link to Stages
ADAS QMS 13 September 2019
Internal SPQE, © Continental AG 26
Understanding your
responsibility for
Level 3
Generic Practices - 3.1 Process definition process
attribute
The process definition process attribute is a measure of the extent to which a standard process is
maintained to support the deployment of the defined process.
› 3.1.1 Define and maintain the standard process that will support › 3.1.4 Identify the required infrastructure and work
the deployment of the defined process environment for performing the standard process.

› Standard Organizational Process and › Resources required for deploying the process to be
defined
› Procedures, Methods and Guidelines defined.
› Tools, Licenses, Budget, time
› Tailoring Guidelines defined
› Workspace, Test Environment etc.
› 3.1.2 Determine the sequence and interaction between processes
so that they work as an integrated system of processes › 3.1.5 Determine suitable methods and measures to monitor
the effectiveness and suitability of the standard process
› Development workflows

› Work breakdown structure to be defined, SW/Algo project › Quantitative methods to identify the effectiveness
schedule to be updated.
such as
› 3.1.3 Identify the roles and competencies, responsibilities, and › Analysis of Trends
authorities for performing the standard process
› Lessons Learnt
› RACI to be defined
› Audits/Assessments
› Training plan to be defined
› Training the resources
› Authorities and operational rights to be defined
Link to Stages

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 28
Generic Practices - 3.2 Process Deployment process
attribute
The process deployment process attribute is a measure of the extent to which the standard process
is deployed as a defined process to achieve its process outcomes .

› 3.2.1 Deploy a defined process that satisfies the context specific › 3.2.4 Provide resources and information to support the
requirements of the use of the standard process performance of the defined process

› Standard defined process to be deployed in the project › Required human resources are made available,
allocated and used.
› Identify the tailoring topics as per the tailoring guidelines
› 3.2.5 Provide adequate process infrastructure to support
› 3.2.2 Assign and communicate roles, responsibilities and the performance of the defined process.
authorities for performing the defined process › All necessary type of resources are made available

› Allocate the resources as per the roles defined in the process › 3.2.6 Collect and analyze data about performance of the
process to demonstrate its suitability and effectiveness
› 3.2.3 Ensure necessary competencies for performing the defined
process › Both Qualitative and Quantitative data is measured,
collected and analysed
› Skill matrix to be defined for the project
› Improvement plans identified and Implemented
› Training plan to be derived from the skill matrix

Link to Stages

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 29
ASPICE Rating

Rating Scale Description

Not achieved There is little or no evidence of achievement of the defined process attribute
in the assessed process.
Partially achieved There is some evidence of an approach to, and some achievement of, the
defined process attribute in the assessed process. Some aspects of
achievement of the process attribute may be unpredictable.

Largely achieved There is evidence of a systematic approach to, and significant achievement
of, the defined process attribute in the assessed process. Some weaknesses
related to this process attribute may exist in the assessed process.

Fully achieved There is evidence of a complete and systematic approach to, and full
achievement of, the defined process attribute in the assessed process. No
significant weaknesses related to this process attribute exist in the assessed
process.

ADAS QMS 13 September 2019

Internal SPQE, © Continental AG 30
Q and A

ADAS QMS 13 September 2019

Internal Thomas Brandstetterr, © Continental AG 31
THANK YOU

ADAS QMS 13 September 2019

Internal Thomas Brandstetterr, © Continental AG 32