Scribd Logo
Upload

Welcome to Scribd!

  • 132 views

    Writing A Test Strategy: Tor Stålhane

    Uploaded by

    Jayson Celadiña
    The document discusses writing a test strategy and provides guidance on key concepts to include. It covers: 1. The purpose of a test strategy is to obtain consensus on test goals and manage expectations from stakeholders. It also helps identify the types of tests to be conducted. 2. A test strategy should define the testing focus or stakeholders, test levels and phases, test environment, and completion criteria. It may also specify resources, processes, and documentation requirements. 3. Software integrity levels like IEEE 1012, ISO 26262, and IEC 61508 influence test activities and define requirements based on the criticality of software failures. The strategy selects appropriate techniques for each integrity level.

    Copyright:

    © All Rights Reserved
    Download as PPT, PDF, TXT or read online from Scribd

    Writing A Test Strategy: Tor Stålhane

    Uploaded by

    Jayson Celadiña
    132 views32 pages
    The document discusses writing a test strategy and provides guidance on key concepts to include. It covers: 1. The purpose of a test strategy is to obtain consensus on test goals and manage expectations from stakeholders. It also helps identify the types of tests to be conducted. 2. A test strategy should define the testing focus or stakeholders, test levels and phases, test environment, and completion criteria. It may also specify resources, processes, and documentation requirements. 3. Software integrity levels like IEEE 1012, ISO 26262, and IEC 61508 influence test activities and define requirements based on the criticality of software failures. The strategy selects appropriate techniques for each integrity level.

    Original Description:

    try this

    Original Title

    Writing a Test Strategy

    Copyright

    © © All Rights Reserved

    Available Formats

    PPT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document discusses writing a test strategy and provides guidance on key concepts to include. It covers: 1. The purpose of a test strategy is to obtain consensus on test goals and manage expectations from stakeholders. It also helps identify the types of tests to be conducted. 2. A test strategy should define the testing focus or stakeholders, test levels and phases, test environment, and completion criteria. It may also specify resources, processes, and documentation requirements. 3. Software integrity levels like IEEE 1012, ISO 26262, and IEC 61508 influence test activities and define requirements based on the criticality of software failures. The strategy selects appropriate techniques for each integrity level.

    Copyright:

    © All Rights Reserved
    Download as PPT, PDF, TXT or read online from Scribd
    Download as ppt, pdf, or txt
    132 views32 pages

    Writing A Test Strategy: Tor Stålhane

    Uploaded by

    Jayson Celadiña
    The document discusses writing a test strategy and provides guidance on key concepts to include. It covers: 1. The purpose of a test strategy is to obtain consensus on test goals and manage expectations from stakeholders. It also helps identify the types of tests to be conducted. 2. A test strategy should define the testing focus or stakeholders, test levels and phases, test environment, and completion criteria. It may also specify resources, processes, and documentation requirements. 3. Software integrity levels like IEEE 1012, ISO 26262, and IEC 61508 influence test activities and define requirements based on the criticality of software failures. The strategy selects appropriate techniques for each integrity level.

    Copyright:

    © All Rights Reserved
    Download as PPT, PDF, TXT or read online from Scribd
    Download as ppt, pdf, or txt
    of 32

    Writing a Test Strategy

    Tor Stlhane

    Why a testing strategy


    We need a testing strategy to help
    System and software testers plus Test-andEvaluation staff to determine the overall
    test strategy for development or
    modification of a software intensive system
    Project stakeholders customers and senior
    management approve the test strategy
    Testers and system and software analysis to
    determine
    Test objectives
    Qualification requirements
    Verification and validation criteria

    Testing strategy concepts


    We will discuss the following concepts:
    Purpose of a test strategy
    Testing focus
    Contents of a test strategy
    Software integrity levels
    Test objectives and priorities

    Purpose of a test strategy


    1
    The test strategy is important to
    Obtain consensus on test goals and
    objectives from stakeholders e.g.
    management, developers, testers,
    customers and users
    Mange expectations right from the start
    Be sure that we are heading in the right
    direction
    Identify the type of tests to be
    conducted at all test levels

    Purpose of a test strategy


    2
    When we write a test strategy is important to
    remember that:
    Whatever we do some kind of test strategy
    will emerge. Thus, we might as well specify
    the one we think is best
    A documented strategy is the most
    effective way to get an early agreement on
    goals and objectives
    We need to address:
    Human factors usability
    Interoperability, except for stand-alone
    systems

    Testing focus
    Our focus will depend on which stakeholder we
    are considering at the moment:
    Users acceptance test and operational tests
    Analysis systems test, qualification tests
    Designer integration tests
    Programmer unit tests
    The main point is that we need to define the
    stakeholder first then the tests to be run.

    Contents of a test strategy


    1

    The following is a list of what can be


    specified in a test strategy. Not all of
    it is needed in all cases only use
    what is necessary.
    Project plan, risks and activities
    Relevant regulations depending of
    application area
    Required processes and standards
    Supporting guide lines

    Contents of a test strategy


    2

    Stakeholders e.g. users, testers,


    maintainers and their objectives
    Necessary resources people,
    computers
    Test levels and phases
    Test environment
    Completion criteria for each phase
    Required documentation and review
    method for each document

    Software integrity level


    There are several ways to define
    software integrity levels. When we
    choose an integrity level this will
    strongly influence the way we do
    testing.
    We will look at three definitions of
    integrity levels:
    IEEE 1012 general software
    ISO 26262 automotive software
    IEC 61508 general safety critical
    software

    IEEE 1012 general


    software
    4, High some functions affect critical
    system performance.
    3, Major some functions affects
    important system performance
    2, Moderate some functions effect
    system performance but workarounds
    can be implemented to compensate.
    1, Low some functions have noticeable
    effect on system performance but
    creates only inconveniences

    V&v Activities
    V&V activity
    SW Integrity Level

    Development
    requirements
    level
    4

    Design
    level

    3 2 1 4 3 2 1

    Implementation
    level

    Test level

    4 3 2

    Acceptance test execution

    X X X

    Acceptance test plan

    X X

    Interface analysis

    X X

    X X X

    X X

    Management and review


    support

    X X

    Management review of
    V&V

    X X

    X X X X X

    X X

    X X
    X

    ISO 26262 automotive


    software
    The ASIL level A, B, C or D is the
    outcome of the combination of three
    factors:
    S Severity. How dangerous is a
    event
    E Probability. How likely is the event
    C Controllability. How easy the
    event to control if it occurs

    Finding the ASIL level


    Severity

    S1

    S2

    S3

    Probability

    C1

    C2

    C3

    E1

    QM

    QM

    QM

    E2

    QM

    QM

    QM

    E3

    QM

    QM

    E4

    QM

    E1

    QM

    QM

    QM

    E2

    QM

    QM

    E3

    QM

    E4

    E1

    QM

    QM

    E2

    QM

    E3

    E4

    Methods for software


    integration testing
    Methods and Measures

    According
    to req.

    ASIL
    A

    Requirements based test

    9.4.4

    ++

    ++

    ++

    ++

    External interface test

    9.4.4

    ++

    ++

    ++

    Fault injection test

    9.4.4

    ++

    ++

    Error guessing test

    9.4.4

    ++

    ++

    Methods for software unit


    testing
    Methods and Measures

    According
    to req.

    ASIL
    A

    Functional tests

    8.4.2

    See table 8.2

    Structural coverage

    8.4.2

    See table 8.3

    Resource usage measurement

    8.4.2

    Back-to-back test between


    simulation model and code, if
    applicable

    8.4.2

    ++ ++

    ++

    IEC 61508 safety critical


    software
    Safety
    integrity
    level

    High demand or continuous mode of


    operation
    (Probability of a dangerous failure per
    hour)

    10 -9 to 10 -8

    10 -8 to 10 -7

    2
    1

    10 -7 to 10 -6
    10 -6 to 10 -5

    PFDavg = Ft / Fnp . The table above, together with this


    value decides the SIL level.

    Detailed design
    Technique/Measure

    SIL1

    SIL2

    SIL3

    SIL4

    HR

    HR

    HR

    HR

    Table B.7

    HR

    HR

    HR

    1c Formal methods including for example,


    CCS, CSP, HOL, LOTOS, OBJ, temporal
    logic, VDM and Z
    2 Computer-aided design tools

    C.2.4

    ---

    HR

    B.3.5

    HR

    HR

    Defensive programming

    C.2.5

    ---

    HR

    HR

    Modular approach

    Table B.9

    HR

    HR

    HR

    HR

    Design and coding standards

    Table B.1

    HR

    HR

    HR

    Structured programming

    C.2.7

    HR

    HR

    HR

    HR

    Use of trusted/verified software modules


    and components (if available)

    C.2.10
    C.4.5

    HR

    HR

    HR

    1a Structured methods including for example,


    JSD, MASCOT, SADT and Yourdon
    1b Semi-formal methods

    Ref
    C.2.1

    Appropriate techniques/measures shall be selected according to the safety integrity level.


    Alternate or equivalent techniques/measures are indicated by a letter following the number. Only
    one of the alternate or equivalent techniques/measures has to be satisfied.

    Module testing and integration


    Technique/Measure
    1
    Probabilistic testing

    Ref
    C.5.1

    SIL1
    ---

    SIL2
    R

    SIL3
    R

    SIL4
    HR

    Dynamic analysis and testing

    B.6.5
    Table B.2

    HR

    HR

    HR

    3
    4

    Data recording and analysis


    Functional and black box testing

    C.5.2
    B.5.1
    B.5.2
    Table B.3

    HR
    HR

    HR
    HR

    HR
    HR

    HR
    HR

    Performance testing

    C.5.20
    Table B.6

    HR

    HR

    Interface testing

    C.5.3

    HR

    HR

    a) Software module and integration testing are verification activities (see table A.9).
    b) A numbered technique/measure shall be selected according to the safety integrity level.
    c) Appropriate techniques/measures shall be selected according to the safety integrity level.

    Test objectives and priorities


    Only in rather special cases can we test all
    input binary input output and few
    parameters. Thus, we need to know
    The overall objective of testing
    The objective of every test case
    The test case design techniques needed
    to achieve our goals in a systematic way.
    The test objectives are our requirements
    specification for testing.

    Test data selection


    One of the important decisions in selecting
    a test strategy is how to select test data.
    We will look at five popular methods
    Random testing
    Domain partition testing
    Risk based testing
    User profile testing
    Bachs heuristic risk-based testing

    Random testing
    The idea of random testing is simple:
    1.Define all input parameters e.g. integer,
    real, string
    2.Use a random test / number generator to
    produce inputs to the SUT
    The main problem with this method is lack of
    an oracle to check the results against. Thus,
    manual checking is necessary.
    The method is mostly used for crash testing
    (robustness testing) will the system
    survive this input?

    Domain partition testing 1


    Definitions:
    A domain is a set of input values for
    which the program performs the same
    computation for every number of the
    set. We want maximal domains so that
    the program performs different
    computations on adjacent domains
    A program is said to have a domain error
    if the program incorrectly performs input
    classification

    Domain testing simple


    example
    aX

    bX c 0

    If a >< 0, this equation has the following solution

    b
    X

    2a

    b
    c

    2
    4a
    a

    Otherwise we have that

    c
    X
    b

    Testing domains
    Aa ><
    =0
    Aa =
    = 00

    D3

    b2
    c

    A = 0 0
    4a 2
    a

    D1
    D2
    b2 c
    A = 0 0
    4a 2 a

    Risk based testing


    The idea of risk based testing is to
    1.Identify the risk or cost of not
    delivering a certain functionality.
    2.Use this info to prioritize tests. We
    will cover this in more details later
    under Test prioritation

    User profile testing


    The main idea with this type of testing is
    to generate tests that mirror the users
    way of using the system.
    Consider a situation where we know that
    the users in 80% of all case
    Fetch a table from the database
    Update one or more info items
    Save the table back to the database

    Then 80% of all tests should test these


    three actions.

    Bachs risk-based testing


    Bachs heuristics is based on his
    experience as a tester. Based on this
    experience he has identified
    A generic risk list things that are
    important to test
    A risk catalogue things that often go
    wrong
    We will give a short summary of the first
    of Bachs lists.

    Bachs generic risk list 1


    Look out for anything that is:
    Complex large, intricate or convoluted
    New no past history in this product
    Changed anything that has been tampered
    with or improved
    Upstream dependency a failure here will
    cascade through the system
    Downstream dependency sensitive to failure in
    the rest of the system
    Critical a failure here will cause serious
    damage

    Bachs generic risk list 2


    Precise must meet the requirements exactly
    Popular will be used a lot
    Strategic of special importance to the users
    or customers
    Third-party developed outside the project
    Distributed spread out over time or space
    but still required to work together
    Buggy known to have a lot of problems
    Recent failure has a recent history of
    failures.

    Test and system level 1

    Test and system level 2


    From the diagram on the previous slide
    we see that we can test on the
    Electronics level e.g. DoorActuator
    sends the right signal
    State / signal level e.g. door is closed
    iff DoorStateClosed
    Logical level e.g. the door remain
    closed as long as the speed is non-zero
    Safety level e.g. the door remain
    closed as long as the train is moving

    Acknowledgement
    The first part of this presentation is
    mainly taken from Gregory T. Daichs
    presentation Defining a Software
    Testing Strategy, 30 April 2002.

    You might also like