Software Quality Assurance for

Process and Product Quality Assurance

Magister Teknologi Informasi

Staged Representation

Provides a defined roadmap for organizational improvement

Level5 Optimizing Level4 Quantitatively Managed

CausalAnalysisand Resolution OrganizationalInnovationand Deployment

Lower level processes are more critical to repeatable success Higher level processes build on the lower level processesLevel3 Defined Level2 Managed

QuantitativeProjectManagement OrganizationalProcessPerformance

Level1 Performed

RequirementsDevelopment TechnicalSolution ProductIntegration Verification Validation RequirementsManagement OrganizationalProcessFocus ProjectPlanning OrganizationalProcessDefinition ProjectMonitoringandControl OrganizationalTraining SupplierAgreement RiskManagement Management IntegratedProjectManagement(forIPPD*) MeasurementandAnalysis IntegratedTeaming* ProcessandProductQuality IntegratedSupplierManagement** Assurance DecisionAnalysisandResolution ConfigurationManagement OrganizationalEnvironmentforIntegration*

Topics

SQA Software Quality Assurance Soft Inspection & Reviews (Verification) SQM Software Quality Management Defect Prevention

CMMI-Process and Product Quality Assurance (PPQA)

The purpose of PPQA is to provide staff and management with objective insight into processes and associated work products The Process and Product Quality Assurance process area supports the delivery of highquality products by providing project staff and managers at all levels with appropriate visibility into, and feedback on, processes and associated work products throughout the life of the project.

CMMI-Process and Product Quality Assurance (PPQA)..

Activities:

Objectively evaluating performed processes and

work products against applicable process descriptions, standards, and procedures Identifying and documenting noncompliance issues Providing feedback to project staff and managers on the results of quality assurance activities Ensuring that noncompliance issues are addressed

Specific Goal and Practice

SG 1 Objectively Evaluate Processes and Work Products

SP 1.1 Objectively Evaluate Processes SP 1.2 Objectively Evaluate Work Products

SG 2 Provide Objective Insight

SP 2.1 Communicate and Resolve

Noncompliance Issues SP 2.2 Establish Records

PPQA vs. Verification

The practices in PPQA ensure that planned processes are implemented, while the practices in the Verification process area ensure that specified requirements are satisfied. These two process areas can on occasion address the same work product but from different perspectives. Projects should take advantage of the overlap to minimize duplication of effort while taking care to maintain separate perspectives

Prologue

Quality Assurance is not a science; it mostly is developed based on best practices (common senses) Such best practices are built over the years, capitalize on vast past experiences Field experiences are essential

Software failures from time to time

RMI (random memory initiation) Boeing 737300 crashed in M1 England, April 1989 Denver International Airport, Feb 1994 Air-Traffic Control System in LA Airport, Sept 04 Southwest Airline crash in Chicago, Dec 05 Software failure at Atlanta's Hartsfield-Jackson Airport, Apr 06 MS Excell 2007 multiplication bugs Volvo Cars: software engine failure, Sept 09

Volvo Recall Sept 2009

Volvo Cars of North America is recalling about 12,000 vehicles, made between March 2007 and August 2009, because of problems with electronic software that could keep the fuel pump from starting. The defect could cause the engine to stall, potentially resulting in a crash, said the National Highway Traffic Safety Administration.

MS-Excel 2007 Multiplication bugs

Impact of software failure

Not just out there

Space shuttle Ariane 5

But also at home

Telephone bill

How about ICR used by the KPU ?

What is Quality ?

Quality is
the totality of features and characteristics of a product that bear on its ability to meet stated or implied needs

Quality is spoken of in terms of

Product quality (software quality) Process quality

Software Quality

Product quality

conformance to requirements fitness for use freedom from errors and failures customer satisfaction

SW quality is normally spoken of in terms of several different dimensions often called parameters

technical quality parameters user quality parameters

SW Quality parameters

Technical parameters

correctness, reliability, capability,

performance, maintainability these are open to objective measures and technical solutions

User parameters

usability, installability, documentation,

availability these often require subjective analysis and non-technical solutions

Technical parameters

Correctness:

measured in terms of defect rate

Reliability

measured in terms of failure rate

Capability

measured in terms of requirement coverage

Maintainability

measured in terms of change log

Performance

measured in terms of speed and resource

usage

User Parameters

(measured in terms of user satisfaction)

Usability

% of users happy with interface & ease of use

Installability

#install problems reported per installation

Documentation

% of users happy with documentation

Availability

% of users reporting access problems

Quality characteristics & features (ISO/IEC 9126)

Functionality Reliability Usability Efficiency Maintainability Portability Can be objective or subjective

Basic principles

Achieving quality

it does not happen by accident, and is not

something that can be added on after the fact so we must plan for it from the beginning and continuously monitor it day by day

Three general principles

know what you are doing know what you should be doing know how to measure the difference

Principle-1: know what you are doing

It means continuously understanding what it is you are building, how you are building it, and what it currently does this requires organization, including having a management structure, reporting policies, regular meeting and reviews, and so on normally addressed by following a SW process with regular milestones, planning, scheduling, and tracking procedures

Principle-2: know what you should be doing

This means having explicit requirements and specifications these must be continuously updated and tracked as part of the development cycle normally addressed by requirements and usecase analysis, explicit acceptance tests with expected results, frequent user feedback particular procedures and methods are usually part of process

Principle-3: know how to measure the difference

This means having standards and best practices achieved using four complementary methods

formal methods testing inspection metric

Software Eng vs traditional eng

Lack of maturity Number of combinations of inputs almost infinite, exhaustive testing impossible Software is discontinuous Exhibits only systematic failures Changes seem to be easy, software is seen as "the easy fix

Software Quality Assurance

Definition

a planned and systematic pattern of actions to

provide adequate confidence that the product conforms to established technical requirements

Purpose

to ensure that the project standard and

procedures are adequate and are adhered to throughout the project

Assuring Quality

Assure that each of the software qualities is met

Goals set in requirements specification Goals realized in implementation Portability versus safety Understandability versus evolvability Size versus correctness

Sometimes easy, sometimes difficult

Sometimes immediate, sometimes delayed Sometimes provable, sometimes doubtful

Complexity of QA

No matter how sophisticated the QA process, the problem of creating the initial specification remains Sometimes, the software system is extremely complicated making it tremendously difficult to perform QA It is difficult to divide the particular responsibilities involved when performing quality assurance Quality assurance has a negative connotation

Story of Software Development

SQ Management Tasks

Improve software quality!!! Know applicable standards and introduce them into company Define company internal software development process Support Software Developers document templates (Design Templates,) work instructions (Coding Standards) tools (e.g. for quality metrics) support with the conduct of requirements and code reviews Provide quality related training for developers For Safety Related Systems: be able to "prove" software safety

Standards

A standard is

"an authoritative or recognised exemplar of correctness, perfection or some definite degree of any quality"

Software development for safety critical systems guided by a huge amount of standards!

"Standards are wonderful things if you don't like one there are so many more to choose from" (Anonymous) between sectors between countries

In practice, little consistency

SWQM Standards Quagmire

For software safety

SQ Factors & QA-S Components

Software quality factors

Product operation factors Product revision factors Product transition factors

Correctness Reliability Efficiency Integrity Usability

Maintainability Flexibility Testability

Portability Reusability Interoperability

Alternative factor models No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Software quality factor

Correctness Reliability Efficiency Integrity Usability Maintainability Flexibility Testability Portability Reusability Interoperability Verifiability Expandability Safety Manageability Survivability

McCalls classic model + + + + + + + + + + +

Evans and Marciniak model

Deutsch and Willis model

+ + + + + + + + + + + +

+ + + + + + + + + + + + + + +

SW-Dev: A complete picture

Sales

Prepare

Develop

Roll-out

Closure

Pre

Post

QA-Coverage

Components of SQA-S

Pre-project components Project life cycle components Infrastructure for error prevention and improvement Quality management Standard, certification and assessment Human resources

OHT 4.41

The Software Quality Shrine

Contract review Ch.5

Project Development plan and Quality Plan Ch.6

Project Life Cycle SQA components

SQA of External Participants Ch 12

Formal Design Reviews Sec. 8.2

Quality Infrastructure components

Procedures Ch. 14 Supporting Devices Ch. 15 Training Instruction Ch. 16 Preventive Actions Ch.17
Configuration Management

Quality Management
Documentation Control Ch. 19 Project Progress Control Ch. 20 Software Quality Metrics Ch. 21 Software Quality Costs Ch. 22

Standards
Quality
Management

Ch. 18

Standards Ch. 23

Project Process Standards Ch.24

Organizational Base Human components

Management - Ch. 25 SQA Unit - Sec. 26.1 SQA Trustees Sec. 26.2 SQA Committees Sec. 26.2 SQA Forums Sec 26.4

Pre-Project Components

Objective: to improve the preparatory steps taken prior to initiating work on the project Elements:

Contract Review Development and quality plan Clarification of customer requirements Project schedule and resources requirement Staffing Evaluate customers capacity to fulfill his obligation Evaluation of development risks

Contract Review:

Common problems in Contract

Vaguely defined requirements

The rights as well as obligations of each parties are not clearly stated, or Perceived differently by both parties Inaccurate cost estimation Shallow and quick resource estimate To save time and resources To increase revenue

Unrealistic budget and schedules

Excessive pressure from management

Participation in a tender Proposal submission according to customers RFP Receipt of an order from a companys customer Internal request from another department in the organization

Proposal draft review + Contract draft review --------------------------Contract review

1. 2. 3.

4. 5. 6. 7. 8. 9.

To make sure that the following activities have been satisfactorily carried out: Customer requirements clarified and documented Alternative approaches for carrying out the project examined Formal aspects of the relationship between the customer and the software firm specified Development risks identified Project resources and timetable adequately estimated The firms capacity with respect to the project examined The customers capacity to fulfill his commitments examined Partner and subcontractors participation conditions defined Protection of proprietary rights defined

To make sure that the following activities have been satisfactorily carried out: 1. No unclarified issues remain in the contract draft 2. All understandings reached subsequent to the proposal are correctly documented 3. No new changes, additions, or omissions have entered the contract draft

(1) Administrative or operative software to be applied internally (2) Software packages originally intended to be sold to the public as off-the-shelf packages (3) Firmware to be embedded in the companys products

Subject (1) Inadequate definition of project requirements (2) Poor estimate of the required resources (3) Poor timetable

Disadvantages to the internal customer * Implementation deviates from the needed applications * Low satisfaction * Unrealistic expectations about project feasibility * Missing scheduled dates for beginning the distribution of new products * Customer unprepared for project risks and their consequences

(4) Inadequate awareness of development risks

Subject

Disadvantages to the internal customer

(1) Inadequate definition * Higher change requirements of project requirements * Wasted resources due to introducing avoidable changes (2) Poor estimate of the * Substantial deviations from budget required resources * Friction between units induced by requirements for budget additions (3) Poor timetable * Development activities are under time pressures and suffer from low quality * Delays in freeing staff for their next project (4) Inadequate * Tardy initiation of efforts to overcome awareness of difficulties development risks

Development & Quality Plans

The DQP document will be used by the development team as the main reference to build the software It is developed based on the approved proposal + schedules, resource estimates, development risk evaluation Should be available before kick off Depending upon the size of the project; Simpler plans for smaller project

The DQP is meant for

Scheduling activities & estimating resources and budget Recruiting and allocating resources Resolving risks Implementing SQA activities Providing management with information for project control

Elements of the Dev. Plan

Project products (doc, sw, training) Project interfaces (with sw/hw) Project methodologies and dev tools SW standard and procedures Mapping of the dev process (inc estimate duration for each phase, logical sequence of activities, resource implication) Project milestones Project staff organization Required dev facilities Dev risks and risks management actions Control methods Project cost estimates

Elements of the Q Plan

Quality goals (e.g. higher profit, efficiency) Review activities

(scope, types, schedule, procedures, PiC)

SW tests Acceptance test Configuration management tools and procedures