IT SOFTWARE VALIDATION

DOCUMENT TYPE : REF. CODE: ISSUE NO: ISSUE DATE:

GUIDANCE QCC-ITSV-001 001 6 DECEMBER 2007

Index
1. Introduction ...................................................................................................................1
2. Aim .............................................................................................................................. 2
3. Scope ........................................................................................................................... 2
4. Definitions ................................................................................................................... 2
5. Software categories ..................................................................................................... 3
5.1 Commercial off-the-shelf (COTS) software .................................................. 3
5.2 Modified off-the-shelf (MOTS) software ...................................................... 3
5.3 Custom software (self-developed software) .................................................. 4
6. Software life cycle ....................................................................................................... 4
7. Validation .................................................................................................................... 5
7.1 Validation diagram ........................................................................................ 5
7.2 Software checklist ......................................................................................... 6
7.3 Validation report ............................................................................................ 6
7.4 Good practices ............................................................................................... 7
8. Configuration management ......................................................................................... 7
9. References ................................................................................................................... 8
Appendix 1 ...................................................................................................................... 9
Appendix 2 .................................................................................................................... 13

1. Introduction
This document presents general validation principles that QCC considers applicable to the
validation of software used in a forensic laboratory. It is based on generally recognized
software validation principles and could therefore be applicable to any software.

Validation is an activity that answers the question “Is the software suitable for its intended
purpose?” Provided the verification activity has been properly performed, that question
translates roughly to “Does the software meet its requirements when it's run?” [11].

It is recommended that validation activities be conducted throughout the entire software life
cycle. The level of validation effort to be applied and validation techniques to be used must be
determined for each software project.

Ref code: QCC-ITSV-001 Issue No. 001 Page: 1/17

2. Aim
The aim of this document is to provide general information on validation of software used in
accredited laboratories. Because detailed papers on software validation have been already
published, the present document provides references to some papers that might be useful
when validating self developed software.

3. Scope
The scope of this document is validation of all types of software used in forensic laboratories.

4. Definitions
Computer system
A group of hardware components and associated software designed and assembled to perform
a specific function or group of functions.

Software
A collection of programs, routines, and subroutines that controls the operation of a computer
or a computerized system.

Software product
The set of computer programs, procedures and associated documentation and data.

Software item
Any identifiable part of software product.

Testing
The process of exercising or evaluating a system component by manual or automated means
to verify that it satisfies requirements or to identify differences between expected and actual
results.

Verification
Confirming that the output from a development phase meets the input requirements for that
phase.

Validation
Establishing by objective evidence that all software requirements have been implemented
correctly and completely and are traceable to system requirements.

Life cycle model

A framework containing the process, activities, and tasks involved in the development and
maintenance of a software product, spanning the life of the software from the definition of its
requirements to the termination of its use, i.e. from concept to retirement.

Ref code: QCC-ITSV-001 Issue No. 001 Page: 2/17

5. Software categories
The software products in computers and automated systems used for the acquisition,
processing, recording, reporting, storage, or retrieval of data can be classified into
3 categories, according to the amount of work required ensuring validation.

• COTS – Commercial off-the-shelf software

• MOTS – Modified off-the-shelf software
• CUSTOM - Self-developed software
These three types of software can reside on local hard drives, network hard drives, embedded
on integrated circuits (IC, ROM, EPROM), or removable disks.

5.1 Commercial off-the-shelf (COTS) software

Commercial off-the-shelf software is most commonly used in forensic laboratories. It is a
code that is purchased without modification and either, cannot, or will not, be modified by the
laboratory and is used “as is”. Almost all software bought by the average computer user fits
into the COTS category: operating systems, office product suites, word processing, and e-mail
programs etc. An example of this would be Microsoft Office or dedicated instrument
interface. This software can be considered adequate by the “mass market” and used as is.
The laboratory should have a set of requirements that describe what the software should do
and there should be testing to ensure that the software meets these requirements.
Embedded IC software inside instruments or automated equipment can be regarded as
validated by the manufacturer and confirmed as adequate during the calibration process. This
assumes the calibration fully exercises the firmware functions. No further action is necessary.
Off-the-shelf operating systems need not be validated as separate programs. However,
system-level validation testing of the application software should address all the operating
system services used, including maximum loading conditions, file operations, handling of
system error conditions and memory constraints that may be applicable to the intended use of
the application program. There should be evidence that any formula strings written by the
laboratory work as expected. Any loadable parameters should be considered as data transfer
and be “checked”.

5.2 Modified off-the-shelf (MOTS) software

A MOTS (either modified or modifiable off-the-shelf) product is typically a COTS product
whose source code can be modified or customised for specific applications by the purchaser,
by the vendor, or by another party to meet the requirements of the customer. The purchased
portion of the software can be considered COTS. The modified or customised portion is
considered CUSTOM and should have evidence of validation. This can be achieved by
documenting the functions and design of the modification. The code or blocks of the
modification should be both documented and annotated showing each function. Finally,
testing should be performed on each block of the modification and evidence given that each
piece works as designed and satisfies each function.

Ref code: QCC-ITSV-001 Issue No. 001 Page: 3/17

5.3 CUSTOM software (self-developed software)
Custom software is a code that is laboratory written or subcontractor written by means of
some commercial standard or configurable software package. Examples of this include
spreadsheets, applications written in MS Visual Basic or C++, SQL database design, etc.
Software products self-developed by the laboratory require full validation. The software
packages themselves do not require validation, but new versions should be always treated
with caution and should be tested and approved before use. Beta-releases should never be
used.

It should especially be noted that spreadsheets are programs, and that they as such require
validation. Spreadsheets may be validated as other programs. Special attention should be paid
to the fact that spreadsheets have a wide-open user interface what makes them very vulnerable
to unintentional changes.

6. Software life cycle

All software has an activity lifecycle. Activities in a typical software life cycle include:
• Management
• Requirements
• Design
• Implementation (Coding)
• Integration and Test
• Installation
• Operation and Support
• Maintenance
Following this lifecycle assures quality and provides validation evidence. Minimally, there
should be evidence of requirements, design, implementation, testing, and installation [10]:
• Requirements – Defining, in user terms the functions of the software (what the software
needs to do). The requirements should be included in a Functional Requirements
document.
• Design – Design of how the software is built, along with any diagrams illustrating the
architecture. The output is a Software Design document.
• Implementation/Coding – Writing the actual source code including header information
and annotations. The code is the document.
• Testing – Software testing according to a test plan of what will be tested and how, with
individual test cases, along with successful completion, which satisfy the plan, and map
directly back to the requirements. Important points to consider are verification of math,
valid/invalid inputs, power failure, and code walkthrough’s1, structural and functional
tests.
• Installation/Checkout – Loading the software onto laboratory hardware.

1
A walkthrough is a term describing the consideration of a process at an abstract level. The term is often
employed in the software industry to describe the process of inspecting algorithms and source code by following
paths through the algorithms or code as determined by input conditions and choices made along the way. The
purpose of such code walkthroughs is generally to provide assurance of the fitness for purpose of the algorithm
or code; and occasionally to assess the competence or output of an individual or team.
Something akin to walkthroughs are used in very many forms of human endeavour since the process is a thought
experiment that seeks to determine the likely outcome(s) of an affair based on starting conditions and the effects
of decisions taken (http://www.encyclopedia.laborlawtalk.com).

Ref code: QCC-ITSV-001 Issue No. 001 Page: 4/17

These processes have to be performed, and documentation updated, each time there is a
change to the software - this is the reason for calling the process a “software life cycle”.
The validation documentation need not be too cumbersome. One document would suffice for
simple software, and it would be separated into several documents for more complex projects.

7. Validation
Software validation should not be left to the end of a project; in fact quite the reverse is
required. The validation should start at the beginning of a project as it is shown in the diagram
below.

7.1 Validation diagram

The following diagram, published by Soft Solution International, shows bad versus good
practice in software validation.

Ref code: QCC-ITSV-001 Issue No. 001 Page: 5/17

7.2 Software checklist
The following is a good checklist to verify that adequate validation activities have occurred
[10]:
1. Has the firmware been validated, via calibration by a cal lab, which has the capability to
thoroughly check the software (i.e. OEM or authorised partner)? (Note: for non-
calibratable device firmware, treat as software)
2. Have all of the “used” firmware parameters been documented and confirmed as correct?
3. Does a Software Requirements document exist for the software?
4. Does a Software Design document exist detailing either the full design or details of the
configuration?
5. Do software testing documents exist describing completed, unique, test cases that exercise
the design both +/- and confirm the requirements?
6. Is there a test log showing test failures and corresponding retests/dispositions?
7. Does evidence exist confirming correct software deployment at each target installation?
8. Has configuration control been applied to all of their software/firmware to ensure that:
a) The software source code location is access controlled.
b) Firmware/software formulae & parameters are “locked” to prevent inadvertent changes.
c) Equipment lists identify software as a separate line item showing correct version and
location.
9. Does evidence exist showing that personnel involved in custom software development
have adequate training?
10. Do Databases and spreadsheets include “audit trails” to avoid previous data being
obscured?
11. Do adequate instructions exist for the operation & maintenance of the software?
12. Does the accuracy of the firmware/software meet or exceed the accuracy required by the
test method?

7.3 Validation report

A software validation report should consist of 5 sections:
1. Objectives and scope of application including (description of the software product, a list
of the involved persons, and specification of the type of software to determine the extent
of the validation);
2. Software life cycle overview (specification of date and signature for the tasks of
preliminary work and the peer reviews assigned to each life cycle phase);
3. Software life cycle activities (information that is relevant for the validation, having all
topics outlined, it should be easier to write the report);
4. Conclusion. (for the persons responsible to conclude and sign the validation report);
5. References and annexes.

A template of such report has been published by Nordtest Software [Method of Software
Validation, NT Techn. Report 535]

Ref code: QCC-ITSV-001 Issue No. 001 Page: 6/17

7.4 Good practices
The following are some good software practices that have been found to assist in managing
validation [10]:
• treat each software product as a piece of calibration equipment that has to be
“recalibrated” each time it is changed or modified;
• place software product masters in a read-only directory;
• network computers that access a shared program on a server;
• lock spreadsheet cells that contain math;
• password protect configuration files or setup screens;
• back up the software;
• plan for hardware/software disaster recovery.

8. Configuration management
Configuration management ensures that all changes to software/hardware are controlled. It
also ensures that all software installations are known, and have periodic checks. To achieve
this, software used in accredited laboratories should be considered with regard to where it fits
into the computing hardware. A software product can be installed on many computer systems.
Many laboratories have one version of software product that is installed on many computer
systems. It is important to separately consider each installation. In all cases, the software
should be under, software configuration management, version and access control. The labs
should have records indicating what versions are current. They should also know which
computer systems have what software products installed. They should also control access to
them so that only authorised individuals have access. There should be a “check” that is
performed periodically to ensure that the correct version is installed and no unauthorised
modifications have occurred [10].

Ref code: QCC-ITSV-001 Issue No. 001 Page: 7/17

9. References
1. ISO/IEC 17025 General Requirements for the Competence of Testing and Calibration
Laboratories, 1999
2. ISO/IEC/IEA/IEEE 12207 – 1995 “Information Technology – Software Lifecycle
Processes”
3. ISO/IEC 12119 – 1994 “Information Technology – Software Packages – Quality
Requirements and Testing”
4. ISO9000-3 “Guidelines for the application of ISO9001 to the development, supply, and
maintenance of software”
5. ASTM E919-96 “Standard Specification for Software Documentation for a Computerized
System”
6. ASTM E 1579 - 93: “Standard Guide for Ensuring Data Integrity in Highly Computerized
Laboratory Operations”
7. ANSI/IEEE Std 1012-1998 “IEEE Standard for Software Verification and Validation
Plans”
8. ANSI/IEEE Std 1059-1993 “IEEE Guide for Software Verification and Validation Plans”
9. EPA 2185 – 1995 “Good Automated Laboratory Practices
10. Gregory D. Gogates, Software Validation in Accredited Laboratories. A Practical Guide.
http://www.a2la.org/guidance/adequate_for_use.pdf
ftp://ftp.fasor.com/pub/iso25/validation/adequate_for_use.pdf,
11. Chuck Taylor, Some Notes on Independent Verification, Validation, and Accreditation of
Computer Software.
12. A96 CAEAL (Canadian Association For Environmental Analytical Laboratories) Policy
on the use of Computers in Accredited Laboratories, Rev 1.2, May 2005,
http://www.caeal.ca/A96-CAEAL%20Lab%20IT%20Pol.pdf

Ref code: QCC-ITSV-001 Issue No. 001 Page: 8/17

APPENDIX 1
COMMON REFERENCES WITHIN ISO/IEC 17025 THAT APPLY TO THE USE OF
ELECTRONIC SYSTEMS IN AN ACCREDITED LABORATORY [12].

Clause Extract / Wording Policy Consideration

4.1.5.c “…shall have policies and procedures to Integrity of data and Access
ensure the protection of its clients’ control
confidential information and proprietary Procedures exist to protect client’s
rights, including procedures for information
protecting the electronic storage and
transmission of results...”
4.3.1 “…shall establish and maintain Integrity of data and Access
procedures to control all documents…. control
….. in this context, “document” could be Procedures to control software
… …. software…. These may be on
various media, whether hard copy or
electronic, …..”
4.3.2.1 “All documents issued… ….shall be… Integrity of data
…reviewed and approved for use….” Quality system reviewed and
approved by authorized personnel by
electronic signatures or password
protection and/or retention of
approval records in hard copy.
4.3.2.2 “The procedure(s) adopted shall ensure Integrity of data and Retrieval of
that: data
a) authorized editions of appropriate Authorized editions of appropriate
documents are available at all documents all locations. (Intranet,
locations…..” NT file Share)
4.3.3.2 “..the altered or new text shall be Integrity of data
identified…” Altered or new text shall be identified
(electronic document)
4.3.3.4 “Procedures shall be established….. Integrity of data
…documents maintained in Procedures shall describe how
computerized systems are made and changes in documents, including
controlled”. software are controlled.
4.12.1.2 “All records… …shall be… …readily Retrieval of data
retrievable…” “…hard copy or Records (electronic media) shall be
electronic media…” stored and maintained so that they are
retrievable
4.12.1.4 “The laboratory shall have procedures to Integrity of data and Access
protect and backup records stored control
electronically and to prevent Procedures to protect and back-up
unauthorized access to or amendment of electronic records.
these records.”
4.12.2.1 “…shall retain records… …to establish Integrity of data and Retrieval of
an audit trail…” data
Retain records for the retention
period (old versions of software also)

Ref code: QCC-ITSV-001 Issue No. 001 Page: 9/17

 Clause Extract / Wording Policy Consideration
4.12.2.2 “Observations, data and calculations Integrity of data
shall be recorded…” Observations shall be recorded at the
time they are made. (electronic).
4.12.2.3 “When mistakes occur in records Integrity of data and Access
,………. In the case of records stored control
electronically, equivalent measures shall Electronic records shall avoid loss to
be taken to avoid loss or change of original data (audit trails) Do
original data”. Databases and spreadsheets include
"audit trails" to not allow previously
recorded data to be obscured?
5.2.1 “The laboratory management shall Validation and Maintenance of
ensure the competence of all who electronic system
operate specific…..” Does evidence show that personnel
involved in use of Custom Software
have adequate training? If the
Custom Software was developed in-
house, is there evidence that they
have adequate training in the
development of these types of
solutions?
5.3.4 Access to and use of areas affecting… Integrity of data and Access
…quality… …shall be controlled. control
Server rooms or server access should
have limited access
5.4.1 “The laboratory shall have instructions Integrity of data Validation and
on the use and operation of Maintenance of electronic system
equipment….” This includes software. Do adequate
instructions exist for the operation &
maintenance of the software?
5.4.7.1 “Calculations and data transfers shall be Integrity of data and Validation of
subject to appropriate checks in a Electronic system
systematic manner.” Calculations (spreadsheet) and data
transfers (tables) shall be subject to
checks. Where other programming
approaches are used to effect data
manipulation and transfer, there must
be some method established to ensure
that these are checked as well.
5.4.7.2 “computer software developed by the Validation of Electronic system
a) user is documented in sufficient detail Software shall be validated and
and suitably validated …..” documented – even if commercial
software is configured for specific
use
5.4.7.2 “procedures are established for Integrity of data and Access
b) protecting data, such procedures shall control
include integrity, confidentiality…” Procedures are established to protect
data

Ref code: QCC-ITSV-001 Issue No. 001 Page: 10/17

 Clause Extract / Wording Policy Consideration
5.4.7.2 “computers and automated equipment Integrity of data and Maintenance
c) are maintained…” of Electronic system
Computer and automated equipment
are maintained
5.4.7.2 Note Validation of Electronic system
“Commercial off-the-shelf software… The software validation note allows
…in general use, within their design labs to take credit for assumed
application range, may be considered validation efforts made by the
suitably validated. However, software manufacturer of purchased software
configuration/ modifications should be but requires that individual
validated as in 5.4.7.2 a)” spreadsheets, macros, and all
configuration / modifications / setups
be validated. This does not apply to
electronic systems used to acquire,
manipulate or reduce data, such as
hard-coded firmwarethat is often
supplied with computer-driven
devices.
5.5.2 “Equipment, and its software…….shall Validation and Maintenance of
be capable of achieving the accuracy Electronic system
required……. Before being placed in Does the accuracy of the
service, equipment (software) shall be Firmware/Software meet or exceed
calibrated or checked to establish that it the accuracy required by the test
meets the labs requirements·…….” method or other relevant
specification? A good test is the
uncertainty contribution of the device
(and its programming) to the overall
uncertainty of the test result. All
deployed software should be verified
prior to being placed in service by
performing some user acceptance
testing such as a comparison of the
requirements against features. This
includes placement into service
following a move or after being
shipped back from calibration or
maintenance.
5.5.4 “Each item of equipment and its Maintenance of Electronic system
software used for testing… …shall… Each item of equipment & software
…be uniquely identified.” shall be uniquely identified.
Procedures should include
documenting the versions in use
(version control)
5.5.5 “Records shall be maintained…” Maintenance of Electronic system
Records shall be maintained of
equipment & software.

Ref code: QCC-ITSV-001 Issue No. 001 Page: 11/17

 Clause Extract / Wording Policy Consideration
5.5.11 “Where calibrations give rise to… Validation of Electronic system
…correction factors… …procedures to Does evidence exist confirming
ensure that copies (e.g. in computer correct software deployment at each
software) are correctly updated.” target installation? Consider the same
approach to software as for other
documents such as document control
(software management), distribution
control.
5.5.12 Test and Calibration equipment, Integrity of data and Maintenance
including software, shall be safeguarded of Electronic system
from adjustments…” Software shall be safeguarded from
adjustments such as password
protection on spreadsheets or other
files.
5.10.1 NOTE 2 Integrity of data
“The test reports or calibration Reports may be issued electronically
certificates may be… ….by electronic
data transfer…”
5.10.2.j “the… …identification of person(s) Integrity of data
authorizing the test report or calibration Reports may contain electronic
certificate.” signatures. LIMS systems should
have established authorization
protocols.
5.10.7 “in the case of transmission of test or Integrity of data
calibration results by… …electronic… Reports may be transmitted
…means, the requirements of this electronically. Whatever method of
International Standard shall be met…” transmission is used, it must provide
an the same level of protection of
integrity of information afforded to
paper documentation.

Ref code: QCC-ITSV-001 Issue No. 001 Page: 12/17

APPENDIX 2
VALIDATION OF MS EXCEL AND OPENOFFICE.ORG CALC SPREADSHEETS

MS Excel is one of the most commonly used computer programs to perform automatic or
semiautomatic calculation and visualisation of data. Excel spreadsheets may be used
whenever a standard calculation has to be repeated over a long period of time. It should be
noted that spreadsheets are programs, and that they as such require validation.
The validation of spreadsheets should ensure that all data generated and all operations and
checks performed with the spreadsheet, as well as online-generated graphs, are valid.

Excel offers several useful tools for spreadsheet validation, e.g.:

- data validation:

Ref code: QCC-ITSV-001 Issue No. 001 Page: 13/17

-conditional formatting:

or formula auditing:

Ref code: QCC-ITSV-001 Issue No. 001 Page: 14/17

which displays relationships between cells.

OpenOffice.org Calc offers Similar tools, though the functions may have different names:
- data validation:

Ref code: QCC-ITSV-001 Issue No. 001 Page: 15/17

- conditional formatting:

Ref code: QCC-ITSV-001 Issue No. 001 Page: 16/17

and formula auditing.

More details about validation of Excel spreadsheets can be found in:

Analytical Method Validation and Instrument Performance Verification

Chung Chow Chan (Editor), Herman Lam (Editor), Y. C. Lee (Editor), Xue-Ming Zhang
(Editor)
ISBN: 0-471-25953-5.
(http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0471259535.html).

Ref code: QCC-ITSV-001 Issue No. 001 Page: 17/17