1

ABSTRACT benefits include the transfer of proven ideas and

Software Development Process Audits - A General

Procedure
Stewart G. Crawford & M. Hosein Fallah
AT&T Bell Laboratories, Holmdel, New Jersey 07733.
USA

technologies in software development from other

To assist development organizations in improving
software quality and productivity, the AT&T Bell
Laboratories Quality Assurance Center created a
Design Quality Group to independently evaluate projects, consolidated upward feedback to project
the software development processes and associated management, and pointers to further opportunities
development products of our AT&T projects. These for education and training.
software development process audits examine
software engineering techniques and tools in Several projects within Bell Labs have requested
practice, as they fit into the overall development these development process audits to date, and
environment. Our strategy behind these audits is to feedback from their management has been .
assemble a team which, with the involvement of the positive. QAC's organizational independence from
developers and their managers, will: characterize the development organizations allows for a
the existing development process; identify project perspective free of the local bias surrounding
strengths and areas for improvements; and development issues; our interviews and
recommend possible improvements. This paper discussions with project management provided
details the general approach behind this strategy. time for them to stop and take a long view of the
global development issues. We can audit a project
1. BACKGROUND AND OVERVIEW at anytime during the project lifecycle, but if done
early enough, the response to the audit findings
During the last few years, the AT&T-Bell can improve the quality and scheduling of the
Laboratories Quality Assurance Center (QAC) product currently under development. In one
conducted several audits of software and hardware particular project audited late in the design phase,
development processes and of documented our findings and recommendations led the project
requirements and craft interface specifications. to enhance the baselining of intermediate lifecycle
Our objectives centered around identifying areas deliverables, comprehensively review the
of potential quality and productivity remaining project schedule, and improve the
improvements. The QAC performed these audits project tracking system; these activities revealed
by invitation from the respective development several trouble areas looming in the near future
organizations, and their positive response led the for that particular software release, and the project
QAC to create a group dedicated to software then addressed the issues before they became
development process audits. This paper defines our serious problems.
general approach; the specifics of any one
particular audit, however, will vary depending on The distinction between a development process
the breadth and depth of the audit scope negotiated audit and a project management audit is primarily
with the software development project. that of focus. The former focuses on the
techniques, tools, documents, and procedures
The audits we describe here mark a departure from through which an idea becomes a supported
the adversarial relationship generally implied by product, while the latter focuses on the
the term "audit". Our primary objective is to assist organization and management of that whole effort.
development organizations in improving software The scope of these two types of audit will overlap
quality and productivity through independent somewhat since by their natures, a process and its
evaluations of their software development process management are closely coupled. Figure 1
and associated development products; additional illustrates these ideas: The upper dashed circle
 2

shows the scope of concerns in a process audit,

while the lower circle depicts the scope of a
project management audit.
 3

DEVELOPMENT PROCESS AUDIT

PROBLEM SOLUTION
SYSTEM SYSTEM

DEVELOPMENT
UNSATISFIED NEED PROCESS SATISFIED NEED
EXISTING
PRODUCT
SYSTEM

SUPPORT ENVIRONMENT

PLAN
& MONITOR
ORGANIZE &
CONTROL
MANAGEMENT
PROCESS

SUPPORT SYSTEM

PROJECT MANAGEMENT AUDITS

Figure 1. Overview of a Development Project

Our strategy behind these audits is to assemble a designated as the team leader. Each team will
team which, with the involvement of the include members with experience and
developers and their managers, will : expertise in process audits, software
- Characterize the existing development development, and software quality assurance.
process by : Additional subject matter experts may be
o Analyzing development called in as necessary.
methodologies, software quality
assurance plans, and related practices 2.1.3 Understand the Project: Understanding
and procedures. the project is a prerequisite. A tutorial or an
o Interviewing supervisory and non- overview presentation to the team by the
supervisory personnel project is a useful start. The team should also
o Attending project meetings, reviews, learn about the project by reading project
and inspections, and descriptions and other related documents that
o Reviewing selected development may be available, and by talking to the project
products; contacts.
- Identify project strengths and areas for
further improvements; and 2.1.4 Understand the Organization: In an
- Provide recommendations for improving the effective audit, it is crucial to know who is
software development process and the doing what in the project. Organization charts
quality of its software products. are often outdated and the group titles are
generally insufficient for identifying the
Positive interactions and close coordination functions and activities of the project staff.
between the projects management and the audit The organization chart should be reviewed
team are important to the effectiveness of these with the project contacts to identify the
audits, and these contributed greatly to our functional responsibilities of the people.
previous successes. The core audit is a time of
the team’s almost continuous presence within 2.1.5 Define the Audit Scope: Negotiation of
the development process; this is the interval the audit's scope with the project's
when we interview the staff and sit in on many management is the foundation for further
project meetings. Our presence will be non- activities. Each audit will be customized to the
threatening only when management support of needs of the particular project, thus, the breath
the audit is understood by the project team. and the depth have to be defined. Depending
on the scope, the audit may focus on some
2. A GENERAL AUDIT PROCEDURE particular phase(s) of the development
process, or its depth could vary depending on
Prepare and Plan the audit objectives, resource constraints, and
timing.
Once a project requests an audit, some
groundwork is needed before the activities can 2.1.6 Develop an Audit Plan: The plan is the
begin. The pre-audit activities include the "road map" for the audit. The plan should
following : reiterate and expand on the agreed scope (the
Why the audit), answering the following
2.1.1 Clarify Global Goals and Objectives: Due questions: What will be audited? Who are the
to the flexibility built into our audit service, the auditors? When can the various audit
initial request may not be sufficient to define the activities be expected to happen and finish?
task adequately. Thus the first planning activity Where and How will the audit activities take
is to clarify what the audit's customer, usually place?. A preliminary draft of the plan should
some level of project management, is seeking be reviewed with the Project's management to
through the audit. assure a mutual -consensus on the audit
Discussion with he project of the managers' activities and schedule.
needs and of our expertise are crucial to the
proper initial focus of our effort. 2.2.Characterize the Development Process

2.1.2 Assemble an Audit Team: Depending on With the initial groundwork laid, the formal
the size of the project, the team may consist of audit activities begin in earnest. Our intent is
three to six people: one person should be to uncover as much as possible within the
established scope and time constraints. The
team should analyze not only the methodology Interviewing people near the project site but
as it is written down, but the "de facto" not in their offices, removes many distractions
methodology, i.e., the methodology as it exists and may help people open up more since
in practice. Our methods of collecting data they're slightly removed from their usual
include analyzing the development process environment.
documentation, interviewing project
personnel, observing project meetings, and We would want to interview people in the major
reviewing development process products. functional areas of software development
included within the scope of the audit. A non-
2.2.1 Analyze methodologies: The first step in inclusive list may include staff and their
characterizing the development process is to managers responsible for:
study the documented methodology. Besides system definition
providing a general process overview and system requirements
background for all the ensuing activities, we system architecture
can study completeness of lifecycle activity software requirements
documentation, evaluate adequacy of the software architecture
written methodology in providing clear and software design
consistent guidelines for the development software code
process, and formulate initial hypotheses to software unit test
follow up in interviews. Any documented software integration test
development methodology, Software Quality system test
Assurance Plan (SQAP), or other documented customer interface
practices and procedures should be included end-user training installation
in this review. software maintenance
problem reporting & corrective action
In analyzing development methodologies, project management & configuration
either in the form of a handbook or a series of management
documents, we would look for: task quality control
definitions, descriptions; and ownership; methodology documentation & updates
solution procedures for tasks; review and developer training
inspection procedures; a well defined
lifecycle, broken into activities; well-defined A primary reason for interviewing people is to
handoffs between activities; and any standards characterize the "de facto" methodology. Does it
currently in use. We would want to look to follow the documented methodology? Is it
any related practices and procedures for consistently applied across people? Is it
measurement and tracking procedures, effective? Additionally, we would want to:
monitoring and control mechanisms, and gather information to support or refute pending
efficient organizational interfaces. The above hypotheses; formulate new hypotheses as new
lists are not meant to be all-inclusive, but they information is gathered; characterize the
are a good starting point. visibility of project tracking and the
effectiveness of project control; and gather
In general, we would get the necessary upward feedback for the project management.
documents through the project interface Basic investigative and journalistic interview
contact or the project librarian. After reading techniques would be the auditors' tools for this
and annotating the documents, all the auditors, task; summary question checklists can help
in a group meeting, would discuss the guide initial interviews.
documents, evaluate their adequacy, and
formulate the first set of hypotheses regarding 2.2.3 Attend Meetings: The auditors should
the process as it is documented. have a blanket invitation to any technical,
administrative, or informative meeting, either
2.2.2- Interview Developers: The core audit is formal or informal, the project holds during the
the main time for interviewing developers on time-frame of the audit. The purpose behind
site and attending any relevant project attending these gatherings is to: observe project
meetings. The auditors should interview as a communication; characterize general
group or in subgroups of at least two : multiple effectiveness of meetings, reviews, and
interviewers provide multiple viewpoints inspections; and characterize compliance of
which helps balance any individual bias.
reviews and inspections to any relevant
standards. • Report strengths as well as areas for
improvement. Although the main objective
The entire team (for large meetings) or a is to identify the areas with potential for
subgroup (for small meetings) would discretely quality and productivity improvement, it is
attend and observe a project meeting. equally important to identify those aspects
Discussing the meeting after it officially ends of the process that are done well. This
with attendees who happen to linger on is often recognizes the successful efforts within the
of value in resolving any questions the auditors project to produce quality software, aids
may have. The meetings observed would the project by itemizing things done well
normally occur during the core audit, but they which shouldn't be changed for the worse,
may occur before or after that time also, and helps us identify good methods and
depending on when representative project practices that have potential for use on
meetings are scheduled. other projects.

2.2.4 Review Selected Development • Separate major from minor concerns. The
Products: The team may decide to review in- team may find many weak spots. The few
depth a sample of products representative of the major problems should be separated from
various lifecycle activities within the scope of the others and given special emphasis and
the audit. Such reviews serve to: examine the discussion; this separation assures they
products' fitness-for-use in downstream don't get lost amidst everything else.
development activities; characterize compliance
of the products to existing standards (project or • State the problems succinctly and state
industry); or explore project details for why they are problems. The problems
necessary background. should be communicated to the project
very clearly. By stating why the team
Individual auditors examine selected products thinks something is a problem, we provide
within their field of expertise. These reviews the project with the assumptions,
would be oriented toward substantive, not perceptions, and the reasoning that led to
editorial or typographical, comments. When the team's conclusions. This information
multiple auditors review the same set of helps the project to assess the audit
documents, they would need to discuss and findings in light of their more detailed
integrate their comments. project knowledge.

2.3 Evaluate and Report • Present recommendations with due

caution. When possible, the team should
The team should get together frequently during recommend viable solutions to problems
the core audit period and review their they find while examining the development
observations. These meetings help ensure that processes. But, the recommendations
the audit is progressing according to the plan, should be expressed as suggestions that the
provide opportunities to formulate and revise project may want to consider in developing
hypotheses, and adjust the planned activities, its own plan of action. The team should be
when necessary. The auditors may decide to cognizant of the complexities of the
include other people in the list of interviewees, technological, functional, and
plan to attend additional meetings, or review organizational aspects of a development
other relevant documents. project as they relate to the recommended
solutions. while the team's
Following the core audit, the teams findings recommendations may provide a good start
should be brought together, combined, for addressing a problem area, the people
evaluated, and documented in an audit report. within the project are in a better position to
As part of this process, any open items should decide what solution is most likely to work.
be followed-up, and the auditors should obtain
additional information and seek clarification on The audit report should be reviewed with the
issues that do not have the team's consensus. project managers before being finalized. Such a
review helps to remove any misunderstanding
In preparing the audit report the following that the team may have. It also provides an
should be considered : opportunity to find out if there are any major
disagreements between the team and the
project managers, and document the opposing
views if necessary in the report. With the
issuance of the final report, the team may also
make a formal presentation of the findings and
provide a forum for discussion of the
recommendations.

The report is meant only for the managers of

the project. It is their prerogative to distribute
the report either up or down managerial lines
or outside the project. The team members will
forward any request for an Audit Report to the
appropriate project manager. The team
members may report some generalities about
the audit to their management as needed to
keep them informed of the audit activities.

2.4 Follow-up

We will continuously try to improve the audit

process. Feedback from the projects provide
valuable information for achieving this
objective. A follow-up with the project right
after an audit may identify some of the changes
that could strengthen the audit process. The
effectiveness of the audit, however, can only be
assessed in due time. Hence, the team leader
should make another follow-up eight to twelve
months after the audit. In this follow-up, we
solicit an assessment of the benefits realized
(or expected to be realized) by the project so
that the cost-effectiveness of the audit can be
evaluated. The project may also have other
suggestions on ways to further improve the
audit process.