Psychological Bulletin: VOL. 51, No. 4

VOL. 51, No.
4 JULY, 1954
Psychological Bulletin
THE CRITICAL INCIDENT TECHNIQUE
JOHN C. FLANAGAN
American Institute for Research and University of Pittsburgh
During the past ten years the leave little doubt concerning its
writer and various collaborators have effects.
been engaged in developing and uti- Certainly in its broad outlines and
lizing a method that has been named basic approach the critical incident
the "critical incident technique." It technique has very little which is
is the purpose of this article to de- new about it. People have been mak-
scribe the development of this meth- ing observations on other people for
odology, its fundamental principles, centuries. The work of many of the
and its present status. In addition, great writers of the past indicates
the findings of a considerable number that they were keen observers of their
of studies making use of the critical fellow men. Some of these writers
incident technique will be briefly re- must have relied on detailed notes
viewed and certain possible further made from their observations. Others
uses of the technique will be indi- may have had unusual abilities to
cated. reconstruct memory images in vivid
The critical incident technique con- detail. Some may have even made a
sists of a set of procedures for col- series of relatively systematic obser-
lecting direct observations of human vations on many instances of a par-
behavior in such a way as to facili- ticular type of behavior. Perhaps
tate their potential usefulness in solv- what is most conspicuously needed to
ing practical problems and develop- supplement these activities is a set
ing broad psychological principles. of procedures for analyzing and syn-
The critical incident technique out- thesizing such observations into a
lines procedures for collecting ob- number of relationships that can be
served incidents having special signifi- tested by making additional observa-
cance and meeting systematically tions under more carefully controlled
defined criteria. conditions.
By an incident is meant any ob-
servable human activity that is suf- BACKGROUND AND EARLY
ficiently complete in itself to permit DEVELOPMENTS
inferences and predictions to be made The roots of the present procedures
about the person performing the act. can be traced back directly to the
To be critical, an incident'must occur studies of Sir Francis Gallon nearly
in a situation where the purpose or 70 years ago, and to later develop-
intent of the act seems fairly clear to ments such as time sampling studies
the observer and where its conse- of recreational activities, controlled
quences are sufficiently definite to observation tests, and anecdotal rec-
327
328 JOHN C. FLANAGAN
ords. The critical incident technique Although in the preparation of these

as such, however, can best be re- reports much greater emphasis was
garded as an outgrowth of studies in given to determining the precise facts
the Aviation Psychology Program of in the case, it was apparent that in
the United States Army Air Forces many instances the official reports
in World War II. The Aviation Psy- did not provide a complete record of
chology Program was established in all the important events. Even with
the summer of 1941 to develop pro- these limitations, the information
cedures for the selection and classifi- given was found to be of considerable
cation of aircrews. value, and the systematic tabulations
One of the first studies (40) carried that were prepared provided the basis
out in this program was the analysis for a series of recommendations that
of the specific reasons for failure in resulted in important changes in Air
learning to fly that were reported for Force selection and training proce-
1,000 pilot candidates eliminated dures.
from flight training schools in the In the summer of 1944 a series of
summer and early fall of 1941. The studies (74) was planned on the prob-
basic source used in this analysis was lem of combat leadership in the
the proceedings of the elimination United States Army Air Forces.
boards. In these proceedings the These represent the first large-scale,
pilot instructors and check pilots systematic effort to gather specific
reported their reasons for eliminating incidents of effective or ineffective
the particular pilot. It was found behavior with respect to a designated
that many of the reasons given were activity. The instructions asked the
cliches and stereotypes such as "lack combat veterans to report incidents
of inherent flying ability" and "in- observed by them that involved be-
adequate sense of sustentation," or havior which was especially helpful
generalizations such as "unsuitable or inadequate in accomplishing the
temperament," "poor judgment," or assigned mission. The statement
"insufficient progress." However, finished with the request, "Describe
along with these a number of specific the officer's action. What did he do?"
observations of particular behaviors Several thousand incidents were col-
were reported. This study provided lected in this way and analyzed to
the basis for the research program on provide a relatively objective and
selecting pilots. Although it was factual definition of effective combat
found very useful, it also indicated leadership. The resulting set of de-
very clearly the need for better pro- scriptive categories was called the
cedures for obtaining a representative "critical requirements" of combat
sample of factual incidents regarding leadership.
pilot performance. Another study (74) conducted in
A second study (13), which empha- the Aviation Psychology Program in-
sized the importance of factual re- volved a survey of disorientation
ports on performance made by com- while flying.1 Disorientation in this
petent observers, was carried out in
1
the winter of 1943-1944 in the 8th, This study was planned by Paul M. Fitts,
9th, 12th, and 15th Air Forces. This Jr., who also contributed to the previously
study collected the reasons for the mentioned USAAF studies and planned and
carried out the interview study with pilots de-
failures of bombing missions as re- scribed below on the design of instruments,
ported in the Group Mission Reports. controls, and arrangements.
THE CRITICAL INCIDENT TECHNIQUE 329
study was defined to include any ex- The principal objective of job analysis pro-
perience denoting uncertainty as to cedures should be the determination of critical
requirements. These requirements include
one's spatial position in relation to those which have been demonstrated to have
the vertical. In this study pilots made the difference between success and fail-
returning from combat were asked ure in carrying out an important part of the
"to think of some occasion during job assigned in a significant number of in-
stances. Too often, statements regarding job
combat flying in which you personal- requirements are merely lists of all the desira-
ly experienced feelings of acute dis- ble traits of human beings. These are prac-
orientation or strong vertigo." They tically no help in selecting, classifying, or
were then asked to describe what they training individuals for specific jobs. To ob-
"saw, heard, or felt that brought on tain valid information regarding the truly
critical requirements for success in a specific
the experience." This study led to a assignment, procedures were developed in the
number of recommendations regard- Aviation Psychology Program for making sys-
ing changes in cockpit and instru- tematic analyses of causes of good and poor
ment panel design and in training in performance.
Essentially, the procedure was to obtain
order to overcome and prevent verti- first-hand reports, or reports from objective
go while flying. records, of satisfactory and unsatisfactory exe-
In a project carried out in the Avia- cution of the task assigned. The cooperating
tion Psychology Program in 1946, individual described a situation in which suc-
Fitts and Jones (12) collected descrip- cess or failure was determined by specific re-
ported causes.
tions of specific experiences from This procedure was found very effective in
pilots in taking off, flying on instru- obtaining information from individuals con-
ments, landing, using controls, and cerning their own errors, from subordinates
using instruments. These interviews concerning errors of their superiors, from su-
pervisors with respect to their subordinates,
with pilots were electrically recorded. and also from participants with respect to co-
They provided many factual inci- participants (13, pp. 273-274).
dents that were used as a basis for
planning research on the design of DEVELOPMENTAL STUDIES AT
instruments and controls and the THE AMERICAN INSTITUTE
arrangement of these within the cock- FOR RESEARCH
pit. At the close of World War II some
In addition to the collection of of the psychologists who had par-
specific incidents and the formulation ticipated in the USAAF Aviation
of critical requirements, as outlined Psychology Program established the
above, the summary volume (13) for American Institute for Research, a
the Aviation Psychology Program nonprofit scientific and educational
Research Reports contained a dis- organization. The aim of this organ-
cussion of the theoretical basis of ization is the systematic study of
procedures for obtaining the critical human behavior through a coordi-
requirements of a particular activity. nated program of scientific research
Perhaps the best method of describ- that follows the same general princi-
ing the status of these procedures at ples developed in the Aviation Psy-
the close of the war is to quote from chology Program. It was in connec-
the discussion in this summary vol- tion with the first two studies under-
ume, which was written in the late taken by the Institute in the spring
spring of 1946. In the section on of 1947 that the critical incident
techniques for defining job require- technique was more formally de-
ments, the present author wrote as veloped and given its present name.
follows: These studies were natural exten-
sions of the previous research in the tories were interviewed. These sci-
Aviation Psychology Program. The entists reported more than 2,500
study reported by Preston (52) dealt critical incidents. The critical be-
with the determination of the critical haviors were used to formulate induc-
requirements for the work of an officer tively a set of 36 categories, which
in the United States Air Force. In constitutes the critical requirements
this study, many of the procedural for the effective performance of the
problems were first subjected to duties of research personnel in the
systematic tryout and evaluation. physical sciences. This initial study
Six hundred and forty officers were provided the basis for the develop-
interviewed, and a total of 3,029 crit- ment of selection tests, proficiency
ical incidents were obtained. This measures, and procedures for evaluat-
led to the development of a set of 58 ing both job performance and the
critical requirements classified into research report.
six major areas. The second study, Another project undertaken by the
reported by Gordon (27, 28), was American Institute for Research in
carried out to determine the critical the spring of 1948 provided valuable
requirements of a commercial airline experience with the critical incident
pilot. In this study, several different technique. This study, reported by
sources were used to establish the Nagay (48), was done for the Civil
critical requirements of the airline Aeronautics Administration under
pilot. These included training rec- the sponsorship of the Committee on
ords, flight check records including Aviation Psychology of the National
the specific comments of check pilots, Research Council. It was concerned
critical pilot behaviors reported in with the air route traffic controller's
accident records, and critical inci- job. One of the innovations in this
dents reported anonymously in in- study was the use of personnel of the
terviews by the pilots themselves. Civil Aeronautics Administration
From this study, 733 critical pilot who had no previous psychological
behaviors were classified into 24 criti- training in collecting critical inci-
cal requirements of the airline pilot's dents by means of personal inter-
job. These were used to develop selec- views. In previous studies all such
tion tests to measure the aptitudes interviewing had been conducted by
and other personality characteristics psychologists with extensive training
found critical for success in the job. in such procedures. In this study,
They also provided the basic data aeronautical specialists from each of
for the formulation of an objective the seven regions conducted the inter-
flight check to determine the eligi- views in their regions after a brief
bility of applicants for the airline training period. An interesting find-
transport rating. ing from this study was the clear re-
The third application of the critical flection of seasonal variations in fly-
incident technique by the staff of the ing conditions in the types of inci-
American Institute for Research was dents reported. The study also
in obtaining the critical requirements demonstrated the selective recall of
for research personnel on a project dramatic or other special types of
sponsored by the Psychological Sci- incidents. This bias was especially
ences Division of the Office of Naval noticeable in the incidents reported
Research. In this study (20), about several months after their occur-
500 scientists in 20 research labora- rence. The incidents obtained in this
study were used to develop proce- only at the end of the two-week peri-
dures for evaluating the proficiency of od appeared to have forgotten 80
air route traffic controllers and also per cent of the incidents observed.
for developing a battery of selection Although it is possible that the find-
tests for this type of personnel. ings may be partially attributed to
In the spring of 1949 the American the fact that the foremen making
Institute for Research undertook a daily records actually observed more
study to determine the critical job critical incidents because of the daily
requirements for the hourly wage reminder at the time of recording, it
employees in the Delco-Remy Divi- is clear that much better results can
sion of the General Motors Corpora- be expected when daily recording is
tion. This study, reported by Miller used.
and Flanagan (46), was the first Another analysis based on data
application of these techniques in an collected at the Delco-Remy Division
industrial situation. Foremen who compared the number of critical in-
were members of a committee ap- cidents of various types obtained
pointed to develop employee evalua- from interviews with those recorded
tion procedures collected 2,500 criti- daily by the foremen on the perform-
cal incidents in interviews with the ance record. Although there were
other foremen in the plants. On the some differences in the relative fre-
basis of these data a form was pre- quencies for specific categories, the
pared for collecting incidents on a general patterns appeared to be quite
day-to-day basis as a continuous similar. These results suggest that
record of job performance. critical incidents obtained from in-
Using this form, the Performance terviews can be relied on to provide
Record for Hourly Wage Employees a relatively accurate account of job
(21), three groups of foremen kept performance if suitable precautions
records on the performance of their are taken to prevent systematic bias.
employees for a two-week period. A In addition to the development of
group of 24 foremen recorded inci- the performance record described
dents daily; another group of 24 above, the critical incidents collected
foremen reported incidents at the in this study were used as the basis
end of each week; and a third group for constructing selection tests cover-
containing the same number of fore- ing both aptitude (18) and attitude
men reported incidents only at the (2) factors.
end of the two-week period. The
three groups of foremen represented STUDIES CARRIED OUT AT THE
comparable conditions of work and UNIVERSITY OF PITTSBURGH
supervision. The foremen reporting A substantial number of studies
daily reported 315 critical incidents; have been carried out in the depart-
the foremen reporting weekly, 155 ment of psychology at the University
incidents; and the foremen reporting of Pittsburgh by students working
only once at the end of two weeks for advanced degrees under the au-
reported 63 incidents. Thus, foremen thor's direction. Most of these studies
who reported only at the end of the had as their objective the determina-
week had forgotten approximately tion of the critical requirements for
one half of the incidents they would a specific occupational group or ac-
have reported under a daily reporting tivity. Many of them also included
plan. The foremen who reported contributions to technique. In 1949
Wagner (66) completed a dissertation tained from questions intended to

on the critical requirements for den- elicit more definitely effective or in-
tists. In this study, critical incidents effective behaviors. Some examples
were obtained from three sources: of these questions are:
patients, dentists, and dental school 1. Think of a time when a foreman has done
instructors. The incidents were clas- something that you felt should be encouraged
sified into four main aspects of the because it seemed to be in your opinion an ex-
ample of good foremanship. (Effective—
dentist's job: (a) demonstrating slight deviation from norm.)
technical proficiency; (&) handling 2. Think of a time when a foreman did
patient relationships; (c) accepting something that you thought was not up
professional responsibility; and (d) to par. (Ineffective—slight deviation from
accepting personal responsibility. As norm.)
3. Think of a time when a foreman has, in
might be expected, the patients did your opinion, shown definitely good foreman-
not report as large a proportion of ship—the type of action that points out the
incidents for demonstrating technical superior foreman. (Effective—substantial de-
proficiency or accepting professional viation from the norm.)
4. Think of a time when a foreman has, in
responsibility as did the other two your opinion, shown poor foremanship—the
groups, and the instructors reported sort of action which if repeated would indicate
only a relatively small proportion of that the man was not an effective foreman.
their incidents in the area of handling (Ineffective—substantial deviation from
patient relationships. norm.)
On the basis of the findings from The frequencies of incidents ob-
this study, a battery of selection tests tained in each of the 40 categories
was developed for use by the Uni- into which the effective behaviors
versity of Pittsburgh School of Den- were classified were compared for
tistry. A number of proficiency tests the questions requesting slight and
for measuring ability with respect substantial deviations from the norm,
to certain of the critical requirements and the significance of the differences
were also developed using these re- was tested by means of the chi-
sults as a basis. square test. Two of the differences
Another dissertation completed in were significant at the 1 per cent
1949 was Finkle's (11) study of the level and one at the 5 per cent level.
critical requirements of industrial Comparisons of the frequencies in
foremen. This study was conducted each of the 40 categories for ineffec-
in the East Pittsburgh plant of the tive incidents failed to reveal any
Westinghouse Electric Corporation. chi squares significant at either the
Critical incidents were obtained from 5 per cent or the 1 per cent level.
foremen, general foremen, and staff The questions involving only a
personnel. A number of points per- slight deviation from the norm re-
taining to technique were studied. sulted in more effective incidents
One finding was in reference to the concerned with gaining the respect
effect on the types of incidents ob- and loyalty of the workers and also
tained of the degree of importance in more incidents that involved
or exceptionalness set up as a criterion making, encouraging, and accepting
for reporting or ignoring incidents. suggestions. They produced signifi-
The incidents obtained from the use cantly fewer incidents regarding fit-
of questions that asked for incidents ting men to jobs. The small number
only slightly removed from the norm of significant differences—only three
were compared with incidents ob- in 80 comparisons—suggests that the
types of incidents obtained are not Weislogel (72) determined the crit-
very greatly changed by variations ical requirements for life insurance
in wording of the questions compa- agency heads. A principal feature
rable to those shown above. It seems of his study related to the compari-
likely that this is at least partially son of two types of agency heads—
due to the fact that the persons in- managers and general agents. It was
terviewed report only incidents that believed that the critical behaviors
represent a fairly substantial devia- for one type of agency head might
tion from the norm regardless of provide a different pattern than that
the precise wording of the question obtained for the other. This hypothe-
asked. sis was not confirmed by the analysis
Another comparison made in this of the obtained incidents. The pat-
study related to the influence of ask- terns of critical requirements were
ing for an effective or an ineffective found to be quite similar for the two
incident first. About 10 per cent more types of administrators.
incidents were obtained from book- Smit (58) carried out a study to
lets requesting effective incidents first determine the critical requirements
than from booklets requesting infor instructors of general psychology
effective incidents first. This differ- courses. Perhaps the finding of most
ence was sufficiently small so that general importance in this study was
it could reasonably be attributed to the existence of substantial differ-
chance sampling fluctuations. ences between the patterns of critical
The incidents collected in this incidents reported by students and
study were used, along with other faculty. The faculty reported a signif-
data, in the preparation of a Perform- icantly larger percentage of effective
ance Record for Foremen and Super- behaviors in the following areas:
visors (23). giving demonstrations or experi-
A study was conducted by Nevins ments, using discussion group tech-
(50) on the critical requirements of niques, encouraging and ascertaining
bookkeepers in sales companies. She students' ideas and opinions.
collected incidents relating to appli- The students, on the other hand,
cants for bookkeeping positions as contributed a larger percentage of
well as for employees working in this behaviors in the following areas: re-
capacity. viewing examinations, distributing
For the collection of the informa- grades, and explaining grades; using
tion about the practicing bookkeep- lecture aids such as drawings, charts,
ers, a modification in the critical inci- movies, models, and apparatus; using
dent technique was made. This was project techniques; giving test ques-
done because, in the bookkeeping tions on assigned material; helping
profession, success and failure are students after class and during class
usually defined in terms of persistent recess; the manner of the instructor.
behavioral patterns. Occasional mis- The faculty reported a larger per-
takes in adding and balancing ac- centage of ineffective behaviors con-
counts are expected, but repeated cerning maintaining order. The in-
errors are considered serious. Instead effective behaviors that were reported
of the single incident, therefore, in a larger percentage by students in-
many of the items included repre- volved these areas: presenting re-
sented either a pattern of behaviors or quirements of the course, using effec-
a series of similar behaviors. tive methods of expression, dealing
with students' questions, pointing Because of the indefinite nature of
out fallacies, reviewing and summa- the concept, it was felt that a check
rizing basic facts and principles, using should be made on the contributor's
project techniques, using verbal diag- understanding of his task. Twenty of
nostic teaching techniques, achieve- the participating persons were asked
ment testing students on assigned to summarize briefly their interpreta-
material, objective type achievement tion of what they had been asked to
testing, using humor. do. This appeared to be very useful
This is a good illustration of the in developing the phrasing of the
problem of the competence of various questions so that they were uniformly
types of available observers to evalu- interpreted by the various observers.
ate the contribution to the general The author of the study classified
aim of the activity of a specific action. all the immaturities on the basis of a
Examination of the reports from stu- classification system developed from
dents indicated a somewhat limited preliminary categorizations prepared
sphere of competence. Apparently by six of the contributors. This clas-
one of the principal reasons for this sification was submitted to 14 psychi-
was the lack of perspective on the atrists for review. They were asked
part of the students and their inabil- to indicate which of the categories
ity to keep the general aim of the they were willing to accept as a type
instructor clearly in mind because of of immaturity as the term had been
its divergence from their own im- defined in an official document.
mediate aims. In many cases, this More than half the categories were
latter aim seemed to be directed accepted by at least 13 of the 14
toward achieving a satisfactory grade judges, and none was rejected by more
in the course. than 50 per cent of the judges. It was
Eilbert (7) developed a functional felt then that the system was accept-
description of emotional immaturity. able.
The contributors of critical incidents This study illustrates the applica-
included psychiatrists, psychologists, tion of the critical incident technique
psychiatric social workers, occupa- to the study of personality. It is
tional therapists, nurses, and corps- believed that this study provides an
men from a military hospital, plus excellent example of the possibilities
13 psychologists in nonmilitary or- for developing more specific behavi-
ganizations. The subjects of the in- oral descriptions.
cidents were primarily patients under Folley (24) reported on the critical
psychiatric care. requirements of sales clerks in depart-
The contributors were given a form ment stores. The behaviors were
that oriented them to the concept abstracted from narrative records of
"emotional immaturity" by sug- individual shopping incidents written
gesting that it was revealed generally by shoppers who were relatively in-
by childlike modes of behavior. The experienced in evaluating sales per-
questions used to elicit incidents sonnel. For various reasons, including
were: Have you recently thought of the competence of the observers,
someone as being emotionally im- their training, and their limited point
mature (regardless of diagnosis)? of view, the resulting description
What specifically happened that gave must be regarded as only partial.
you this impression? What would In the past few years, many other
have been a more mature reaction to individuals and groups have made
the same situation? use of the techniques described
above, or modifications of them, in a types of judgments are required re-
wide variety of studies. Some of these garding the relevance of various con-
studies on which reports are being ditions and actions on the observed
published will be reviewed briefly in success in attaining the defined pur-
the section on applications. pose for this activity.
It is believed that a fair degree of
THE PROCEDURE IN ITS success has been achieved in develop-
PRESENT FORM ing procedures that will be of assist-
From the foregoing discussion, it is ance in gathering facts in a rather
clear that the critical incident tech- objective fashion with only a mini-
nique is essentially a procedure for mum of inferences and interpreta-
gathering certain important facts tions of a more subjective nature.
concerning behavior in defined situa- With respect to two other steps that
tions. It should be emphasized that are essential if these incidents are to
the critical incident technique does be of value a comparable degree of
not consist of a single rigid set of objectivity has not yet been ob-
rules governing such data collection. tained. In both instances, the sub-
Rather it should be thought of as a jective factors seem clearly due to
flexible set of principles which must current deficiencies in psychological
be modified and adapted to meet the knowledge.
specific situation at hand. The first of these two other steps
The essence of the technique is consists of the classification of the
that only simple types of judgments critical incidents. In the absence of
are required of the observer, reports an adequate theory of human be-
from only qualified observers are havior, this step is usually an in-
included, and all observations are ductive one and is relatively subjec-
evaluated by the observer in terms of tive. Once a classification system has
an agreed upon statement of the pur- been developed for any given type of
pose of the activity. Of course, sim- critical incidents, a fairly satisfactory
plicity of judgments is a relative degree of objectivity can be achieved
matter. The extent to which a re- in placing the incidents in the defined
ported observation can be accepted categories.
as a fact depends primarily on the The second step refers to inferences
objectivity of this observation. By regarding practical procedures for
objectivity is meant the tendency for improving performance based on the
a number of independent observers observed incidents. Again, in our
to make the same report. Judgments present stage of psychological knowl-
that two things have the same effect edge, we are rarely able to deduce or
or that one has more or less effect predict with a high degree of con-
than the other with respect to some fidence the effects of specific selection,
defined purpose or goal represent the training, or operating procedures on
simplest types of judgments that can future behaviors of the type observed.
be made. The accuracy and therefore The incidents must be studied in the
the objectivity of the judgments de- light of relevant established principles
pend on the precision with which the of human behavior and of the known
characteristic has been defined and facts regarding background factors
the competence of the observer in and conditions operating in the spe-
interpreting this definition with rela- cific situation. From this total pic-
tion to the incident observed. In this ture hypotheses are formulated. In
latter process, certain more difficult only a few types of activities are there
both sufficient established principles precisely what it is necessary to do
and sufficient information regarding and not to do if participation in the
the effective factors in the situation activity is to be judged successful or
to provide a high degree of confidence effective. It is clearly impossible to
in the resulting hypotheses regarding report that a person has been either
specific procedures for improving the effective or ineffective in a particular
effectiveness of the results. activity by performing a specific act
In the sections which follow, the unless we know what he is expected
five main steps included in the pres- to accomplish. For example, a super-
ent form of the procedures will be visor's action in releasing a key
described briefly. In order to pro- worker for a half a day to participate
vide the worker with maximum flexi- in a recreational activity might be
bility at the present stage, in addition evaluated as very effective if the
to examples of present best practice, general aim of the foreman was to get
the underlying principles for the step along well with the employees under
will be discussed and also the chief him. On the other hand, this same
limitations with, wherever possible, action might be evaluated as ineffec-
suggestions for studies that may re- tive if the primary general aim is the
sult in future improvements in the immediate production of materials or
methods. services.
In the case of the usual vocational
1. General Aims activities the supervisors can be ex-
A basic condition necessary for any pected to supply this orientation. In
work on the formulation of a func- certain other types of activities, such
tional description of an activity is a as civic, social, and recreational ac-
fundamental orientation in terms of tivities, there frequently is no super-
the general aims of the activity. No visor. The objectives of participation
planning and no evaluation of specific in the activity must then be deter-
behaviors are possible without a mined from the participants them-
general statement of objectives. The selves. In some instances, these may
trend in the scientific field toward not be verbalized to a sufficient ex-
operational statements has led a tent to make it possible to obtain
number of writers to try to describe them directly.
activities or functions in terms of the Unfortunately, in most situations
acts or operations performed, the there is no one general aim which is
materials acted on, the situations in- the correct one. Similarly, there is
volved, the results or products, and rarely one person or group of persons
the relative importance of various who constitute an absolute, authori-
acts and results. These analyses have tative source on the general aim of
been helpful in emphasizing the need the activity. In a typical manufac-
for more specific and detailed de- turing organization the foreman, the
scriptions of the requirements of ac- plant manager, the president, and the
tivities. Typically, however, such stockholders might define the general
discussions have failed to emphasize aim of the workers in a particular
the dominant role of the general aim section somewhat differently. It is
in formulating a description of suc- not possible to say that one of these
cessful behavior or adjustment in a groups knows the correct general aim
particular situation. and the others are wrong. This does
In its simplest form, the functional not mean that one general aim is as
description of an activity specifies good as another and that it is unim-
portant how we define the purpose of ment should be referred either to

the activity. It does mean that we these authorities or to others to ob-
cannot hope to get a completely obtain a final statement of the general
jective and acceptable general aim for aim that is acceptable to them.
a specific activity. The principal Necessary revisions should be made
criterion in formulating procedures as indicated by these discussions.
for establishing the general aim of the Usually considerable effort is required
activity should be the proposed use to avoid defeating the purpose of the
of the functional description of the
activity which is being formulated.
Unless the general aim used is accept- OUTLINE FOR INTERVIEW TO
able to the potential users of the de- ESTABLISH THE GENERAL
tailed statement of requirements, the AIM FOR AN ACTIVITY
whole effort in formulating this state- 1. Introductory statement: We are mak-
ment will have been wasted. ing a study of (specify activity). We
believe you are especially well quali-
The most useful statements of aims fied to tell us about (specify ac-
seem to center around some simple tivity).
phrase or catchword which is slogan- 2. Request for general aim: What would
like in character. Such words provide you say is the primary purpose of
(specify activity) ?
a maximum of communication with 3. Request for summary: In a few
only a minimum of possible misinter- words, how would you summarize
pretation. Such words as "apprecia- the general aim of (specify activ-
tion," "efficiency," "development," ity)?
"production," and "service" are
likely to be prominent in statements FIG. 1. SAMPLE FORM FOR USE IN OBTAIN-
of general aims. For example, the ING GENERAL AIM
general aim of a teacher in elemen-
tary school art classes might be the general aim by cluttering up the
development of an appreciation of statement with specific details and
various visual art forms on the part qualifying conditions.
of the students. The general aim of In summary, the general aim of an
the good citizen might be taken as activity should be a brief statement
effective participation in the develop- obtained from the authorities in the
ment and application of the rules and field which expresses in simple terms
procedures by which individuals and those objectives to which most people
groups are assisted in achieving their would agree. Unless a brief, simple
various goals. statement has been obtained, it will
With the aid of a form of the type be difficult to get agreement among
shown in Fig. 1, the ideas of a num- the authorities. Also it will be much
ber of well-qualified authorities can harder to convey a uniform idea to
be collected. It is expected that in the participants. This latter group
response to the question on the pri- will get an over-all impression and
mary purpose of the activity many this should be as close to the desired
persons will give a fairly lengthy and general aim as possible.
detailed statement. The request to
summarize is expected to get them to 2. Plans and Specifications
condense this into a brief usable To focus attention on those aspects
statement. These should be pooled of behavior which are believed to be
and a trial form of the statement of crucial in formulating a functional
general aim developed. This state- description of the activity, precise
338 JOHN C, FLANAGAN
instructions must be given to the cific behavior should be observed and re-
observers. It is necessary that these corded.
b. Relevance to the general aim. After the de-
instructions be as specific as possible cision has been made that a particular situa-
with respect to the standards to be tion is an appropriate one for making observa-
used in evaluation and classification. tions, the next step is to decide whether or not
The group to be studied also needs a specific behavior which is observed is rele-
to be specified. vant to the general aim of the activity as de-
fined in the section above. For example, if the
One practical device for obtaining general aim of the activity was defined as sus-
specific data is to obtain records of tained high quality and quantity of produc-
"critical incidents" observed by the tion, it might be difficult to decide whether or
reporting personnel. Such incidents not to include an action such as encouraging
an unusually effective subordinate to get train-
are defined as extreme behavior, ing that would assist him in developing his
either outstandingly effective or in- ability in an avocational or recreational activ-
effective with respect to attaining the ity not related to his work. In this case, it
general aims of the activity. The might be specified that any action which either
directly or indirectly could be expected over a
procedure has considerable efficiency long period of time to have a significant effect
because of the use of only the ex- on the general aim should be included. If it
tremes of behavior. It is well known could not be predicted with some confidence
that extreme incidents can be more whether this effect would be good or bad, it
accurately identified than behavior should probably not be considered.
The extent of detail required to obtain ob-
which is more nearly average in jectivity with respect to this type of decision
character. depends to a considerable degree on the back-
One of the primary aims of scien- ground and experiences of the observers with
tific techniques is to insure objectiv- respect to this activity. For example, super-
ity for the observations being made visors with substantial experience in a particu-
lar company can be expected to agree on
and reported. Such agreement by whether or not a particular behavior is rele-
independent observers can only be vant to the attainment of the general aim.
attained if they are all following the On the other hand, if outside observers were
same set of rules. It is essential that to be used, it would probably be necessary to
specify in considerable detail the activities
these rules be clear and specific. In that can be expected to have an effect on the
most situations the following speci- general aim.
fications will need to be established c. Extent of effect on the general aim. The re-
and made explicit prior to collecting maining decision that the observer must make
is how important an effect the observed inci-
the data: dent has on the general aim. It is necessary to
a. The situations observed. The first neces- specify two points on the scale of importance:
sary specification is a delimitation of the situa- (a) a level of positive contributions to the gen-
tions to be observed. This specification must eral aim in specific terms, preferably including
include information about the place, the per- a concrete example, and (6) the corresponding
sons, the conditions, and the activities. Such level of negative effect on the general aim ex-
specifications are rather easily defined in many pressed in similar terms.
instances. For example, such brief specifica- A definition which has been found useful is
tions as observations of "the behavior in class- that an incident is critical if it makes a "sig-
rooms of regularly employed teachers in a nificant" contribution, either positively or
specified high school while instructing students negatively, to the general aim of the activity.
during class periods," constitute a fairly ade- The definition of "significant" will depend on
quate definition of a situation of this type. the nature of the activity. If the general aim
In complex situations it is probably essen- of the activity is in terms of production, a sig-
tial not only that the specifications with re- nificant contribution might be one which
spect to the situation be relatively complete caused, or might have caused, an appreciable
and specific, but also that practical examples change in the daily production of the depart-
be provided to assist the observer in deciding ment either in the form of an increase or a de-
in an objective fashion whether or not a spe- crease.
In certain specific situations, it might be de- 3. Collecting the Data
sirable and possible to set up a quantitative
criterion such as saving or wasting IS minutes If proper plans and specifications
of an average worker's production. In some are developed, the data collection
situations, a definition of significance might be phase is greatly simplified. A neces-
set up in terms of dollars saved or lost both di-
rectly and indirectly. sary condition for this phase is that
Actions which influence the attitudes of oth- the behaviors or results observed be
ers are more difficult to evaluate objectively. evaluated, classified, and recorded
Perhaps the best we might be able to do is to while the facts are still fresh in the
state it in terms of a probability estimate. For mind of the observer. It would be
example, one such criterion might be that the
minimum critical level would be an action that desirable for these operations to be
would have an influence such that at least one
person in ten might change his vote on an issue
of importance to the company. Specifications Regarding Observations
d. Persons to make the observations. One ad-
ditional set of specifications refers to the selec- 1. Persons to make the observations.
tion and training of the observers who are to a. Knowledge concerning the activ-
make and report the judgments outlined in the ity.
steps above. b. Relation to those observed.
Wherever possible, the observers should be c. Training requirements.
selected on the basis of their familiarity with 2. Groups to be observed.
the activity. Special consideration should be a. General description.
given to observers who have made numerous b. Location.
observations on persons engaged in the activ- c. Persons.
ity. Thus, for most jobs, by far the best ob- d. Times.
servers are supervisors whose responsibility it e. Conditions.
is to see that the particular job being studied is 3. Behaviors to be observed.
done. However, in some cases very useful ob- a. General type of activity.
servations can be contributed by consumers of b. Specific behaviors.
the products and services of the activity. For c. Criteria of relevance to general
example, for a study of effective sales activi- aim.
ties, the customers may have valuable data to d. Criteria of importance to general
contribute. For a study of effective parental aim (critical points)
activity, the children may be able to make val-
uable contributions.
In addition to careful selection of the per- FIG. 2. FORM FOR DEVELOPING SPECIFICA-
sons to make observations, attention should TIONS REGARDING OBSERVATIONS
be given to their training. Minimal training
should include a review of the nature of the
general aim of the activity and a study of the performed at the time of observation
specifications and definitions for the judg- so that all requisite facts could be
ments they will be required to make. Where determined and checked. Memory is
the situation is complex or the observers are improved if it is known in advance
not thoroughly familiar with the activity, su- that the behavior to be observed is to
pervised practice in applying these definitions be remembered. It is greatly im-
should be provided. This can be done by pre-
paring descriptions of observations and asking proved if the specific aspects of what
the observers to make judgments about these is to be observed are defined and if
materials. Their judgments can be immedi- the operations to be performed with
ately confirmed or corrected during such su- respect to evaluation and classifica-
pervised practice periods. tion are clearly specified.
In Fig. 2 is shown a form for use in develop- The critical incident technique is
ing specifications regarding observations. The
use of this form in making plans for the collec-
frequently used to collect data on
tion of critical incidents or other types of ob- observations previously made which
servational data should aid in objectifying are reported from memory. This is
these specifications. usually satisfactory when the in-
cidents reported are fairly recent and production requirements and accept-
the observers were motivated to ing changes in jobs are higher in
make detailed observations and eval- terms of the recorded than the re-
uations at the time the incident called incidents. The fact that items
occurred. such as wasting time and assisting on
The importance of obtaining re- problems are lower for the recalled
cent incidents to insure that the in- incidents suggests that part of this
cidents are representative of actual discrepancy lies in the interpretations
happenings was demonstrated in the of the category definitions. The
study on air route traffic control- classifying of recorded incidents was
lers by Nagay (48) reported above. done by the foremen, while the classi-
However, as also discussed in that fication of the recalled incidents was
study, in some situations adequate done by the research workers. In
coverage cannot be obtained if only fairness, it should also be noted that
very recent incidents are included. the definitions used by the research
Evidence regarding the accuracy workers were rewritten before they
of reporting is usually contained in were incorporated in the foremen's
the incidents themselves. If full and manuals.
precise details are given, it can On the whole, it seems reasonable
usually be assumed that this informa- to assume that, if suitable precau-
tion is accurate. Vague reports sug- tions are taken, recalled incidents can
gest that the incident is not well re- be relied on to provide adequate data
membered and that some of the data for a fairly satisfactory first approxi-
may be incorrect. In several situa- mation to a statement of the require-
tions there has been an opportunity ments of the activity. Direct obser-
to compare the types of incidents vations are to be preferred, but the
reported under two conditions (a) efficiency, immediacy, and minimum
from memory and without a list of demands on cooperating personnel
the types of incidents anticipated, which are achieved by using recalled
and (b) those reported when daily incident data frequently make their
observations were being made in a use the more practical procedure.
routine work situation, and the eval- Another practical problem in col-
uations and classifications were made lecting the data for describing an
and recorded on a prepared check list activity refers to the problem of how
within 24 hours of the time of obser- it should be obtained from the ob-
vation. The results of one such com- servers. This applies especially to
parison were discussed briefly above the problem of collecting recalled
in connection with the American data in the form of critical incidents.
Institute for Research study of fac- Four procedures have been used and
tory employees. will be discussed briefly below:
During the observational period a
negligible number of incidents were a. Interviews. The use of trained personnel
to explain to observers precisely what data are
reported by the foremen as not fitting desired and to record the incidents, making
into the general headings included on sure that all necessary details are supplied, is
the list. Although the proportions of probably the most satisfactory data collection
incidents for the various items on the procedure. This type of interview is somewhat
list are not identical, they are reason- different from other sorts of interviews and a
brief summary of the principal factors involved
ably close for most of the items. will be given.
Items on such matters as meeting (i) Sponsorship of the study. If a stranger to
the observers is collecting the data, it is ordi- velopes, avoidance of identifying information,
narily desirable to indicate on what authority the mailing of data immediately to a distant
the interview is being held. This part should point for analysis, and similar techniques are
be as brief as possible to avoid any use of time helpful in establishing the good faith of the in-
for a prolonged discussion of a topic irrelevant terviewer in taking all possible precautions to
to the purpose of the interview. In many in- safeguard the incidents reported.
stances all that needs to be said is that some- (v) The question. The most crucial aspect
one known and respected by the observer has of the data collection procedure is the ques-
suggested the interview. tions asked the observers. Many studies have
(ii) Purpose of the study. This should also shown that a slight change in wording may
be brief and ordinarily would merely involve a produce a substantial change in the incidents
statement that a study was being made to de- reported. For example, in one study the last
scribe the requirements of the activity. This part of one of the specific questions asked was,
would usually be cast in some such informal "Tell just how this employee behaved which
form as, "We wish to find out what makes a caused a noticeable decrease in production."
good citizen," or, "We are trying to learn in This question resulted in almost all incidents
detail just what successful work as a nurse in- reported having to do with personality and at-
cludes." In cases where there is some hesita- titude behaviors. This part of the question was
tion about cooperating or a little more expla- changed to, "Tell just what this employee did
nation seems desirable, a statement can be which caused a noticeable decrease in produc-
added concerning the value and probable uses tion." This second question produced a much
of the results. This frequently takes the form broader range of incidents. To the question
of improving selection and training proce- writer "how he behaved" and "what he did"
dures. In some instances, it would involve im- seemed like about the same thing. To the fore-
proving the results of the activity. For ex- men who were reporting incidents "how he be-
ample, the interviewer might say, "In order to haved" sounded as if personality and atti-
get better sales clerks we need to know just tudes were being studied. The subtle biases
what they do that makes them especially ef- involved in the wording of questions are not
fective or ineffective," or, "If parents are to always so easily found. Questions should al-
be more effective, we need to be able to tell ways be tried out with a small group of typical
them the things they do that are effective and observers before being put into general use in
ineffective." a study.
(iii) The group being interviewed. If there is The question should usually refer briefly to
any likelihood of a person feeling, "But, why the general aim of the activity. This aim might
ask me?" it is desirable to forestall this by be discussed more fully in a preliminary sen-
pointing out that he is a member of a group tence. It should usually state that an incident,
which is in an unusually good position to ob- actual behavior, or what the person did is de-
serve and report on this activity. The special sired. It should briefly specify the type of be-
qualifications of members of this group as ob- havior which is relevant and the level of im-
servers can be mentioned briefly, as, "Super- portance which it must reach to be reported.
visors such as yourself are constantly observ- It should also tie down the selection of the in-
ing and evaluating the work of switchboard cidents to be reported by the observer in some
girls," or, "Students are in an unusually good way, such as asking for the most recent obser-
position to observe the effectiveness of their vation, in order to prevent the giving of only
teachers in a number of ways." the more dramatic or vivid incidents, or some
(iv) The anonymity of the data. Especially other selected group, such as those which fit
for the collection of information about inef- the observer's stereotypes.
fective behavior, one of the principal prob- An effective procedure for insuring that the
lems is to convince the observer that his report interpretation of the persons being interviewed
cannot harm the person reported on in any is close to that intended is to request a sample
way. Usually he also needs to be convinced of persons typical of those to be interviewed to
that the person reported on will never know state in their own words what they understand
that he has reported the incident. Assurances they have been asked to do. These persons
are not nearly so effective in this situation as should be selected so as to represent all types
actual descriptions of techniques to be used in who will be interviewed. From a study of their
handling the data, which enable the observer interpretations, necessary revisions can be
to judge for himself how well the anonymity of made to insure that all interviewees will be in
the data will be guarded. Under no circum- agreement as to the nature of the incidents
stances should the confidences of the reportees they are to provide.
be violated in any way. The use of sealed en- (vi) The conversation. The interviewer
should avoid asking leading questions after
the main question has been stated. His re-
marks should be neutral and permissive and "Think of the last time you saw one
should show that he accepts the observer as of your subordinates do something that
the expert. By indicating that he understands was very helpful to your group in
what is being said and permitting the observer meeting their production schedule."
to do most of the talking, the interviewer can (Pause till he indicates he has such an
usually get unbiased incidents. If the question incident in mind.) "Did his action re-
does not seem to be understood, it can be result in increase in production of as much
peated with some reference to clarifying just as one per cent for that day?—or some
what is meant by it. If the observer has given similar period?"
what seems like only part of the story, he (If the answer is "no," say) "I won-
should be encouraged by restating the essence der if you could think of the last time
of his remarks. This usually tends to encour- that someone did something that did
age him to continue and may result in his have this much of an effect in increasing
bringing out many relevant details that the in- production." (When he indicates he has
terviewer did not know the situation well such a situation in mind, say) "What
enough to ask for. In some cases, it is desira- were the general circumstances leading
ble to have the interviews recorded elec- up to this incident?"
trically and transcribed. This increases the
work load substantially, and trained inter-
viewers can usually get satisfactory reports at
the time or by editing their notes shortly after "Tell me exactly what this person did
the interview. that was so helpful at that time."
Usually the interviewer should apply cer-
tain criteria to the incidents while they are be-
ing collected. Some of the more important cri-
teria are: (a) is the actual behavior reported; "Why was this so helpful in getting
(6) was it observed by the reporter; (c) were all
relevant factors in the situation given; (d) has your group's job done?"
the observer made a definite judgment regard-
ing the criticalness of the behavior; (e) has the
observer made it clear just why he believes the
behavior was critical. "When did this incident happen?"
In Fig. 3 is shown a sample of the type of
form used by interviewers to collect critical in-
cidents. Of course the form must be adapted "What was this person's job?"
to the needs of the specific situation.
b. Group interviews. Because of the cost in
time and personnel of the individual interview, "How long has he been on this job?"
a group interview technique has been devel-
oped. This retains the advantages of the indi-
vidual interview in regard to the personal con- 'How old is he?"
tact, explanation, and availability of the inter-
viewer to answer questions. To some extent it
also provides for a check on the data supplied FIG. 3. SAMPLE OF A FORM FOR USE BY AN
by the interviewees. Its other advantages are INTERVIEWER IN COLLECTING EFFECTIVE
that the language of the actual observer is pre- CRITICAL INCIDENTS
cisely reproduced and the time for editing the
interviews is virtually eliminated. ble for the interviewer to read the responses of
The method consists of having the inter- each member of the group to the first question
viewer give his introductory remarks to a and make sure that he understands what is
group very much as he would do in an individ- wanted. There seems to be a certain amount
ual interview. There is an opportunity for of social facilitation, and the results in most
questions and clarification. Then each person situations have been excellent. In the report
is asked to write incidents in answer to specific of the first use of this procedure by Wagner
questions contained on a specially prepared (65), the amount of interviewer time required
form. The size of the group which can be han- per usable incident was 4.3 minutes for the
dled effectively will vary with the situation. group interview procedure as compared with
If the group is fairly small, it is usually possi- 15.7 minutes for individual interviews. The
quality of these incidents, obtained from offi- tion system with each additional 100 inci-
cers in the United States Air Force, appeared dents. For most purposes, it can be considered
to be about the same for the two situations. that adequate coverage has been achieved
c. Questionnaires. If the group becomes when the addition of 100 critical incidents to
large, the group interview procedure is more in the sample adds only two or three critical be-
the nature of a questionnaire procedure. haviors. For jobs of a supervisory nature, it
There are, of course, all types of combinations appears that between 2,000 and 4,000 critical
of procedures that can be used. The one that incidents are required to establish a compre-
is most different from those discussed is the hensive statement of requirements that in-
mailed questionnaire. In situations where the cludes nearly all of the different types of criti-
observers are motivated to read the instruc- cal behaviors. For semiskilled and skilled jobs
tions carefully and answer conscientiously, between 1,000 and 2,000 incidents seem to be
this technique seems to give results which are adequate to cover the critical behaviors.
not essentially different from those obtained Coverage of all or nearly all of the various
by the interview method. Except for the ad- critical behaviors is not the only criterion as to
dition of introductory remarks, the forms used whether or not a sufficient number of critical
in collecting critical incidents by means of incidents has been collected. If a relatively
mailed questionnaires are about the same as precise definition of each critical behavior cat-
those used in group interviews. egory is required, it may be necessary to get at
d. Record forms. One other procedure for least three or four examples of each critical be-
collecting data is by means of written records. havior. Similarly, if the critical incidents are
There are two varieties of recording: one is to to be used as a basis for developing selection
record details of incidents as they happen. tests, training materials, and proficiency
This situation is very similar to that described measures, more incidents may be required to
in connection with obtaining incidents by in- provide a sufficient supply of usable ideas for
terviews above, except that the observation the development of these materials.
and giving of incidents are delayed following In summary, although there is no simple
the introductory remarks and the presentation formula for determining the number of critical
of the questions until an incident is observed incidents that will be required, this is a very
to happen. important consideration in the plan of the
A variation of this procedure is to record study; checks should be made both on the first
such incidents on forms which describe most of hundred or so incidents and again after ap-
the possible types of incidents by placing a proximately half of the number of incidents
check or tally in the appropriate place. believed to be required have been obtained in
As additional information becomes availa- order to make it possible to revise the prelimi-
ble on the nature of the components which nary estimates, if necessary, with a minimum
make up activities, observers may thus collect loss in effort and time.
data more efficiently by using forms for record
ing and classifying observations. In the mean-
time, because of the inadequacy of the infor- 4. Analyzing the Data
mation currently available regarding these The collection of a large sample of
components, it seems desirable to ask observ-
ers to report their observations in greater de- incidents that fulfill the various con-
tail and have the classification done by spe- ditions outlined above provides a
cially trained personnel. functional description of the activity
Size of sample. A general problem which in terms of specific behaviors. If the
overlaps the phases of collecting the incidents
and analyzing the data relates to the number sample is representative, the judges
of incidents required. There does not appear well qualified, the types of judgments
to be a simple answer to this question. If the appropriate and well defined, and the
activity or job being defined is relatively sim- procedures for observing and report-
ple, it may be satisfactory to collect only SO or
100 incidents. On the other hand, some types ing such that incidents are reported
of complex activity appear to require several accurately, the stated requirements
thousand incidents for an adequate statement can be expected to be comprehensive,
of requirements. detailed, and valid in this form.
The most useful procedure for determining There is only one reason for going
whether or not additional incidents are needed
is to keep a running count on the number of further and that is practical utility.
new critical behaviors added to the classifica- The purpose of the data analysis
stage is to summarize and describe development of procedures for evaluating on-
the data in an efficient manner so the-job effectiveness. For selection purposes,
the most appropriate classification system is a
that it can be effectively used for psychological one. The main headings have to
many practical purposes. do with types of psychological traits that are
In the discussion which follows, it utilized in the selection process. For training
should be kept in mind that the uses, the best classification system follows a
set of headings that is easily related to training
process of description has been com- courses or broad training aims. For pro-
pleted. The specific procedures to be ficiency measurement, the headings tend to be
discussed are not concerned with im- similar to those for training except that there
proving on the comprehensiveness, is less attention to possible course organization
specificity of detail, or validity of the and aims and greater attention to the com-
ponents of the job as it is actually performed.
statement of the requirements of the For the development of procedures for evalu-
activity. Rather, they are concerned ating on-the-job effectiveness to establish a
with making it easier to report these criterion of success, the classification system is
requirements, to draw inferences necessarily directed at presenting the on-the-
job behaviors under headings that represent
from them, and to compare the ac- either well-marked phases of the job or pro-
tivity with other activities. vide a simple framework for classifying on-the-
The aim is to increase the useful- job activities that is either familiar to or easily
ness of the data while sacrificing as learned by supervisors.
Similarly, in nonvocational activities the
little as possible of their compre- frame of reference depends on the uses planned
hensiveness, specificity, and validity. for the findings. For example, if a study is be-
It appears that there are three pri- ing made to define immaturity reactions in
mary problems involved: (a) the military personnel, the frame of reference
selection of the general frame of would depend somewhat on whether the func-
tional description is to be used primarily to
reference that will be most useful for identify personnel showing this type of malad-
describing the incidents; (&) the in- justment or whether the principal use will be
ductive development of a set of to try to prepare specifications for types of
major area and subarea headings; situations in which immaturity reactions
would not lead to serious difficulties.
and (c) the selection of one or more b. Category formulation. The induction of
levels along the specificity-generality categories from the basic data in the form of
continuum to use in reporting the incidents is a task requiring insight, experi-
requirements. Each of these prob- ence, and judgment. Unfortunately, this pro-
lems will be discussed below: cedure is, in the present stage of psychological
knowledge, more subjective than objective.
a. Frame of reference. There are countless No simple rules are available, and the quality
ways in which a given set of incidents can be and usability of the final product are largely
classified. In selecting the general nature of dependent on the skill and sophistication of
the classification, the principal consideration the formulator. One rule is to submit the ten-
should usually be that of the uses to be made tative categories to others for review. Al-
of the data. The preferred categories will be though there is no guarantee that results
those believed to be most valuable in using the agreed on by several workers will be more use-
statement of requirements. Other considera- ful than those obtained from a single worker,
tions are ease and accuracy of classifying the the confirmation of judgments by a number of
data, relation to previously developed defini- persons is usually reassuring. The usual pro-
tions or classification systems, and considera- cedure is to sort a relatively small sample of
tions of interpretation and reporting, which incidents into piles that are related to the
will be discussed in a later section. frame of reference selected. After these tenta-
For job activities, the choice of a frame of tive categories have been established, brief
reference is usually dominated by considera- definitions of them are made, and additional
tions of whether the principal use of the re- incidents are classified into them. During this
quirements will be in relation to selection, process, needs for redefinition and for the de-
training, measurement of proficiency, or the velopment of new categories are noted. The
tentative categories are modified as indicated 5. Interpreting and Reporting

and the process continued until all the inci-
dents have been classified. It is never possible in practice to
The larger categories are subdivided into obtain an ideal solution for each of
smaller groups and the incidents that describe the practical problems involved in
very nearly the same type of behavior are obtaining a functional description of
placed together. The definitions for all the
categories and major headings should then be an activity. Therefore, the statement
re-examined in terms of the actual incidents of requirements as obtained needs
classified under each. interpretation if it is to be used
c. General behaviors. The last step is to de- properly. In many cases, the real
termine the most appropriate level of specific-
ity-generality to use in reporting the data. errors are made not in the collection
This is the problem of weighing the advan- and analysis of the data but in the
tages of the specificity achieved in specific inci- failure to interpret them properly.
dents against the simplicity of a relatively Each of the four preceding steps, (a)
small number of headings. The level chosen
might be only a dozen very general behaviors the determination of the general aim,
or it might be several hundred rather specific (6) the specification of observers,
behaviors. Practical considerations in the im- groups to be observed, and observa-
mediate situation usually determine the opti- tions to be made, (c) the data collec-
mal level of generality to be used. tion, and (d) the data analysis, must
Several considerations should be kept in
mind in establishing headings for major areas be studied to see what biases have
and in stating critical requirements at the se- been introduced by the procedures
lected level of generality. These are listed be- adopted. If there is a division of
low: opinion as to the general aim and one
(i) The headings and requirements should
indicate a clear-cut and logical organization. of the competing aims is selected,
They should have a discernible and easily re- this should be made very clear in the
membered structure. report. If the groups on whom the
(ii) The titles should convey meanings in observations are made are not repre-
themselves without the necessity of detailed sentative of the relevant groups in-
definition, explanation, or differentiation.
This does not mean that they should not be volved, they must be described as
defined and explained. It does mean that these precisely as possible. The aim of the
titles, without the detailed explanation, study is usually not a functional de-
should still be meaningful to the reader. scription of the activity as carried on
(iii) The list of statements should be ho-
mogeneous; i.e., the headings for either areas by this sample but rather a state-
or requirements should be parallel in content ment relating to all groups of this
and structure. Headings for major areas type. In order to avoid faulty in-
should be neutral, not defining either unsatis- ferences and generalizations, the limi-
factory or outstanding behaviors. Critical re- tations imposed by the group must
quirements should ordinarily be stated in posi-
tive terms. be brought into clear focus. Simi-
(iv) The headings of a given type should all larly, the nature of judgments made
be of the same general magnitude or level of in collecting and analyzing the data
importance. Known biases in the data causing must be carefully reviewed.
one area or one requirement to have a dispro-
portionate number of incidents should not be While the limitations need to be
reflected in the headings. clearly reported, the value of the
(v) The headings used for classification and results should also be emphasized.
reporting of the data should be such that find- Too often the research worker shirks
ings in terms of them will be easily applied and his responsibility for rendering a
maximally useful. judgment concerning the degree of
(vi) The list of headings should be compre-
hensive and cover all incidents having signifi- credibility which should be attached
cant frequencies. to his findings. It is a difficult task,
but if the results are to be used, and a check list or some similar type
someone will have to make such a of procedure for evaluating the typi-
judgment, and the original investi- cal performance of persons engaged
gator is best prepared to make the in this activity. If an observational
necessary evaluations either for the check list that includes all of the im-
general case or for certain typical portant behaviors for the activity is
specific examples. available, the performance of the in-
dividual can be objectively evaluated
USES OF THE CRITICAL INCIDENT and recorded by merely making a
TECHNIQUE single tally mark for each observa-
The variety of situations in which tion. Such records provide the essen-
the collection of critical incidents will tial basis for criterion data which are
prove of value has only been par- sufficiently detailed and specific for
tially explored. In the approximately special purposes but at the same time
eight years since the writer and his can be combined into a single over-all
colleagues began a systematic for- evaluation when this is desirable.
mulation of principles and procedures Such a procedure was first suggested
to be followed in collecting this type and tried out in connection with de-
of data, a fairly large number of ap- velopmental studies of the American
plications has been made. The appli- Institute for Research. These in-
cations will be discussed under the cluded: Preston's study of officers for
following nine headings: (a) Meas- the United States Air Force (52);
ures of typical performance (criteria); Nagay's study on air route traffic
(J) measures of proficiency (standard controllers for the Civil Aeronautics
samples); (c) training; (d) selection Administration (49); and M. H.
and classification; (e) job design and Weislogel's study on research per-
purification; (/) operating procedures; sonnel for the Office of Naval Re-
(g) equipment design; (h) motivation search (69). Another American In-
and leadership (attitudes); (i) coun- stitute for Research study was re-
seling and psychotherapy. ported by R. B. Miller and the
Space is not available here to de- present author (21). This was a per-
scribe these various applications in formance record form for hourly
detail. However, a brief description wage employees developed in coopera-
of the types of application that have tion with personnel of the Delco-
been made, along with brief illustra- Remy Division of the General Motors
tive examples and references, will be Corporation, the Employment Prac-
presented. Some of the studies in- tices Division of that corporation,
volve several of the types of applica- and the Industrial Relations Center
tions to be discussed. The presenta- of the University of Chicago. The
tion is not intended to be complete, same authors have developed similar
but rather to give the reader inter- performance records for salaried em-
ested in further study some orienta- ployees, and foremen and supervisors
tion and guidance. (22, 23). The principles and pro-
Measures of typical performance cedures underlying this type of evalu-
(criteria). The simplest and most ation of performance have been pub-
natural application of a systematic- lished elsewhere (14, 15, 17).
ally collected set of critical incidents A number of important contribu-
is in terms of the preparation of a tions to the development of func-
statement of critical requirements tional descriptions and standards of
performance have been made by —they are a means of sharing the more essen-
other groups using the critical inci- tial discipline of examining professional expe-
rience, forming hypotheses about professional
dent technique. One of the most conduct, and testing these hypotheses by ref-
notable of these is the development erence to the welfare of the people affected by
by Hobbs et al. (3, 31), of Ethical them (3, p. v).
Standards of Psychologists. More
than 1,000 critical incidents involving In addition to the study by Smit
ethical problems of psychologists mentioned in a previous section (58),
were contributed by the members of several other studies on the use of the
the American Psychological Associa- critical incident procedures as a
tion. It is believed that this repre- basis for evaluating teaching effec-
sents the first attempt to use empiri- tiveness have been reported. One of
cal methods to establish ethical these was a study conducted under
standards. Because of the impor- the joint sponsorship of the Educa-
tance of this study, and the generality tional Research Corporation and the
of some of the problems involved, Harvard University Graduate School
certain of the conclusions reported by of Education with funds provided by
the Committee on Ethical Standards the New England School Develop-
for Pyschology in their introductory ment Council and the George F.
statement will be quoted here. Milton Fund. This was an explora-
First, it is clear that psychologists believe tory study of teacher competence
that ethics are important; over two thousand reported by Domas (6). Approxi-
psychologists were sufficiently concerned with mately 1,000 critical incidents were
the ethical obligations of their profession to collected from teachers, principals,
contribute substantially to the formulation of and other supervisors. Although this
these ethical standards. Second, psychologists
believe that the ethics of a profession cannot be was an exploratory study, it was felt
prescribed by a committee; ethical standards that it made an important contribu-
must emerge from the day-by-day value com- tion to the general problem of relat-
mitments made by psychologists in the prac- ing salary to teacher competence.
tice of their profession. Third, psychologists
share a conviction that the problems of men, The second of these studies was
even those involving values, can be studied conducted as part of the teacher
objectively; this document summarizes the re- characteristics study sponsored by
sults of an effort to apply some of the tech- the American Council on Education
niques of social science to the study of ethical
behavior of psychologists. Fourth, psycholo- and subsidized by the Grant Founda-
gists are aware that a good code of ethics must tion. This study is reported by
be more than a description of the current sta- Jensen (32). Teachers, administra-
tus of ethics in the profession; a code must em- tors, and teachers in training in the
body the ethical aspirations of psychologists Los Angeles area contributed more
and encourage changes in behavior, bringing
performance ever closer to aspiration. Fifth, than 1,500 critical incidents of teacher
psychologists appreciate that process is often behavior. The incidents were classi-
more important than product in influencing fied under personal, professional, and
human behavior; the four years of widely- social qualities. The category formu-
shared work in developing this code are counted
on to be more influential in changing ethical lation indicated that there were about
practices of psychologists than will be the pub- 20 distinct critical requirements.
lication of this product of their work. Finally, These were recommended as a basis
psychologists recognize that the process of for teacher evaluation and as an aid
studying ethical standards must be a continu- to the in-service growth of teachers.
ing one; occasional publications such as this
statement mark no point of conclusion in the Another study was that of Smith
ongoing process of denning ethical standards and Staudohar (59), which deter-
mined the critical requirements for Another type of application of the

basic training of tactical instructors critical incident technique to the de-
in the United States Air Force. From velopment of bases for evaluating be-
130 training supervisors, 555 tactical havior is the previously mentioned
instructors, and 3,082 basic trainees, study of Eilbert (7). His list of 51
a total of 6,615 usable incidents were types of immature reaction based on
obtained. The authors comment a collection of several hundred criti-
that: cal incidents describing manifesta-
The training supervisors report a predomi- tions of emotional immaturity is be-
nance of ineffective incidents in the major lieved to provide a useful guide to
areas of: Sets a good example and maintains further investigation and appraisal of
effective personal relations. The tactical in- persons with behavior problems. It
structors report more ineffective incidents in
the area of Makes his expectations clear. Ba- is believed that the results of this
sic trainees show a predominance of ineffec- study provide substantial encourage-
tive incidents in three areas: Sets a good ex- ment to the application of the critical
ample, Considers trainee's needs, and Main- incident technique to similar prob-
tains effective personal relations (59, p. 5). lems in the field of clinical diagnosis
Another study on the evaluation of and evaluation.
instructor effectiveness was carried Measures of proficiency (standard
out by Konigsburg (33). This study samples). A closely related use of
involved the development of an in- critical incidents is to provide a basis
structor check list for college infor evaluating the performance of
structors based on the critical in- persons by use of standard samples of
cident technique and a comparison behavior involving important as-
of techniques for recording observa- pects of the activity. Such evalua-
tions. Its principal findings were the tions are called proficiency measures
very low correlation coefficients be- and are differentiated from the evalu-
tween the total scores from the ation of typical performance on the
Purdue Rating Scale for Instruction job primarily on the basis that a test
and the instructor check list. When situation rather than a real job situa-
these two instruments were each tion is used. Measures of this sort are
given to half the class on the same especially useful at the end of train-
day, the average correlation coeffi- ing courses as checks on the main-
cient was found to be .29. The other tenance of proficiency, and when the
principal finding is that the planned tasks assigned to participants vary a
performances of a total of 46 pre- great deal in difficulty or are not di-
determined behaviors were better rectly observed by the supervisors.
reflected by the results obtained on One of the first applications of
the instructor check list than by the critical incidents to the development
results on the Purdue Rating Scale. of proficiency measures was Gordon's
A somewhat related study has been study on the development of a stand-
reported by Barnhart (4). This ard flight check for the airline trans-
study collected a large number of port rating (28, 29). This study was
critical incidents for the purpose of done by the American Institute for
establishing the critical requirements Research under the sponsorship of
for school board membership. The the National Research Council Com-
author applied his findings to the mittee on Aviation Psychology with
problem of evaluating the effective- funds provided by the Civil Aero-
ness of school board members. nautics Administration. In this
study data from analyses of airline dents were classified under three
accidents were combined with critical main headings: (a) proficiency as a
incidents reported by airline pilots to pilot; (6) proficiency as a teacher;
provide the basis for developing an and (c) proficiency in maintaining
objective measure of pilot profi- effective personnel relations. The
ciency. The flight check consisted of proficiency measures developed in
the presentation of situations pro- connection with this study included
viding uniformly standardized op- paper-and-pencil tests presenting cri-
portunities to perform the critical tical situations and requiring the
aspects of the airline pilot's job as instructor to select one of several
indicated from the study of the acci- proposed solutions.
dents and critical incidents reported. Another development of this type
The new check was found to yield 88 carried on at the American Institute
per cent agreement on the decision to for Research was the construction of
pass or fail a particular pilot when tests for evaluating research pro-
examined on flights on successive ficiency in physics and chemistry for
days by different check pilots. The the Office of Naval Research by
previous flight check when used on M. H. Weislogel (71). This study
the same flights gave only 63 per cent was based on the critical incidents for
agreement, which was little better research personnel (20) discussed in
than chance under the conditions of a previous section. The items for
the study. these proficiency measures were
Similar studies on the development based on detailed rationales. The
of flight checks at the American In- items described a practical research
stitute for Research have been carried situation in considerable detail and
out by Marley (36, 37), G. S. Miller outlined five specific choices concern-
(39), and Ericksen (9). These studies, ing such matters as the best thing to
sponsored by the United States Air do next, suggestions for improving
Force and the Civil Aeronautics Ad- the procedure as reported, etc. The
ministration, were concerned re- critical behaviors tested in the items
spectively with objective flight were taken directly from the critical
checks for B-29 bombing crew mem- incidents. The method of developing
bers, B-36 bombing crew members, tests through the use of comprehen-
and private pilots flying light civilian sive rationales has been discussed
aircraft. Ericksen also developed a generally in another paper (16).
light plane proficiency check to pre- Three studies have been reported
dict military flying success (10) on by the American Institute for Re-
a similar project sponsored by the search in which critical incidents
United States Air Force Human Re- were used as a basis for developing
sources Research Center. situational performance tests for
A similar set of proficiency meas- measuring certain aspects of the pro-
ures was developed by Krumm for ficiency of military personnel. These
Air Force pilot instructors (34, 35), included the study of Sivy and Lange
also under the sponsorship of the on the development of an objective
Human Resources Research Center. form of the Leaders Reaction Test
These measures were based on more for the Personnel Research Branch,
than 4,000 critical incidents collected Department of the Army (57). This
from student pilots, flight instructors, test included 20 situational problems
and supervisors. The critical inci- based on the critical requirements of
the noncommissioned combat infan- Air Force Human Factors Operations
try leader as determined on the basis Research Laboratory. On the basis
of critical incidents collected in mili- of several thousand incidents re-
tary maneuvers and during combat ported by aircrew personnel regard-
operations at the front in Korea. A ing emergencies, three evaluation
second proficiency measure of a devices were prepared. These in-
somewhat similar sort was developed volved a conventional type multiple-
for other types of personnel by R. L. choice test; a special multiple-choice
Weislogel (73). The third study of test designed to measure the indi-
this type was carried out by Suttell vidual's information concerning the
(61) for the Human Resources Re- important cues in the emergency
search Center. This study was based situation, the appropriate actions to
on critical incidents collected in pre- be taken, and the basic troubles or
vious studies of the American Insti- causes of the emergency; and a
tute for Research and reported the "flight check" to be used in evaluat-
development and preliminary evalua- ing the performance of aircrew mem-
tion of the Officer Situations Test. bers in electronic flight simulators.
This test was designed to measure The obvious relevance of the be-
nonintellectual aspects of officer per- haviors involved in critical incidents
formance through the use of 16 and the specific details included make
situational problems requiring about such incidents an ideal basis for de-
six hours of testing time. veloping training programs and train-
Because of the great difficulty in ing materials.
obtaining valid and reliable measures A recent study by Collins (5) uses
of typical performance, accurate critical incidents as a basis for evalu-
measures of proficiency are essential ating the effectiveness of a training
for many types of activities. It is program. The types of incidents re-
apparent that a comprehensive set of ported by mothers after a two-week
critical incidents can be of great training course were significantly dif-
value in constructing such measures. ferent from those reported at the
Training. Many of the applica- beginning of the program in a number
tions of the critical incident technique of aspects relevant to the objectives
to training problems have been car- of the program. The critical inci-
ried out for the military in special dents appeared to provide a much
situations so that the reports are more sensitive basis for revealing
classified security information. In changes than other procedures used.
addition to work by Preston, Glaser, Selection and classification. Until
and R. L. Weislogel, R. B. Miller and recently, the customary approach of
Folley have utilized critical inci- the research psychologist to the de-
dents in establishing training require- velopment of tests for selection and
ments for specific types of main- classification purposes has been as
tenance mechanics (47) in a study for follows: A very brief period was given
the Human Resources Research Cen- to study of the job. Following this,
ter. a wide variety of selection procedures
Similarly, Ronan has used critical was administered to a group of ap-
incidents as a basis for developing a plicants or employees, and follow-up
program of training for emergency data were gathered. Since the re-
procedures in multi-engine aircraft search psychologist had little con-
(54) in a study for the United States fidence in the accuracy of his analysis
of the psychological elements re- classified into 24 job elements. These
quired by the job, there was a tend- job elements were inductively formu-
ency to try everything that was lated from the critical incidents and
available and seemed even remotely were grouped under the four area
related to the tasks involved. This headings: (a) learning and thinking;
has been called the "shotgun ap- (b) observation and visualization;
proach." It was hoped that with a (c) sensory-motor coordination; and
wide scatter at least a few of the tests (d) motives, temperament, and lead-
would pay off. The critical incident ership.
technique has lent substantial sup- The development of more than 100
port to the more thorough study of proficiency tests to measure each of
the job prior to initiating testing pro- the various critical behaviors in-
cedures. There is increasing feeling cluded in the 24 tentatively proposed
at the present time that a much job elements was reported by Hahn
larger percentage of the investiga- (30) for the School of Aviation Medi-
tor's time should be spent on deter- cine. These tests were administered
mining the critical requirements of to a group of approximately 500 high
the job, so that the psychologist will school senior boys, and the intercor-
have sufficient confidence in his relations were used to reformulate
tentative conclusions as to the nature the tentative job elements. In a
of the important selection procedures study just completed by Taylor (62)
to permit their use on a tentative for the Human Resources Research
basis prior to the collection of em- Center, the results of applying an
pirical follow-up data. This is espe- analytical procedure developed by
cially important in those situations Horst to study the interrelationships
where the follow-up requires a very involved are reported. This analysis
long period of time or where the led to the formulation of a new set of
number of cases that can be followed 20 job elements for each of which a
up is so small that definitive findings selection test has been developed.
cannot be anticipated. These tests have been administered
One of the most important re- to several hundred aviation cadets
quirements for developing a system and follow-up data on their success in
of job analysis that will facilitate a aircrew training should be available
relatively accurate identification of soon.
the important job elements for a A similar project based on critical
specific task is to establish a clear and incidents collected from various ci-
specific set of definitions for these job vilian jobs has been reported by the
elements in behavioral terms. The present author (2, 18, 19). The
American Institute for Research has Flanagan Aptitude Classification
carried out a series of projects on this Test Series, published in 1953, pro-
problem. The first of these was a vides aptitude measures for 14 criti-
study undertaken by Wagner under cal job elements. The Applicant
the sponsorship of the United States Inventory, also published in 1953,
Air Force School of Aviation Medi- measures attitudes predictive of job
cine to define the requirements of air- adjustment for hourly wage em-
crew jobs in terms of specific job ployees.
elements (67, 68). Several thousand An effort to adapt the critical in-
critical incidents were gathered from cident technique to the problem of
aircrew members, and these were developing civil service examinations
is reported by Wager and Sharon several tasks are involved. If the
(64). In an exploratory study they jobs have been studied by use of the
collected about 100 incidents regard- critical incident technique, it may be
ing on-the-job behaviors of main- possible to select and train each team
tenance technicians. These incidents member for only two or three of the
were used as a basis for determining critical job elements. This tends to
job requirements in terms of be- maximize the effectiveness of per-
havior, and test items were developed formance with respect to each of the
for use in selecting applicants who various types of tasks. Although
could be expected to meet these re- such procedures have nearly always
quirements. been informally used in planning the
Another study that used critical work of teams, the critical incident
incidents as a basis for developing technique facilitates the collection of
tests to predict performance was car- the data essential to this type of job
ried out by O'Donnell (51). His test, purification.
designed to predict success in den- Some preliminary work on this
tistry, was based on critical incidents problem has been carried out at the
collected by Wagner. The test in- American Institute for Research.
cludes items designed to predict, in Recommendations resulting from
part, the following three general these studies for reducing the number
areas: (a) demonstrating technical of job elements required in certain
proficiency; (b) handling patient re- common maintenance jobs are ex-
lationships; and (c) accepting pro- pected to lead to a saving of millions
fessional responsibility. A follow-up of dollars in training costs as well as
study indicated moderate validity for to improving the effectiveness of job
these materials. performance.
One of the few studies known to Operating procedures. Another ap-
the author in which the critical inci- plication of critical incidents which
dent technique was used in a project has not been adequately exploited is
carried on outside the United States the study of operating procedures.
is Emons' doctor's dissertation (8). Detailed factual data on successes
This study, carried out at the Uni- and failures that can be systematic-
versity of Liege, investigated the ap- ally analyzed are of great importance
titudes of effective sales personnel in in improving the effectiveness and
a large department store. A group of efficiency of operations. Such infor-
40 supervisors provided 228 critical mation can be efficiently collected by
incidents. Nine categories were for- means of the critical incident tech-
mulated from this group of incidents nique.
and recommendations made for an Examples of such studies are pro-
aptitude test to improve current vided by a series of three projects
selection procedures. carried out by the American Institute
5. Job design and purification. In- for Research under the sponsorship
adequate attention has been given to of the United States Air Force
the scientific design of jobs to pro- School of Aviation Medicine. The
mote over-all efficiency. Where a first of these involves the collection of
team has several different types of critical incidents relating to near ac-
tasks to perform, it is frequently pos- cidents in flying reported by Vasilas,
sible to design each of the team mem- Fitzpatrick, DuBois, and Youtz (63).
ber's jobs so that only a few of the More than 1,700 critical incidents
were collected from pilots and other representative of operating experi-

aircrew members by procedures de- ence provides a sound basis for modi-
veloped for this study. These inci- fying existing equipment and design-
dents pointed to possible improve- ing new models.
ments in training job design and In the study by Fitts and Jones
equipment design as well as in operat- (12), mentioned above, which was
ing procedures. carried out at the Aero-Medical
The second of these studies was Laboratory, 270 critical incidents
specifically concerned with the effect relating to errors in reading and in-
of the age of pilots and other crew terpreting aircraft instruments were
members on aircrew operations. This collected and analyzed. These led to
study was reported by Shriver (56), a number of specific suggestions re-
and included tentative suggestions garding modifications in instrument
regarding various modifications in displays.
operating procedures. Other recent studies conducted at
The third study in this series, re- the American Institute for Research
ported by Goodenough and Suttell have used data from the critical in-
(26), involved the collection of criti- cident technique along with other
cal incidents regarding the impair- sources to develop procedures for
ment of human efficiency in emer- designing jobs. The reports on these
gency operations. These incidents projects are classified for military
provide a detailed statement of both security reasons.
the types of stresses that impair per- Other projects at the American
formance and the types of perform- Institute for Research have used the
ance that are impaired under specific critical incident technique as a sup-
conditions. More than 2,000 critical plemental procedure for task analysis
incidents were collected in which of equipment in the design stage of
impairment in performance on opera- development (9, 10, 34, 35, 39).
tional assignments was observed. These procedures have been found
These incidents were collected in very effective when used by psy-
Alaska and the Far East as well as in chologists working closely with en-
operational commands in the United gineers on the preparation of design
States. This report contains sugges- specifications for new equipment.
tions for improving operations in Motivation and leadership. The
emergency situations. study of attitudes has been somewhat
Equipment design. An application limited and difficult to interpret be-
closely related to that just discussed cause of the almost exclusive reliance
involves the collection of critical in- on verbal statements of opinions and
cidents to improve the design of preferences. The critical incident
equipment. Reports of specific in- technique has been applied in a few
cidents from the field have always instances to gather factual data re-
been a basis for equipment modifica- garding specific actions involving de-
tions. The critical incident technique cisions and choices. These studies
facilitates the collection and process- suggest that critical incidents of this
ing of this type of information. Too type may be a very valuable supple-
often in the past action was taken on mentary tool for the study of atti-
the basis of informal reports from tudes.
operating personnel. The collection A recent study carried out by
of large numbers of critical incidents Preston of the American Institute for
Research for the Air Force's Human veloping objective measures of im-
Resources Research Center (53) used provement but also in experimental
critical incidents as a basis for study- studies of the types of improvement
ing decisions of airmen to re-enlist in resulting from the therapists' use of
the Air Force. It is believed that specific procedures.
these specific incidents provide valu- A somewhat related type of study
able information not contained in initiated by Diederich and reported
studies utilizing only data on opin- by Allen (1) describes the use of the
ions. technique to obtain critical incidents
A series of reports by Ruch (55) from students reporting things that
contains critical incidents on combat caused them to like a fellow high
leadership collected from senior offi- school student either more or less
cers in the Far East Air Forces. than before. This study is being con-
These incidents provide a factual tinued to provide the basis for tests
basis for the study of motivation and of specific value areas. An incidental
leadership of Air Force personnel finding of the study was that when
engaged in combat operations. an example of the kind of incident
Counseling and psychotherapy. An- desired was shown on the form, S3
other field in which current tech- per cent of the positive and 23 per
niques emphasize over-all impres- cent of the negative behaviors re-
sions, opinions, and reports of single ported were in the same category as
cases is counseling and psychother- the example given.
apy. There appears to be a trend,
however, in this field toward em- SUMMARY AND CONCLUSIONS
phasizing the collection of factual This review has described the de-
incidents. This suggests that the velopment of a method of studying
critical incident technique may be activity requirements called the criti-
useful in this area also. cal incident technique. The tech-
Exploratory work has recently nique grew out of studies carried out
been done at the University of Pitts- in the Aviation Psychology Program
burgh with the critical incident tech- of the Army Air Forces in World
nique to establish areas of change War II. The success of the method
accompanying psychotherapy. A in analyzing such activities as com-
series of three master's theses were bat leadership and disorientation in
carried out by Speth, Goldfarb, and pilots resulted in its extension and
Mellett (25, 38, 60). They collected further development after the war.
243 critical incidents from 11 psycho- This developmental work has been
therapists. These incidents were col- carried out primarily at the American
lected about patients who had shown Institute for Research and the Uni-
improvement and were replies to the versity of Pittsburgh. The reports of
question, "What did the patient do this work are reviewed briefly.
that was indicative of improve- The five steps included in the crit-
ment?" Although these studies were ical incident procedure as most
primarily exploratory in nature, the commonly used at the present time
tentative finding that different thera- are discussed. These are as follows:
pists stress different criteria of im- (a) Determination of the general aim
provement and nonimprovement sug- of the activity. This general aim
gests that the critical incident ap- should be a brief statement obtained
proach may be of use not only in de- from the authorities in the field
which expresses in simple terms those It should be emphasized that criti-
objectives to which most people cal incidents represent only raw data
would agree, (b) Development of and do not automatically provide
plans and specifications for collecting solutions to problems. However, a
factual incidents regarding the ac- procedure which assists in collecting
tivity. The instructions to the per- representative samples of data that
sons who are to report their observa- are directly relevant to important
tions need to be as specific as possible problems such as establishing stand-
with respect to the standards to be ards, determining requirements, or
used in evaluating and classifying the evaluating results should have wide
behavior observed, (c) Collection of applicability.
the data. The incident may be re- The applications of the critical
ported in an interview or written up incident technique which have been
by the observer himself. In either made to date are discussed under the
case it is essential that the reporting following nine headings: (a) Meas-
be objective and include all relevant ures of typical performance (criteria);
details, (d) Analysis of the data. (b) measures of proficiency (standard
The purpose of this analysis is to samples); (c) training; (d) selection
summarize and describe the data in and classification; (e) job design and
an efficient manner so that it can be purification; (/) operating proced-
effectively used for various practical ures; (g) equipment design; (K) moti-
purposes. It is not usually possible to vation and leadership (attitudes);
obtain as much objectivity in this (i) counseling and psychotherapy.
step as in the preceding one. (e) In- In summary, the critical incident
terpretation and reporting of the technique, rather than collecting
statement of the requirements of the opinions, hunches, and estimates,
activity. The possible biases and im- obtains a record of specific behaviors
plications of decisions and procedures from those in the best position to
made in each of the four previous make the necessary observations and
steps should be clearly reported. The evaluations. The collection and tabu-
research worker is responsible for lation of these observations make it
pointing out not only the limitations possible to formulate the critical
but also the degree of credibility and requirements of an activity. A list of
the value of the final results obtained. critical behaviors provides a sound
It should be noted that the critical basis for making inferences as to re-
incident technique is very flexible and quirements in terms of aptitudes,
the principles underlying it have training, and other characteristics.
many types of applications. Its two It is believed that progress has been
basic principles may be summarized made in the development of pro-
as follows: (a) reporting of facts re- cedures for determining activity re-
garding behavior is preferable to the quirements with objectivity and pre-
collection of interpretations, ratings, cision in terms of well-defined and
and opinions based on general im- general psychological categories.
pressions; (b) reporting should be Much remains to be done. It is
limited to those behaviors which, hoped that the critical incident tech-
according to competent observers, nique and related developments will
make a significant contribution to provide a stable foundation for pro-
the activity. cedures in many areas of psychology.
356 JOHN C, FLANAGAN
REFERENCES
1. ALLEN, C. D. Critical requirements in in- ington: U. S. Government Printing Of-
terpersonal behavior. Unpublished sen- fice, 1947. (AAF Aviat. Psychol. Pro-
ior thesis, Princeton Univer., 1950. gram Res. Rep. No. 1.)
2. AMERICAN INSTITUTE FOR RESEARCH. 14. FLANAGAN, J. C. A new approach to eval-
The applicant inventory. Pittsburgh: uating personnel. Personnel, 1949, 26,
American Institute for Research, 1954. 35-42.
3. AMERICAN PSYCHOLOGICAL ASSOCIATION. 15. FLANAGAN, J. C. Critical requirements: a
COMMITTEE ON ETHICAL STANDARDS new approach to employee evaluation.
FOR PSYCHOLOGY. Ethical standards of Personnel Psychol., 1949, 2, 419-425.
psychologists. Washington, D. C.: 16. FLANAGAN, J. C. Use of comprehensive
Amer. Psychol. Ass., 1953. rationales. Educ. psychol. Measmt,
4. BARNHART, R. E. The critical require- 1951, 11, 151-155.
ments for school board membership 17. FLANAGAN, J. C. Principles and proce-
based upon an analysis of critical inci- dures in evaluating performance. Per-
dents. Unpublished doctor's disserta- sonnel, 1952, 28, 373-386.
tion, Indiana Univer., 1952. 18. FLANAGAN, J. C. Flanagan aptitude classi-
5. COLLINS, MARJORIE G. Selected methods fication tests. Chicago: Science Research
applied to the evaluation of a parent Associates, 1953.
training course in child management. 19. FLANAGAN, J. C. Improving personnel se-
Unpublished doctor's dissertation, lection. Publ. Personnel Rev., 1953, 14,
Univer. of Pittsburgh, 1954. 107-112.
6. DOMAS, S. J. Report of an exploratory 20. FLANAGAN, J. C., et al. Critical require-
study of teacher competence. Cambridge: ments for research personnel. Pitts-
New England School Development burgh : American Institute for Research,
Council, 1950. 1949.
7. EILBERT, L. R. A study of emotional im- 21. FLANAGAN, J. C., MILLER, R. B., BURNS,
maturity utilizing the critical incident R. K., HENDRIX, A. A., STEWART, B.,
technique. Univer. Pittsburgh Bull., PRESTON, H. O., & WEST, E. D. The
1953, 49, 199-204. performance record for hourly employes.
8. EMONS, V. L'analyse de la fonction de Chicago: Science Research Associates,
vendeuse de grand magasin par la 1953.
me'thode des exigencies critiques. Un- 22. FLANAGAN, J. C., MILLER, R. B., BURNS,
published doctor's dissertation, Univer. R. K., HENDRIX, A. A., STEWART, B.,
of Liege, Belgium, 1952. PRESTON, H. 0., & WEST, E. D. The
9. ERICKSEN, S. C. Development of an ob- performance record for non-supervisory
jective proficiency check for private pi- salaried employes. Chicago: Science Re-
lot certification. Washington: Civil search Associates, 1953.
Aeronautics Administration, 1951. 23. FLANAGAN, J. C., MILLER, R. B., BURNS,
(Div. of Res., Rep. No. 95.) R. K., HENDRIX, A. A., STEWART, B.,
10. ERICKSEN, S. C. Development of a light PRESTON, H. O., & WEST, E. D. The
plane proficiency check to predict mili- performance record for foremen and su-
tary flying success. USAF, Hum. Re- pervisors. Chicago: Science Research
sour. Res. Cent. Tech. Rep., 1952, No. Associates, 1953.
52-6. 24. FOLLEY, J. D., JR. Development of a list
11. FINKLE, R. B. A study of the critical re- of critical requirements for retail sales
quirements of foremanship. Univer. personnel from the standpoint of cus-
Pittsburgh Bull., 1950,46,291-297. (Ab- tomer satisfaction. Unpublished mas-
stract) ter's thesis, Univer. of Pittsburgh, 1953.
12. FITTS, P. M., & JONES, R. E. Psychologi- 25. GOLDFARB, A. Use of the critical incident
cal aspects of instrument display. 1. technique to establish areas of change
Analysis of 270 "pilot error" experi- accompanying psychotherapy: II. Re-
ences in reading and interpreting air- lationship to diagnostic group. Unpub-
craft instruments. Dayton, O.: U. S. lished master's thesis, Univer. of Pitts-
Air Force, Air Materiel Command, burgh, 1952.
Wright-Patterson Air Force Base, 1947. 26. GOODENOUGH, D. R., & SUTTELL, BAR-
(Mem. Rep. TSEAA-694-12A.) BARA J. The nature and extent of impair-
13. FLANAGAN, J. C. The aviation psychology ment of human efficiency in emergency op-
program in the Army Air Forces. Wash- erations: the field study. Pittsburgh:
American Institute for Research, 1952, 40. MILLER, N. E. Psychological research on
27. GORDON, T. The airline pilot: a survey of pilot training. Washington: U. S. Gov-
the critical requirements of his job and ernment Printing Office, 1947. (AAF
of pilot evaluation and selection pro- Aviat. Psychol. Program. Res. Rep. No.
cedures. Washington: Civil Aeronau- 8.)
tics Administration, 1947. (Div. of Res., 41. MILLER, R. B. Anticipating tomorrow's
Rep. No. 73.) maintenance job. USA F, Hum. Resour.
28. GORDON, T. The development of a stand- Res. Cent., Res. Rev., 1953, No. 53-1.
ard flight-check for the airline transport 42. MILLER, R. B. A method for determining
rating based on the critical require- human engineering design requirements
ments of the airline pilot's job. Wash- for training equipment. Dayton, O.:
ington: Civil Aeronautics Administra- U. S. Air Force, Air Development Cen-
tion, 1949. (Div. of Res., Rep. No. 85.) ter, Wright-Patterson Air Force Base,
29. GORDON, T. The development of a meth- 1953. (Tech. Rep. 53-135.)
od of evaluating flying skill. Personnel 43. MILLER, R. B. Handbook on training and
PsychoL, 1950, 3, 71-84. training equipment design. Dayton, O.:
30. HAHN, C. P. Measurement of individual U. S. Air Force, Air Development Cen-
differences with respect to critical job ter, Wright-Patterson Air Force Base,
requirements. USA F Sch. Aviat. Med., 1953. (Tech. Rep. 53-136.)
1954, Proj. No. 21-29-014, No. 2. 44. MILLER, R. B. A method for man-ma-
31. HOBBS, N. The development of a code of chine task analysis. Dayton, 0.: U. S.
ethical standards for psychology. A mer. Air Force, Air Development Center,
Psychologist, 1948, 3, 80-84. Wright-Patterson Air Force Base, 1953.
32. JENSEN, A. C. Determining critical re- (Tech. Rep. 53-137.)
quirements for teachers. J. exp. Educ., 45. MILLER, R. B. Human engineering de-
1951, 20, 79-86. sign schedule for training equipment.
33. KONIGSBURG, D. The development and Dayton, O.: U. S. Air Force, Air Devel-
preliminary evaluation of an instructor opment Center, Wright-Patterson Air
check list based on the critical incident Force Base, 1953. (Tech. Rep. 53-138.)
technique. Unpublished doctor's dis- 46. MILLER, R. B., & FLANAGAN, J. C. The
sertation, Univer. of Pittsburgh, 1954. performance record: an objective merit-
34. KRUMM, R. L. Critical requirements of rating procedure for industry. Amer.
pilot instructors. USA F, Hum. Resour. Psychologist, 1950, S, 331-332. (Ab-
Res. Cent., Tech. Rep., 1952, No. 52-1. stract)
35. KRUMM, R. L. Development of a measure of 47. MILLER, R. B., & FOLLEY, J. D., JR. The
pilot instructor proficiency based on the validity of maintenance job analysis from
critical requirements of the instructor's the prototype of an electronic equipment.
job. Pittsburgh: American Institute for Pittsburgh: American Institute for Re-
Research, 1953. search, 1952.
36. MARLEY, F. W. Individual differences in 48. NAGAY, J. A. The development of a pro-
critical aircrew elements. I. The deter- cedure for evaluating the proficiency of
mination of critical proficiency require- air route traffic controllers. Washing-
ments for B-29 combat crews. USAF ton: Civil Aeronautics Administration,
Sch. Aviat. Med., 1952, Proj. No. 21-29- 1949. (Div. of Res., Rep. No. 83.)
014, No. 1. 49. NAGAY, J. A. The airline tryout of the
37. MARLEY, F. W. The development of per- standard flight-check for the airline
formance flight checks for B-29 combat transport rating. Washington: Civil
crews. USAF, Hum. Resour. Res. Lab., Aeronautics Administration, 1949. (Div.
1952, Rep. No. 19. of Res., Rep. No. 88.)
38. MELLETT, T. P. Use of the critical inci- 50. NEVINS, CHARLOTTE I. An analysis of
dent technique to establish areas of reasons for the success or failure of
change accompanying psychotherapy: bookkeepers in sales companies. Un-
III. Differences among therapists. Un- published master's thesis, Univer. of
published master's thesis, Univer. of Pittsburgh, 1949.
Pittsburgh, 1952. 51. O'DoNNELL, R. J. The development and
39. MILLER, G. S. Development of B-36 evaluation of a test for predicting den-
bombardment objective flight checks and tal student performance. Univer. Pitts-
proficiency measures for use with recon- burgh Bull., 1953, 49, 240-245. (Ab-
naissance crews. Pittsburgh: American stract)
Institute for Research, 1953. 52. PRESTON, H. O. The development of a pro-
cedure for evaluating officers in the 64. WAGER, C. E., & SHARON, M. I. Defining
United States Air Force. Pittsburgh: job requirements in terms of behavior.
American Institute for Research, 1948. Personnel Admin., 1951, 14, 18-25.
53. PRESTON, H. 0. Events affecting reenlist- 65. WAGNER, R. F. A group situation com-
ment decisions. Pittsburgh: American pared with individual interviews for se-
Institute for Research, 1953. curing personnel information. Person-
54. RONAN, W. W. Training for emergency nel Psychol., 1948,1,93-107.
procedures in multi-engine aircraft. 66. WAGNER, R. F. A study of the critical re-
Pittsburgh: American Institute for Re- quirements for dentists. Univer. Pitts-
search, 1953. burgh Bull., 1950, 46, 331-339. (Ab-
55. RUCH, F. L. Incidents of leadership in stract)
combat. USAF, Hum. Resour. Res. 67. WAGNER, R. F. Development of stand-
Inst., Res. Memo, 1953, No. 3. ardized procedures for defining the re-
56. SHRIVER, BEATRICE M. Age and behav- quirements of aircrew jobs in terms of
ior: a study of the effects of aging on air- testable traits. USAF Sch. Aviat. Med.,
crew performance. USAF Sch. Aviat. 1951, Proj. No. 21-29-010, No. 1.
Med., 1953, Proj. No. 21-0202-0005, 68. WAGNER, R. F. Using critical incidents to
No. 3. determine selection test weights. Per-
57. SIVY, J., & LANGE, C. J. Development of sonnel Psychol., 1951, 4, 373-381.
an objective form of the Leaders Reac- 69. WEISLOGEL, MARY H. Development of a
tion Test. A GO, Personnel Res. Branch, test for selecting research personnel. Pitts-
PRS Rep. 1952, No. 930. burgh: American Institute for Research,
58. SMIT, Jo ANNE, A study of the critical re- 1950.
quirements for instructors of general 70. WEISLOGEL, MARY H. Procedures for
psychology courses. Univer. Pittsburgh evaluating research personnel with a per-
Bull., 1952, 48, 279-284. (Abstract) formance record of critical incidents.
59. SMITH, R. G., JR., & STAUDOHAR, F. T. Pittsburgh: American Institute for Re-
Technical requirements of basic train- search, 1950.
ing tactical instructors. USAF, Hum. 71. WEISLOGEL, MARY H. The development of
Resour. Res. Cent., Tech. Rep., 1954, No. tests for evaluating research proficiency in
54-7. physics and chemistry. Pittsburgh:
60. SPETH, E. W. The use of the critical inci- American Institute for Research, 1951.
dent technique to establish areas of 72. WEISLOGEL, R. L. Critical requirements
change accompanying psychotherapy: for life insurance agency heads. Univer.
I. Function of age and education. Un- Pittsburgh Bull., 1952, 48, 300-305.
published master's thesis, Univer. of (Abstract)
Pittsburgh, 1952. 73. WEISLOGEL, R. L., & SCHWARZ, P. A.
61. SUTTELL, BARBARA J. Development of a Some practical and theoretical prob-
situation test to measure non-intellectual lems in situational testing. Unpublished
officer qualities. Pittsburgh: American manuscript, Pittsburgh, American In-
Institute for Research, 1953. stitute for Research, 1953.
62. TAYLOR, M. V., JR. The development of 74. WICKERT, F. Psychological research on
aircrew job element aptitude tests, Pitts- problems of redistribution. Washing-
burgh: American Institute for Re- ton: U. S. Government Printing Of-
search, 1953. fice, 1947. (AAF Aviat. Psychol. Pro-
63. VASILAS, J. N., FITZPATRICK, R., DuBois, gram Res. Rep. No. 14.)
P. H., & YOUTZ, R. P. Human factors
in near accidents. USAF Sch. Aviat.
Med., 1953, Proj. No. 21-1207-0001,
No. 1. Received November 10, 1953.

Psychological Bulletin: VOL. 51, No. 4

Uploaded by

Copyright:

Available Formats

Psychological Bulletin: VOL. 51, No. 4

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Psychological Bulletin: VOL. 51, No. 4

Uploaded by

Copyright:

Available Formats

VOL. 51, No.

ords. The critical incident technique Although in the preparation of these

Wagner (66) completed a dissertation tained from questions intended to

portant how we define the purpose of ment should be referred either to

tentative categories are modified as indicated 5. Interpreting and Reporting

mined the critical requirements for Another type of application of the

were collected from pilots and other representative of operating experi-

You might also like