Qualitative Research and Evaluation Methods PDF

rvalua+ion
Metkods
Michael Quinn Patton

]
To the faculty and learners of The Union Institute, Cincinnati, Ohio,
for their friendship and scholarship, and their commitments to
methodological eclecticism, interdisciplinary inquixy, integration of
theory and practice, valuing both reflection and action, scholarship
that is socially relevant and meaningful, individualized professional
and personal development, lifelong learning/ social justice and equity,
human diversty and global community; a scholarly community
governed by principies and processes rather than rules and
regulations, and iimovations in learning-centered, nontraditional
doctoral education, including faculty meetings that are interesting
and important, an indication of innovation of the highest order
KF/^
F A C U L D A D E DE n-wc
DESPORTO t D E E D C a A
U N I V E R S I D A D E DO P O R T C
BBLOTHC.
EDITION

F A C U L D A D E PE CINCIAS DO
HPORTO DESPORTO E D- EDUCAO FSiCA
FACULDADE DE DESPORTO
O
UNIVERSIDADE DO PORTO
UNV
fERSD
f ARE 00 PORTO
I
015774
| /Q^
/vSage Publications
4 SC I X I International Educational and ProfessionaI Publisher
Thousand Oaks London New Delhi
Ty/fTM FCDEF
Copyright 2002 by Sage Publications, Inc.
Ali rights reserved. No part of this book may be reproduced or utilize d in any form or by
any means, electronic or mechanical, including photocopying, recording, orby any infor-
ma tion storage and retrieval system, without permission in writing from the publisher.
For information:
Sage Publications, Inc.

2455 Teller Road
Thousand Oaks, Califrnia 91320
E-mail: [email protected]
Sage Publications Ltd.
6 Bonhill Street
London EC2A 4PU
United Kingdom
Sage Publications ndia Pvt. Ltd.
M-32 Market
Greater Kailash I
New Delhi 110 048 ndia
Printed in the United States of America
Library of Congress Cataloging-in-Publication Data
Patton, Michael Quinn.

Qualitative research and evaluation methods / by Michael Quinn
Patton. 3rd ed.
p. cm.
Rev. ed. of: Qualitative evaluation and research methods. 2nd ed. 1990. )
Includes bibliographical references and index.
ISBN 0-7619-1971-6
1. Social sciencesMethodology. 2. Evaluation research (Social
action programs). I. Patton, Michael Quinn. Qualitative evaluation and
research methods. II. Title.
H62.P3218 2001
300'.7'23dc21 2001005181
04 05 06 10 9 8 7 6 5 4
Acquiring Editor: C. Deborah Laughton

Editorial Assistant: Vernica Novak
Production Editor: Diana E. Axelsen
Editorial Assistant: Kathryn Joumey
Copy Editor: Kate Peterson
Typesetter/Desigtier: Janelle LeMaster
Cover Designer: Michelle Lee
BHef (Confervfs
Preface xxi
PjA"RTT 1 . Conceptual Issues in Qualitative Inquiry 1
1. The Nature of Qualitative Inquiry 3

2. Strategic Themes in Qualitative Inquiry 37
3. Varie ty in Qualitative Inquiry: Theoretical Orientations 75
4. Particularly Appropriate Qualitative Applications 143
P A R T 2 . Qualitative Designs and Data Collection 207

5. Designing Qualitative Studies 209
6. Fieldwork Strategies and Observation Methods 259
7. Qualitative Interviewing (^339^
P j A R T 3 . Analysis, Interpretation, and Reporting 429
8. Qualitative Analysis and Interpretation (431^)
9. Enhancing the Quality and Credibility of Qualitative Analysis
References RI
Author Index 11
Subject Index 113
About the Author Al

Defailed (Zov\ie.rs
Preface xxi
P . A R T 1 , Conceptual Issues in Qualitative Inquiry 1
1. The Nature of Qualitative Inquiry 3

The Fruit of Qualitative Methods 3
|ggSj||i Three Kinds of Qualitative Data 4
Recognizing Qualitative Data 4
Qualitative Findirigs: Themes, Patterns/ Concepts, Insights, Understandings 5
E H 1 H Women's Ways of Knowing: An Example of

Qualitative Frndings 7
mHnmBn
Coming-of-Age Paradigms 9
Different Purposes of and Audiences for Qualitative Studies:
Research, Evaluation, Dissertations, and Personal Inquiry 9
Making Methods Decisions 12
Methods Choices: Contrasting Qualitative and Quantitative Emphases 12
nsmnnHn
m U l i ^ l U B i Some Guiding Questions and Options for Methods Decisions 13
Comparing Two Kinds of Data: An Example 14
The Power of Qualitative Data 17
Face Validity and Credibility 20
The Purpose of Open-Ended Responses 20
Inquiry by Observation 21
The Raw Data of Qualitative Inquiry 26
People-Oriented Inquiry 27
The Fruit of Qualitative Methods Revisited 28
^ ^ [ f f l g Internet E-mail Discussion Groups (listservs)
on Qualitative Methods 29
Between-Chapters Interlude:
Top Ten Pieces of Advice to a Graduate Student
Considering a Qualitative Dissertation 31
2. Strategic Themes in Qualitative Inquiry 37

General Principies 37
The Purpose of a Strategic Framework 38
Design Strategies for Qualitative Inquiry 39
Naturalistic Inquiry 39
|X]J|||]||9I Themes of Qualitative Inquiry 40
Emergent Design Flexibility 43
Purposeful Sampling 45
Data Collection and Fieldwork: Strategies for Qualitative Inquiry 47
Qualitative Data 47
Direct Personal Experience and Engagement: Going Into the Field 47
Empathic Neutrality 49
Empathy and Insight 51
A Dynamic, Developmental Perspective 54
Analysis Strategies for Qualitative Inquiry 55
Unique Case Orientation 55
Inductive Analysis and Creative Synthesis 55
Holistic Perspective 58
Context Sensitivity 61
Voice and Perspective: Reflexivity 63
i i Reflexive Questions: Triangulated Inquiry 66
From Strategic Ideais to Practical Choices 66
Beyond Competing Inquiry Paradigms 68
Pragmatism 71
Ideal Conditions for Research: A Cautionary Tale 72
3. Variety in Qualitative Inquiry: Theoretical Orientations 75
Special Gifts 75
From Core Strategies to Rich Diversity 76
Which Approach Is Right? 77
Alternative Ways of Distinguishing Qualitative Traditions 79
Foundational Questions 80
Theoretical Traditions and Orientations 81
Ethnography 81
B J M I Culture, Culture Everywhere: Sample of Media Headlines 83
Autoethnography and Evocative Forms of Inquiry 84
l52jm[|lBS9 Varieties of Autoethnography: A Partial Lexicology 85
Truth and Reality-Oriented Correspondence Theory:
Positivist, Realist, and Analytic Induction Approaches 91
Social Construction and Constructivism 91
Constructivism Versus Constructionism 97
Phenomenology 104
Heuristic Inquiry 107
Qualitative Heuristics: A German Alternative Tradition 109
Ethnomethodology 110
Symbolic Interaction 112
Hermeneutics 113
Narra tology or Narrative Analysis 115
Ecological Psychology 118
A Systems Perspective and Systems Theory 119
Chos and Complexity Theory: Nonlinear Dynamics 123
Grounded Theory 124
Complexity (Chos) Theory Precepts and Qualitative
Inquixy Implications 126
Orientational Qualitative Inquiry: Feminist Inquiry Criticai Theory
and Queer Theory as Examples 129
Variety in Qualitative Inquiry: Different Answers to Core Questions 131
XO^KKI Variety in Qualitative Inquiry: Theoretical Traditions 132
Pragmatism 135
B B Sample Internet E-mail Discussion Groups (listservs) and

Sites Relevant to Qualitative Inquiry and Theory 136
The Apple of Your Eye 137
APPENDIX 3.1. Example of Autoethnographic Writing 138
4. Particularly Appropriate Qualitative Applications 143
Apprenticeship in Pragmatism 143
Practical Purposes and Concrete Questions 145
A Focus on Quality 145
Quality Assurance and Program Evaluation 147
^^^QQOm Comparing Program Evaluation and Quality Assurance 149
Evaluation Applications 151
Outcomes Evaluation 151
Evaluating Individualized Outcomes 152
I I B e h i n d the Numbers of an Employment Program:
The Story of Li 155
Process Studies 159
Implementation Evaluation 161
Logic Models and Theories of Action 162
Evaluability Assessments 163
Comparing Programs: Focus on Diversity 163
155181131111 Types of Teacher Centers 166
Prevention Evaluation 166
Documenting Development Over Time and Investigating
System Changes 167
From Evaluation Issues to Evaluation Models 168
Evaluation Models 169
Goal-Free Evaluation 169
Transaction Models: Responsive and Illuminative Evaluation 171
Connoisseurship Studies 172
Utilization-Focused Evaluation 173
Interactive and Participatory Applications 175
Personalizing and Humanizing Evaluation 175
Harmonizing Program and Evaluation Values 176
Developmental Applications: Action Research, Action Learning,
Reflective Practice, and Learning Organizations 177
j^QQJ^yjQQI Common Principies Undergirding Qualitative Inquiry
and Humanistic Values 177
Matching Program Philosophy and Evaluation Approach:
An Illustration 178
Appreciative Inquiry 181
Participatory Research and Evaluation: Valuing and Facilitating
Collaboration 182
Principies of Fully Participatory and Genuinely
Collaborative Inquiry 185
Supporting Democratic Dialogue and Deliberation 185
Supporting Democracy Through Process Use: Helping the
Citizenry Weigh Evidence and Think Evaluatively 187
Special Applications 191
The Need for Unobtrusive Measures 191
State-of-the-Art Considerations: Lack of Proven Quantitative
Instrumentation 192
Confirmatory and Elucidating Research: Adding Depth, Detail,
and Meaning to Quantitative Analyses 193
Rapid Reconnaissance 194
Capturing and Communicating Stories 195
Example of a "Most Significant Change" Story 197
Legislative Auditing and Monitoring 198
Futuring Applications: Anticipatory Research and Prospective
Policy Analysis 200
Breaking the Routine: Generating New Insights 202
Summary: A Vision of the Utility of Qualitative Methods 202
Qualitative Inquiry Applications: Summary Checklist
of Particularly Appropriate Uses of Qualitative Methods 204
EXHIBIT 4.9 Sample Internet E-mail Discussion Groups (listservs)
and Sites Relevant to Qualitative Applications and Practice 205
P A R T 2 . Qualitative Designs and Data Collection 207
5. Designing Qualitative Studies 209

The First Evaluation 209
A Meta-Evaluation 211
Clarity About Purpose: A Typology 213
Basic Research 215
EXHIBIT 5.1 Fundamental Disciplinary Questions 216
Applied Research 217
Sample Interdisciplinary Applied Research Questions 218
Evaluation Research: Summative and Formative 218
Action-Oriented, Problem-Solving Research 221
The Purpose of Purpose Distinctions 222
Examples of Types of Research Questions: A Family Research Example 223
Criticai Trade-Offs in Design 223
A Typology of Research Purposes 224
^QQQQÎjQQI Family Research Example: Research Questions

Matched to Research Category 225
Breadth Versus Depth 227
Units of Analysis 228
Purposeful Sampling 230
f ^ j ^ j j j Q Q Examples of Units of Analysis for Case Studies and
Comparisons 231
Information-Rich Cases 242
Sample Size 242
E&UBBB^^BBBSBBB^BBB
B a Sampling Strategies 243
Emergent Designs and Protection of Human Subjects 246
Methodological Mixes 247
Triangulation 247
Mixing Data, Design, and Analysis Approaches 248
The Case of Operation Reach-Out: Variations in Program
Evaluation Design 249
Altemative Pure and Mixed Strategies 251
E 3 J Measurement, Design, and Analysis:
Pure and Mixed Combinations 252
Design and Methods Decisions 253
Design Issues and Options 254
mSBSBiuSfinSfiSEmSSHS ^
Choices 257
6. Fieldwork Strategies and Observation Methods 259

To Understand the World 260
Folk Wisdom About Human Observation 261
The Value of Direct Observations 264
Observation-Based Evaluation and Applied Research in a
Political World 265
Variations in Observational Methods 265
Variations in Observer Involvement: Participant or Onlooker or Both? 265
Insider and Outsider Perspectives: Emic Versus Etic Approaches 267
Who Conducts the Inquiry? Solo and Team Versus Participatory
and Collaborative Approaches 269
Overt Versus Covert Observations 273
Variations in Duration of Observations 273
Variations in Observational Focus 275
Dimensions Along Which Fieldwork Varies: An Overview 276
What to Observe: A Sensitizing Framework 276
KQ^njnnQn Dimensions Showing Fieldwork Variations 277
Sources of Data 279
E P ^ ^ m Examples of Sensitizing Concepts 280
The Setting 280
B H J i n Example of Combining Description and Metaphor
to Provide a Sense of Place 282
The Human, Social Environment 283
Historical Perspectives 284
Planned Program Implementation Activities and Formal Interactions 285
Informal Interactions and Unplanned Activities 285
The Native Language of the Program 288
Nonverbal Communication 290
Unobtrusive Observations 291
Documents 293
Observing What Does Not Happen 295
Nested and Layered Case Studies Dnring Fieldwork 297
Observing Oneself 299
Nested, Layered, and Overlapping Mini-Case Studies
During Fieldwork: Example From the Wilderness
Education Program Evaluation 300
Sources of Data Reviewed 301
Creativity in Fieldwork 302
Doing Fieldwork: The Data-Gathering Process 302
Field Notes 302
EXHIBIT 6.5 Fieldnotes Comparisons 304
Procedurally Speaking 305
Observations, Interviews, and Documentation: Bringing Together
Multiple Perspectives 306
The Technology of Fieldwork and Observation 307
Stages of Fieldwork 310
Entry Into the Field 310
What You Say and What You Do 314
Routinization of Fieldwork: The Dynamics of the Second Stage 317
Key Informants 321
Bringing Fieldwork to a Close 322
Evaluation Feedback 324
The Observer and What Is Observed: Unity and Separation 326
The Personal Experience of Fieldwork 329
A Part of and Apart From the World Observed 329
Summary Guidelines for Fieldwork 330
^ ^ Q H Q Summary Guidelines for Fieldwork 330
Outside to Inside, Inside to Outside: Shifting Perspectives 333
Preface 335
"Nothing About Us, Without Us" 335
Barbara Lee
Qualitative Interviewing 339
Beyond Silent Observation 339
Rigorous and Skillful Interviewing 340
Inner Perspectives 340
Variations in Qualitative Interviewing 341
The Informal Conversational Interview 342
The Interview Guide 343
The Standardized Open-Ended Interview 344
QJQ^QQH Evaluation Interview Guide for Participants in an
Employment Training Program 345
Combining Approaches 347
Summary of Interviewing Strategies 348
Question Options 348
Experience and Behavior Questions 348
M i Variations in Interview Instrumentation 349
Opinion and Values Questions 350
Feeling Questions 350
Knowledge Questions 350
Sensory Questions 350
Background/Demographic Questions 351
Distinguishing Question Types 351
The Time Frame of Questions 351
HxffiHHHKB A Matrix of Question Options 352
Sequencing Questions 352
Wording Questions 353
Asking Truly Open-Ended Questions 353
The Horns of a Dichotomy 354
Asking Singular Questions 358
Clarity of Questions 361
Why to Take Care Asking "Why?" 363
Rapport and Neutrality 365
Neutral Questions 365
Using Illustrative Examples in Questions 366
Role-Playing and Simulation Questions 367
Presupposition Questions 369
Alternative Question Forma ts 370
Prefatory Statements and Announcements 370
Probes and Follow-Up Questions 371
Process Feedback During the Interview 374
Support and Recognition Responses 375
Maintaining Control and Enhancing the Quality of Responses 375
The One-Shot Question 378
The Final or Closing Question 379
Beyond Technique 379
Mechanics of Gathering Interview Data 380
Recording the Data 380
BSSBFFHHHI Tins fnr Taoe-Rerordine: Interviews: How to Keeo
Transcribers Sane 382
Taking Notes During Interviews 383
After the Interview 383
Special Applications and Issues 385
Think-Aloud Protocol Interviewing 385
Focus Group Interviews 385
Group Interviews 390
Cross-Cultural Interviewing 391
Language Differences 392
Differing Norms and Values 393
Beyond Standard Interviewing: Creative Qualitative Modes of Inquiry 394
Participant Interview Chain 396
Data Collection by Program Staff 397
j M S I U I I I ^ S Training Nonresearchers as Focus Group Interviewers:
Women Leaving Prostitution 399
Interactive Group Interviewing and Dialogues 400
Creativity and Data Quality: Qualitative Bricolage 400
Specialized and Targeted Interview Approaches 402
Ethical Challenges in Qualitative Interviewing 405
Informed Consent and Confidentiality 407
i s a y n n i l i s i Ethical Issues Checklist 408
New Directions in Informed Consent: Confidentiality
Versus People Owning Their Own Stories 411
Reciprocity: Should Interviewees Be Compensated? If So, How? 412
How Hard Should Interviewers Push for Sensitive Information? 415
Be Careful. It's Dangerous Out There. 415
Personal Reflections on Interviewing 416
An Interview With the King of the Monkeys 417
Halcolm on Interviewing 418
APPENDIX 7.1. Sample of a Detailed Interview Guide 419
APPENDIX 7.2. Examples of Standardized Open-Ended Interviews 422
P;A~RT 3 . Analysis, Interpretation, and Reporting 429
8. Qualitative Analysis and Interpretation 431

The Complete Analysis Isn't 431
The Challenge 432
Purpose as Context 434
When Does Analysis Begin? 436
Thick Description 437
Options for Organizing and Reporting Qualitative Data 439
Organizing the Data 440
Protecting Data 441
Computer-Assisted Qualitative Data Management and Analysis 442
KMHHffifli Examples of Software Programs for Qualitative Analysis 444
I M l i l i Internet Resources and E-mail Discussion Groups (listservs)

on Qualitative Analysis 445
Case Studies 447
UJ23QE&I Case Study: Layers of Possible Analysis 448

From Data to Case Study 449
i B i l i i The Process of Constructing Case Studies 450
Pattern, Theme, and Content Analysis 452
Inductive and Deductive Qualitative Analyses 453
Indigenous Concepts and Practices 454
Sensitizing Concepts 456
Indigenous Typologies 457
Analyst-Constructed Typologies 458
||||]||||!gj Qualitative Analysis of Ancestry at the U.S. Census 461
The Intellectual and Mechanical Work of Analysis 462
Coding Data, Finding Patterns, Labeling Themes, and Developing
Category Systems 462
IMUflCT First-Cut Coding Examples: Sample Codes From the
Field Note Margins 464
Convergence and Divergence in Coding and Classifying 465
Determining Substantive Significance 467
Logical Analysis 468
^TQnnQgl An Empirical Typology of Teacher Roles in Dealing With
High School Dropouts 469
A Process/Outcomes Matrix 471
E B E B I Mapping Stakeholders' Stakes 472
Conceptual Guide for Data Collection and Analysis:
Utilization of Planmng, Evaluation, and Reporting 473
^yOQQQIP Matrix of Linkages Between Program Processes and Impacts 474
An Analysis Example: Recognizing Processes, Outcomes, and
Linkages in Qualitative Data 474
Interpreting Findings 477
Interpreting for Meaning 477
Comparisons, Causes, Consequences, and Relationships 478
Theory-Based Analysis Approaches 481
Phenomenological Analysis 482
Grounded Theory 487
Qualitative Comparative Analysis 492
Analytic Induction 493
Special Analytical Issues and Frameworks 494
Reflexivity and Voice 494
Collaborative and Participatory Analyses 496
The Hermeneutic Circle and Interpretation 497
Analyzing Institutional Documents 498
Dramaturgical Analysis 499
Finding Nothing 500
Synthesizing Qualitative Studies 500
Reporting Findings 502
Balance Between Description and Interpretation 503
Communicating With Metaphors and Analogies 504
Drawing Conclusions 506
Special Issues in Evaluation Reporting and an Example 506
Feedback and Analysis 506
Evaluative Feedback Using Indigenous Typologies 507
QJJ^QjJQ Distinguishing Observations From Perceived Impacts 509
To Write a Report or Not to Write a Report? 510
Focus 511
The Executive Summary and Research Abstract 511
I B g M i l l Utilization-Focused Evaluation Reporting 512
Carpe Diem Briefings 512
The Creativity of Qualitative Inquiry 512
The Past and the Future: Deciding in Which Direction to Look 515
APPENDIX 8.1. Excerpts From a Codebook for Use by Multiple Coders 516
APPENDIX 8.2. Mike: An Dlustrative Case Study 518
APPENDIX 8.3. Excerpts From an Illustrative Interview Analysis:
Reflections on Outcomes From Participants in a
Wilderness Education Program 525
Riddles of Qualitative Inquiry: Who Am I? 537
Gary D. Shank
9. Enhancing the Quality and Credibility of Qualitative Analysis 541

Interpreting Truth 541
Alternative Criteria for Judging Quality 542
^^QQQyQQ Alternative Sets of Criteria for Judging the Quality
and Credibility of Qualitative Inquiry 544
Traditional Scientific Research Criteria 544
Social Construction and Constructivist Criteria 546
Artistic and Evocative Criteria 547
Criticai Change Criteria 548
Evaluation Standards and Principies 549
Clouds and Cotton: Mixing and Changing Perspectives 551
Credibility 552
Rigor: Strategies for Enhancing the Quality of Analysis 553
Integrity in Analysis: Generating and Assessing Rival Conclusions 553
Negative Cases 554
Triangulation 555
ISfflMKEl A Story of Triangulation: Testing Conclusions With
More Fieldwork 559
Design Checks: Keeping Methods and Data in Context 563
High-Quality Lessons Learned 564
I^Sn^QEQI High-Quality Lessons Leamed 565
The Credibility of the Researcher 566
Considering Investiga tor Effects: Varieties of Reactivity 567
Intellectual Rigor 570
The Paradigms Debate and Credibility 570
Beyond the Numbers Game 572
Beyond Objectivity and Subjectivity: New Concepts, New Language 574
Reflections on Truth and Utility as Criteria of Quality 577
From Generalizations to Extrapolations and Transferability 581
The Credibility Issue in Retrospect: Increased Legitimacy for
Qualitative Methods 584
Beyond the Qualitative-Quantitative Debate 584
Matching Claims and Criteria 587
APPENDIX 9.1. Case Study: A Documenter's Perspective 589
Refere nc es RI
Author Index II
Subject Index 113
About the Author Al

Preface.
T he story is told that at the conclusion of a rigorous course in philoso-

phy, one of the students lamented: "Professor, you have knocked a
hole in everything I've ever believed in, but you have given me nothing to
take its place."
To which the philosopher replied: "You will recall that among the labors
of Hercules he was required to clean out the Augean stables. He was not, let
me point out, required to fill them,"
In doing this revision, I reviewed over a
W hile part of the task of this revi-

sion has been to clean out the
qualitative Augean stables, the
truly Herculean task has been deciding
thousand new books on qualitative meth-
ods, program evaluation, case studies,
monographs, and related works published
what to add. Unlike the professor who can in the last decade, as well as hundreds of ar-
be content with getting the stables cleaned, ticles scattered through scores of journals
the author of a revision bears responsibility covering the full range of disciplines and
for restocking the stables with fresh nutri- professions. Two important new qualita-
ents and feed, a task made especially chal- tive journals Qualitative Inquiry and Field
lenging because of the unprecedented Methodsbegan publication, as did spe-
blossoming of qualitative inquiry in recent cialized qualitative journals m a number of
years. professions (e.gv health, nursing, social
m. xxi
xxii 13. QUALITATIVE RESEARCH AND EVALUATION METHODS
work, organizational development) and proaches are needed and credible, that
some devoted to specific approaches (e.g., mixed methods can be especially valuable,
Grounded Theory Reviezu). The Handbook of and that the challenge is to appropriately
Qualitative Research was published (1994), as match methods to questions rather than ad-
was a revision (2000), and the Handbook of hering to some narrow methodological or-
Methods in Cultural Anthropology (1998) thodoxy. With less need to establish the
made its debut. Sophisticated new software value of qualitative inquiry by debating
programs have been developed to support those of quantitative and experimental per-
qualitative analysis. Internet listservs have suasion, qualitative inquirers have turned
emerged to facilitate dialogue. The Hercu- their attention to eacli other, noticing that
lean challenge has been analyzing this geo- they are engaging in different kinds of quali-
metric growth to determine primary trends, tative inquiry from widely different per-
patterns, and themes. The results of that spectives. Qualitative methodologists and
analysis are reflected throughout this new theorists have thus taken to debating each
edition. other. The upshot of ali the developmental
The first edition of this book (1980), enti- work in qualitative methods is that there is
tled Qualitative Evaluation Methods, focused now as much variation among qualitative
on the variety of ways in which qualitative researchers as there is between qualitatively
methods were being applied in the then and quantitatively oriented scholars and
still-emergent profession of program evalu- evaluators. A primary purpose of this new
ation. (The American Evaluation Associa- edition is to sort out the major perspectives
tion was not established until 1984.) That in that debate, portray the diversity of quali-
edition appeared in the midst of the heated tative approaches now available, and exam-
qualitative-quantitative debate about the ine the influences of this diversity on appli-
relative value of different methods and al- cations, especially but not exclusively in
ternative paradigms. The second edition program evaluation, which has experienced
(1990), entitled Qualitative Evaluation and Re- a parallel flowering of diversity and atten-
search Methods, was influenced by maturing dant controversies about new directions.
of the paradigms debate. It included much
more attention to the ways in which differ-
ent theoretical and philosophical perspec-
tives influenced qualitative inquiry, as well !=l Organization of This Edition
as the greater range of applications in eval-
uation as that profession blossomed. This Chapter 1 provides a range of examples of
latest edition involves yet another change qualitative findings. I begin by presenting a
of title, Qualitative Research and Evaluation number of significant illustrations of the
Methods, reflecting the degree to which de- fruit of qualitative inquiry, in order to give a
velopments in qualitative inquiry during taste of what results from qualitative stud-
the la st decade have been driven by a diver- ies and help those new to such inquiry know
sifying research agenda and scholarly dia- where they are headed and what they are
logue, much of which has found its way into trying to produce. Chapter 2 reviews and
evaluation, to be sure. adds to the primary strategic themes that
The classic qualitative-quantitative de- define qualitative inquiry. Chapter 3 exam-
bate has been largely resolved with recogni- ines different qualitative approaches, in-
tion that a variety of methodological ap- cluding several that have emerged dis-
Preface |J. xxiii
tinctly in the last decade. Chapter 4 presents edition and traveled many qualitative
a wide range of qualitative applications, miles, the list of those to whom I am in-
many of them new, in evaluation, action re- debted is too long and the danger of leaving
search, and organizational, community, and out important influences too great for me to
international development. Chapters 5, 6, include such traditional acknowledge-
and 7 cover design and data gathering, of- ments here. I can only refer the reader to the
fering guidance in purposeful sampling, references and stories in the book as a start-
mixed methods, fieldwork, observational ing point.
approaches, and interviewing, with special I must, however, acknowledge cartoonist
attention directed to the skills and compe- Michael Cochran of Tupper Lake, New York,
tencies needed to gather high-quality data. who drew the many new illustrations in-
Chapter 8 provides direction and processes clude d here to lighten the reader's way
for analyzing qualitative data, always the along this journey. Our collaboration began
most challenging aspect of this work. at a Union Institute research methods semi-
Finally, Chapter 9 de ais with paradigms, nar he took while pursuing his doctorate in
politics, and ways of enhancing the credibil- professional psychology. His evaluation of
ity of qualitative inquiry. This chapter also the seminar include d cartoons. I liked his
presents what I consider to be the five dis- wit and style, so I offered ideas for cartoons
tinct and competing frameworks for under- on qualitative inquiry and he turned them
taking and judging the quality of qualitative into art. Doing the cartoons, he told me, was
studies: traditional scientific research crite- a wonderful distraction from writing his dis-
ria; social construction and constructivist sertation. Fm grateful for his humor and tal-
criteria; artistic and evocative criteria; criti- ent.
cai change criteria; and pragmatic, utility- The editorial and production staff at Sage
oriented evaluation standards and princi- Publications deserve special mention. Only
pies. Along the way I've added, as is my fellow authors who have struggled with edi-
wont, hundreds of new stories and exam- tors of limited vision and understanding can
ples. I've also created over 50 new exhibits fully appreciate what it means to work with
that summarize and illuminate major C. Deborah Laughton, an experienced and
points. knowledgeable editor who not only knows
qualitative methods and evaluation as
deeply as any practitioner of these arts, but
also has significantly shaped those fields by
tI. A c k n o w l e d g m e n t s conceptualizing works that she saw a need
for and then nurtuiing authors, which she
I began this preface by noting the Herculean does better than any editor I've ever known,
task of revision given the enormous growth to assure that those works came to fruition.
and increased diversity of qualitative in- That she is also a writer and designer has
quiry. One task proved more than Hercu- made our working together a genuine col-
lean, and I could not complete it. I began to laboration. Kate Peterson's skilled copyedit-
list the many colleagues and evaluation cli- ing added appreciably to the final product as
ents to whom I am indebted and who de- she made many suggestions for improving
serve acknowledgement for their contribu- clarity and readability, and I came to trust
tions to my understanding and writing over and even rely on her unusual eye for detail
the years. Now that I have reached this third Janelle Lemaster's interior design work con-
xxiv s QUALITATIVE RESEARCH AND EVALUATION METHODS
verted the raw maimscript into the carefully newborn child. (Having been left out of that
crafted book you now hold. Diana Axelsen decision, the newborn child subsequently
pulled it ali together as production editor to made it clear he didn't always agree.) The con-
get the book launched on schedule. I came to tribution of Jeanne to the book exemplifies
count on not only her great management why the personal and professional sometimes
competence but also her good humor. cannot and ought not be separated. Jeanne's
Finally, in acknowledging the superb Sage reflections on her own evaluation fieldwork
production team, I should also reprise the and interviewing experiences helped me clar-
preface to the first edition in which I noted ify and break through some particularly diffi-
that my initial foray into qualitative writing cult sections of the book. Her editorial advice
was due entirely to the persuasive powers was invaluable. Those were her tangible con-
of Sara Miller McCune, co-founder of Sage tributions; the intangibles she contributed are
Publications, who had shepherded the first the things that made the book happen.
edition of Utilization-Focused Evaluation (1978)
into print and, based on the perspective in
that book, urged me during a trip to Minne- Those intangibles and Jeanne's ongoing
sota in 1978 to write a qualitative compan- support have remained the mainstay of my
ion. Her vision and follow-through have writing. Meanwhile, the newborn child re-
made Sage Publications the leading pub- ferred to above, Quinn Campbell, has com-
lisher of both evaluation and qualitative in- pleted a master's degree in engineering, and
quiry books. his younger sister, Charmagne Campbell-
With the reader's indulgence, and by way Patton, is about to complete college. As this
of further providing a historical context for was being completed, their older brother,
this third edition, permit me to include an Braxidon Patton, participated in a two-day
excerpt from that first preface so maxty years workshop I conducted on qualitative meth-
ago: ods in preparation for his first evaluation
fieldwork, a sideline he has tumed to as a
As other authors know, there is no way to re- way of supporting his real passion, writing
ally recognize the contribution of one's family and performing rock music. Thus have the
to a book like this, the writing of which was a years passed, love maturing and children
struggle and matter of endurance for both growing, bringing forth the need to revise
family and author. While Sara Miller McCune the old, celebrate the new, and clean out the
was persuading me that the book should be qualitative Augean stables while restocking
written, Jeanne was persuading me that we them with fresh nutrients. It is those nutri-
could nurture together both a new book and a ents that follow.
pyvRT 1
Conceptual Issues
in Qualitative Inquiry
Psychometricians try to measure it

Experimentalists try to control it.
Interviewers ask questions about it.
Observers watch it.
Participant observers do it.
Statisticians count it.
Evaluators value it.
Qualitative inquirers find meaning in it.
When in doubt, observe and ask questions.

When certain, observe at length and ask many more questions.
Gigo's law of deduction: Garbage in,, garbage out.

Halcolm/s law of induction: No new experience, no new insight.
Qualitative inquiry cultivates the most useful of ali human capacities:

The capacity to learn.
m. 1
2 [. CONCEPTUAL ISSUES IN QUALITATIVE INQUIRY
Innovators are told: "Think outside the box."

Qualitative scholars tell their students: "Study the box. Observe it. Inside. Out-
side. From inside to outside, and outside to inside. Where is it? How did it get
there? What's around it? Who says it's a 'box'? What do they mean? Why does it
matter? Or does it? What is not 'box'? Ask the box questions. Question others
about the box. What's the perspective from inside? From outside? Study diagrams
of the box. Find documents related to the box. What does thinking have to do with
the box anyway? Understand this box. Study another box. And another. Under-
stand box. Understand. Then you can think inside and outside the box. Perhaps.
For awhile. Until it changes. Until you change. Until outside becomes inside
again. Then start over. Study the box."
There is no burden of proof. There is only the world to experience and understand.
Shed the burden of proof to lighten the load for the journey of experience.
From Halcolm's Laws of Inquiry
The Nature of Qualitative Inquiry
TThe F ^ u i t o f Q u a l i t a t i v e j M e ^ o d s
There once lived a man in a country with no fruit trees. A scholar, he spent a
great deal of time reading. He often carne across references to fruit. The descrip-
tions enticed him to undertake a journey to experience fruit for himself.
He went to the marketplace and inquired where he could find the land of
fruit. After much searching he located someone who knew the way. After a long
and arduous journey he came to the end of the directions and found himself at
the entrance to a large apple orchard. It was springtime and the apple trees were
in blossom.
The scholar entered the orchard and, expectantly, pulled off a blossom and
put it in his mouth. He liked neither the texture of the flower nor its taste. He
went quickly to another tree and sampled another blossom, and then another,
and another. Each blossom, though quite beautiful, was distasteful to him. He
left the orchard and returned to his home country, reporting to his fellow villag-
ers that fruit was a much overrated food.
Being unable to recognize the difference between the springblossom and the
summer fruit, the scholar never realized that he had not experienced what he
was looking for.
From Halcolm's Inquiry Parables
S 3
Three Kinds of Qualitative Data
Interviews
Open-ended questions and probes yield in-depth responses about people's experiences, per-
ceptions, opinions, feelings, and knowledge. Data consist of verbatim quotations with sufficient
context to be interpretable.
Observatons
Fieldwork descriptions of activities, behaviors, actions, conversations, interpersonai interac-
tions, organizational or community processes, or any other aspect of observable human experi-
ence. Data consist of field notes: rich, detailed descriptions, including the context within which the
observatons were made.
Documents
Written materiais and other documents from organizational, clinicai, or programs records;
memoranda and correspondence; offcial publications and reports; personal diaries, letters, artis-
tic works, photographs, and memorabilia; and written responses to open-ended surveys. Data con-
sist of excerpts from documents captured in a way that records and preserves context.
Recognizing Qualitative Data processes that are part of observable hu-

man experience. Document analysis includes
This book discusses how to collect, ana- studying excerpts, quotations, or entire pas-
lyze, and use qualitative data. To begin, let's sages from organizational, clinicai, or pro-
examine the fruit of qualitative methods. It is gram records; memoranda and correspon-
important to know what qualitative data dence; official publications and reports;
and findings look like so that you will know personal diaries; and open-ended written
what you are seeking. It will also be impor- responses to questionnaires and surveys.
tant to consider criteria for judging the qual- (See Exhibit 1.1.)
ity of qualitative data. Apples come to mar- The data for qualitative analysis typically
ket sorted by type (Red Delicious/ Golden), come from fieldwork. During fieldwork, the
purpose (e.g., cooking or eating), and qual- researcher spends time in the setting under
ity. Likewise, qualitative studies vary by studya program, an organization, a com-
type, purpose, and quality. munity, or wherever situations of impor-
Qualitative findings grow out of three tance to a study can be observed, people in-
kinds of data collection: (1) in-depth, open- terviewed, and documents analyzed. The
ended interviews; (2) direct observation; researcher makes firsthand observa tions of
and (3) written documents. Interviews yield activities and interactions, sometimes en-
direct quotations from people about their ex- gaging personally in those activities as a par-
periences, opinions, feelings, and knowl- ticipant observer. For example, an evaluator
edge. The data from observatons consist of might participa te in ali or part of the pro-
detailed descriptions of people's activities, gram under study, participating as a regular
behaviors, actions, and the full range of inprogram member, client, or student. The
terpersonai interactions and organizational qualitative researcher talks with people
The Nature of Qualitative Inquiri/ [J. 5
about their experiences and perceptions. The quality of qualitative data depends
More formal individual or group interviews to a great extent on the methodological
may be conducted. Relevant records and skill, sensitivity, and integrity of the re-
documents are examined. Extensive field searcher. Systematic and rigorous obser-
notes are collected through these observa- vation involves far more than just being
tions, interviews, and document reviews. present and looking around. Skillful inter-
The voluminous raw data in these field viewing involves much more than just
notes are organized into readable narrative asking questions. Content analysis re-
descriptions with major themes, categories, quires considerably more than just read-
and illustrative case examples extracted ing to see what's there. Generating useful
through content analysis. The themes, pat- and credible qualitative findings through
terns, understandings, and insights that observation, interviewing, and content
emerge from fieldwork and subsequent analysis requires discipline, knowledge,
analysis are the fruit of qualitative inquiry. training, practice, creativity, and hard
Qualitative findings may be presented work.
alone or in combination with quantitative This chapter provides an overview of
data. Research and evaluation studies em- qualitative inquiry. La ter chapters exam-
ploying multiple methods, including combi- ine how to choose among the many op-
nations of qualitative and quantitative data, tions available within the broad range of
are common. At the simplest levei, a ques- qualitative methods, theoretical perspec-
tionnaire or interview that asks both fixed- tives, and applications; how to design a
choice (closed) questions and open-ended qualitative study; how to use observa-
questions is an example of how quantitative tional methods and conduct in-depth,
measurement and qualitative inquiry are of- open-ended interviews; and how to ana-
ten combined. lyze qualitative data to generate findings.
& Qualitative Findings: T h e m e s , Patterns,

Concepts, Insights, U n d e r s t a n d i n g s
ewton and the apple. Freud and anxiety. Jung and dreams. Piaget
and his children. Darwin and Galapagos tortoises. Marx and
England's factories. Whyte and street corners. What are you obsessed with?
Halcolm
Mary Field Belenky and her colleagues gether, informed partly by previous re-
set out to study women's ways of knowing. search but ultimately basing the analysis
They conducted extensive interviews with on their own collective sense of what cate-
135 women from diverse backgrounds prob- gories best captured what they found in
ing how they thought about knowledge, au- the narrative data. They argued with each
thority, truth, themselves, life changes, and other about which responses belonged in
life in general. They worked as a team to which categories. They created and aban-
group similar responses and stories to- doned categories. They looked for com-
Discovery of an early qualitative evaluation report
monalities and differences. They worked irtforming variations in ways of knowing.

hard to honor the diverse points of view they After painstaking analysis, they ended up
found while also seeking patterns across sto- with the five categories of knowing summa-
ries, experiences, and perspectives. One rized in Exhibit 1.2, a framewbrk that be-
theme emerged as particularly powerful: caine very influential in women's studies
"Again and again women spoke of 'gaining and represents one kind of fruit from quali-
voice' " (Belenky et al. 1986:16). Voice versus tative inquiry.
silence emerged as a central metaphor for
WomerTs Ways of Knowing:

An Example of Qualitative Findings
Silence: A position in which women experience themselves as mindless and voiceless and
subject to the whims of externai authority.
Received knowledge: Women conceive of themselves as capable of receiving, even reproducing

knowledge from externai authorities, but not capable of creating knowledge on their own.
Subjective knowledge: A perspective from which truth and knowledge are conceived as
persona!, private, and subjectively known or intuited.
Procedural knowledge: Women are invested in learning and apply objective procedures for
obtaining and communicatng knowledge.
Constructedknowledge: Women view ali knowledge as contextual, experience themselves as

creators of knowledge, and value both subjective and objective strategies for knowing.
SOURCE: Belenky et al. (1986:15).
One of the best-known and most influen- Stephen Covey (1990) used this same
tial books in organizational development sampling approach in doing case studies of
and management is In Search of Excellence: "highly effective people." He identified
Lessons From America's Best-Run Companies. seven habits these people practice: (1) being
Peters and Waterman (1982) based the book proactive; (2) beginning with the end in
on case studies of 62 highly regarded com- mind; (3) putting first things first; (4) think-
panies. They visited companies, conducted ing win/ win; (5) seeking first to understand,
extensive interviews, and studied corporate then seeking to be understood; (6) syner-
documents. From that massive amount of gizing, or engaging in creative cooperation;
data they extracted eight attributes of excel- and (7) self-renewal.
lence: (1) abias for action; (2) close to the cus- Both of these best-selling books, In Search
tomer; (3) autonomy and entrepreneurship; of Excellence and The 7 Habits of Highly Effec-
(4) productivity through people; (5) hands- tive People, distill a small number of impor-
on, value-driven; (6) stick to the knitting; (7) tant lessons from a huge amount of data
simple form, lean staff; and (8) simultaneous based on outstanding exemplars. It is com-
loose-tight properties. Their book devotes a mon in qualitative analysis for mounds of
chapter to each theme with case examples field notes and months of work to reduce to a
and implications. Their research helped small number of core themes. The quality of
launch the quality movement that has now the insights generated is what matters, not
moved from the business world to not-for- the number of such insights. For example, in
profit organizations and government. This an evaluation of 34 programs aimed at peo-
study also illustrates a common quaHtative ple in poverty, we found a core theme that
sampling strategy: studying a relatively separated more effective from less effective
small number of special cases that are suc- programs: How people are treated affects
cessful at something and therefore a good how they treat others. If staff members are
source of lessons learned. treated autocratically and insensitively by
management, with suspicion and disre- tinguish, and elaborate three primary pro-
spect, staff will treat clients the same way. cesses that contribute to the development of
Contrariwise, responsiveness reinforces re- a relationship: "Being-In," "Being-For," and
sponsiveness, and empowerment breeds "Being-With."
empowerment. These insights became the Being-In involves immersingoneself in
centerpiece of subsequent cross-project, col- another's world: listening deeply and atten-
laborative organizational and staff develop- tively so as to enter into the other person's
ment processes. experience and perception. "I do not select,
A different kind of qualitative finding is interpret, advise, or direct. . . . Being-In the
illustrated by Angela Browne's book When world of the other is a way of going wide
Battered Women Kill (1987). Browne con- open, entering in as if for the first time, hear-
ducted in-depth interviews with 42 women ing just what is, leaving out my own
from 15 states who were charged with a thoughts, feelings, theories, biases I enter
crime in the death or serious injury of their with the intention of understanding and ac-
mates. She was often the first to hear these cepting perceptions and not presenting my
women's stories. She used one couple's his- own view or reactions 1 only want to en-
tory and vignettes from nine others, repre- courage and support the other person's ex-
sentative of the entire sample, to illuminate pression, what and how it is, how it came to
the progression of an abusive relationship be, and where it is going." (Moustakas 1995:
from romantic courtship to the onset of 82-83)
abuse through its escalation until it was on- Being-For involves taking a stand in
going and eventually provoked a homicide. support of the other person, being there for
Her work helped lead to legal recognition the other. "I am listening. I am also offering a
of battered women's syndrome as a legiti- position, and that position has an element of
mate defense, especially in offering insight my being on that person's side, against ali
into the common outsider's question: Why others who would minimize, deprecate, or
doesn't the woman just leave? An insider's deny this person's right to be and to grow....
perspective on the debilitating, destructive, I become an advocate of the person with ref-
and all-encompassing brutality of battering erence to his or her frustrations and prob-
reveals that question for what it is: the facile lems in dealing with others." (Moustakas
judgment of one who hasn't been there. The 1995:83)
effectiveness of Browne's careful, detailed, Being-With involves being present as
and straightforward descriptions and quo- one's own person in relation to another per-
tations lies in their capacity to take us inside son, bringing one's own knowledge and ex-
the abusive relationship. Offering that in- perience into the relationship. "This may in-
side perspective powers qualitative report- volve disagreeing with the other's ways of
ing. interpreting or judging or presenting some
Clark Moustakas (1995), a humanistic spect of the world. Being-With means lis-
psychologist and phenomenologist, also tening and hearing the other's feelings,
gives us an insider's perspective: his thoughts, objectives, but it also means offer-
own. An astute and dedicated observer of ing my own perceptions and views. There is,
relationships, especially therapeutic rela- in Being-With, a sense of joint enterprise
tionships, he drew deeply on his own expe- two people fully involved, struggling, ex-
riences and clinicai cases to identify, dis- ploring, sharing." (Moustakas 1995:84)
IjfPlIH''"1' Coming-of-Age Paradigms
Dimensions
of Comporiso n Tribal Initiotion Modern Corning ofAg
View of life passages One-time transition from child Multiple passages over a lifetime
to aduit journey
Territory Tribal territory Earth: Global community
Ancestry Creation myth Evolutionary story of humankind
Identity Becoming a man or woman Becoming a complete person
Approach Standardized Individualized
Outcome Tribe-based identity Person a lity identity: Sense of self
Message You are first and foremost a You are first and foremost a person
member of the tribe in your own right
SOURCE: Patton (I999a:333, 335).
Qualitative findings of ten have this sim- sis that leads to a new synthesis. In philoso-
ple yet elegant and insightful character. This phy such contrasts derive from the rumina-
straightforward yet nuanced framework tions of philosophers; in qualitative research
represents a creative synthesis of years of such thematic contrasts emanate from and
participant observation and personal in- are grounded in fieldwork.
quiry. Through cases, dialogues, quotations, This quick sampling of the fruit of quali-
cases, and introspective reflections, Mous- tative inquiry is meant, like a wine tasting, to
takas illuminates the process of moving demonstrate choices toward developing a
froin Being-In to Being-For and ultimately more sophisticated palate, or like appetiz-
Being-With. His work exemplifies the con- ers, as an opening to the fuller feast yet to
tribution of phenomenological inquiry to come. The next section discusses some of the
humanistic psychology. different research and evaluation purposes
Still a different format for capturing and that affect what kind of fruit results from
reporting qualitative findings is illustrated qualitative inquiry and how the quality of
by my own inquiry into alternative coming- that fruit is judged.
of-age approaches. I used the device of con-
structing ideal-typical alternative para-
digms to compare and contrast what I Different Purposes of and
learned (Patton 1997a). Exhibit 1.3 provides Audiences for Qualitative Studies:
a sampling of contrasts between traditional Research, Evaluation, Dissertations,
tribe-centered initiations and modern youth- and Personal Inquiry
centered coming-of-age celebrations. These
kinds of polar contrasts can sometimes set As the title of this book indicates, qualita-
up a Hegelian dialectic of thesis and antithe- tive methods are used in both research and
evaluation. But because the purposes of re- Qualitative methods are often used in
search and evaluation are different, the crite- evaluations because they tell the program's
ria for judging qualitative studies can vary story by capturing and communicating the
depending on purpose. This point is impor- participants' stories. Evaluation case studies
tant. It means one can't judge the appropri- have ali the elements of a good story. They
ateness of the methods in any study or the tell what happened, when, to whom, and
quality of the resulting findings without with what consequences. Many examples in
knowing the study's purpose, agreed-on this book are drawn from program evalua-
uses, and intended audiences. Evaluation tion, policy analysis, and organizational de-
and research typically have different pur- velopment. The purpose of such studies is to
poses, expected uses, and intended users. gather information and generate findings
Dissertations add yet another layer of com- that are useful. Understanding the pro-
plexity to this mix. Let's begin with evalua- gram^ and participants' stories is useful to
tion. the extent that they illuminate the processes
Program evaluation is the systematic col- and outcomes of the program for those who
lection of information about the activities, must make decisions about the program. In
characteristics, and outcomes of programs Utization-Focused Evaluation (Patton 1997a),
to make judgments about the program, im- I presented a comprehensive approach to
prove program effectiveness, and/or inform doing evaluations that are useful, practical,
decisions about future prograrmning. Pol- ethical, and accurate. The primary criterion
icies, organizations, and personnel can also for judging such evaluations is the extent to
be evaluated. Evaluative research, quite which intended users actually use the find-
broadly, can include any effort to judge or ings for decision making and program im-
enhance human effectiveness through sys- provement. The methodological implication
tematic data-based inquiry. Human beings of this criterion is that the intended users
are engaged in ali kinds of efforts to make must value the findings and find them credi-
the world a better place. These efforts in- ble. They must be interested in the stories,
clude assessing needs, formulating policies, experiences, and perceptions of program
passing laws, delivering programs, manag- participants beyond simply knowing how
ing people and resources, providing ther- many came into the program, how many
apy, developing communities, changing or- completed it, and how many did what after-
ganizational culture, educating students, ward. Qualitative findings in evaluation il-
intervening in conflicts, and solving prob- luminate the people behind the numbers
lema. In these and other efforts to make the and put faces on the statistics, not to make
world a better place, the question of whether hearts bleed, though that may occur, but to
the people involved are accomplishing what deepen understanding.
they want to accomplish arises. When one Research, especially fundamental or basic
examines and judges accomplishments and research, differs from evaluation in that its
effectiveness, one is engaged in evaluation. primary purpose is to generate or test theory
When this examination of effectiveness is and contribute to knowledge for the sake of
conducted systematically and empirically knowledge. Such knowledge, and the theo-
through careful data collection and thought- ries that undergird knowledge, may subse-
ful analysis, one is engaged in evaluation re- quently inform action and evaluation, but
search. action is not the primary purpose of funda-
UAA.'-' . ; . j : .:, .,.,V

tfLftt k>$X"- !;=: !;:
^rrtsThil. 8 st! .'.:, .-iV"i L-I !.'i.'! u !:'* . , ' , ! ' ':,:; . 1 o r -vj ? 1 < " ' rr> m ; i
flWJ-fOT-n Ti Vjji:" iVy.Ti ji ': ! 'v!:.' i w! i - u i t :. ' ":!:> V : ; '! , i 1 : " f 11-3-=- [ !<' 1
VI' I. i >
i.i-.iiv.li,::'
;,!=:; j:-n j;! jT!t:! !;= i f t j ? ^ f f c s - ri h ! >n !' i ! ! , ii r 1V1 m \ \ 'i; 'V',iVi!:oi:r."!-. mvi-'' j
.'-=.'= j:" !.!:::-."-.;!!" i-,v:V: , i ijYi v; ,i.v. jjVi!.v i! nJ i iW!.',.;"';,:::;::,:,! yj.y .
V M-V
-.fl Ii-I j:;; iYi;Mi!::!;h!ni:ir!tfktiyr:siri" i n - ii r :.=: !.Vi:. U.i ri t: IV!; ri ,1 ,! ,!.;: j -v. ,, ,-i....,,-j- vi ,,,,.,,,
h:i -inl^r-^r-jii.H^^rh^n^^rr î-i-i: !-!." ;. v l Y : i-vkz! q.!.a IY-.h.;
3
11 oIA'', i!* 1 ".: !: -'! ::', m iyi i i = : n 11
V':i!;::: ^ . n a /-.'n, . ..VMV s:-! r.ii.' ::;i.::M;.' r:'.! "-: .: .r- r
1
'.YPOV :;":' rivrjiM.- umIY! t- ms.. : cti11.: 1 :.i:V!!! i:,11 : vr!T! : \ i 'o ! i v, i =i i vi i
i!l'i -!:!' -=:"':'!= \>!l =!'=:.!:liY/rvi|i ! i = " i ! = i ' wrt
:
.-'IA-;!.;!.n.:iiV!..; 'j.'i- I:I.-i "
!: ' ; : i".;-!,!":: : ! v; - S t i " =-:;!-t: n m nVi-iv.ri.i.1 v- i!:>i ! i li!'.!!!!!!!.' ! P ! . ! ! i ' " t ' ! : * 1 " > ' ' i .
X-'.;>::;. :iT= -v;: !:= ! 6 * 7 ' ' r v t V " , i V r - i ,i ; i i :. !:;.'! -. ^ o > !'' :;jxia > .:;>.
i=r.':-i:'!' n r r o '; M t t f i t f . ; d :ni I * 1 !" i i'!l
mental research. Qualitative inquiry is espe- ing the quality of the methodological proce-
cially powerful as a source of grounded the- dures followed and the analysis done. Qual-
ory, theory that is inductively generated itative dissertations, once quite rare, have
from fieldwork, that is, theory that emerges become increasingly common as the criteria
from the researcher's observations and infor judging qualitative contributions to
terviews out in the real world rather than in knowledge have become better understood
the laboratory or the academy. The primary and accepted. But those criteria are not abso-
audiences for research are other researchers lute or universally agreed on. As we shall
and scholars, as well as policymakers and see, there are many varieties of qualitative
others interested in understanding some inquiry and multiple criteria for judging
phenomenon or problem of interest. The re- quality, many of which remain disputed.
search training, methodological prefer- While the precedmg discussion of evalua-
ences, and scientific values of those who use tion, research, and dissertations has empha-
research will affect how valuable and credi- sized taking into account externai audiences
be they find the empirical and theoretical and consumers of qualitative studies, it is
fruit of qualitative studies. also important to acknowledge that you may
Dissertations and graduate theses offer be the primary intended audience for your
special insight into the importance of atten- work. You may study something because
tion to audience. Savvy graduate students you want to understand it. As my children
learn that to complete a degree program, the grew to adulthood, I found myself asking
studenfs committee must approve the questions about coming of age m modern
work. The particular understandings, val- society so I undertook a personal inquiry that
ues, preferences, and biases of committee became a book (Patton 1997a), but I didn't
members come into play in that approval start out to write a book. I started out trying
process. The committee will, in essence, to understand my own experience and the
evaluate the studenfs contribution, includ- experiences of my children. That is a form of
qualitative inquiry. While doing interviews Making Methods Decisions

with recipients of MacArthur Foundation
Fellowships (popularly called "Genius The implication of thinking about pur-
Awards"),1 was told by a social scientist that pose and audience in designing studies is
her fieldwork was driven by her own search that methods, no less than knowledge, are
for understanding and that she disciplined dependent on context. No rigid rules can
herself to not even think about publication prescribe what data to gather to investigate a
while engaged in interviewing and observ- particular interest or problem. There is no
ing because she didn't want to have her in- recipe or formula in making methods deci-
quiry affected by attention to externai audi- sions. Widely respected psychometrician
ences. She wanted to know because she wanted Lee J. Cronbach has observed that design-
to know, and she had made a series of career ing a study is as much art as science. It is
and professional decisions that allowed her "an exercise of the dramatic imagination"
to focus on her personal inquiry without be- (Cronbach 1982:239). In research as in art,
ing driven by the traditional academic ad- there can be no single, ideal standard.
monition to "publish or perish." She didn't Beauty no less than "truth" is in the eye of
want to subject herself to or have her work the beholder, and the beholders of research
influenced by externai criteria and judg- and evaluation can include a plethora of
ment. stakeholders: scholars, policymakers, fund-
In summary, ali inquiry designs are af- ers, program managers, staff, program par-
fected by intended purpose and targeted au- ticipants, journalists, critics, and the general
dience, but purpose and audience deserve public. Any given design inevitably reflects
special emphasis in the case of qualitative some imperfect interplay of resources, capa-
studies, where the criteria for judging qual- bilities, purposes, possibilities, creativity, and
ity may be poorly understood or in dispute, personal judgments by the people involved.
even among qualitative methodologists. Research, like diplomacy, is the art of the
This book cannot resolve these debates, but possible. Exhibit 1.4 provides a set of ques-
it will illuminate the methodological options to consider in the design process, re-
tions and their implications. (Chapter 9 dis- gardless of type of inquiry. With that back-
cusses alternative criteria for judging the ground, we can tum to consideration of the
quality of qualitative studies.) relative strengths and weaknesses of quali-
tative and quantitative methods.
13. M e t h o d s Choices: Contrasting

Qualitative and Quantitative E m p h a s e s
ot everything that can be counted counts, and not everything that

counts can be counted.
Albert Einstein
Some Guiding Questions and Options for

EXHIBIT 1.4 Methods Decisions
1. What are the purposes of the inquiry?

Research: Contribution to knowledge
Evaluation: Program improvement and decision making
Dissertation: Demonstrate doctoral-levei scholarship
Personal inquiry: Find out for oneself
2. Who are the primary audiences for the findings?

Scholars, researchers, academicians
Program funders, administrators, staff, participants
Doctoral committee
Oneself, friends, family, lovers
3. What questions wll guide the inquiry?

Theory-derived, theory-testing, and/or theory-oriented questions
Practical, applied, action-oriented questions and issues
Academic degree or discipine/specialization priorities
Matters of personal interest and concern, even passion
4. What data will answer or illuminate the inquiry questions?

Qualitative: Interviews, field observatons, documents
Quantitative: Surveys, tests, experments, secondary data
Mixed methods: What kind of mix? Which methods are primary?
5. What resources are avaiiable to support the inquiry?

Financial resources
Time
People resources
Access, connections
6. What criteria will be used to judge the quality of the findings?

Traditional research cr/ter/o; Rigor, validty, relabilty, generalizabiity
Evaluatian standards: Utility, feasiblty, propriety, accuracy
Nontraditional criteria: Trustworthiness, diversity of perspectives, clarity of voce, credibility
of the inquirer to primary users of the findings
Thinking about design alternatives and know if they're obese, measure body fat in
methods choices leads directly to consider- relation to height and weight and compare
ation of the relative strengths and weak- the results to population norms. If you want
nesses of qualitative and quantitative data. to know what their weight means to them,
The approach here is pragmatic. Some ques- how it affects them, how they think about it,
tions lend themselves to numerical answers; and what they do about it, you need to ask
some don't. If you want to know how much them questions, find out about their experi-
people weigh, use a scale. If you want to ences, and hear their stories. A comprehen-
sive and multifaceted understanding of a distraction. Guba and Lincoln (1981) have
weight in people's lives requires both their commented on this aspect of qualitative
numbers and their stories. Doctors who look research:
only at test results and don't also listen to
their patients are making judgments with Fatigue, shifts in knowledge, and cooptation,
inadequate knowledge, and vice versa. as well as variations resulting from differences
Qualitative methods facilitate study of is- in training, skill, and experience among differ-
sues in depth and detail. Approaching field- ent "instruments," easily occur. But this loss in
work without being constrained by prede- rigor is more than offset by the flexibility, in-
termined categories of analysis contributes sight, and ability to build on tacit knowledge
to the depth, openness, and detail of qualita- that is the peculiar province of the human in-
tive inquiry. Quantitative methods, on the strument. (p. 113)
other hand, require the use of standardized
measures so that the varying perspectives Because qualitative and quantitative
and experiences of people can be fit into a methods involve differing strengths and
limited number of predetermined response weaknesses, they constitute alternative, but
categories to which numbers are assigned. not mutually exclusive, strategies for re-
The advantage of a quantitative approach search. Both qualitative and quantitative
is that it's possible to measure the reactions data can be collected in the same study. To
of a great many people to a limited set of further illustrate these contrasting ap-
questions, thus facilitating comparison and proaches and provide concrete examples of
statistical aggregation of the data. This gives the fruit of qualitative inquiry, the rest of this
a broad, generalizable set of findings pre- chapter presents select excerpts from actual
sented succinctly and parsimoniously. By studies.
contrast, qualitative methods typically pro-
duce a wealth of detailed information about
a much smaller number of people and cases. Comparing Two Kinds
This increases the depth of understanding of of Data: An Example
the cases and situations studied but reduces
generalizability. The Technology for Literacy Center was a
Validity in quantitative research depends computer-based adult literacy program in
on careful instrument construction to ensure Saint Paul, Minnesota. It operated out of a
that the instrument measures what it is storefront facility in a lower-socioeconomic
supposed to measure. The instrument must area of the city. In 1988, after three years of
then be administered in an appropriate, pilot operation, a major funding decision
standardized manner according to pre- had to be made about whether to continue
scribed procedures. The focus is on the mea- the program. Anticipating the funding deci-
suring instrumentthe test items, survey sion, a year earlier local foundations and the
questions, or other measurement tools. In public schools had supported a summative
qualitative inquiry, the researcher is the inevaluation to determine the overall outcomes
strument. The credibility of qualitative and cost-effectiveness of the center. The
methods, therefore, hinges to a great extent evaluation design included both quantita-
on the skill, competence, and rigor of the tive and qualitative data.
person doing fieldworkas well as things The quantitative testing data showed
going on in a person's life that might prove great variation. The statistics on average
achievement gains masked great differences death from hepatitis. During the week she
among participants. The report concluded seldom gets more than three hours of sleep
that although testing showed substantial each night. At the time of the case study, she
achievement test gains for the treatment had spent 15 months in the program and
group versus the control group, the more progressed from not reading at ali (second-
important finding concerned the highly in- grade levei) to being a regular library user
dividualized nature of student progress. The (and testing a grade levei higher than where
report concluded, "The data on variation in she began). She developed an interest in
achievement and instructional hours lead to Black history and reported being particu-
a very dramatic, important and significant larly pleased at being able to read the Bible
finding: there is no average student at TLC" on her own. She described what it was like
(Patton and Stockdill 1987:33). not being able to read:
This finding highlights the kind of pro-
gram or treatment situation where qualita- Where do you go for a job? You can't make out
tive data are particularly helpful and appro- an application. You go to a doctor and you
priate. The Technology for Literacy Center can't fill out the forms, and it's very embar-
has a highly individualized program in rassing. You have to depend on other people
which learners proceed at their own pace to do things like this for you. Sometimes you
based on specific needs and interest. The don't even want to ask your own kids because
students come in at very different leveis, it's just like you're depending too much on
with a range of goals, participate in widely people, and sometimes they do it willingly,
varying ways, and make very different and sometimes you have to beg people to
gains. Average gain scores and average help
hours of instruction provide a parsimonious Ali the progress has made me feel lots
overview of aggregate progress, but such better about myself because I can do some of
statistical data do little to help funders un- the things I've been wanting to do and I could-
derstand what the individual variation n't do. It's made me feel more independent to
means. To get at the meaning of the program do things myself instead of depending on
for individual participants, the evaluation other people to do them for me.
included case studies and qualitative data
from interviews. A second contrasting case tells the story
of Sara Johnson, a 42-year-old Caucasian
woman who dropped out of school in the
INDIVIDUAL CASE EXAMPLES lOth grade. She is a clerical office manager.
She tested at 12th-grade levei on entry to the
One case is the story of Barbara Jenkins, a program. After 56 hours of study over 17
65-y ear-old Black grandmother who came to days, she received her general equivalency
Minnesota after a childhood in the deep diploma (GED), making her a high school
South. She works as a custodian and house graduate. She immediately entered college.
cleaner and is proud of never having been on She said that the decision to return for her
welfare. She is the primary breadwinner for GED was
a home with five children spanning three
generations, including her oldest daugh- an affirmation, as not having a diploma had
ter's teenage children for whom she has really hurt me for a long time It was always
cared since her daughter's unexpected scary wondering if somebody actually found
out that I was not a graduate that they would To get the perspective of students, I con-
fire me or they wouldn't accept me because I ducted group interviews. "Groups are not
hadn't graduated. The hardest thing for me to just a convenient way to accumulate the in-
do was tell my employer. He s very much into dividual knowledge of their members. They
education and our company is education- give rise synergistically to insights and solu-
oriented. So the hardest thing I ever had to do tions that would not come about without
was tell him I was a high school dropout. I them" (Brown, Collins, and Duguid 1989:
needed to tell because I needed time to go and 40). In group interviews I asked students to
take the test. He was just so understanding. describe the program's outcomes in per-
I couldn't believe it. t was just wonderful. sonal terms. I asked, "What difference has
I thought he was going to be disappointed in what you are learning made in your lives?"
me, and he thought it was wonderful that I Here are some responses.
was going back. He came to graduation.
I love the newspaper now, and actually read it.
Yeah, I love to pick up the newspaper now. I
These short excerpts from two contrast-
used to ha te it. Now I love the newspaper.
ing cases illustrate the value of detailed, de-
scriptive data in deepening our understand- I can follow sewing directions. I make a gro-
ing of individual variation. Knowing that cery list now, so I'm a better shopper. I don't
each woman progressed about one grade forget things.
levei on a standardized reading test is only a
small part of a larger, much more complex Yeah, you don't know how einbarrassing it is
picture. Yet, with over 500 people in the pro- to go shopping and not be able to read the
gram, it would be overwhelming for wife's grocery list. It's helped me out so much
funders and decision makers to attempt to in the grocery store.
make sense of 500 detailed case studies
Helps me with my medicine. Now I can read
(about 5,000 double-spaced pages). Statisti-
the bottles and the directions! I was afraid to
cal data provide a succinct and parsimoni-
give the kids medicine before because I wasn't
ous summary of major patterns, while select
sure.
case studies provide depth, detail, and indi-
vidual meaning. I don't get lost anymore. I can find my way
around. I can make out directions, read the
map. I work construction and we change loca-
OPEN-ENDED INTERVIEWS
tions a lot. Now I can find my way around. I
Another instructive contrast is to com- don't get lost anymore!
pare closed-ended questionnaire results
with responses to open-ended group inter- Just getting a driver's license will be wonder-
views. Questionnaire responses to quantita- ful. I'm 50. If I don't get the GED, but if I can
tive, standardized items indicated that 77% get a license...! I can drive well, but I'm scared
of the adult literacy students were "very to death of the written test. Just getting a
happy" with the Technology for Literacy driver's license . . . , a driver's license.
Center program; 74% reported learning "a
Now I read outdoor magazines. I used to just
great deal." These and similar results re-
read the titles of booksnow I read the books!
vealed a general pattern of satisfaction and
progress. But what did the program mean to I was always afraid to read at school and at
students in their own words? church. I'm not afraid to read the Bible now at
Bible class. It's really important to me to be achievement tests administered in both fali
able to read the Bible. and spring, criterion-referenced tests devel-
oped by teachers, performance objectives,
I can fill out applications now. You have to teacher peer ratings, student ratings of
know how to fill out an application in this teachers, parent ratings of teachers, princi-
world. I can look in the Yellow Pages. It used to pal ratings of teachers, and teacher self-
be so embarrassing not to be able to fill out ap- ratings.
plications, not to be able to find things in the
The Kalamazoo accountability system be-
Yellow Pages. I feel so much better now. At
gan to attract national attention. For exam-
least my application is filled out right, even if I
ple, the American School Board Journal re-
don't get the job, at least my application is
ported in April 1974 that "Kalamazoo
filled out right.
schools probably will have one of the most
comprehensive computerized systems of
I'm learning just enough to keep ahead of my personnel evaluation and accountability yet
kids. My family is my motivation. Me and my devised" (p. 40). In the first of a three-part se-
family. Once you can read to your kids, it ries on Kalamazoo, the American School
makes ali the difference m the world. It helps Board Journal asserted: "Take it from Kala-
you to want to read and to read more. When I mazoo: a comprehensive, performance-based
can read myself, I can help them read so they system of evaluation and accountability can
can have a better life. The kids love it when I work" ("Kalamazoo Schools" 1974:32).
read to them.
Not everyone agreed with that positive
assessment, however. The Kalamazoo Edu-
These group interview excerpts provide cation Association charged that teachers
some qualitative rnsights into the individ- were being demoralized by the accountabil-
ual, personal experiences of adults learning ity system. Some school officials, on the
to read. The questionnaire results (77% satis- other hand, argued that teachers did not
fied) provided data on statistically general- want to be accountable. In the spring of 1976,
izable patterns, but the standardized ques- the Kalamazoo Education Association, with
tions only tap the surface of what it means assistance from the Michigan Education As-
for the program to have had "great per- sociation and the National Education Asso-
ceived impact." The much smaller sample of ciation, sponsored a survey of teachers to
open-ended interviews adds depth, detail, find out the teachers' perspective on the ac-
and meaning at a very personal levei of ex- countability program (Perrone and Patton
perience. Another example will show that 1976).
qualitative data can yield not only deeper
The education association officials were
understanding but also political action as
interested primarily in a questionnaire con-
the depth of participants' feelings is re-
sisting of standardized items. One part of
vealed.
the closed-ended questionnaire provided
teachers with a set of statements with which
The Power of Qualitative Data they could agree or disagree. The question-
naire results showed that teachers felt the ac-
In the early 1970s, the school system of countability system was largely ineffective
Kalamazoo, Michigan, implemented a new and inadequate. For example, 90% of the
accountability system. It was a complex sys- teachers disagreed with the school adminis-
tem that included using standardized tration's published statement "The Kala-
mazoo accountability system is designed to 2. Finally, we'd like you to use this space to
personalize and individualize education"; add any additional comments you'd
88% reported that the system does not assist like to make about any part of the
teachers to become more effective; 90% re- Kalamazoo accountability system.
sponded that the accountability system has
not improved educational planning in A total of 373 teachers (70% of those who
Kalamazoo; and 93% believed, "Account- responded to the questionnaire) took the
ability as practiced in Kalamazoo creates an time to respond to one of these open-ended
undesirable atmosphere of anxiety among questions. Ali of the comments nade by
teachers." And 90% asserted, "The account- teachers were typed verbatim and included
ability system is mostly a public relations ef- in the report. These open-ended data filled
fort." Nor did teachers feel that the account- 101 pages. When the school officials and
ability system fairly reflected what they did school board members rejected the ques-
as teachers, since 97% of them agreed, "Ac- tionnaire data, rather than argue with them
countability as practiced in Kalamazoo about the meaningfulness of teacher re-
places too much emphasis on things that can sponses to the standardized items, we asked
be quantified so that it misses the results of them to turn to the pages of open-ended
teaching that are not easily measured." teacher comments and simply read at ran-
It is relatively clear from these statements dom what teachers said. Examples of the
that most teachers who responded to the comments they read, and could read on vir-
questionnaire were negative about the actually any page in the report, are reproduced
countability system. When school officials below in six representative responses from
and school board members reviewed the the middle pages of the report.
questionnaire results, however, many of
Teacher Response No. 284: "I don't feel that
them immediately dismissed those results
fear is necessary in an accountability situation.
by arguing that they had never expected
The person at the head of a school system has
teachers to like the system, teachers didn't
to be human, not a machine. You just don't
really want to be accountable, and the teach-
treat people like they are machines!
ers' unions had told their teachers to re-
"The superintendent used fear in this sys-
spond negatively anyway. In short, many
tem to get what he wanted. That's very hard to
school officials and school board members
explain in a short space. It's something you
dismissed the questionnaire results as bi-
have to live through to appreciate. He lied on
ased, inaccurate, and the results of teacher
many occasions and was very deceitful. Teach-
union leaders telling teachers how to re-
ers need a situation where they feel comfort-
spond in order to discredit the school au-
able. I'm not saying that accountability is not
thorities.
good. I am saying the one we have is lousy. It's
The same questionnaire included two hurting the studentsthe very ones we're
open-ended questions. The first was placed supposed to be working for."
midway through the questionnaire, and the
second came at the end of the questionnaire. Teacher Response No. 257: "This system is
creating an atmosphere of fear and intimida-
1. Please use this space to make any fur- tion. I can only speak for the school I am in, but
ther comments or recommendations people are tense, hostile and losing their hu-
concerning any component of the ac- manity. Gone is the good will and teain spirit
countability system. of administration and staff and I believe this
ali begins at the top. One can work in these teachers and administration and the Board of
conditions but why, if it is to 'shape up' a few Education ali had a good working relation-
poor teachers. Instead, it's having disastrous ship. In the past 4 yearsunder the present
results on the whole faculty community." superintendentI find the atmosphere dete-
rioratmg to the point where teachers distrust
Teacher Response No. 244: "In order to fully
each other and teachers do not trust adminis-
understand the oppressive, stifling atmo-
trators at ali! We understand the position the
sphere in Kalamazoo you have to ^be in the
administrators have been forced into and feel
trenches'the classrooms. In 10 years of
compassion for themhoweverwe still
teaching, 1 have never ended a school year as
have no trust! Going to school each morning is
depressed about 'education' as 1 have this
no longer an enjoyable experience."
year. lf things do not improve in the next two
years, I will leave education. The Kalamazoo
Teacher Response No. 261: "A teacher
accountability system must be viewed in its
needs some checks and balances to function
totality and not just the individual component
effectively; it would be ridiculous to think
parts of it. In to to, it is oppressive and stifling.
otherwiseif you are a concerned teacher. But
"In teaching government and history, stu-
in teaching you are not turning out neatly
dents often asked what it was like to live in a
packaged little mechanical products ali alike
dictatorship. I now know firsthand.
and endowed with the same qualities. This
"The superintendent with his accountabil-
nonsensical accountability program we have
ity model and his abrasive condescending
here makes the superintendent look good to
manner has managed in three short years to
the community. But someone who is in the
destroy teacher morale and effective creative
classroom dealing with ali types of kids, some
classroom teaching.
who cannot read, some who hardly ever come
"Last evening my wife and 1 went to an end
to school, some who are in and out of jail, this
of the school year party. The atmosphere there
teacher can see that and the rigid accountabil-
was strangelittle exuberance, laughter or re-
ity model that neglects the above mentioned
lease. People who in previous years laughed,
problems is pur 'BULLSHIT!' "
sang and danced were unnaturally quiet and
somber. Most people went home early. The
Teacher Response No. 251: " 'Fear' is the
key topic was the superintendent, the school
word for 'accountability' as applied in our sys-
board election, and a millage campaign. Peo-
tem. My teaching before 'Accountability' is
ple are still tense and uncertain.
the same as now. 'Accountability' is a political
"While the school board does not 'pay us to
ploy to maintain power. Whatever good there
be happy' it certainly must recognize that
may have been in it in the beginning has been
emotional stability is necessary for effective
destroyed by the awareness that each new ed-
teaching to take place. The involuntary trans-
ucational 'system' has at its base a political
fers, intimidation, coercion and top to bottom
motive. Students get screwed... . The bitter-
'channelized' communication in Kalamazoo
ness and hatred in our system is incredible.
must qualify this school system for the list of
What began as 'noble' has been destroyed.
'least desirable' school systems in the nation."
You wouldn'tbelieve the new layers of admin-
Teacher Response No. 233: "1 have taught in istration that have been created just to keep
Kalamazoo for 15 years and under five super- this monster going.
intendents. Until the present superintendent, I "Our finest compliment around our state is
found working conditions to be enjoyable and that the other school systems know what is go-
ing on and are having none of it. Lucky people.

Come down and visit in hell sometime."
T H n i i n H T I - i:: r f c = A : ; H n
Face Validity and Credibility V 'ii''..rr i ! . . !' J..I- : = ..!'! h-I S i.

sar i/.w!- i ui-Ji in:-! !Vi- =i ::!.!1
;
What was the impact of the qualitative :' : - i ! i V A ' i- ! ! I U ! i ! '.. i 11 i ! : i" ! i . i - ! i ! 11 .-. L-
data collected from teachers in Kalamazoo? i.-i!. v i : 1 ! * 1 1 i : : ij 5 "i:-!* .. >. !.; . i v : i r.-."i i . ! : i !: j . l \ i -i
You will recall that many of the school board V . ir,!. r i.'i. 1 ' ' I ! :. ! i:' 1 i : ' . ! : .'i' i ! = ! ' : . \\i'-1 i',
members initially dismissed the standard- l*r I l i - ! ' . ! | !.'. ii' i ::!= I ! I ' . ' y ! j M '
ized questionnaire responses as biased, : : ; . ' ! " ' < : !':/i.! . V i i . i i : i ' i i I M ! . - " i i . '
!
rigged, and the predictable result of the un- "iinkl ii!' MIv- ,I..!,:' \l ,\ iij- Ji, M " Vil1
ion's campaign to discredit school officials. I ! ! i ! m i: -|i- .V j.. .'! i' . r m Y ! r l : . i\ : i V I i/.'.'i I T
However, after readmg through a few pages Lnl lv > M -i i'T=" Vi }-, i . ! Vi' i i vl-I l'i '.Vi.'i !,=
of the teachers' own personal comments, af- i !.': !:.= .vil !1
i r r ! =" -."> s* 'ii.r Vi:. ii i 'i ! i'il' i.i \U
ter hearing about teachers' experiences with :!:- ! ri Jal-iiS i' }':-,* IRIRU .*=iJ-: !I .''
the accountability system in their own i -i i T ' N ! ! ! .T = T :; i. 1'Vi 1 ! N 1 ,',T! ' r
words, the tenor of the discussion about the Jt i" i '.ri :'i!' i l/i .'l i '||.
! ' 'i IV! 'i i! A I- i i' ! I VI i1 !.'
evaluation report changed. School board = != >!!' i-i-iiilrvi .:".'!! i'! ^vl' -i iVi-'
members could easily reject what they per- !!: nr-D-V, ''"" - w s ^ ir p * m' /-vy-'!^
ceived as a "loaded" questionnaire. They .v r it i;v! :
! ,'i -i 11; ;,yl i 'i ( ' . i 1 \ i ' ' 'i l' i h ) r
could not so easily dismiss the anguish, fear, H -Vlro" hr !Oi mi'i-rv hi - p r
and depth of concern revealed m the teach- i : "ii-3-:= :: N-V .*'
ers' own reflections. The teachers' words on|i !"i'i .!':;!>"! -.V.?- r l n j l l - i r i " '
had face validity and credibility. Discussion IlIllIllIlISiSSSSiSSiSSiSSSiiSSSIiSSSiS!
of the evaluation results shifted from an at-
tack on the measures used to the question:
"What do you think we should do?"
During the summer of 1976, following teachers' own words became part of the im-
discussion of the evaluation report, the su- petus for change in Kalamazoo.
perintender "resigned." The new superin-
tendent and school board in 1976-1977 used
The Purpose of
the evaluation report as a basis for starting
Open-Ended Responses
fresh with teachers. A year la ter teacher as-
sociation officials reported a new environ- The preceding example illus trates the dif-
ment of teacher-administration cooperation ference between qualitative inquiry based
in developing a mutually acceptable ac- on responses to open-ended questions and
countability system. The evaluation report quantitative measurement based on scales
did not directly cause these changes. Many composed of standardized questionnaire
other factors were involved in Kalamazoo at items. Quantitative measures are succinct,
that time. However, the qualitative informa- parsimonious, and easily aggregated for
tion in the evaluation report revealed the full analysis; quantitative data are systematic,
scope and nature of teachers' feelings about standardized, and easily presented in a
what it was like to work in the atmosphere short space. By contrast, the qualitative find-
created by the accountability system. The ings are longer, more detailed, and variable
depth of those feelings as expressed in the m content; analysis is difficult because re-
The Nature of Qualitative Inquiri/ [J. 2 1
sponses are neither systematic nor standard- the open-ended comments of the Kalama-
ized. Yet, the open-ended responses permit zoo teachers illustrate the fruit of qualitative
one to understand the world as seen by the methods.
respondents. The purpose of gathering re- While the Kalamazoo example illustrates
sponses to open-ended questions is to en- the most elementary form of qualitative in-
able the researcher to understand and cap- quiry, namely, responses from open-ended
ture the points of view of other people questionnaire items, the major way in which
without predetermining those points of qualitative researchers seek to understand
view through prior selection of question- the perceptions, feelings, and knowledge of
naire categories. As Lofland (1971) put it: people is through in-depth, intensive inter-
"To capture participants 'in their own terms' viewing. The chapter on interviewing will
one must learn their categories for rendering discuss ways of gathering high-quality in-
explicable and coherent the flux of raw real- forma tion from peopledata that reveal ex-
ity. That, indeed, is the first principie of qual- periences with program activities and per-
itative analysis" (p. 7, emphasis added). spectives on treatment impacts from the
Direct quotations are a basic source of raw points of view of participants, staff, and oth-
data in qualitative inquiry, revealing respon- ers involved in and knowledgeable about
dents' depth of emotion, the ways they have the program or treatment being evaluated.
organized their world, their thoughts about
what is happening, their experiences, and Inquiry by Observation
their basic perceptions. The task for the qual-
itative researcher is to provide a framework What people say is a major source of qual-
within which people can respond in a way itative data, whether what they say is ob-
that represents accurately and thoroughly tained verbally through an interview or in
their points of view about the world, or that written form through document analysis or
part of the world about which they are talk- survey responses. There are limitations,
ingfor example, their experience with a however, to how much can be learned from
particular program being evaluated. Too of- what people say. To understand fully the
ten social scientists "enter the field with pre- complexities of many situations, direct par-
conceptions that prevent them from allow- ticipation in and observation of the phenom-
ing those studied to 'tell it as they see it' " enon of interest may be the best research
(Denzin 1978b:10). method. Howard S. Becker, one of the lead-
I have included the Kalamazoo evalua- ing practitioners of qualitative methods in
tion findings as an illustration of qualita- the conduct of social science research, ar-
tive inquiry because open-ended responses gues that participant observation is the most
on questionnaires represent the most ele- comprehensive of ali types of research strat-
mentary form of qualitative data. There are egies.
severe limitations to open-ended data col-
lected in writing on questionnaires, limi- The most complete form of the sociological da-
tations related to the writing skills of re- tum, after ali, is the form in which the partici-
spondents, the impossibility of probing or pant observer gathers it: an observation of
extending responses, and the effort required some social event, the events which precede
of the person completing the questionnaire. and follow it, and explana tions of its meaning
Yet, even at this elementary levei of inquiry, by participants andspectators,before, during,
the depth and detail of feelings revealed in and after its occurrence. Such a datum gives us
Certain really discriminating people like nothing better than to relax on the
beach with a good, in-depth, and detailed qualitative study in hand.
moreinformation about the event under study Observational data, especially partici-
than data gathered by any other sociological pant observation, permit the evaluation re-
method. (Becker and Geer 1970:133) searcher to understand a program or treat-
ment to an extent not entirely possible using ents so that they could make conscious
only the insights of others obtained through choices about their own parenting styles and
interviews. Of course, not everything can increase their confidence about the choices
be directly observed or experienced, and they make. Parents were also to be treated
participant observation is a highly labor- with respect and to be recognized as the pri-
intensiveand, therefore, relatively expensive mary educators of their childrenin other
research strategy. In a later chapter, strate- words, the early childhood educators were
gies for using observational methods, in- not to impose their expertise upon parents
cluding both participant and nonparticipant but, instead, to make clear that parents are
approaches, will be discussed at length. My the real experts about their own children.
purpose at this point is simply to give the Site visits were made to ali programs, and
reader another taste of the fruit of qualitative parenting discussions were observed on
methods. Before discussing how to collect each site visit. Descriptions of these sessions
observational evaluation data, it is helpful to then became the primary data of the evalua-
know what such data should look like. tion. In short, the evaluators were to be the
The purpose of observational analysis is eyes and ears of the legislature and the state
to take the reader into the setting that was program staff, permitting them to under-
observed. This means that observational stand what was happening in various parent
data must have depth and detail. The data sessions throughout the state. Descriptive
must be descriptivesufficiently descrip- data about the sessions also provided a mir-
tive that the reader can understand what oc- ror for the staff who conducted those ses-
curred and how it occurred. The observer's sions, a way of looking at what they were do-
notes become the eyes, ears, and perceptual ing to see if that was what they wanted to be
senses for the reader. The descriptions must doing.
be factual, accurate, and thorough without What follows is a description from one
being cluttered by irrelevant minutiae and such session. The criterion that should be
trivia. The basic criterion to apply to a re- applied in reading this description is the ex-
corded observation is the extent to which the tent to which sufficient data are provided to
observation permits the reader to enter the take you, the reader, into the setting and per-
situation under study. mit you to make your own judgment about
The observation that follows is meant to the nature and quality of parent education
illustrate what such a descriptive account is being provided.
like. This evaluation excerpt describes a
two-hour observation of mothers discussing
their child rearing in a parent education pro- OBSERVATION DATA ILLUSTRATED:
gram. The purpose of the program, one of 22 A DISCUSSION FOR MOTHERS
such state-supported programs, was to in- OF TWO- YEAR-OLDS
crease the skills, knowledge, and confidence
of parents. The program was also aimed at The group discussion component of this
providing a support group for parents. In parent education program operates out of a
funding the program, legislators empha- small classroom in the basement of a church.
sized that they did not want parents to be The toddler center is directly overhead on
told how to rear their children. Rather, the the first floor so that noises made by the chil-
purpose of the parent education sessions dren these mothers have left upstairs can be
was to increase the options available to par- heard during the discussion. The room is
just large enough for the 12 mothers, one other mothers will talk about their own ex-
staff person, and me to sit along three sides periences as they want to. For example, one
of the room. The fourth side is used for a of the topics is the problem a mother is hav-
movie screen. Some mothers are smoking. ing with her child urinating in the bathtub.
(The staff person told me afterward that Other mothers share their experiences with
smoking had been negotiated and agreed on this problem, ways of handling it, and
among the mothers.) The seats are padded whether or not to be concerned about it. The
folding chairs plus two couches. A few col- crux of that discussion seems to be that it is
orful posters with pictures of children play- not a big deal and not something that the
ing decorate the walls. Small tables are avail- mother ought to be terribly concerned
able for holding coffee cups and ashtrays about. It is important not to make it a big
during the discussion. The back wall is lined deal for the child; the child will outgrow it.
with brochures on child care and child de- The discussion turns to things that
velopment, and a metal cabinet in the room two-year-olds can do around the house to
holds additional program materiais. help their mothers. This is followed by some
The session begins with mothers watch- discussion of the things that two-year-olds
ing a 20-minute film about preschool chil- can't do and some of their frustrations in try-
dren. The film forms the basis for getting dis- ing to do things. There is a good deal of
cussion started about "what two-year-olds laughing, sharing of funny stories about
do." Louise, a part-time staff person in her children, and sharing of frustrations about
early 30s who has two young children of her children. The atmosphere is informal and
own, one a two-year-old, leads the discus- there is a good deal of intensity in listening.
sion. Louise asks the mothers to begin by Mothers seem especially to pick up on
picking out from the film. things that their things that they share in common about the
own children do, and talking about the way problems they have with their children.
some of the problems with children were Another issue from another mother is the
handled in the film. For the most part, the problem of her child pouring out her milk.
mothers share happy, play activities their She asks, "What does it mean?" This ques-
children like. "My Johnny loves the play- tion elicits some suggestions about using
ground just like the kids in the film." "Yeah, water aprons and cups that don't spill and
mine could live on the playground." other mothers' similar problems, but the dis-
The focus of the discussion turns quickly cussion is not focused and does not really
to what happens as children grow older, come to much closure. The water apron sug-
how they change and develop. Louise com- gestion brings up a question about whether
ments, "Don't worry about what kids do at a or not a plastic bag is okay. The discussion
particular age. Like don't worry that your turns to the safety problems with different
kid has to do a certain thing at age two or else kinds of plastic bags. About 20 minutes of
he's behind in development or ahead of de- discussion have now taken place. (At this
velopment. There's just a lot of variation in point, one mother leaves because she hears
the ages at which kids do things." her child crying upstairs.)
The discussion is free flowing and, once The discussion returns to giving chil-
begun, is not directed much by Louise. dren baths. Louise interjects, "Two-year-
Mothers talk back and forth to each other, olds should not be left alone in the bathtub."
sharing experiences about their children. A With reference to the earlier discussion
mother will bring up a particular point and about urinating in the bathtub, a mother in-
The Nature of Qualitative Inquiri/ [J. 2 5
teijects that water with urine in it is prob- the 11 mothers have participated, most of
ably better than the lake water her kids swim them actively. Four mothers have not partic-
in. The rnother with the child who urinates ipated.)
in the bathtub says again, "It really bugs me Another mother brings up a new prob-
when he urinates in the tub." Louise re- lem. Her child is destroying her plants,
sponds, "It really is your problem, not his. dumping plants out, and tearing them up.
If you can calm yourself down, he'll be "I really get mad." She says that the tech-
okay." nique she has used for punishment is to iso-
At a lull in the discussion, Louise asks, late the child. Then she asks, "How long do
"Did you agree with everything in the you have to punish a two-year-old before it
movie?" The mothers talk a bit about this starts working?" This question is followed
and focus on an incident in the movie where by intense discussion with several mothers
one child bites another. Mothers share sto- making comments. (This discussion is re-
ries about problems they've had with their produced in full to illustrate the type of dis-
childrenbiting. Louise inteijects, "Biting can cussion that occurred.)
be dangerous. It is important to do some-
thing about biting." The discussion turns to Mother No. 2: "Maybe he needs his own
what to do. One mother suggests biting the plant. Sometimes it helps to let a child
child back. Another mother suggests that have his own plant to take care of and
kids will work it out themselves by biting then he comes to appreciate plants."
each other back. Mothers get very agitated,
Mother No. 3: "Maybe he likes to play in the
more than one mother talks at a time. Louise
dirt. Does he have his own sand or dirt to
asks them to "cool it," so that only one per-
play in around the house?"
son talks at a time. (The mother who had left
returns.) Mother No. 4: "Oatmeal is another good
The discussion about biting leads to a dis- thing to play in."
cussion about child conflict and fighting in Louise: "Rice is another thing that children
general, for example, the problem of chil- like to play in and it's clean, good to use
dren hitting each other or hitting their moth- indoors."
ers. Again, the question arises about what to
do. One mother suggests that when her child Mother No. 5: "Some things to play in would
hits her, she hits him back, or when her child be bad or dangerous. For example, pow-
bites her, she bites him back. Louise inter- dered soap isn't a good thing to let kids
jects, "Don't modelbehavior you don't like." play in."
She goes on to expiam that her philosophy is Mother No. 2: "Can you put the plants where
that you should not do things as a model for he can't get at them?"
children that you don't want them to do. She
says that works best for her; however, other Mother with problem: "I have too many plants,
mothers may find other things that work I can't put them ali out of the way."
better for them. Louise comments that hit- Louise: "Can you put the plants somewhere
ting back or biting back is a technique sug- else or provide a place to play with dirt or
gested by Dreikurs. She says she disagrees rice?" (Mother with problem kind of
with that technique, "but you ali have to de- shakes her head no. Louise goes on.) "An-
cide what works for you." (About 40 min- other thing is to tell the kid the plants are
utes have now passed since the film, and 7 of alive, to help him learn respect for living
things. Tell him that those plants are alive time. Several mothers give their points of
and that it hurts them. Give him his own view.
plant that he can get an investment in." Louise: "The person who owns the house
sets the rales. Two-year-olds can learn to be
Mother with problem: "IT1 try it."
careful. But don't go around ali day long
Mother No. 2: "You've got to be fair about a saying, "No, no.' "
two-year-old. You can't expect them not
to touch things. It's not fair. I try hanging The time had come for the discussion to
ali my plants." end. The mothers stayed around for about 15
minutes, interacting informally and then go-
Louise: "Sometimes just moving a child
ing upstairs to get their children into their
bodily away from the thing you don't
winter coats and hats for the trip home. They
want him to do is the best technique."
seemed to have enjoyed themselves and
Mother No. 4: "They'11 outgrow it anyway." continue d talking informally. One mother
with whom Louise had disagreed about the
Mother with problem: "Now he deliberately issue of whether it was ali right to bite or hit
dumps them and I really get angry." children back stopped to continue the dis-
Louise: "Maybe he feels a rivalry with the cussion. Louise said:
plants if you have so many. Maybe he's
trying to compete." I hope you know that I respect your right to
have your own views on things. I wasn't try-
Mother No. 3: "Let him help with the plants.
ing to tell you what to do. I just disagreed, but I
Do you ever let him help you take care of
definitely feel that everybody has a right to
the plants?"
their own opinion. Part of the purpose of the
Mother No. 6: "Some plants are dangerous to group is for everyone to be able to come to-
help with." gether and appreciate other points of view and
understand what works for different people.
Louise: "Some dangerous house plants are
poison."
The mother said that she certainly didn't
feel bad about the disagreement and she
Louise reaches up and pulls down a bro- knew that some things that worked for other
chure on plants that are dangerous and says people didn't work for her and that she had
she has brochures for everyone. Several peo- her own ways but that she really enjoyed the
ple say that they want brochures and she group.
goes to the cabinet to make them available. Louise cleaned up the room, and the ses-
One mother who has not participated version ended.
bally up to this point specifically requests a
brochure. This is followed by a discussion of
child-proofing a house as a method of child The Raw Data of
rearing versus training the child not to touch Qualitative Inquiry
things, but with less emphasis on child-
proofing, that is, removing temptation ver- The description of this parenting session
sus teaching children to resist temptation. is aimed at permitting the reader to under-
One parent suggests, in this context, that stand what occurred in the session. These
children be taught one valuable thing at a data are descriptive. Pure description and
m a w i n L m a i L ^ E S : . q u a o t k &s w e l i h d s c or o m M
i.' v I : V;;-;1.".'!" v i y i i v r YXÛa rr: r ! . ; i ! i r=,V.V:. n-.f ' i'.v:! vi Vi -i!i!i-j'! ! ii j-- -V
:!!.= ':; i":;-'Vi PTO"! !"" :>"11'! ,;.=:::'!; rw: 1 ? r- ! 'T'-!.!"*
!
: = rv| V- - , v
:".'!! :V: v.ifi'V! ! V! !| jVi!! l'j'i:".":i" " l"l!"| jVi " i= h=' i.M-yLi i.V \:biY. i r!Jri j!i Vili y.if rrifia'.Own
u M ! ! ' ! | ! = : : ! * i i " V | : : ! : ! | ! ! = : : i > i-1 V i ! ; . ; :i:T:I.I::" [rwyf
| "'! '.-] 11 "Ti V !! Vi .V! ! v.- r Vi :i i i i' ::v,'i rs.!
i,=vi..: li
: !!=, ir.M r. :;=:m:i: rt r ="i ! i i." :! SlSllSISSSSSlllilIillilIIillIllIll
! ' :,; \m .'h- v; ;= 1 I! |.-i! >:!iVi
;; '
!"'!!:Yi :;y r r - ! m= : : y : : r . ': r - : i ; . - ' i u - m ,
>-i i'i1:;i:i,:|:'i:!i.i!'n =v'..vi!':i"i.i:'-:i.:!.'::! i.V::=-S.!
!! n:v!. n.;; rw;: !=.: iV: :;=.! :; = ! i ; v S ivy-..i i i: h ,'i
M iV!1- : -|.:"s 3.!.P:V \ i- UK-S '! ii-i:'"'-i:!'1 i!!!
;!.! !==: i i v : - : !: l v !.n'V i " - : ! 1j - d =:.==.!
! !
li/' i'i!.bi:'r:"
!
: r' ==::-!: -:T VT' y^îi i'

:::;!'i'!'-^"''!:'::-.- rp.SPJi.M " ! ! " M i 1 ;!.!;,:: i^M-iVi n.;;T! j,'! jV i.!:'" >:U 'Vi jJ"i -
:;!i:v::::yi!V!v;j;= !!.;:!: .'|V! : i? w n"-" .s'! :;!'=:: 5.--I=: J- iniratim I ' =V;I!l-5 i s;="="s !'i v:i'iyO U
:;::. i J: zrvkor* .^vir niikip -r-i' i1! il1:'i Vi lVV!:'! S!' /il-.r'!! M Vti? :ii! -.Vin V
; v i : ! .! : ;. =;=-.! : n \ >: r : i f y m -'n l!! r . m v :;=!:, u n .'i-i:!!!;!!: iWiVii1 .vi nu ^.'iiVni! i:M:I.! r-i
ii:11 i-1! .';!;!: 'I! J; =:!: f:!.;: sT!i.!:ij:i VJ;: ^.jivi;' hi;Vi'!;'i 5V; .'!!'"ii.;r !ri;v"
zaxrw, 00 r nrn -i:; ,-cv 1 - i : ; ! a ! v ; V ! :i !=
: ihMi i r : ! i ' ! i ! ! i j k U ! ' ' ! ! Z iJ,:!:!'! i-i
!
i.'i i j i" 1 v ! 1V! !i'ij.'i ". .. m i i K' !.;i.V I i.',1' iV,''i

V!; i v[ i." i " i:! t<i1.? . i.!iVii-ii=: iV:"i:-i (ivu/irrr- rffr, jrirr!";!I1 !V!;;;;'iV-jj! rc!IM;j-vv1 ' i'
Ll:,'i!'.Vi :i.:!! a! !! i l 'V:!-: iVfti:. f ,':i .ti :!'!! i! W !=:!'!!f'M'i'=. iÎj,L-I,.: j.|iHTV =V
quotations are the raw data of qualitative in- sented as separa te and distinct from each
quiry. other. In practice, they are often fully inte-
The description is meant to take the grated approaches. Becoming a skilled ob-
reader into the setting. The data do not in- server is essential even if you concentrate
clude judgments about whether what oc- primarily on interviewing because every
curred was good or bad, appropriate or in- face-to-face interview also involves and re-
appropriate, or any other interpretive quires observation. The skilled interviewer
judgments. The data simply describe what is thus also a skilled observer, able to read
occurred. State legislators, program staff, nonverbal messages, sensitive to how the in-
parents, and others used this description, terview setting can affect what is said, and
and descriptions like this from other pro- carefully attuned to the nuances of the inter-
gram sites, to discuss what they wanted the viewer-interviewee interaction and relation-
programs to be and do. The descriptions ship.
helped them make explicit their own judg- Likewise, interviewing skills are essential
mental criteria. for the observer because during fieldwork,
In later chapters, guidance on interpret- you will need and want to talk with people,
ing qualitative data will be offered in depth. whether formally or informally. Participant
observers gather a great deal of information
People-Oriented Inquiry through informal, naturally occurring con-
versations. Understanding that interview-
Thus far, the examples of observation and ing and observation are mutually reinforc-
interviewing in this chapter have been pre- ing qualitative techniques is a bridge to
understanding the fundamentally people- The Fruit of Qualitative

oriented nature of qualitative inquiry. Methods Revisited
Sociologist John Lofland has suggested
that there are four people-oriented man- This chapter began with the parable of the
dates in collecting qualitative data. First, the man who traveled far in search of a widely
qualitative methodologist must get close proclaimed food called "fruit." When finally
enough to the people and situation being directed to a fruit tree, he confused the
studied to personally understand in depth spring blossom of the tree with the fruit of
the details of what goes on. Second, the qual- the tree. Finding the blossom to be tasteless,
itative methodologist must aim at capturing he dismissed ali he had heard about fruit as a
what actually takes place and what people hoax and went on his way. This chapter has
actually say: the perceived facts. Third, qual- described qualitative data so that the person
itative data must include a great deal of pure in search of the fruits of qualitative methods
description of people, activities, interac- will know what to look forand know
tions, and settings. Fourth, qualitative data when the real thing has been attained. Ex-
must include direct quotations from people, hibit 1.5 lists Internet resources for those
both what they speak and what they write who want to carry on this search for qualita-
down. tive fruit in virtual space. To close this chap-
ter, it may be instructive to consider two
other short parables about the search for
The commitment to get close, to be factual, de- fruit.
scriptive and quotive, constitutes a significant While the first seeker after fruit arrived
commitment to represent the participants in too early to experience the ripened delicacy
their own terms. This does not mean that one and tasted only the blossom, a second seeker
becomes an apologist for them, but rather that after fruit arrived at a tree that had been im-
one faithfully depicts what goes on in their properly cultivated, so that its fruit was
lives and what life is like for them, in such a shriveled and bitter. This bad fruit had been
way that one's audience is at least partially left to rot. Not knowing what good fruit
able to project themselves into the point of looked like, he sampled the bad. "Well, I've
view of the people depicted. They can "take seen and tasted fruit," he said, "and I can tell
the role of the other" because the reprter has you for sure that it's terrible. I've had it with
given them a living sense of day-to-day talk, fruit. Forget it. This stuff is awful." He went
day-to-day activities, day-to-day concerns on his way and his journey was wasted.
and p r o b l e m s , . . . One can hope that such a foolish mis take
A major methodological consequence of is less likely today, because early in school
these commitments is that the qualitative students are taught the danger of generaliz-
study of people in situ is a process of discovery. It ing from limited cases. Yet, rumors persist
is of necessity a process of learning what is that some people continue to reject ali quali-
happening. Since a major part of what is hap- tative data as worthless (and "rotten"), hav-
pening is provided by people in their own ing experienced only bad samples produced
terms, one must find out about those terms with poor methods.
rather than impose upon them a preconceived A third seeker after fruit arrived at the
or outsider's scheme of what they are about. It same tree that produced the shriveled and
is the observer's task to find out what is funda- bitter fruit. He picked some of the rotting
mental or central to the people or world under fruit and examined it. He took the fruit to a
observation. (Lofland 1971:4) farmer who cultivated fruit trees with great
Internet E-mail Discussion Groups (listservs) on

Qualitative Methods
1. [email protected]: Qualitative Research for the Human Sciences; to subscribe,

send this message to [email protected]: subscribe QUALRS-L yourname
2. [email protected]: Qualitative Research in Management and Organization Studies;

to subscribe, send this message to [email protected]: subscribe qualnet
3. [email protected]: Qualitative Research List, initiated by Penn State, but immediately

attracted a broader audience; to subscribe, send this message to [email protected]:
subscribe QUAL-L firstname lastname
Other resources for qualitative evaluation and research:
4. [email protected]: American Evaluation Association (AEA) Discussion List; to

subscribe, send this message to [email protected]: subscribe evaltalk ourname
AEA home page with links to evaluation organizations, training programs, and
Internet resources: www.eval.org
5. [email protected]: A list for social science research methods instructors; to subscribe,

send this message to [email protected]: join methods yourname
NOTE: Thanks to Judith Preisse, Aderhold Distinguished Professor, Social Foundationsof Education, University
of Gergia, for list subscription details. These sites and subscription details may change, and this list is not ex-
haustive. This list is meant to be suggestive of the qualitative resources availabie through the Internet. See
Chapter3, Exhibit 3.7;Chapter4, Exhibit 4.9; and Chapter8, Exhibit 8.3,for addtional, morespecialized quali-
tative resources through the Internet.
success. The farmer peeled away the rotten the stonelike thing he held in his hand was a
exterior and exposed what looked like a seed, ali he had to do was plant it, tend prop-
stone inside. The farmer told him how to erly the tree's growth, and work for the
plant this hard core, cultivate the resulting eventual harvestthe fruit. Though there
trees, and harvest the desired delicacy. The was much work to be done and there were
farmer also gave him a plump, ripe sample many things to be leamed, the resulting
to taste. Once the seeker after fruit knew high-quality fruit was worth the effort.
what fruit really was, and once he knew that
Between-Chapters Interlude
!
i
Top Ten Pieces of Advice to a
Graduate Student Considering
a Qualitative Dissertation
3. Top Ten Responses
T he following query was posted on

an Internet listserv devoted to dis-
cussing qualitative inquiry:
I am a new graduate student thinking about

1. Be sure that a qualitative approach fits
your research questions: questions about
doing a qualitative dissertation. I know you
people's experiences; inquiry into the
are ali busy, but I would appreciate an an-
meanings people make of their expe-
swer to only one question.
riences; studying a person in the con-
If you could give just one bit of advice to a
text of her or his social/interpersonal
student considering qualitative research for a
environment; and research where not
dissertation, what would it be?
enough is known about a phenomenon
for standardized instruments to have
The responses below carne from differ- been developed (or even to be ready to
ent people. I've combined some responses, be developed).
edited them (while trying to maintain the (Chapter 2 will help with this by
flavor of the postings), and arranged them presenting the primary themes of qual-
for coherence. itative inquiry.)
!J. 33
2. Study qualitative research. There are 5. Practice interviewing and observation

lots of different approaches and a lot to skills. Practice! Practice! Practice! Do
know. Study carefully a couple of the lots of intervie ws. Spend a lot of time do-
books that provide an overview of dif- ing practice fieldwork observations. Get
ferent approaches, then go to the origi- feedback from someone who's really
nal sources for the design and analysis good at interviewing and observations.
details of the approach you decide to There's an amazing amount to learn.
use. And it's not just head stuff. Qualitative
(Chapter 3 covers different qualita- research takes skill. Don't make the
tive approaches.) mis take of thinking it's easy. The better I
get at it, the more I realize how bad I was
3. Find a dissertation adviser who will when I started.
support your doing qualitative re- (Chapters 6 and 7 cover the skills of
search. Otherwise, it can be a long, qualitative inquiry.)
tough haul. A dissertation is a big com-
mitment. There are other practical ap- 6. Figure out analysis before you gather
proaches to using qualitative methods data. I've talked with lots of advanced
that don't involve ali the constraints of grad students who rushed to collect
doing a dissertation, things like pro- data before they knew anything about
gram evaluation, action research, and analyzing itand lived to regret it, big
organizational development. You can time. This is true for statistical data and
still do lots of great qualitative work quantitative data, but somehow people
without doing a dissertation. But if you seem to think that qualitative data are
can find a supportive adviser and com- easy to analyze. No way. That's a big-
mittee, then, by ali means, go for it. time NO WAY. And don't think that the
(Chapter 4 covers particularly ap- new software will solve the problem.
propriate practical applications of quali- Another big-time NO WAY. You, that's
tative methods.) YOU, still have to analyze the data.
(Chapter 8 covers analysis.)
4. Really work on design. Qualitative de-
signs follow a completely different logic 7. Be sure that you're prepared to deal
from quantitative research. Completely with the controversies of doing qualita-
different. Are you listening? Com- tive research. People on this listserv are
pletely different. Especially sampling. constantly sharing stories about people
This is not the same as questionnaires who don't "get" qualitative research
and tests and experiments. You can and put it down. Don't go into it naively.
combine designs, like quant and qual Understand the paradigms and politics.
approaches, but that gets really compli- (Chapter 9 deals with paradigms,
cated. Either way, you have to figure out politics, and ways of enhancing the
what's unique about qualitative de- credibility of qualitative inquiry.)
signs.
(Chapter 5 covers qualitative de- 8. Do it because you want to and are con-
signs.) vinced it's right for you. Don't do it be-
Top Ten Pieces ofAdvice !}. 35
cause someone told you it would be storm, and problem solve, as well as
easier. It's not. Try as hard as possible share in each other 's successes, ali in a
to pick/negotiate dissertation research more relaxed environment that helps
questions that have to do with some take some of the edge off the stress
passion/interest in your professional (for example, you might have potluck
life. Qualitative research is time-con- meals at different homes?). This can be
suming, intima te, and intenseyou will tremendously liberating (even on a less
need to find your questions interesting than regular basis). Take care of your-
if you want to be at ali sane during the self.
processand still sane at the end.
10. Prepare to be changed. Looking deeply
9. Find a good mentor or support group. at other people's lives will force you to
Or both. In fact, find several of each. If look deeply at yourself.
you can, start a small group of peers in (See the discussions "Voice, Perspec-
the same boat, so to speak, to talk about tive, and Reflexivity" in Chapter 2 and
your research together on a regular "The Observer and What Is Observed:
basisyou can share knowledge, brain- Unity and Separation" in Chapter 6.)
Strategic Themes in
Qualitative Inquiry
rand strategy should guide tactical decisions. Within a grand strat-

egy ali manner of tactical errors may be made, and indeed, are inevi-
table, but can be corrected as long as the strategic vision remains true and fo-
cused. At least that's the theory. In practice . . . ? Try it and see.
Halcolm
eneâl T-Vinciples
Strategos is a Greek word meaning "the thinking and action of a general."

What it means to be strategic is epitomized by that greatest of Greek generais,
Alexander. He conducted his first independent military operation in northern
Macedonia at age 16. He became the ruler of Macedonia after his father, Philip,
was assassinated in 336 B.C. Two years later, he embarked on an invasion of Pr-
sia and conquest of the known world. In the Battle of Arbela, he decisively de-
feated Darius III, King of Kings of the Persian Empire, despite being outnum-
bered 5 to 1 (250,000 Persians against Alexander and fewer than 50,000 Greeks).
Alexander's military conquests are legend. What is less known and little ap-
preciated is that his battlefield victories depended on in-depth knowledge of the
I psychology and culture of the ordinary people and military leaders in opposing
I armies. He included in his military intelligence information about the beliefs,
| worldview, motivations, and patterns of behavior of those he faced. Moreover,
his conquests and subsequent rule were more economic and political in nature
than military. He used what we would now understand to be psychological, so-
ciological, and anthropological insights. He understood that lasting victory de-
pended on the goodwill of and alliances with non-Greek peoples. He carefully
studied the customs and conditions of people he conquered and adapted his
policiespolitically, econornically, and culturallyto promote good conditions
in each locale so that the people were reasonably well-disposed toward his rule
(Garcia 1984).
In this approach, Alexander had to overcome the arrogance and ethnocen-
trism of his own training, culture, and Greek philosophy. Historian C. A. Robin-
son, Jr. explained that Alexander was brought up in Plato's theory that ali
non-Greeks were barbarians, enemies of the Greeks by nature, and Aristotle
taught that ali barbarians (non-Greeks) were slaves by nature. But
Alexander had been able to test the smugness of the Greeks by actual contact with
the barbarians,. . . and experience had apparently convinced him of the essential
sameness of ali people. (Robinson 1949:136)
In addition to being a great general and enlightened ruler, Alexander appears

to have been an extraordinary ethnographer, a qualitative inquirer par excel-
lence, using observa tions and firsthand experience to systematically study and
understand the peoples he encountered and to challenge his own culture's prej-
udices.
And as Halcolm finished telling the story of Alexander the Great, he re-
minded those assembled that skills in observation and interviewing are life
skills for experiencing the world. "One can say of qualitative inquiry what Mar-
eei Proust said of art, Thanks to this, instead of seeing one world, our own, we
see it multiplied. So many worlds are at our disposal.' "
From Halcolm's Historical Biographies
The Purpose of a Strategic Framework
i P;
a ^ s erception is strong and sight weak. In strategy it is important to see
distant things as if they were close and to take a distanced view of
close things.
Miyamoto Musashi (1584-1645),

Japanese warrior, strategist
Strategic Themes in Qualitative Inquiri/ J. 39
D on't mistake a clear view for a short distance.
Grand Canyon hiking advice
verybody has a plan until they've been hit.
Old boxing saying
A well-conceived strategy, by providing these researchers have in common? They are

overall direction, provides a framework for in the field studying the real world as it un-
decision making and action. It permits folds.
seemingly isolated tasks and activities to fit Qualitative designs are naturalistic to the
together, integra ting separate efforts toward extent that the research takes place in real-
a common purpose. Specific study design world settings and the researcher does not
and methods decisions are best made within attempt to manipulate the phenomenon of
an overall strategic framework. This chap- interest (e.g., a group, event, program, com-
ter offers 12 major themes or principies of munity, relationship, or interaction). The
qualitative inquiry that, taken together, con- phenomenon of interest unfolds naturally in
stiitute a comprehensive and coherent stra- that it has no predetermined course estab-
tegic framework for qualitative inquiry, in- lished by and for the researcher such as
cluding fundamental assumptions and would occur in a labor ator y or other con-
epistemological ideais. Exhibit 2.1 summa- trolled setting. Observations take place in
rizes those themes in three basic categories: real-world settings and people are inter-
design strategies, data collection and field- viewed with open-ended questions in places
work strategies, and analysis strategies. and under conditions that are comfortable
for and familiar to them.
Egon Guba (1978), in his classic treatise on
Design Strategies for naturalistic inquiry, identified two dimen-
Qualitative Inquiry sions along which types of scientific inquiry
can be described: (1) the extent to which the
Naturalistic Inquiry scientist manipulates some phenomenon in
advance in order to study it and (2) the ex-
An anthropologist studies initiation rites tent to which constraints are placed on out-
among the Gourma people of Burkina Faso puts, that is, the extent to which predeter-
in West frica. A sociologist observes inter- mined categories or variables are used to
actions among bowlers in their weekly describe the phenomenon under study. He
league games. An evaluator participates then defined "naturalistic inquiry" as a
fully in a leadership training program she is "discovery-oriented" approach that mini-
documenting. A naturalist studies bighorn mizes investigator manipulation of the
sheep beneath Powell Plateau in the Grand study setting and places no prior constraints
Canyon. A policy analyst interviews people on what the outcomes of the research will be.
living in public housing in their homes. An Naturalistic inquiry contrasts with con-
agronomist observes farmers' spring plant- trolled experimental designs where, ideally,
ing practices in rural Minnesota. What do the investigator controls study conditions
Themes of Qualitative Inquiry
Desiqn Strategies
1. Naturaistic inquiry Studying real-world situations as they unfold naturalfy;
nonmanipulative and noncontrolling; openness to whatever
emerges (lack of predetermined constraints on findings).
2. Emergent design Openness to adapting inquiry as understanding deepens and/or
flexibility situations change; the researcher avoids getting locked into rigid
designs that eliminate responsiveness and pursues new paths of
discovery as they emerge.
3. Purposeful sampling Cases for study (e.g., people, organizations, communities,
cultures, events, criticai incidences) are selected because
they are "information rich" and illuminative, that is, they offer
useful manifestations of the phenomenon of interest; sampling,
then, is aimed at insight about the phenomenon, not empirical
generalization from a sample to a popuation.
Data Coilection and Fieldwork Strategies

4. Qualitative data Observations that yield detailed, thick description; inquiry in
depth; interviews that capture direct quotations about people's
personal perspectives and experiences; case studies; careful
document review.
5. Personal experience The researcher has direct contact with and gets close to the
and engagement people, situation, and phenomenon under study; the researcher's
personal experiences and insights are an important part of
the inquiry and criticai to understanding the phenomenon.
6. Empathic neutrality An empathic stance in interviewing seeks vicarious under-
and mindfulness standing without judgment (neutrality) by showing openness,
sensitivity, respect, awareness, and responsiveness; in
observation it means being fully present (mindfulness).
7. Dynamic systems Attention to process; assumes change as ongoing whether
focus is on an individual, an organization, a community, or an
entire culture; therefore, mindful of and attentive to system
and situation dynamics.
by manipulating, changing, or holding con- ence between asking, "Tell me about your
stant externai influences and where a very experience in the program" and "How satis-
limited set of outcome variables is mea- fied were you? Very, somewhat, little, not at
sured. Open- ended, conversation-like inter- ali."
views as a form of naturalistic inquiry con- In the simplest form of controlled experi-
trast with questionnaires that have prede- mental inquiry the researcher enters the
termined response categories. It's the differ- program at two points in time, pretest and
Anaiysis Strateqies
8. Unique case Assumes each case is special and unique; the first levei of
orientation analysis is being true to, respecting, and capturing the detas
of the individual cases being studied; cross-case analysis
follows from and depends on the quality of individual case
studies.
9. nductive analysis Immersion in the details and specifics of the data to discover
and creative synthesis important patterns, themes, and interrelationships; begins by
exploring, then confirming; guided by analytical principies
rather than ruies; ends with a creative synthesis.
10. Holistic perspective The whole phenomenon under study is understood as a complex
system that is more than the sum of its parts; focus on complex
interdependencies and system dynamics that cannot meaning-
fully be reduced to a few discrete variables and linear, cause-
effect relationships.
11. Context sensitivity Places findings in a social, historical, and temporal context;
careful about, even dubous of, the possibility or meaningful-
ness of generalizations across time and space; emphasizes
instead careful comparative case anayses and extrapolating
patterns for possible transferability and adaptation in new
settings.
12. Voice, perspective, The qualitative analyst owns and is reflective about her or his
and reflexivity own voice and perspective; a credible voice conveys authentic-
ity and trustworthiness; complete objectivity being impossible
and pure subjectivty undermtning crediblity, the researcher's
focus becomes balanceunderstanding and depicting the world
authenticaliy in ali its complexity while being self-analytical,
politically aware, and reflexive in consciousness.
posttest, and compares the treatment group treatment remains relatively constant and
to some control group on a limited set of unchanging.
standardized measures. Such designs as- While there are some narrow, carefully
sume a single, identifiable, isolated, and controlled, and standardized treatments
measurable treatment. Moreover, such de- that fit this description, in practice human
signs assume that, once introduced, the interventions (programs) are often quite
comprehensive, variable, and dynamic
changing as practitioners learn what does ments can involve comparing two groups,
and does not work, developing new ap- one of which experiences some change
proaches and realigning priorities. This, of while the other doesn't. What makes this
course, creates considerable difficulty for naturalistic inquiry is that real-world partic-
controlled experimental designs that need ipants direct the change, not the researcher,
specifiable, unchanging treatments to relate as in the laboratory.
to specifiable, predetermined outcomes. However, the distinction is not as simple
Controlled experimental evaluation designs as being in the field versus being in the labo-
work best when it is possible to limit pro- ratory; rather, the degree to which a design is
gram adaptation and improvement so as not naturalistic falls along a continuum with
to interfere with the rigor of the research de- completely open fieldwork on one end and
sign. completely controlled laboratory control on
By contrast, under real-world conditions the other end, but with varying degrees of
where programs are subject to change and researcher control and manipulation be-
redirection, naturalistic inquiry replaces the tween these end points. For example, the
fixed treatment/outcome emphasis of the very presence of the researcher, asking ques-
controlled experiment with a dynamic, pro- tions, or as in the case of formative program
cess orientation that documents actual oper- evaluation, providing feedback, can be an
ations and impacts of a process, program, or intervention that reduces the natural un-
intervention over a period of time. The eval- folding of events. Unobtrusive observations
uator sets out to understand and document are needed as an inquiry strategy when the
the day-to-day reality of participants in the inquirer wants to minimize data collection
program, making no attempt to manipulate, as an intervention. Nor are laboratory condi-
control, or eliminate situational variables or tions found only in buildings. Field experi-
program developments, but accepting the ments are common in agriculture where re-
complexity of a changing program reality. searchers want to introduce a considerable
The data of the evaluation include whatever amount of control, reduce variation in extra-
emerges as important to understanding neous variables, and focus on a limited set of
what participants experience. predetermined measures, as in crop fertil-
Natural experiments occur when the ob- izer studies.
server is present during a real-world change Let me conclude this discussion of natu-
to document a phenomenon before and after ralistic inquiry with two examples to illus-
the change. Durrenberger and Erem (1999) trate variations in this design strategy. In
documented "a natural experiment in evaluating a wilderness-based leadership
thought and structure" when, because of a training program, I participated fully in the
change at a hospital they were studying, 10-day wilderness experience, guided in my
they were able to contrast two different observations by nothing more than the sen-
structures of leadership in a union worksite. sitizing concept "leadership." The only "un-
They had already documented the degree natural" elements of my participation were
and nature of "union consciousness" before that (1) everyone knew I was taking notes to
the change, so by repeating their observa- document what happened and (2) at the end
tions after the change in a hospital structure, of each day I conducted open-ended, con-
they were able to. take advantage of a natu- versational interviews with staff. While this
rally occurring experiment. Natural experi- constitutes a relatively pure naturalistic in-
Strategic Themes in Qualitative Inquiri/ J. 4 3
<u
"4<1d
quiry strategy, my presence, note taking, tivities. For 736 consecutive nights, Strass-
and interviews must be presumed to have mann kept track of ali the women who used
altered somewhat the way the program un- the hut. This allowed her to collect statis-
folded. I know, for example, that the debrief- tics on the frequency and length of menstru-
ing questions I asked staff in the evenings ation among the Dogon women, but with a
got them thinking about things they were completely naturalistic inquiry strategy, il-
doing that led to some changes along the lustrating how both quantitative and quali-
way in how they conducted the training. tative data can be collected within a natural-
The second example comes from the istic design strategy. There's no reason to
fieldwork of Beverly Strassmann among the believe that her presence over this long pe-
Dogon people in the village of Sangui in the riod changed the women's menstruation
Sahel, about 120 miles south of Tombouctou patterns.
in Mali, West frica (Gladwell 2000). Her
study focused on the Dogon tradition of
Emergent Design Flexibility
having menstruating women stay in small,
segregated adobe huts at the edge of the vil-
lage. She observed the comings and goings In the wilderness leadership training pro-
of these women and obtained urine samples gram I evaluated, halfway through the
from them to be sure they were menstruat- 10-day experience the group I was with un-
ing. The women only slept in the huts. Dur- expectedly split into two subgroups. I had to
ing the day, they went about their normal ac- make an in-the-field, on-the-spot decision
about which group to follow and how to get search. How will they know what will result
interviews with the others at a later time. from the inquiry if the design is only par-
Naturalistic inquiry designs cannot usu- tially specified? The answer is: They won't
ally be completely specified in advance of know with any certainty. Ali they can do is
fieldwork. While the design will specify an look at the results of similar qualitative m-
initial focus, plans for observations, and ini- quiries, inspect the reasonableness of the
tial guiding interview questions, the natu- overall strategies in the proposed design,
ralistic and inductive nature of the inquiry and consider the capacity of the researcher
makes it both impossible and inappropriate to fruitfully undertake the proposed study.
to specify operational variables, state test- As with other strategic themes of qualita-
able hypotheses, or finalize either instru- tive inquiry, the extent to which the design is
mentation or sampling schemes. A natural- specified in advance is a matter of degree.
istic design unfolds or emerges as fieldwork Doctoral students doing qualitative disser-
unfolds. ta tions will usually be expected to present
Lincoln and Guba (1985) made an exten- fairly detailed fieldwork proposals and in-
sive comparison of the design characteristics terview schedules so that the approvmg
of qualitative/naturalistic inquiry in con- doctoral committee can guide the student
trast to quantitative/experimental methods. and be sure that the proposed work will lead
They concluded: to satisfying degree requirements. Many
funders will fund only detailed proposals.
What these considera tions add up to is that the As an ideal, however, the qualitative re-
design of a naturalistic inquiry (whether researcher needs considerable flexibility and
search, evaluation, or policy analysis) cannot openness. The fieldwork approach of an-
be given in advance; it must emerge, develop, thropologist Brackette F. Williams repre-
unfold The call for an emergent design by sents the ideal of emergence in naturalistic
naturalists is not simply an effort on their part mquiry.
to get around the "hard thinking" that is sup- Williams has focused on issues of cultural
posed to precede an inquiry; the desire to per- identity and social relationships. Her work
mit events to unfold is not merely a way of has mcluded in-depth study of ritual and
rationalizing what is at bottom "sloppy in- symbolism in the construction of national
quiry/' The design specifications of the con- identity in Guyana (1991), and the ways that
ventional paradigm form a procrustean bed of race and class function in the national con-
such a nature as to make it impossible for the sciousness of the United States. In 1997, she
naturalist to lie in itnot only uncomfortably, received a five-year MacArthur Fellowship,
but at ali (p. 225) which has allowed her to pursue a truly
emergent, naturalistic design in her current
Design flexibility stems from the open- fieldwork on the phenomenon of killing in
ended nature of naturalistic inquiry as well America. I had the opportunity to interview
as pragmatic considerations. Being open her about her work and am including sev-
and pragmatic requires a high tolerance for eral excerpts from that interview1 through-
ambiguity and uncertainty as well as trust in out this chapter to illustrate actual scholarly
the ultimate value of what inductive analy- implementa tion of some of the strategic ide-
sis will yield. Such tolerance, openness, and ais of qualitative inquiry. Here she describes
trust create special problems for dissertation the necessity of an open-ended approach to
committees and funders of evaluation or re- her fieldwork because her topic is broad and
she needs to follow wherever the phenome- col of questions and issues to pursue. It was
non takes her. general sampling to get a sense of what I
wanted to know. At other times, it's just to get
I'm tracking somethingkillingthat's mov- a general opinion from Jolin Q. Public about a
ing very rapidly in the culture. Every time I question that I've gotten ali kinds of official re-
talk to someone, there's another set of data, an- sponses to, but I want to know what people in
other thing to look at. Anything that happens general think. In an airport, I may get an op-
in America can be relevant, and thafs the ex- portunity to talk to 5 or 10 people. If I have sev-
hausting part of it. It never shuts off. You listen eral stops, I may get 15 or 20 by the time I come
to the radio. You watch television. You pass a home.
billboard with an advertisement on it. There's I fashion the research as I want to fashion it
no such thing as something irrelevant when based on what I think this week as opposed to
you're studying something like this or maybe what I thought Iast week. I don't follow some
just studying the society that you're in. You proposal. I don't have in mind that this has to
don't always kno w exactly how it's going to be be a book that's going to have to come out a
relevant, but somehow it just strikes you and certain way. I'm following where the data take
you say to yourself: I should document the me, where my questions take me.
date of when I saw this and where it was and
what was said because it's data. Few qualitative studies are as fully emergent
I don't follow every possible lead people and open-ended as the fieldwork of Wil-
give me. But generally, it is a matter in some liams. Her work exemplifies the ideal of
sense of opportunity sampling, of serendipity, emergent design flexibility.
whatever you want to call it, I key into things
that turn out to be very important six months Purposeful Sampling
la ter.
I do impromptu interviews. I don't have In 1940, eminent sociologist Kingsley
some target number of interviews in mind or Da vis published what was to become a clas-
predetermined questions. It depends on the sic case study, the story of Anna, a baby kept
person and the situation. Airports, for examin nearly total isolation from the time of her
ple, are a good place for impromptu inter- birth until she was discovered at age six. She
views with people. So some times, instead of had been deprived of human contact, had
using airport time to write, I interview people acquired no language skills, and had re-
about the death penalty or about killing or ceived only enough care to keep her barely
about death in their life. It's called opportunity alive. This single case, horrifying as was the
sampling, I begin with a general description. abuse and neglect, offered a natural experi-
You're such and such an age. You come from ment to study socialization effects and the
such and such a place and, by the way, what do relative contributions of nature and nurture
you think about ali this killing? And I sort of to human development. In 1947, Davis pub-
launch into a conversation. Sometimes the in- lished an update on Anna and a comparison
terview goes on for a couple of hours and case of socialization isolation, the story of
sometimes, maybe 10 or 15 minutes. I just say, Isabelle. These two cases offered consider-
"You wouldn't mind if I record this, would able insight into the question of how long a
you?" If they say no, I take notes. human being could remain isolated before
I did a lot of that kind of impromptu inter- "the capacity for full cultural acquisition"
viewing m the first year to formulate a proto- was permanently damaged (Davis 1940,
1947). The cases of Anna and Isabelle are ex- which one can learn a great deal about issues
treme examples of purposeful case sam- of central importance to the purpose of the
pling. research, thus the term purposeful sampling.
Unusual clinicai cases in medicine and For example, if the purpose of an evaluation
psychology, instructive precisely because is to increase the effectiveness of a program
they are unusual, offer many examples of in reaching lower-socioeconomic groups,
purposeful sampling. Neurologist Oliver one may leam a great deal more by focusing
Sacks (1985) presents a number of such cases in depth on understanding the needs, inter-
in his widely read and influential book The ests, and incentives of a small number of
Man Who Mistook His Wifefor a Hat, the very carefully selected poor families than by
title of which hints at the uniqueness of the gathering standardized information from a
cases examined. While one cannot general- large, statistically significant sample. The
ize from single cases or very small samples, cases sampled can be individual people,
one can leam from themand learn a great families, organizations, cultures, incidents,
deal, often opening up new territory for fur- or activities, to mention examples. But re-
ther research, as was the case with Piagefs gardless of the kind of unit of analysis (e.g.,
detailed and insightful observations of his an athlete or a sports team, a teacher or a
own two children. classroom), the purpose of purposeful sam-
Perhaps nowhere is the difference be- pling is to select information-rich cases
tween quantitative and qualitative methods whose study will illuminate the questions
better captured than in the different strate- under study.
gies, logics, and purposes that distinguish Chapter 5 will review several different
statistical probability sampling from quali- strategies for purposefully selecting infor-
tative purposeful sampling. Qualitative in- mation-rich cases. In my interview with her,
quiry typically focuses on relatively small Brackette F. Williams offered an example of
samples, even single cases (N = 1) such as an information-rich case from her ongoing
Anna or Isabelle, selected purposefully to per- study of killing in America.
mit inquiry into and understanding of a phe-
nomenon in depth. Quantitative methods I've been tracking information on a serial
typically depend on larger samples selected killersomeone who has just been identified
randomly in order to generalize with confi- as a "serial killer" in Louisianawho's killing
dence from the sample to the population young Black men, shooting them up with
that it represents. Not only are the tech- drugs and taking one of their tennis shoes,
niques for sample selection different, but the sometimes both. Now, I'm interested in the
very logic of each approach is distinct be- fact that as society more and more identifies
cause the purpose of each strategy is differ- young Black men as sort of the quintessential
ent. bad guys, this serial killer picks a bad guy. For
The logic and power of probability sam- contrast, look at serial killers who picked
pling derive from its purpose: generaliza- women at a certain period of time, about 15-20
tion. The logic and power of purposeful years ago, because they wore, in his estima-
sampling derive from the emphasis on tion, a size 13. Now, track our obsession with
in-depth understanding. This leads to se- obesity. How a serial killer picks his victims
lecting information-rich cases for study in can tell you something important about
depth. Information-rich cases are those from what's going on in society.
3. Data Collection and I was down in Texas interviewing last March,

Fieldwork: Strategies thinking about my research and interviewing
for Qualitative Inquiry people, and there was a childhood memory
that I had of an electrocution of a man that was
the son of a woman who lived across the field
Qualitative Data from us. Now a rumor about this had always
been in the back of my mind. Whenever I'd
Qualitative data consist of quotations, ob- hear about a death penalty case over the years,
servations, and excerpts from documents. I would thihk about this man having been
The first chapter provided several examples electrocuted. I thought he was electrocuted be-
of qualitative data. Deciding whether to use cause he raped this White woman. So I'm sit-
naturalistic inquiry or an experimental ap- ting in my cousin's kitchen after I had done
proach is a design issue. This is different some of these interviews and another woman,
from deciding what kind of data to collect an older woman who was a relative of hers,
(qualitative, quantitative, or some combina- came in and the conversation goes around. I
tion), although design and data altematives happen to mention this memory of mine. I
are clearly related. Qualitative data can be asked, "Is that just something that I concocted
collected in experimental designs where out of having read a book or something, but it
participants have been randomly divided never happened?" She answered, "Oh, no, it
into treatment and control groups. Likewise, happened. You only have one part of the story
some quantitative data may be collected in wrong. He didn't rape her. He looked at her."
naturalistic inquiry approaches. Neverthe- You know, you read about these things in
less, controlled experimental designs pre- history books and then ali of a sudden, it's Rke
dominantly aim for statistical analyses of a part of a world that you existed in. These
quantitative data, while qualitative data are things happened around you and yet some-
the primary focus in naturalistic inquiry how there was so much of a distance, you
This relationship between design and mea- couldn't touch it. I knew about this man ali my
surement will be explored at greater length life, but in ali the reading and ali the history
in the chapter on design. books, I couldn't touch that. Doing this project
Qualitative data describe. They take us, the way I'm doing it allozvs me to touch things that
as readers, into the time and place of the ob- otherwise I would never touch.
serva tion so that we know what it was like to
have been there. They capture and commu-
Direct Personal Experience and
nicate someone else's experience of the
Engagement: Going Into the Field
world in his or her own words. Qualitative
data tell a story. In the excerpt below, from
my interview with her, Williams tells the The preceding quotation from Williams
story of checking out a childhood memory. exemplifies the personal nature of qualita-
This story gives us insight into the nature of tive fieldwork. Getting close to her subject
her naturalistic inquiry and open-ended in- matter, including using her own experi-
terviewing, shows how a criticai incident ences, both from childhood and day-to-day
can be a purposeful sample, and, in the story in her adult life, illustrates the all-encom-
itself, offers something of the flavor of quali- passing and ultimately personal nature of
tative data. in-depth qualitative inquiry. Traditionally,
social scientists have been warned to stay other people, assume that in order to know or
distant from those they studied to maintain understand others one is well-advised to give
"objectivity." But that kind of detachment some conscious attention to that effort in
can limit one's openness to and understand- face-to-face contacts. They assume, too, that
ing of the very nature of what one is study- the internai world of sociologyor any other
ing, especially where meaning-making and social worldis not understandable unless
emotion are part of the phenomenon. Look one has been part of it in a face-to- face fashion
closely at what Williams says about the ef- for quite a period of time. How utterly para-
fects of immersing herself personally in her doxical, then, for these same persons to tum
fieldwork, even while visiting relatives: around and make, by implication, precisely
"Doing this project the way Tm doing it al- the opposite claim about people they have
lows me to touch things that otherwise I never encountered face-to-facethose people
would never touch." appearing as numbers in their tables and as
Fieldwork is the central activity of quali- correlations in their matrices! (Lofland 1971:3)
tative inquiry. "Going into the field" means
having direct and personal contact with peo- Qualitative inquiry means going into the
ple under study in their own environments fieldinto the real world of programs, orga-
getting close to the people and situa- nizations, neighborhoods, street corners
tions being studied to personally under- and getting close enough to the people and
stand the realities and minutiae of daily life, circumstances there to capture what is hap-
for example, life as experienced by partici- pening. To immerse oneself in naturally oc-
pants in a welfare-to-work program. The in- curring complexity involves what qualita-
quirer gets close to the people under study tive methodologist Norman Denzin (1978a)
through physical proximity for a period of has called "the studied commitinent to ac-
time as well as through development of tively enter the worlds of interacting in-
closeness in the social sense of shared expe- dividuais" (pp. 8-9). This makes possible
rience, empathy, and confidentiality. That description and understanding of both exter-
many quantitative methodologists fail to nally observable behaviors and internai
ground their findings in personal qualitative states (worldview, opinions, values, atti-
understanding poses what sociologist John tudes, and symbolic constructs). Given the
Lofland (1971) called a major contradiction qualitative emphasis on striving for depth of
between their public insistence on the ade- understanding, in context, attitude surveys
quacy of statistical portrayals of other hu- and psychological tests are inadequate for
mans and their personal everyday dealings revealing inner perspectives. "The inner
with and judgments about other human perspective assumes that understanding
beings. can only be achieved by actively participai-
ing in the life of the observed and gaining in-
In everyday life, statistical sociologists, like sight by means of introspection" (Bruyn
everyone else, assume that they do not know 1963:226).
or understand very well people they do not Actively participating in the life of the ob-
see or associate with very much. They assume served means going where the action is, get-
that knowing and understanding other people ting one's hands dirty, participating where
require that one see them reasonably often and possible in actual program activities, and
in a variety of situations relative to a variety of getting to know program staff and partici-
issues. Moreover, statistical sociologists, like pants on a personal leveiin other words,
getting personally engaged so as to use ali of It is important to note that the admonition
one's senses and capacities, including the ca- to get close to the data is in no way meant to
pacity to experience affect no less than cog- deny the usefulness of quantitative meth-
nition. Such engagement stands in sharp ods. Rather, it means that statistical portray-
contrast to the professional comportment of als must always be interpreted and given
some in the field, for example, supposedly human meaning. I once interviewed an eval-
objective evaluators, who purposely project uator of federal health programs who ex-
an image of being cool, calm, externai, pressed frustration at trying to make sense
and de tache d. Such detachment is pre- out of statistical data from over 80 projects
sumed to reduce bias. However, qualitative after site visit funds had been cut out of the
methodologists question the necessity and evaluation: "There's no way to evaluate
utility of distance and detachment, asserting something thafs just data. You know, you
that without empathy and sympathetic in- have to go look."
trospection derive d from personal encoun- Going into the field and having personal
ters, the observer cannot fully understand contact with program participants is not the
human behavior. Understanding comes only legitimate way to understand human
from trying to put oneself in the other per- behavior. For certain questions and for situa-
son's shoes, from trying to discern how oth- tions involving large groups, distance is in-
ers think, act, and feel. evitable, perhaps even helpful, but to get at
In a classic study, educational evaluator deeper meanings and preserve context,
Edna Shapiro (1973) studied young chil- face-to-face interaction is both necessary
dren in classrooms in the national Follow and desirable. This returns us to a recurrent
Through program using both quantitative theme of this book: matching research meth-
and qualitative methods. It was her close- ods to the purpose of a study, the questions
ness to the children in those classrooms that being asked, and the resources available.
allowed her to see that something was hap- In thinking about the issue of closeness to
pening that was not captured by standard- the people and situations being studied, it is
ized tests. She could see differences in chil- useful to remember that many major contri-
dren, observe their responses to diverse butions to our understanding of the world
situations, and capture the varying mean- have come from scientists' personal experi-
ings they attached to common events. She ences. One finds many instances where close-
could feel their tension in the testing situa- ness to sources of data made key insights
tion and their spontaneity in the more natu- possible Piagefs closeness to his children,
ral classroom setting. Had she worked solely Freud's proximity to and empathy with his
with data collected by others or only at a dis- patients, Darwin's closeness to nature, and
tance, she would never have discovered the even Newton's intimate encounter with an
crucial differences in the classroom settings apple. In short, closeness does not make bias
she studieddifferences that actually al- and loss of perspective inevitable; distance
lowed her to evaluate the innovative pro- is no guarantee of objectivity.
gram in a meaningful and relevant way.
Where standardized tests showed no differ- Empathic Neutrality
ences between classrooms using different
approaches, her direct observations docu- If, as the previous section has discussed,
mented important and significant program naturalistic inquiry involves fieldwork that
impacts. puts one in close contact with people and
their problems, what is to be the researcher's of a supposedly value-free social science,

cognitive and emotional stance toward subjectivity is the very antithesis of scientific
those people and problems? No universal inquiry.
prescription can capture the range of possi- Objectivity has been considered the
bilities, for the answer will depend on the strength of the scientific method. The pri-
situation, the nature of the inquiry, and the mary methods for achieving objectivity in
perspective of the researcher. But thinking science have been conducting blind experi-
strategically, I offer the phrase "empathic ments and quantification. "Objective tests"
neutrality" as a point of departure. It sug- gather data through instruments that, in
gests that there is a middle ground between principie, are not dependent on human skill,
becoming too involved, which can cloud perception, or even presence. Yet, it is clear
judgment, and remaining too distant, which that tests and questionnaires are designed
can reduce understanding. What is em- by human beings and therefore are subject to
pathic neutrality? Consider this anecdote by the intrusion of the researcher's biases by
way of illustration. the very questions asked. Unconscious bias
Pragmatst philosopher William James, in the skillful manipula tion of statistics to
also a scholar of anatomy and psychology, prove a hypothesis in which the researcher
had a great capacity for empathy, as dis- believes is hardly absent from hypothetical-
played in his classic study The Varieties o/Re- deductive inquiry.
ligious Experience ([1902] 1999). Editor Clif- Part of the difficulty in thinking about the
ton Fadiman (1985:305) recounts that while fieldwork stance of the qualitative mquirer
he was teaching at Radcliffe, Gertrude Stein is that the terms objectivity and subjectivity
took a course from him in which, having at- have become so loaded with negative con-
tended the opera and then partied into the notations and subject to acrimonious debate
wee hours the night before an exam, she (e.g., Scriven 1972a; Borman and Goetz 1986;
wrote, "Dear Professor James, I am so sorry Krenz and Sax 1986; Guba 1991) that neither
but I do not feel a bit like writing an exami- term any longer provides useful guidance.
nation paper today." James is said to have These terms have been politicized beyond
written back: "Dear Miss Stein, I understand utility. To claim the mantle of "objectivity" in
perfectly. I often feel like that myself." Had the postmodern age is to expose oneself as
he added, but the exam is still due, instead of embarrassingly naive. The ideais of absolute
ordinary sympathy he would have dis- objectivity and value-free science are impos-
played extraordinary empathic neutrality. sible to attam in practice and are of question-
Methodologists and philosophers of sci- able desirability in the first place since they
ence debate what the researcher's stance ignore the intrinsically social nature and hu-
should be vis--vis the people being studied. man purposes of research. On the other
Critics of qualitative inquiry have charged hand, subjectivity has such negative connota-
that the approach is too subjective, in large tions in the public mind that to admit being
part because the researcher is the instrument subjective may undermine one's credibility
of both data collection and data interpreta- with audiences unsophisticated about
tion and because a qualitative strategy in- phenomenological assumptions and nu-
cludes having personal contact with and ances. In short, the terms objectivity and sub-
getting close to the people and situation un- jectivity have become ideological ammuni-
der study. From the perspective of advocates tion in the methodological paradigms
debate. My pragmatic solution is to avoid plexities and multiple perspectives as they

using either word and to stay out of futile de- emerge, and be balanced in reporting both
bates about subjectivity versus objectivity. confirmatory and disconfirming evidence
Qualitative research in recent years has with regard to any conclusions offered.
moved toward preferring such language as Neutrality is not an easily attainable
trustworthiness and authenticity. Evaluators stance, so ali credible research strategies in-
aim for "balance/' "fairness," and "com- clude techniques for helping the investiga-
pleteness" (Patton 1997a:282). Chapter 9 will tor become aware of and deal with selective
discuss these terms and the stances they im- perception, personal biases, and theoretical
ply at greater length. At this point, I simply predispositions. Qualitative inquiry, be-
want to note the strategic nature of the issue cause the human being is the instrument of
of inquirer stance and add empathic neutral- data collection, requires that the investigator
ity to the emerging lexicon that attempts to carefully reflect on, deal with, and report po-
supersede the hot button term objective and tential sources of bias and error. Systematic
the epithet subjective. data collection procedures, rigorous train-
Any research strategy ultimately needs ing, multiple data sources, triangulation, ex-
credibility to be useful. No credible research ternai reviews, and other techniques to be
strategy advocates biased distortion of data discussed in this book are aimed at produc-
to serve the researcher's vested interests and ing high-quality qualitative data that are
prejudices. Both qualitative/naturalistic incredible, trustworthy, authentic, balanced
quiry and quantitative/experimental in- about the phenomenon under study, and
quiry seek honest, meaningful, credible, and fair to the people studied.
empirically supported findings. Any credi- The livelihood of evaluators and re-
ble research strategy requires that the inves- searchers depends on their integrity and
tigator adopt a stance of neutrality with re- credibility. Independence and neutrality,
gard to the phenomenon under study. This then, are serious issues.
simply means that the investigator does not However, neutrality does not mean de-
set out to prove a particular perspective or tachment. It is on this point that qualitative
manipulate the data to arrive at predisposed inquiry makes a special contribution. Quali-
truths. Theneutral investigator enters the re- tative inquiry depends on, uses, and en-
search arena with no ax to grind, no theory hances the researcher's direct experiences in
to prove (to test but not to prove), and no the world and insights about those ex-
predetermined results to support. Rather, periences. This includes learning through
the investigator's commitment is to under- empathy.
stand the world as it unfolds, be true to com-
Empathy and Insight
T he idea of acquiring an "inside" understandingthe actors' definitions

of the situationis a powerful central concept for understanding the
purpose of qualitative inquiry.
Thomas A. Schwandt (2000:102)

Empathy develops from personal contact

with the people interviewed and observed
during fieldwork. Empathy involves being
able to take and understand the stance, posi- iVi i: to 'n"i'i;;i:;! SFV ::.:;: iid * > u-, kx .VSH'!:';

tion, feelings, experiences, and worldview .ji.;j r&i |;!"!':!i! i! ir.V:! r-h-, V . 'iVn r/iu!
of others. Put metaphorically, empathy is ! ! , !':" ! ' :!! i : -iii' I '! !; ! ; I;'I ! !:!!I;".' ! i.:J" n n l V i ! i'-
"like being able to imagine a life for a spider, A ; \ V J ! ; TF! R. Y:'.-; I V A I : ! -.-JYV. (> RI-N <".'
a maker's life, or just some aliveness in its ':i,i;!!:M;:!'\!!!n:,!:Mi!i:v. irjsYn!;! k\ mlv i ^-i-'
wide abdmen and delicate spinnerets so & !i, >Vi k : T . ??;!!! K I - V i A M - N h : V " I i i W -
!
you take it outside in two paper cups instead W - .Mv hi! : I: H;. OA r' :',!
of stepping on it" (Dunn 2000:62). Empathy 1 P J ! ' I ! " i " !L|: RT-D* L . ' ! . I : i " ! . J ' I T ! ':: ! R I,:;.
combines cognitive understanding with af- :.!'!:' V i V i !.';!;! J I T F ! IV; ! I L I ' ! M' ri 1 !.!. W , NNHN ri
fective connection, and in that sense differs WJS V!:'RVSN1;::NYJ.r, IIIXHRIY''!" W M ! !|
from sympathy, which is primarily emo- ; : ; I R ; 1 ; H ' " ; ! . TH-H V/!';:! F:-I" . T T i.VI W : , I H T !I I;'
tional (Wisp 1986). Ki' TIJ. II.I,\<:!:'i:I a.*:- 1'!!-:' D -A V I.-RI-I*-, !::M;IJ: N 1 ; P;
The value of empathy is emphasized in .! : 'i i 'i '.] i' j!i-ii FRI"1 ! ! !:.'' M S ! I . V ^ ' . , Y Y v ]i ri.;R N ; v i : ! !'
the phenomenological doctrine of Verstehen . V i HL' i V i {! :i i'i.' I! Y L " ' J i.i > J I."I .! Y r M V I ! rvri,
that undergirds much qualitative inquiry. *T i i i ! I ' H \ iii R':I::.I" IV!T; RRN-J.''
Verstehen means "understanding" and refers

to the unique human capacity to make sense
of the world. This capacity has profound im- ences need methods different from those
plications for how one studies human be- used in agricultural experimentation and
ings. The Verstehen doctrine presumes that physical sciences because human beings are
since human beings have a unique type of different from plants and nuclear particles.
consciousness, as distinct from other forms The Verstehen tradition stresses understand-
of life, the study of humanbeings willbe dif- ing that focuses on the meaning of human
ferent from the study of other forms of life behavior, the context of social interaction, an
and nonhuman phenomena. The capacity empathic understanding based on personal
for empathy, then, is one of the major assets experience, and the connections between
available for human inquiry into human af- mental states and behavior. The tradition of
fairs. Verstehen places emphasis on the human ca-
The Verstehen premise asserts that human pacity to know and understand others
beings can and must be understood in a through empathic introspection and reflec-
manner different from other objects of study tion based on direct observation of and in-
because humans have purposes and emo- teraction with people. "Verstehen thus en-
tions; they make plans, construct cultures, tails a kind of empathic identification with
and hold values that affect behavior. Their the actor. It is an act of psychological
feelings and behaviors are influenced by reenactment getting inside the head of an
consciousness, deliberation, and the capac- actor to understand what he or she is up to in
ity to think about the future. Human beings terins of motives, beliefs, desires, thoughts,
live in a world that has special meaning to and so on" (Schwandt 2000:192).
them, and because their behavior has mean- Max Weber brought the term empathy into
ing, "human actions are intelligible in ways social science to emphasize the importance
that the behavior of nonhuman objects is of comprehending the motives and feelings
not" (Strike 1972:28). Human and social sci- of people in a social-cultural context.
Both Verstehen and empathy depend largely deeply and attentively so as to enter into the
on qualitative data. Verstehen is an attempt to other person's experience and perception.
"crack the code" of the culture, that is, detect
the categories into which a culture codes ac- I do not select, interpret, advise, or d i r e c t . . . .
tions and thoughts Empathy in evaluation Being-In the world of the other is a way of go-
is the detection of emotions manifested in the ing wide open, entering n as if for the first
program participants and staff, achieved by time, hearing just what is, leaving out my own
evaluators' becoming aware of similar or com- thoughts, feelings, theories, b i a s e s . . . . I enter
plementary emotions in themselves. (Meyers with the intention of understanding and ac-
1981:180) cepting perceptions and not presenting my
own view or r e a c t i o n s . . . . I only want to en-
A qualitative strategy of inquiry proposes courage and support the other person's ex-
an active, involved role for the social scien- pression, what and how it is, how it came to be,
tist. "Hence, insight may be regarded as the and where itis going. (Moustakas 1995:82-83)
core of social knowledge. It is arrived at by
being on the inside of the phenomena to be At first, the phrase "empathic neutrality"
observed It is participation in an activity may appear to be an oxymoron, combining
that generates interest, purpose, point of contradictory ideas. Empathy, however, de-
view, value, meaning, and intelligibility, as scribes a stance toward the people one en-
well as bias" (Wirth 1949:xxii). This is a quite countersit communicates understanding,
different scientific process from that envi- interest, and caring. Neutrality suggests a
sioned by the classical, experimental ap- stance toward their thoughts, emotions, and
proach to science, but it is still an empirical, behaviorsit means being nonjudgmental.
(i.e., data-based), scientific perspective. The Neutrality can actually facilitate rapport
qualitative perspective "in no way suggests and help build a relationship that supports
that the researcher lacks the ability to be sci- empathy by disciplining the researcher to be
entific while collecting the data. On the con- open to the other person and nonjudgmental
trary, it merely specifies that it is crucial for in that openness. Rapport and empathy,
validityand, consequently, for reliability however, must not be taken for granted, as
to try to picture the empirical social world Radhika Parameswaran (2001) found in do-
as it actually exists to those under investiga- ing fieldwork among young middle-class
tion, rather than as the researcher imagines it women in urban ndia who read Western ro-
to be" (Filstead 1970:4), thus the importance mance fiction.
of such qualitative approaches as partici-
pant observation, depth interviewing, de- Despite their eventual willingness to share
tailed description, and case studies. their fears and complaints about gendered so-
These qualitative inquiry methods pro- cial pressures, I still wonder whether these
vide opportunities to achieve empathy and young women would have been more open
give the researcher an empirical basis for about their sexuality with a Westerner who
describing the perspectives of others. Chap- might be seen as less likely to judge them
ter 1 cited the framework of humanis tic psy- based on cultural expectations of women's be-
chologist Clark Moustakas, who has de- havior in Indian society. The well-known
scribed this nonjudgmental empathic stance word rapport, which is often used to signify
as "Being-In" another's worldimmers- acceptance and warm relationships between
ing oneself in another's world by listening informants and researchers, was thus some-
thing I could not take for granted despite be- data interpretation and formulating recom-
ing an insider; ali I could claim was an mendations, but during fieldwork, the focus
imperfect rapport. (Parameswaran 2001:69) should be on rigorously observing and inter-
viewing to understand the people and situa-
Evaluation presents special challenges tion being studied. This nuanced relation-
for rapport and neutrality as well. After ship between neutrality and empathy will
fieldwork, an evaluator may be called on to be discussed further in both the data collec-
render judgments about a program as part of tion and analysis chapters.
A Dynamic, Developmental Perspective
"D here is nothing permanent except change.
Heraclitus (Ancient Greece)
A questionnaire is like a photograph. A in participants' experiences undermine the

qualitative study is like a documentary film. logic of an experimental design because
Both offer images. One, however the pho- these developmentsali natural, even inev-
tographcaptures and freezes a moment in itable, in real-world programscall into
time, like recording a respondenfs answer question what the "treatment" or experi-
to a survey question at a moment in time. ment actually is.
The otherthe film offers a fluid sense of Naturalistic inquiry assumes the ever-
development, movement, and change. changing world posited by the observation
Qualitative evaluation researchers, for in the ancient Chinese proverb that one
example, conceive of programs as dynamic never steps into the same river twice.
and developing, with "treatments" chang- Change is a natural, expected, and inevita-
ing in subtle but important ways as staff ble part of human experience, and docu-
learns what does and doesn't work, as cli- menting change is a natural, expected, and
ents move in and out, and as conditions of intrinsic part of fieldwork. Rather than try-
deli very are altered. A primary challenge, ing to control, limit, or direct change, natu-
then, becomes describing and understand- ralistic inquirers expect change, anticipate
ing these dynamic program processes and the likelihood of the unanticipated, and are
their holistic effects on participants so as to prepared to go with the flow of change. One
provide information for program improve- gets this sense of pursuing change in the
ment. In contrast, an experimental design comment by anthropologist Williams cited
for an evaluation typically conceives of the earlier: 'Tm tracking somethingkilling
program as a fixed thing, like a measured that's moving very rapidly in the culture."
amount of fertilizer applied to a cropa Part of her inquiry task is to track cultural
treatment, an interventionthat has prede- changes the way an epidemiologist tracks a
termined, measurable outcomes. Inconsis- disease. As a result, reading a good qualita-
tency in the treatment, instability in the tive case study gives the sense of reading a
intervention, changes in the program, vari- good story. It has a beginning, a middle, and
ability in program processes, and diversity an endingthough not necessarily an end.
3. Analysis Strategies evaluation. That is what is meant by the

for Qualitative Inquiry "unique case orientation" of qualitative in-
quiry.
Uni que Case Orientation Case studies are particularly valuable in
program evaluation when the program is in-
"Six windows on respect" is how Har- dividualized, so the evaluation needs to be
vard sociologist Sara Lawrence-Lightfoot attentive to and capture individual differ-
(2000:13) describes the six detailed case ences among participants, diverse experi-
studies, each a full chapter, she presents in ences of the program, or unique variations
her book Respect. The cases offer different from one program setting to another. As
angles on the meaning and experience of re- noted earlier, a case can be a person, an
spect in inodern society as illuminated by a event, a program, an organization, a time
nurse-inidwife, a pediatrician, a teacher, an period, a criticai mcident, or a community.
artist, a law school professor, and a pastoral Regardless of the unit of analysis, a qual-
therapist/AIDS activist. Before drawing itative case study seeks to describe that unit
themes and contrasts froin this small, pur- in depth and detail, holistically, and in
poseful sample, and before naming the six context.
angles they represent, Lawrence-Lightfoot
had the task of constructing the unique cases
Inductive Analysis
to tell these distinct stories. Her first task,
and Creative Synthesis
then, was to undertake the "art and science
of portraiture" (Lawrence-Lightfoot and Benjamin Whorf's development of the fa-
Da vis 1997). From these separate portraits, mous Whorf hypothesisthat language
she fashions a stained glass mosaic that de- shapes our experience of the environment
picts and illuminates Respect and that words shape perceptions and ac-
I undertook a study of a national fellow- tions, a kind of linguistic relativity theory
ship award program that had had more than (Schultz 1991)provides an instructive ex-
600 recipients over a 20-year period. A sur- ample of inductive analysis. Whorf was an
vey had been done to get the fellows' opin- insurance investigator assigned to look into
ions about select issues, but the staff wanted explosions in warehouses. He discovered
more depth, richness, and detail to really un- that truck drivers were entering "empty"
derstand patterns of fellowship use and im- warehouses smoking cigarettes and cigars.
pact. With a team of researchers, we con- The warehouses, it turned out, often con-
ducted 40 in-depth, face-to-face interviews tained invisible, but highly flammable
and wrote case studies. Through inductive gases. He interviewed truckers and found
analysis, we subsequently identified dis- that they associated the word empty with
tinct enabling processes and impacts, and harmless and acted accordingly. From these
we created a frainework that depicted rela- specific observations and findings, he induc-
tionships between status at the time of the tively formulated his general theory about
award, enabling processes, and impacts. The language and perception that has informed
heart of the study remained the 40 case stud- a half-century of communications scholar-
ies. To read only the framework analysis ship (Lee 1996).
without reading the case studies would be to Qualitative inquiry is particularly ori-
lose much of the richness, depth, meaning, ented toward exploration, discovery, and in-
and contribution of qualitative research and ductive logic. Inductive analysisbegins with
ables are important and what relationships

among those variables can be expected.
^biSyjTTVi!!
AMPARA TIVE ANAL/fiiSi The strategy of inductive designs is to al-
low the important analysis dimensions to
vi :::;:::!!= :: I: :'I o emerge from pattems found m the cases un-
V: i 0 r ,;i lyj :.>. .V! "; (Vi. ; ; : der study without presupposing in advance
what the important dimensions will be. The
I.=::i:!-::.!; YI --s:---:i:s rr?-=:-=-.' r c :;' qualitative analyst seeks to understand the
multiple interrelationships among dimen-
i = n C^:;yy:!: :: !=-= :;='= ::=i.:: sions that emerge from the data without
:: m ; !--' n! i'..i -ni .;! i & ' !=.= making prior assumptions or specifying hy-
erO?J blh--n VI !! == :==!'.:! !"J :=: ! ! potheses about the linear or correlative rela-
fei iii-- !=!':=" i !=::! =:: >
: > tionships among narrowly defined, opera-
!"3W.i ri:.:; t u s e = = .lu tionalized variables. For example, an induc-
Hard Wai HHCTIS in !'=::prarr iihtuaitai tive approach to program evaluation means
rn.;;;: ".::.':! iihsi ^(v^ wt that understanding the nature of the "inter-
cra-rto ii i''i!ii=:' iliîlsi!;'-!' !i:i. vention" emerges from direct observations
.-.1-rtl of program activities and interviews with
.V!IY! 1:iMM- ::::!IttlF!.:= !!!SYÎ^ ,'i:!11 participants. In general, theories about what
! - m U ! : n V v ! ! ' c w m w . . - * a ;:! :
i!:;/ YM -::=Y!>= is happening in a setting are grounded in
Y' ^ I Y V I ! ' i Y i k ; ; -! R== i: i l V ^ i h - i f EIIYC-I and emerge from direct field experience
rather than being imposed a priori as is the
!V;:=i !. :;! li M iY ^ fi' .vi .! W.,"^ case in formal hypothesis and theory testing.
The straightforward contrast between
: t ti ti iji : ii [liiiliii closed-ended questionnaires and open-
=:IPC J I T A ^ S :. I:;:.;
! ; : V I Y ! Y G T F P T '
. : ! i [ L i : ! j ! Y ' i ! V i i=i=j.:J: t;i:i:r- r i / f iiitiiiiiiiiiiiiiiiiiiiiiiiiitiiiiiiiiii;

i ^ ^ ended interviews in Chapter 1 illustrated
the difference between deductive and in-
ductive approaches at the simplest levei. A
specific observations and builds toward structured, multiple-choice questionnaire
general pattems. Categories or dimensions requires a deductive approach because
of analysis emerge from open-ended obser- items mustbe predetermined based on some
vations as the inquirer comes to understand theory or preordinate criteria, for example,
pattems that exist in the phenomenon being program goals about what is important to
investigated. measure. An open-ended interview, by way
Inductive analysis contrasts with the hy- of contrast, permits the respondent to de-
pothetical-deductive approach of experi- scribe what is meaningful and salient with-
mental designs that require the specification out being pigeon hole into standardized cat-
of main variables and the statement of spe- egories.
cific research hypotheses before data collec- In practice, these approaches are often
tion begins. A specification of research hy- combined. Some evaluation or research
potheses based on an explicit theoretical questions may be determined deductively,
framework means that general constructs while others are left sufficiently open to
provide the framework for understanding permit inductive analyses based on direct
specific observations or cases. The investiga- observations. While the quantitative/exper-
tor must then decide in advance what vari- imental approach is largely hypothetical-
Pigeon Holing
deductive and the qualitative/naturalistic separate cases. Once that is done, cross-case
approach is largely inductive, a study can in- analysis can begin in search of patterns and
clude elements of both strategies. Indeed, themes that cut across individual experi-
over a period of inquiry, an investigation ences. The initial focus is on full understand-
may flow from inductive approaches, to find ing of individual cases before those unique
out what the important questions and vari- cases are combined or aggregated themati-
ables are (exploratory work), to deductive cally. This helps ensure that emergent cate-
hypothesis-testing or outcome measure- gories and discovered patterns are grounded
ment aimed at confirming and/or generaliz- in specific cases and their contexts (Glaser
ing exploratory findings, then back again to and Strauss 1967).
inductive analysis to look for rival hypo- Just as writers report different creative
theses and unanticipated or unmeasured processes, so too qualitative analysts have
factors. different ways of working. Although soft-
The precise nature of inductive analysis ware programs now exist to facilitate work-
depends, in part, on the purpose of the anal- ing with large amounts of narrative data and
ysis and the number and types of cases in a substantial guidance can be offered about
study. Where there are several cases to be the steps and processes of content analysis,
compared and contrasted, an inductive ap- making sense of multiple interview tran-
proach begins by constructing individual scrip ts and pages of field notes cannot be re-
cases, without pigeon holing or categorizing duced to a formula or even a standard series
those cases. That is, the first task is to do a of steps. There is no equivalent of a statistical
careful job independently writing up the significance test or factor score to tell the an-
alyst when results are important or what that I should rewrite this part of this chapter. I
quotations fit together under the same had completely forgotten about this tape. It
theme. Finding a way to creatively synthesize was done in early '98 or late '97 and maybe I
and present findings is one of the challenges hadn't listened to it or looked at the transcript
of qualitative analysis, a challenge that will for a while, and I've just finished a chapter or
be explored at length in Part 3 of this book. section of a chapter. I pull that tape off the
For the moment, I can offer a flavor of that shelf. I listen to it. I go back to the transcript
challenge with another excerpt from my in- and I start writing again. I start revising in
terview with anthropologist Williams. Here ways that it seems to me that tape emands.
she describes part of her own unique ana-
lytic process. As Williams describes her analysis and
writing process, she offers insight into what
My current project folio ws up work that I have it means when qualitative researchers say
always done, which is to study categories and they are "working to be true to the data" or
classifications and their implications. Right that their analytical process is "data driven."
now, as I said, the focus of my work is on kill- Williams says, "I start revising in ways that it
ing and the desire to kill and the categories seems to me that tape demands." It is com-
people create in relation to killing. Part of it mon to hear qualitative analysts say that, as
right now focuses on the death penalty, but they write their conclusions, they keep go-
mainly on killing. My fascination is with the ing back to the cases; they reread field notes;
links between category distinctions, commit- and they listen again to interviews. Induc-
ments, and the desire to kill for those comm.it- tive analysis is built on a solid foundation
ments. That's what I study. of specific, concrete, and detailed observa-
I track categories, like "serial killers" or tions, quotations, documents, and cases.
"death row inmates." The business of con- As thematic structures and overarching
stantly transforming people into acts and acts constructs emerge during analysis, the
into people is part of the way loyaltes, com- qualitative analyst keeps returning to field-
mitments, and hatreds are generated. So I'm a work observations and interview tran-
classifier. I study classificationtheories of scripts, working from the bottom up, stay-
classification. A lot of categories have to do ng grounded in the foundation of case
with very abstract things; others have to do write-ups, and thereby examining emergent
with very concrete things like skin color. But themes and constructs in light of what they
ultimately, the classification of a kill is what illuminate about the case descriptions on
I'm focusing on now. I've been asking myself which they are based. That is inductive
lately, for the chapter I've been working on, "Is analysis.
there a fundamental difference, for example,
in the way we classify to kill?" Consider Holistic Perspective
thepercentage of people classified as "death
worthy"the way we classify to justify the Holography is a method of photography
death penalty. in which the wave field of light scattered by
As I write, moving back and forth between an object is captured as an interference pat-
my tapes and my interviews, I don't feel that I tern. When the photographic recordthe
have to follow some fixed outline or that I hologramis illuminated by a laser, a three-
have to code things to come out a certain way. dimensional image appears. Any piece of a
Sometimes I listen to a tape and I start to think hologram will reconstruct the entire image.
This has become a metaphor for thinking in quires operationalization of independent

new ways about the relationships between and dependent variables with a focus on
parts and wholes. The interdependence of their statistical covariance. Outcomes must
flora, fauna, and the physical environment be identified and measured as specific vari-
in ecological systems offers another meta- ables. Treatments and programs must also
phor for what it means to think and analyze be conceptualized as discrete, independent
holistically. variables. The characteristics of program
Researchers and evaluators analyzing participants are also described by standard-
qualitative data strive to understand a phe- ized, quantified dimensions. Sometimes the
nomenon or program as a whole. This variables of interest are derived from pro-
means that a description and interpretation gram goals, for example, student achieve-
of a person's social environment, or an orga- ment test scores, recidivism statistics for a
nizations externai context, is essential for group of juvenile delinquents, sobriety rates
overall understanding of what has been ob- for participants in chemical dependency
serve d during fieldwork or said in an inter- treatment programs. At other times, the
view. This holistic approach assumes that variables measured are indicators of a larger
the whole is understood as a complex sys- construct. For example, community well-
tem that is greater than the sum of its parts. being may be measured by such rates for de-
The analyst searches for the totality or unify- linquency, infant mortality, divorce, unem-
ing nature of particular settingsthegestalt. ployment, suicide, and poverty (Brock,
Psychotherapist Fritz Perls (1973) made the Schwaller, and Smith 1985). These variables
term gestalt equivalent with a holistic per- are statistically manipulated or added to-
spective inpsychology. He use d the example gether in some linear fashion to test hypoth-
of three sticks that are just three sticks until eses and draw inferences about the relation-
one places them together to form a triangle. ships among separate indicators, or the
Then they are much more than the three sep- statistical significance of differences be-
arate sticks combined: They form a new tween measured leveis of the variables for
whole. different groups. The essential logic of this
approach is as follows: (1) Key program out-
A gestalt may be a tangible thing, such as a tri- comes and processes can be represented by
angle, or it may be a situation. A happening separate independent variables, (2) these
such as a meeting of two people, their conver- variables canbe quantified, and (3) relation-
sation, and their leave-taking would consti- ships among these variables are best por-
tute a completed situation. If there were an trayed statistically.
interruption in the middle of the conversadon, The primary critique of this logic by qual-
it would be an incomplete gestalt. (Brown itative-naturalistic evaluators is that such an
1996:36) approach (1) oversimplifies the complexities
of real-world programs and participants' ex-
The strategy of seeking gestalt units and periences, (2) misses major factors of impor-
holistic understandings in qualitative analy- tance that are not easily quantified, and (3)
sis contrasts with the logic and procedures fails to portray a sense of the program and its
of evaluation studies conducted in the ana- impacts as a whole. To support holistic anal-
lytical tradition of "let's take it apart and see ysis, the qualitative inquirer gathers data on
how it works." The quantitative-experimen- multiple aspects of the setting under study
tal approach to evaluation, for example, re- to assemble a comprehensive and complete
HaLisric \ j t o b m T A m \ m . !N ^ LTICS
H W Z R K ::;' ; N-RH ^ R - :
: I :'-. I U V U I V R R I I I I ii' i. i'I :'. !: -li vi i, I vi li :"! Or' ri .v j iV.'
U.1" n r f jV:. foitf ! i:v VV. L j !i i! ! i:"i H , ! >..! V' i " T C i'i:;i Vi '>..'.:b Vri!!.'.i. :iIIiJ'i-3'!=!VRRT vM\\\
iV:n liVsy ^;:!!::M |;''!''<: ,> ! r.i ,' ' u iym' .tith- - : rM.V-ii i:! rn iVi." iVll! Q % OA! S I "
T-' !'v W, :\:KhMf " -fritS Vi !I:V.!Vi\
1 . . I'I. '..II"IJ"!ri!V : M, R.'
J P ! ! ! - ! ! I;'R ;'.;.;I:.;:N
hl!! ri!!!!' " I n C:!;V P I iVfl !.'.!!' i:>,\\ v 'nr i.- > r, Iiis' II""rV.:1 tri."rj.! i-Vs?i:;.;! Liki'" Cr.Vii- Imf
[ii|iHtlf || i| l llll Ml^ ill lllll II il l il lllll iil [ Ml[ [ ll [ [ ! o I nli nl INI1 n] Ii.NIi:iV:' III Ni:i!.'.n.i
M il Iii'I i:!!.'l 1HJ" ' lr.Wii
H nl M =.i" ITI ' 11,=.' ri =;. : V
V.T; N ; Y ! , : ) ' I V I .,:!,Vi ..'.I.' --\fJ ;:';! '.V"-
VII" .II!::II, :'I!' .II* I W I
! V J H F ;:!RN>::!.:'. "-VI|T
; :;V. ! hi.- IJ:""I"!.':' \>V>? l:^:1 IVismaiM ,VVM-
rAan Ti':-!! i^";]! n :^-::;'!' ;^" . iMfl-
"':;.;i;;-':'I.=r-.;',ryi;",;!V." .j.;:
tv&it \v !;'':!:;s!i!:!J irnivv: :!.V vh^-^d n .
!
iVi.ji.ii i-,."iH !-V!|-=;iri-î^-^:1!:^:!1 Ih:' iAf.Vv-nr;,
iiiHUPita.'. |:!.'jj"i'ii'i-U.jiVr" v ! , 1 i^.Ti.Vii
.Vra.i n I !i. I ! "S V;T:; j' i j !'. V i i r . V ! j j =<''! Vi i ' i ! ' ! j : l . ' i . IY
- ! :
? ilVV - i"! T U "UVHP N p"." i !I i 1 ; ; V ! ;i IIIJ-! ! J 1
!!.!:!! iVniV .jf 1

:i i . V ' r ' . ' ' ! r W i
i"r!.'i'ir-iir !iir/iVi:!1 Li =-.i /'!:. .V:iVr ,'irr!:;i'Vii"i'.
O Y O M IYM' ! !
i . " i . i !.' u ! r . ' ; ! ; ! " ; r i ! ! i . ; j ! |;
iV'nhrf nw ": .; ::!!r=:!i;." .;;.:-..;';I:!.I!; .!-; !iv
r i v i ! :". ; n . ; : i r -
! :
1' C P ; ' ! U V i t E J s p V . Y ! ^ . ' i i':i,
;i I:I a i r, \ ';.=r,' !ri I :. I I HV -J R,'! ";.. ;;-RI.;; ri.
:!!' isM ftit nt- i : ".W t^ -vg.^T.Wvi J O ! ! !i?iu-j:' - =--:Ih;:-H sv .is:iis.'!1 > v A ; i-=--
s : !:
j:'iii i'i*Mo - ii Huni<. r r i r!V 'i""r! r r i ! !;: s- : = :

.
i V I' i' i Viri-i I. ;VI'! : ' A R \ \ I I | ;: : . !:.'! L I-I: , Y!-: i'
1
r. !!' rr-ri IVvJr-J "

1 - . . .. J W H C '!' J I - I I ' Y ! !
: I ' I I ' ! ! . : ' I"I'.'IRI>r-v. "VN':!'
.!< :'!:= ii;'!=:;ri': :i :':- .i liIT mriYrg
! ! i'
- ni-.;Y"i:r>.:iV !' ! > Q \ i i;;.},' .. , '.: ! > u vi i Vr
l
i',!i I u ,i!i ! ; Sfr i! i!!!:" f6> k . : "-siT!i.;: .i:.:;.|! J'S:-M:!S il1.1. i'i..' i.r !nr::i;vo!";rri:r .Ji :: .; i\i :j
U"n- I; :'.p y !". TH!:..:;'. n t TD.1 lVU.1 A i rrsn !Y:: .".'Sm.TiV'!!! , t ! 1 IHR ;i.yrv,i::;V
: ! ' : V S : I nhU: VI :r iini.vi . MIM ! I M .VS.V V W
! ! i'i i1"!1'!.-'! :i/!Vi ri rrhi'.!!.") i!; ri.;-. fiiiVri-a l".'i vi
R ' I ! ; : . ' E I K R F ,M:i.!! 'VIR ! > | :I!"<!' v.ijV, ;V!:i a; xiy! -=,:"i. i' !i i- Vi-! '/ij.' :;'.;-
::i'i.::,T! i l'. ! v jVi-!! .;'!! W. \ îYDi" !i '!? 11.
1
picture of the social dynamic of the particu- surement), then statistical portrayals can be
lar situation or program. This means that at quite powerful and succinct. The advan-
the time of data collection, each case, event, tages of qualitative portrayals of holistic set-
or setting under study, though treated as a tings and impacts are that greater attention
unique entity with its own particular mean- can be given to nuance, setting, interdepen-
ing and its own constellation of relation- dencies, complexities, idiosyncrasies, and
ships emerging from and related to the con- context. John Dewey (1956) articula ted what
text within which it occurs, is also thought of a holistic approach means for both teaching
as a window into the whole. Thus capturing and research if one wants to gain insight into
and documenting history, interconnections, and understand the world of the child:
and system relationships are part of field-
work. The child's life is an integral, a total one. He
The advantages of using quantitative passes quickly and readily from one topic to
variables and irvdicators are parsimony, pre- another, as from one spot to another, but is not
cision, and ease of analysis. Where key ele- conscious of transition or break. There is no
ments can be quantified with validity, reli- conscious isolation, hardly conscious distinc-
ability, and credibility, and where necessary tion. The things that occupy him are held to-
statistical assumptions can be met (e.g., lin- gether by the unity of the personal and social
earity, normality, and independence of mea- interests which his life carries along.. . . [His]
universe is fluid and fluent; its contents dis- How many program staffs have complained
solve and reform with amazing rapidity. But of the evaluation research monster?
after ali, it is the child's own world. It has the It is no simple task to undertake holistic
unity and completeness of his own life. analysis. The challenge is "to seek the es-
(pp. 5-6) sence of the life of the observed, to sum up,
to find a central unifying principie" (Bruyn
Qualitative sociologist Irwin Deutscher 1966:316). Again, Shapiro's (1973) work in
(1970) commented that despite the totality of evaluating innovative Follow Through class-
our personal experiences as living, working rooms is instructive. She found that stan-
human beings, social scientists have tended dardized test results could not be inter-
to focus their research on parts to the virtual preted without understanding the larger
exclusion of wholes: cultural and institutional context in which
the individual child is situated. Taking con-
text seriously, the topic of the next section, is
We knew that human behavior was rarely if an important element of holistic analysis.
ever directly influenced or explained by an An illuminative example of holistic think-
isolated variable; we knew that it was imposing came to me from a Portuguese colleague.
sible to assume that any set of such variables He told of driving in a remote area of his
was additive (with or without weighting); we country when he came upon a sizable herd
knew that the complex mathematics of the in- of sheep being driven along the road by a
teraction among any set of variables was in- shepherd. Seeing that he would be delayed
comprehensible to us. In effect, although we until the sheep could be turned off the road,
knew they did not exist, we defined them into he got out of the car and struck up a conver-
being. (p. 33) sation with the shepherd.
"How many sheep do you have?" he
While many would view this intense cri- asked.
tique of variable analysis as too extreme, the "I don't know," responded the young
reaction of many program staff to scientific man.
research is like the reaction of Copernicus to Surprised at this answer, the traveler
the astronomers of his day: "With them/' he asked, "How do you keep track of the flock if
observed, "it is as though an artist were to you don't know how many sheep there are?
gather the hands, feet, head, and other mem- How would you know if one was missing?"
bers for his images from diverse models, The shepherd seemed puzzled by the
each part excellently drawn, but not related question. Then he explained, "I don't need
to a single body, and since they in no way to count them. I know each one and I know
match each other, the result would be mon- the whole flock. I would know if the flock
ster rather than man" (from Kuhn 1970:83). was not whole."
Context Sensitivity
ny single act from any single person, put out of context, is

damnable.
Actor Kevin Spacey accepting the 2000 Academy Award

for Best Performance by an Actor in the film
American Beauty, explaining the film's message
Let's move, now, from sheep to elephants. text and as part of an ecological system in re-
One of the classic tales used to illustrate the la tion to other flora and fauna, in its natural
relationship between parts and wholes is the environment.
story of the nine blind people and the ele- When we say to someone, "You've taken
phant. Each person touches only one part of my comment out of context," we are saying,
the elephant and therefore knows only that You have distorted what I said, changed its
part. The person touching the ears thinks an meaning by omitting criticai context.
elephant is like a large, thin fan. The person In Victor Hugo's great classic Les
touching the tail thinks the elephant is like a Misrables, we first encounter Jean Valjean as
rope. The person touching the truck thinks a hardened criminal and common thief; then
of a snake. The legs feel like tree trunks, the we learn that he was originally sentenced to
elephanfs side like a tall wall. And so it five years in prison for stealing a loaf of
goes. The holistic point is that one must put bread for his sister's starving family. That
ali of these perspectives together to get a full added context for his "crime" changes our
picture of what an elephant actually looks understanding. The battle over standard-
like. ized sentencing guidelines in the criminal
But such a picture will still be limited, justice system is partly a debate about how
even distorted, if the only place one sees the much to allow judges sway in taking into ac-
elephant is in the 200 or at the circus. To un- count context and individual circumstances
derstand the elephanthow it developed, in pronouncing sentences.
how it uses its trunk, why it is so largeone Naturalistic inquhy preserves natural
must see it on the African savanna or in the context. Social psychology experiments un-
Asian jungle. In short, one must see it in con- der laboratory conditions strip observed ac-
tions from context. But that is the point of were obtained on 120 students from four
such laboratory experimentsto generate classrooms. The extraction of significant pre-
fndings that are context free. The scientific dictor variables is the purpose of the final
ideal of generalizing across time and space analysis. Interviews were conducted with
is the ideal of identifying principies that do teachers and principais to determine how test
not depend on context. In contrast, qual- scores were used. The analysis concludes with
itative inquiry elevates context as criticai to the researcher's interpreta tions. The research-
understanding. Portraitist Sara Lawrence- er wishes to thank those who cooperated in
Lightfoot (1997) explains why she finds con- this study.
text "crucial to the documentation of human
experience and organizational culture": This journal article abstract represents ac-
ademic writing as I was taught to do it in
By context, I mean the settingphysical, geo- graduate school. This writing style still pre-
graphic, temporal, historical cultural, aesthetic dominates in scholarly journals and books.
within which action takes place. Contextbe- No human being is visible in this writing.
comes the framework, the reference point, the The passive voice reigns. Instruments were
map, the ecological sphere; it is used to place selected; decisions were made; a model was
people and action in time and space and as a constructed; records were reviewed and
resource for understanding what they say and coded; data were obtained; predictor vari-
do. The context is rich in clues for interpreting ables were extracted; interviews were con-
the experience of the actors in the setting. We ducted. The warmth of thanks is extended
have no idea how to decipher or decode an by a role, the researcher: "The researcher
action, a gesture, a conversation, or an excla- wishes to thank those who cooperated." The
ma tion unless we see it embedded in context third-person, passive voice communicates a
(p- 41) message: This work is about procedures not
people. This academic style is employed to
project a sense of objectivity, control, and au-
Voice and Perspective:
thority. The overall impression is mechani-
Reflexivity
cal, robotlike, distant, detached, systematic,
and procedural. The research is the object of
ABSTRACT OF A SCHOOL attention. Any real, live human being, sub-
ACHIEVEMENT STUDY ject to ali the usual foibles of being human, is
barely implied, generally disguised, hidden
This study will delineate the major factors that away, and kept in the background.
affect school achievement Instruments were Contrast that academic voice with my
selected to measure achievement based on va- explana tion of how I analyzed a 10-day
lidity and reliability criteria. Decisions were coining-of-age experience with my son in
ma de about administering the tests in con- the Grand Canyon. (I presented part of the
junction with administrators taking into ac- analysis of that experience as Exhibit 1.3 in
count time and resource constraints. A the first chapter.) Here's an excerpt in which
regression model was constructed to test rela- I describe the analytical process.
tionships between various background vari-
ables and demonstrated achievement. School I'm not sure when the notion first took hold of
records were reviewed and coded to ascertain me that articulating altemative coming of age
students' background characteristics. Data paradigms might help elucidate our Canyon
experience. Before formally conceptualizmg notes, asks interview questions, and inter-
contrasting paradigm dimensions, I experi- prets responses. Self-awareness, then, can be
enced them as conflicting feelings that ema- an asset in both fieldwork and analysis. De-
nated from my struggle to sort out what I veloping appropriate self-awareness can be
wanted my son's initiation to be, while also a form of "sharpening the instrument"
grappling with defining my role in the pro- (Brown 1996:42). The methods section of a
cess. I suppose the idea of alternative para- qualitative study reports on the researcher's
digms first emerged the second night as I training, preparation, fieldwork procedures,
paced the narrow beach where White Creek and analytical processes. This is both the
intersects Shinumo and pondered the Great strength and weakness of qualitative meth-
Unconformity [a geologic reference] as meta- ods, the strength in that a well-trained, expe-
phor for the gap between tribal approaches to rienced, and astute observer adds value and
initiation and coming of age for contemporary credibility to the inquiry, while an ill-pre-
youth. In the weeks and months after our Can- pared, inexperienced, and imperceptive ob-
yon experience, far from languishing in the server casts doubt on what is reported. Judg-
throes of retox as I expected, the idea of con- ments about the significance of findings are
trasting paradgms stayed with me, as did the thus inevitably connected to the researcher's
Canyon experience. I started listing themes credibility, competence, thoroughness, and
and matching them with incidents and turn- integrity. Those judgments, precisely be-
ing points along the way. The sequence of inci- cause they are acknowledged as inevitably
dents became this book and the contrasting personal and perspective dependent, at least
themes became the basis for this closing chap- to some extent, invite response and dia-
ter, a way for me to figure out how what logue, rather than just acceptance or rejec-
started out as an initiation become a humanist tion.
coming of age celebra tion. (Patton 1999a:332) Reflexivity has entered the qualitative lex-
icon as a way of emphasizing the impor-
The contrast between the traditional aca- tance of self-awareness, political/cultural
demic voice and the personal voice of quali- consciousness, and ownership of one's per-
tative analysis recalls philosopher and theo- spective.
logian Martin Buber's (1923) influential
distinction between "I-It" and "I-Thou" re- In the rush of interest in qualitative research in
lationships. An I-It relationship regards the past 15 years, few topics have developed
other human beings from a distance, from a as broad a consensus as the relevance of ana-
superior vantage point of authority, as ob- lytic "reflexivity." By most accounts, reflexivi-
jects or subjects, things in the environment ty is a deconstructive exercise for locating the
to be examined and placed in abstract intersections of author, other, text, and world,
cause-effect chains. An I-Thou perspective, and for penetrating the representational exer-
in contrast, acknowledges the humanity of cise itself. (MacBeth 2001:35)
both self and others and implies relation-
ship, mutuality, and genuine dialogue. Being reflexive involves self-questioning
The perspective that the researcher brings and self-understanding, for "ali under-
to a qualitative inquiry is part of the context standing is self-understanding" (Schwandt
for the findings. A human being is the in- 1997a:xvi). To be reflexive, then, is to under-
strument of qualitative methods. A real, live take an ongoing examination of what I know
person makes observations, takes field and how I know it, "to have an ongoing con-
versation about experience while simulta- account, and communicate perspective and
neously living in the moment" (Hertz 1997: voice. Balancing criticai and creative analy-
viii). Reflexivity reminds the qualitative in- ses, description and interpretation, or direct
quirer to be attentive to and conscious of the quotation and synopsis also involves issues
cultural, political, social, linguistic, and of perspective, audience, purpose, and
ideological origins of one's own perspective voice. No rules or formula can tell a qualita-
and voice as well as the perspective and tive analyst precisely what balance is right
voices of those one interviews and those to or which voice to use, only that finding both
whom one reports. Exhibit 2.2 depicts this balance and voice is part of the work and
reflexive triangulation. challenge of qualitative inquiry, what Lewis
Writing in the first-person, active voice (2001) has acknowledged as "the difficulty
communicates the inquirer's self-aware role of trying to situate the I in narrative re-
in the inquiry: "I started listing themes and search" (p. 109).
matching them with incidents and turning In addition to finding voice, the criticai and
points along the way." The passive voice creative writing involved in qualitative
does not: "Themes were listed and matched analysis and synthesis challenge the in-
to incidents and turning points along the quirer to own onefs voice and perspective. Here,
way." Judith Brown (1996) captured the im- we owe much to feminist theory for high-
portance of the first-person voice in the title lighting and deepening our understanding
of her book The I in Science: Training to Utilize of the intricate and implicate relationships
Subjectivity in Research. By subjectivity she between language, voice, and consciousness
means "the domain of experiential self- (e.g., Gilgan 1982; Minnich 1990). We are
knowledge" (p. 1). Voice reveals and com- challenged by postmodern critiques of
municates this domain. knowledge to be clear about and own our
But voice is more than grammar. A credi- authorship of whatever we propound, to be
ble, authoritative, authentic, and trustwor- self-reflective, to acknowledge biases and
thy voice engages the reader through rich limitations, and to honor multiple perspec-
description, thoughtful sequencing, appro- tives (Greene 1998a, 1998b; Mabry 1997)
priate use of quotes, and contextual clarity while "accepting incredulity and doubt as
so that the reader joins the inquirer in the modal postmodern responses to ali at-
search for meaning. And there are choices of tempts to explain ourselves to ourselves"
voice: the didactic voice of the teacher; the (Schwandt 1997b:102). From struggles to lo-
searching, logical voice of the sleuth; the nar- cate and acknowledge the mevitably politi-
rator voice of the storyteller; the personal cal and moral nature of evaluative judg-
voice of the autoethnographer; the doubting ments, we are challenged to connect voice
voice of the skeptic; the intimacy of the in- and perspective to praxisacting in the
sider's voice; the detachment of the out- world with an appreciation for and recog-
sider's voice; the searching voice of uncer- nition of how those actions iriherently ex-
tainty; and the excited voice of discovery, to press social, political, and moral values
offer but a few examples. Just as point of (Schwandt 1989, 2000) and to personalize
view and voice have become focai points of evaluation (Kushner 2000), both by owning
writing good fiction and nonfiction, as in our own perspective and by taking seriously
Nancy Mairs's (1997) Voice Lessons: On Be- the responsibility to communicate authenti-
coming a (Woman) Writer, so too qualitative cally the perspectives of those we encounter
analysts are having to learn about, take into during our inquiry. These represent some
mSjaiMMiim Reflexive Questions: Triangulated Inquiry
Those studied \ / Those receiving the

(participants): \ Reflexive screens: / study (audience):
How do they know what \ Culture, age, gender, class, / How do they make
they know? What shapes and \ social status, education, / make sense of what 1 give
has shaped their woridview? \ family, politicai praxis, / them? What perspectives
How do they perceive me? \ language, vaiues / do they bring to the findings
Why? How do 1 know? 1 offer? How do they perceive
How do 1 perceive them? me? How do 1 perceive them?
Myself:
(as qualitative inquirer):
What do 1 know?
How do 1 know what know?
What shapes and has shaped my perspective?
With what voice do 1 share my perspective?
What do 1 do with what i have found?
of the more prominent contextual forces real-world observations through naturalis-

that have elevated the mportance of own- tic inquiry; openness, responsiveness, and
ing voice and perspective in qualitative flexibility through emergent designs; focus
analysis. through purposeful sampling; richness and
It takes no great self-awareness or self- depth through qualitative data; use of ali of
confidence to report a statistically signifi- one's capacities through personal experi-
cant t test with confidence intervals based on ence and engagement; balancing the criticai
a formula and calculations easily replicated and creative through a stance of empathic
and confirmed. It can take considerable neutrality; sensitivity to dynamic processes
self-awareness and confidence to report: I and systems; appreciation of idiosyncrasies
coded these 40 interviews, these are the through a unique case orienta tion; insight
themes I found, here is what I think they and understanding through inductive anal-
mean, and here is the process I undertook to ysis, contextual sensitivity, and a holistic
arrive at those meanings. The latter state- perspective; and authenticity and trustwor-
ment calls for, even demands, a sense of thiness through ownership of voice and per-
voice and perspective. spective. These are not absolute and univer-
sal characteristics of qualitative inquiry, but
rather strategic ideais that provide a direc-
S From Strategic Ideais tion and framework for developing specific
to Practical Choices designs and concrete data collection tactics.
Ideally, a pure qualitative inquiry strat-
The 12 themes of qualitative inquiry re- egy includes ali the themes and dimensions
viewed in this chapter are strategic ideais: identified in this chapter. For example, in an
ideal naturalistic/inductive inquiry the re- begin to affect the program quite directly
searcher neither manipulates the setting un- and intentionally (given the job of helping
der study nor predetermines what variables improve the program), thus moving away
or categories are worth measuring. In prac- from a purely naturalistic approach. As
tice, however, it is important to recognize evaluative feedback is used to improve the
that actually conducting holistic-inductive program, the evaluator may then move back
analysis and implementing naturalistic in- into a more naturalistic stance to observe
quiry are always a matter of degree. In mak- how the feedback-induced changes in the
ing this point, Guba (1978) has depicted the program unfold.
practice of naturalistic inquiry as a wave on In the same vein, the attempt to under-
which the investigator moves from varying stand a program or treatment as a whole
degrees of a "discovery mode" to varying does not mean that the investigator never
emphasis of a "verification mode" in at- becomes involved in component analysis or
tempting to understand the real world. As in looking at particular variables, dimen-
fieldwork begins, the inquirer is open to sions, and parts of the phenomenon under
whatever emerges from the data, a discov- study. Rather, it means that the qualitative
ery or inductive approach. Then, as the in- inquirer consciously works back and forth
quiry reveals patterns and major dimen- between parts and wholes, separate vari-
sions of interest, the investigator will begin ables, and complex, interwoven constella-
to focus ori verifying and elucidating what tions of variables in a sorting-out then
appears to be emerginga more deductive putting-back-together process. While stay-
approach to data collection and analysis. In ing true to a strategy that emphasizes the im-
essence, what is discovered may be verified portance of a holistic picture of the program,
by going back to the world under study and the qualitative evaluator recognizes that
examining the extent to which the emergent certain periods of fieldwork may focus on
analysis fits the phenomenon and zuorks to component, variable, and less-than-the-
explain what has been observed. Glaser and whole kinds of analysis.
Strauss (1967), in their classic framing of The practice and practicalities of field-
grounded theory, described what it means work also mean that the strategic mandate
for results to fit and work: "By 'fit' we mean to "get close" to the people and setting un-
that the categories must be readily (not forc- der study is neither absolute nor fixed.
ibly) applicable to and indicated by the data Closeness to and involvement with the peo-
under study; by 'work' we mean that they ple under study are most usefully viewed as
must be meaningfully relevant to and be variable dimensions. The personal styles
able to explain the behavior under study" and capabilities of evaluators will permit
(p. 3). Discovery and verification mean mov- and necessitate variance along these dimen-
ing back and forth between induction and sions. Variations in types of programs and
deduction, between experience and reflec- evaluation purposes will affect the extent to
tion on experience, and between greater de- which an evaluator can or ought to get close
grees and lesser degrees of naturalistic in- to the program staff and participants. More-
quiry. over, closeness is likely to vary over the
In program evaluation in particular, the course of an evaluation. At times the eval-
evaluator may, through feedback of initial uator may become totally immersed in the
findings to program participants and staff, program experience. These periods of im-
mersion may be followed by times of Beyond Competing

withdrawal and distance (for personal as Inquiry Paradigms
well as methodological reasons), to be fol-
lowed still later by new experiences of im- Having presente d the strategic ideais of
mersion in and direct experience with the qualitative inquiry and noted variations in
program. their practical implementation and attain-
Nor is it necessary to be a qualitative ment, before closing this chapter I want to
methods purist. Qualitative data can be acknowledge and comment on the contro-
collected and used in conjunction with versy that sometimes engulfs qualitative
quantitative data. Today's evaluator must methods. Students attempting to do qualita-
be sophisticated about matching research tive dissertations can get caught up in and
methods to the nuances of particular evalua- may have to defend, philosophically as well
tion questions and the idiosyncrasies of spe- as methodologically, the use of qualitative
cific stakeholder needs. Such an evaluator inquiry to skeptical committee members
needs a large repertoire of research methods who define doctoral-level work as rigorous
and techniques to use on a variety of prob- hypothesis testing. Evaluators may encoun-
lems. Thus, an evaluator may be called on to ter policymakers and funders who dismiss
use any and ali social science research meth- qualitative data as mere anecdote. The sta-
ods, including analyses of quantitative data, tistically addicted may poke fun at what
questionnaires, secondary data analysis, they call the "softness" of qualitative data.
cost-benefit and cost-effectiveness analyses, (In Western society, where anything can be
standardized tests, experimental designs, and often is sexualized, the distinction be-
unobtrusive measures, participant observa- tween "hard" data and "soft" data has addi-
tion, and in-depth interviewing. The evalua- tional nuances of meaning and innuendo.)
tion researcher works with intended users of Such encounters derive from a long-stand-
the findings to design an evaluation that ining methodological paradigms war. Though
cludes any and ali data that will help shed many have pronounced the war and even
light on important evaluation questions, the debate over (cf. Cook 1995; Greene
given constraints of resources and time. 1998a:36; Patton 1997a: 290-95), not every-
Such an evaluator is committed to research one has adopted a stance of methodological
designs that are relevant, meaningful, un- enlightenment and tolerance, namely, that
derstandable, and able to produce useful re- methodological orthodoxy, superiority, and
sults that are valid, reliable, and believable. purity should yield to methodological ap-
On many occasions a variety of data collec- propriateness, pragmatisin, and mutual re-
tion techniques and design approaches may spect. Therefore, a brief review of the para-
be used together. Multiple methods and a digms debate is in order. (Elsewhere I have
variety of data types can contribute to meth- provided a more extensive review of the
odological rigor. The ideal in evaluation de- methodological paradigms debate; Patton
signs is methodological appropriateness, 1997a: 265-99,1988c.)
design flexibility, and situational respon-
Philosophers of science and method-
siveness in the service of utility (Patton
ologists have been engaged in a long-stand-
1997a)not absolute allegiance to some
ing epistemological debate about the nature
ideal standard of paradigm purity and
of "reality" and knowledge. Thatphilosoph-
methodological orthodoxy.
ical debate finds its way into research and
evaluation in arguments over the goals of the variety of inquiry approaches has ex-
empirical studies and differences of opinion panded well beyond the simplistic
about what constitutes "good" research. In dichotomy between quantitative and quali-
its simplest and most strident formula tion, tative paradigms. In contrast to these two
this debate has centered on the relative value classically opposed orthodoxies, this book
of two different and competing inquiry par- offers a pragmatic strategy of matching con-
adigms: (1) using quantitative and experi- crete methods to specific questions, includ-
mental methods to generate and test hypo- ing the option of tactically mixing methods
thetical-deductive generalizations versus as needed and appropriate. My practical
(2) using qualitative and naturalistic ap- (and controversial) view is that one can learn
proaches to inductively and holistically un- to be a good interviewer or observer, and
derstand human experience and con- learn to make sense of the resulting data,
structed meanings in context-specific without first engaging in deep
settings. For example, Taylor and Bogdan epistemological reflection and philosophi-
(1984) contrast the Verstehen tradition, cal study. Such reflection and study can be
rooted in qualitative phenomenology, to
helpful to those so inclined, but it is not a
measurement-oriented positivism as fol-
prerequisite for fieldwork. Indeed, it can be
lows:
a hindrance. Getting some field experience
first, then studying philosophy of science,
Two major theoretical perspectives liave dom-
has much to recommend it as a learning
inated the social science scene. The irst, posi-
strategy. Otherwise, it's ali abstractions.
tivism, traces its origins in the social sciences to
Still, the paradigms debate is part of our
the great theorists of the riineteenth and early
methodological heritage and knowing a bit
twentieth centuries and especially to Auguste
about it, and its distortions (Shadish 1995b,
Comte and Emile Durkheim. The positivist
1995c), may deepen appreciation for the im-
seeks the facts or causes of social phenomena
portance of a strategic approach to methods
apart from the subjective states of individuais.
Durkheim told the social scientist to consider
decision making.
social facts, or social phenomena, as "things" A paradigm is a worldviewa way of
that exercise an externai influence on people. thinking about and making sense of the
The second theoretical perspective, which, complexities of the real world. As such, par-
following the lead of Deutscher, we will de- adigms are deeply embedded in the social-
scribe as phenomenological, has a longhistory in ization of adherents and practitioners. Para-
philosophy and sociology. The phenomen- digms tell us what is important, legitimate,
ologist is committed to understanding social and reasonable. Paradigms are also norma-
phenomena from the actor's own perspective. tive, telling the practitioner what to do with-
He or she examines how the world is experi- out the necessity of long existential or
enced. The important reality is what people epistemological consideration. But it is this
perceive it to be. (pp. 1-2) aspect of paradigms that constitutes both
strength and weaknessa strength in that it
Debate about these contrasting and com- makes action relatively easy, a weakness in
peting perspectives has been an important that the very reason for action is hidden in
part of the history of research and evalua- the unquestioned assumptions of the para-
tion, but, as Chapters 3, 4, and 9 will show, digm.
Now Calm down a bit Fm not criticizing your operation

down here... I'm just asking... .in your entry evaluation process,
do you operate from a qualitative or quantitative paradigm?
Are there paradigms after death?
Scientists work from models acquired through ask or debate what makes a particular prob-
education and through subsequent exposure lem or solution legitimate tempts us to sup-
to the literature often without quite knowing pose that, at least intuitively, they know the
or needing to know what characteristics have answer. But it may only indicate that neither
given these models the status of community the question nor the answer is felt to be rele-
paradigms. . . . That scientists do not usually vant to their research. Paradigms may be prior
to, more binding, and more complete than any make researchers biased in favor of and
set of rules for research that could be unequiv- against certain approaches.
ocally abstracted from them. (Kuhn 1970:46) While one may still encounter people
who rigidly confess allegiance to only quan-
But what does ali this matter to the stu- titative or qualitative methods, most practi-
dent interested in pursuing some research or tioners appear to have become eclectic and
evaluation question? It matters because pragmatic. Looking back, we can now see
paradigm-derived biases are the source of that the qualitative-quantitative debate
the distinctions mentioned earlier between oversimplified and often confused method-
"hard" data and "soft" data, empirical stud- ological and philosophical issues. For exam-
ies versus "mere anecdotes," and "objec- ple, the notion that some combinations of
tive" research versus "subjective" studies. methods and philosophy ever constituted
These labels reveal value-laden prejudices consistent, coherent, and stable paradigms
about what constitute credible and valuable has proved problematic. Shadish (1995c), for
contributions to knowledge. Such preju- example, in introducing an important set of
dices and paradigmatic blinders limit meth- articles aimed at "de-Kuhnifying" the de-
odological choices, flexibiHty, and creativity. bate, concluded that "there is little empirical
Adherence to a methodological paradigm evidence in support of such a Kuhnian para-
can lock researchers into unconscious pat- digm portrayal. . . . [T]he relevant concep-
terns of perception and behavior that dis- tual and philosophical issues are far more
guise the biased, predetermined nature of complex than the simple quantitative-quali-
their methods "decisions." Methods deci- tative dichotomy implies" (p. 48). Chapter 9
sions tend to stem from disciplinary pre- will revisit the quantitative-qualitative para-
scriptions, concerns about scientific status, digms debate in more depth as part of our
old methodological habits, and comfort examination of issues that affect judgments
with what the researcher knows best. about the quality and credibility of qualita-
Training and academic socialization tend to tive methods.
Pragmatism
/ - very thinker puts some portion an apparently stable world in peril.
John Dewey (1929)
While a paradigm offers a coherent assert that randomized experiments are "the
worldview, an anchor of stability and cer- standard against which other designs for
tainty in the real world sea of chos, operat- impact evaluation are judged" (p. 21). My
ing narrowly within any singular paradigm pragmatic stance aims to supersede one-
can be quite limiting. As a pragmatist, I take sided paradigm allegiance by increasing the
issue as much with the purist, one-sided ad- concrete and practical methodological op-
vocacy of Lincoln and Guba (1985), who be- tions available to researchers and evalua-
lieve that naturalistic inquiry is the only tors. Such pragmatism means judging the
valid and meaningful way to study human quality of a study by its intended purposes,
beings, as I do with the narrow, intolerant available resources, procedures followed,
stance of Boruch and Rindskopf (1984), who and results obtained, ali within a particular
context and for a specific audience. When a on strategic choices has conveyed, I hope,
new drug is tested before being made avail- the idea that a wide range of possibilities ex-
able to the general population, a dou- ists when selecting methods. The point is to
ble-blind randomized experiment to deter- do what makes sense, report fully on what
mine efficacy is the design of choice, with was done, why it was done, and what the
careful attention to controlled and carefully implications are for findings. Chapter 5 is
measured dosage and outcome interactions, devoted to design issues, including design
including side effects. But if the concern is flexibility, using multiple methods, and
whether people take the new drug appropri- making practical decisions.
ately, and one wants to know what people in A Sufi story about the wise fool Mulla
a group think about the new drug (e.g., an Nasrudin illustrates the importance of un-
antidepressant), how they make sense of derstanding the connections between strate-
taking or not taking it, what they believe gic ideais and practical tactics in real-world
about themselves as a result of experiencing situations. Real-world situations seldom re-
the drug, and how those around them deal semble the theoretical ideais taught in the
with it, then in-depth interviews and ob- classroom.
servations are the place to start. The impor-
tance of understanding alternative research Ideal Conditions for
paradigms is to sensitize researchers and Research: A Cautionary Tale
evaluators to the ways in which their meth-
odological prejudices, derived from their In his youth, Nasrudin received training
disciplinary socialization experiences, may in a small monastery noted for its excellence
reduce their methodological flexibility and in the teaching of martial arts. Nasrudin be-
adaptability. came highly skilled in self-defense and after
I reiterate: Being pragmatic allows one to two years of training both his peers and his
eschew methodological orthodoxy in favor teachers recognized his superior abilities.
of methodological appropriateness as the pri- Each day, it was the responsibility of one
mary criterion for judging methodological of the students to go to the village market to
quality, recognizing that different methods beg for alms and food. It happened that a
are appropriate for different situations. Situ- small band of three thieves moved into the
ational responsiveness means designing a area. They observed how the monastery ob-
study that is appropriate for a specific in- tained food daily and began hiding along
quiry situation or interest. A major purpose the path the students had to take back to the
of this book, and the focus of Chapter 4, is to monastery. As a returning student returned,
identify the kinds of research questions and laden with food and alms, the thieves would
program evaluation situations for which attack. After three days of such losses, the
qualitative inquiry is the appropriate meth- monastery's few supplies were exhausted. It
od of choice. was Nasrudin's turn to go to the village mar-
Paradigms are really about epistemology, ket. His elders and peers were confident that
ontology, and philosophy of science. As Nasrudin's martial arts skills were more
such, paradigms are important theoretical than sufficient to overcome the small band
constructs for illuminating fundamental as- of thieves.
sumptions about the nature of reality. But at At the end of the day, Nasrudin returned
the pragmatic levei of making concrete ragged, beaten, and empty-handed. Every-
methods decisions, this chapter's emphasis one was amazed. Nasrudin was taken im-
mediatelybefore the Master. "Nasrudin," he and conditions in which qualitative strate-

asked, "how is it that with ali your skill in gies and methods offer advantages. In
our ancient arts of defense, you were over- Chapter 4,1 will present a range and variety
come?" of situations and inquiry problems that par-
"But I did not use the ancient arts," re- ticularly lend theinselves to qualitative in-
plied Nasrudin. quiry. Chapter 5 will then discuss in more
Ali present were dumbfounded. An ex- detail some of the methodological trade-offs
planation was demanded. involved in adapting the strategic ideais of
"Ali of our competitions are preceded by qualitative methods to the practical realities
great and courteous ceremony," Nasrudin of conducting research and evaluation in the
explained. "We have learned that the open- field. Chapters devoted to observation, in-
ing prayers, the ceremonial cleansing, the terviewing, analysis, and enhancing the
bow to the Eastthese are essential to the quality and credibility of qualitative studies
ancient ways. The ruffians seemed not to un- follow that design chapter. To lay the
derstand thenecessity for these things. 1 did- groundwork for in-depth review of applica-
n't find the situa tion ideal enough to use the tions and methods, the next chapter exam-
methods you have taught us, Master." ines alternative theoretical frameworks that
are closely associated with and used to
guide qualitative inquiry.
On more than one occasion, researchers

or evaluators have told me of their belief in
the potential usefulness of qualitative meth- S Note
ods, but they tell me, "I just haven't found
the ideal situation in which to use them." 1. Excerpts in this chapter from the interview
Ideal situations are rare, but we will con- with Brackette F. Williams are used with her per-
sider throughout this book the questions mission.
Variety in Qualitative Inquiry
Theoretical Orientations
S p e c i a l (C\i-f+s
"Tell us again, Master, how it was in the beginning."

"In the beginning special gifts were given to different groups of people. The
caregivers were endowed with compassion for the less fortunate. The engineers
were given the ability to see what was not yet there. The carpenters were given
patience to set straight lines and perfect angles. The technicians were provided
with diligence so that they might conscientiously follow the blueprints and de-
tailed directions of others. The experimental scientists were given the certain be-
lief that the world could be manipulated according to their vision of it. The qual-
itative inquirers were gifted with a passion for depth, detail, and understanding
meanings. And so it went until, finally, there remained one last group and one
last gift. These were the explorers. To them was given the gift of curiosity that
they might forever see new worlds and uncover the many wonders of the
world."
"But what of the evaluators?" the children asked. "You have not mentioned
their special gift."
Halcolm smiled. "The evaluators, dear children, were spread throughout ali
the other groups, each endowed with the special gift of his or her own group,
and each using that gift in a special way."
"But does that not make for much arguing among evaluators about who has
the most special gift of ali?"
Halcolm grinned.
From Halcolm/s Origins of Human Species
13. 75
From Core Strategies to Rich cussing qualitative research as if it were one

Diversity approach. (Jacob 1988:16)
The last chapter presented 12 primary Major social sciences have drawn on and
threads that are woven through the tapestry contributed to qualitative methods in differ-
of qualitative inquiry. A central point of that ent ways depending on the interests of theo-
chapter was that different purposes, situa- rists and methodologists in a particular dis-
tions, questions, and resources will affect the cipline (cf. Brizuela et ai. 2000; Kuhns and
degree to which such qualitative ideais as Martorana 1982). The language of discourse
naturalistic inquiry, a holistic perspective, also varies. As Schwandt (1997a) has ob-
and inductive analysis can be realized in served in his very useful dictionary of quali-
practice. Yet, despite variation along the sev- tative terminology:
eral dimensions of qualitative inquiry, there
are still core strategies and directions that Qualitative inquiry... is a set of multiple prac-
differentiate a qualitative/naturalistic strat- tices in which words in methodological and
egy from a quantitative/experimental one, philosophical vocabularies acquire different
as well as places where they can usefully be meanings in their use or in particular acts of
combined to complement each other (e.g., speaking about the meariing of the practice.
Tashakkori and Teddlie 1998). This chapter These different ways of speaking form some-
will present the rich menu of alternative pos- thing more like a constellation of contested
sibilities zvithin qualitative research by fo- practices than an integrated, readily survey-
cusing on different theoretical perspectives able order. There are multiple sources and
that are associated with qualitative inquiry. kinds of disputes, but generally they involve
Qualitative inquiry is not a single, mono- different ways of conceiving of the aim of
lithic approach to research and evaluation. qualitative inquiry stemming from different
Discussions such as that in Chapter 2 that fo- traditions of thought. (p. xiv)
cus on differentiating primary strategies of
qualitative/naturalistic methods from those Those coming new to qualitative inquiry
of quantitative/experimental methods can are understandably confused and even
leave the impression that there are only two discombobulated by the diverse terminol-
methodological or paradigmatic alterna- ogy and contested practices they encounter.
tives. In fact, as we "turn inward in qualita- Phenomenology. Hermeneutics. Ethnometh-
tive research," we find "an exhilarating and odology. Semiotics. Heuristics. Phenomen-
at times exhausting proliferation of types ography. Such language! Exhibit 3.1 repro-
zvithin the qualitative paradigm (Page duces a letter of lamentation I received fol-
2000:3). io wing publication of the first edition of this
book, which did not include the current
When one looks more c l o s e l y . . . the apparent chapter.
unity of the qualitative approach vanishes, This chapter sorts through some of the
and one sees considerable diversity. What has major perspectives and traditions that in-
been called "qualitative research" conveys dif- form the rich variety that is qualitative in-
ferent meanings to different people. Needless quiry. We shall look at how varying theoreti-
to say, this has ca used considerable confusion. cal traditions emphasize different questions
. . . A major source of the confusion lies in dis- and how these particular emphases can
Variety in Qualitative Inquiri/ J. 77
E X H B I T 3.1 Which Approach Is Rght?
Helpl
Dear Dr. Patton:
I desperately need your help. I am a graduate student in education, planning to do

my dissertation observing classrooms and teachers identified as innovative and ef-
fective. want to see if they share any common approachesorwisdom that might be
considered "best practices." I took this idea to one professor who asked me if was
proposng a phenomenologicai or grounded theory study. When i asked what the dif-
ference was, he said it was myjob to find out. I've read about both but am stii con-
fused. Another professortold me i could do a qualitative study, but that asking about
"best practices" meant that I was a positivist not a phenomenologist. Another grad
student was tod to "use a hermeneuticframing," butshe'sin a different department
with a differenttopic. I'm a formerschooiteacherand, Ithink, a prettygoodobserver
and interviewer. gotvery excited reading your book about the valueof in-depth ob-
servations and interviewing, and that's where I got the idea for my dissertation, but
now Tm being toid I have to fit into one of these categories. Please tell me which one
is rightformy study. I don'tcare which one itis. Ijustwanttogeton withstudying in-
novative classrooms. I feel lost and am on the verge of just doing a questionnaire
where these phiiosophy questions don't seem to get asked. But if you can tel me
which approach is right, I might still be able to do what l want to do. Help!!!
Dear :
Yourdilemma is common.The distinctionsyoiTre being asked to makeare, indeed,

difficult-and not everyone agrees about what these terms and traditions mean. I
didn't include them in my book in the hopes that the methods of in-depth interview-
ing and observation could stand on their own. As youVe discovered, you don't need a
class in phiiosophy to design good questionnaires, though an argument can be made
that people using questionnaires and statistics would benefit from reflection on their
epistemological (nature and justification of knowledge) and ontological (nature of
reality) assumptions. Unfortunately, a lot of qualitative courses spend more time on
epistemology than methods, which may make students better philosophers than in-
terviewers. Some balance is needed. Your professors are doing you a service by having
you struggle with understanding different qualitative schools of thought because
what approach you take does make a difference-and students of qualitative inquiry
should be expected to know at least the major competing and contrasting traditions,
just as those doing statistical tests need to understand what different tests do. In the
next edition of my book Ml include a chapter reviewing major phlosophical and
methodological traditions. But that won't help you now.
Toansweryourquestion directly, there is no "right" approach any more than there
is a "right" fruit-apples, oranges, passion fruit. What you eat is a matter of personal
taste, availability, prce, history, and preference. Since you are also serving others
(your doctoral committee), their preferences come into play, as you well know. Each
tradition of qualitative inquiry offers a different emphasis, framework.or focus. I am
reluctant to offer a recommendation about which tradition fits your work best, but
(continued)
EXHIBIT 3.1 Continued
until the nextedition ofmy book is out with a new chapter that sorts through thevari-
ous traditions, i feel obliged to offeryou some guidance. So, here are three alternatives
to consider.
First, because you portray yourself as a pragmatic, experienced practitioner, you
could frameyour5tudy as qualitative, utilization-focused evaluation research [Patton
1997a]. You have to specify intended users for your study (for example, innovative
teachers and curriculum designers) and intended uses (facilitating discussing about
"effective practices"). This puts you in a tradition of generating practical and useful
knowledge for action in the tradition of refiective practice [Schon 1983]. Yourfocus
would be perceived patterns of effectiveness.
If that doesn't work and your committee insists on a more explicitly phiiosophical
or theoretical framework for your inquiry, you might consider either "social
constructionism" or "realism," which are two of the most general (and contradictory)
of the traditions informing qualitative inquiry. I must warn you that there are compet-
ing versions of constructionism and realism (academics without arguments are like
paraders without costumes or sports teams without u n i f o r m s - i f s how the players
differentiate themselves and figure out who to applaud). Either of these traditions will
guide you in thinking about how people in particular contexts (in your case, schools)
individualy and collectiveiy construct meaning and knowledge (in your case, effective
or "best" practices).
The third alternative involves a change of topic, which may sound like bad news.
The good news isthatyouVe already collected a lot ofthe data. You could do a disser-
tation on the social constructions of qualitative paradigms using your professors as
subjects. Obviously, you've already been doing participant observation on this topic.
Or you might do a hermeneutic study of qualitative terminology. Or a phenom-
enological study on the experience of graduate students trying to frame a qualitative
study. Or a heuristic inquiry into your own experience of qualitative design. Or but
thafs where you started out, sn't it.
Best wishes, whateveryou decide.
affect the analytical framework that guides This chapter will be of particular interest
fieldwork and interpretation. Understand- to social scientists conducting basic or ap-
ing the divergent theoretical and phiiosoph- plied research, and students doing disserta-
ical traditions that have influenced qualita- tions, because their work is typically based
tive inquiry is especially important in the on and aimed at contributing to theory. The
design stage when the focus of fieldwork next chapter, in contrast, will focus on practi-
and interviewing is determined. Weaving cal and concrete evaluation and action re-
together theory-based inquiry traditions search questions appropriate for qualitative
and qualitative methods will reveal a rich inquiry, though theoretical understandings
tapestry with many threads of differing tex- can be important for practitioners and pol-
ture, color, length, and purpose. icy analysts because "theoretical concep-
tions shape public arguments, giving people their introduction to the Handbook of Qualita-
the concepts they use and shaping the alter- tive Research trace six phases of qualitative
na tives they consider" (Nussbaum 2001:35). research history that help explain the dra-
Taken together, this chapter and the next of- matically varying conceptions of what con-
fer a broad range of goals for and ap- stitutes qualitative research.
proaches to qualitative research. Chapter 5
will then integrate theoretical and practical 1. During the "traditional period" of colo-
concerns by introducing a typology of re- nial research (up to World War II),
search purposes to elucidate the design, ethnographers, influenced by positiv-
methods, and analysis implica tions of vary- ism, strove for objectivity in their field-
ing purposes. work and reports.
2. The "modernist phase" (to the 1970s)

Alternative Ways was a time in which qualitative re-
of Distinguishing searchers emphasized methodological
Qualitative Traditions rigor and procedural formalism as they
sought acceptance within social science
There is no definitive way to categorize and reacted against postpositivism's
the various philosophical and theoretical emergent emphasis on interpretivism.
perspectives that have influenced and that
distinguish types of qualitative inquiry. Lin- 3. During the "blurred genres phase"
coln and Guba (2000) identify five "alterna- (1970-1986), a large number of alterna-
tive inquiry paradigms": positivism, post- tive approaches emerged, creating com-
positivism, criticai theory, constructivism, peti tion and confusion, the legacy of
and participatory. Schwandt (2000) dis- which remains in the daunting jargon
cusses "three epistemological stances for and labels of qualitative perspectives:
qualitative inquiry: interpretivism, herme- structuralism, symbolic interactionism,
neutics, and social constructionism." Crotty phenomenology, ethnomethodology, crit-
(1998) also offers three primary epistemo- icai theory, semiotics, neopositivism,
logical influences: objectivism, construc- micro-macro descriptivism, neo-Marx-
tionism, and subjectivism; these, he posits, ism, poststructuralism, naturalism, con-
have influenced in varying degrees different structionism, and deconstructionism.
theoretical perspectives: positivism (and
postpositivism), interpretivism (symbolic 4. Next carne the "crisis of representation"
interaction, phenomenology, hermeneu- that focused on issues of reflexivity,
tics), criticai inquiry, feminism, and post- power, privilege, race, gender, and so-
modernism. Creswell (1998) distinguishes cioeconomic classali of which under-
"five qualitative traditions of inquiry": biog- mined traditional notions of validity
raphy, phenomenology, grounded theory, and neutrality.
ethnography, and case study.
While there is some overlap among these 5. The "fifth moment" describes recent
frameworks, there are also important differ- history and is characterized as "a triple
ences reflecting varying experiences with crisis of representation, legitimation,
and emphases within the history of qualita- and praxis" (p. 17) in which the inevita-
tive research. Denzin and Lincoln (2000b) in bly creative and interpretive nature of
qualitative writing is put under the mi- plines given birth by the mother of ali dis-
croscope, mcluding the perspective of ciplines, philosophy, can be distinguished
the qualitative writer, and searching by their core burning questions. For sociol-
questions are raised about how to evalu- ogy, the burning question is the Hobbesian
ate the quality of qualitative research question of order: What holds society or so-
and evaluation. During this period, cial groups together? What keeps them from
more activist, explicitly political, and falling apart? Psychology asks: Why do in-
participatory approaches sought legiti- dividuais think, feel, and act as they do? Po-
macy as, for example, in "empower- litical science asks: What is the nature of
ment evaluation" (Fetterman, Kaftar- power, how it is distributed, and with what
ian, and Wandersman 1996) and using consequences? Economics studies how re-
qualitative/interpretive writing "to ad- sources are produced and distributed.
vance the promises of radical demo- Disciplines and subdisciplines reveal lay-
cratic racial justice embodied m the ers of questions. Biologists inquire into the
post-civil rights, Chicana/Chicano and nature and varie ty of life. Botanists ask how
Black Arts Aesthetic movements" (Den- plants grow, while agriculturists investigate
zin 2000a:256). producing food, and agronomists narrow
their focus still further to field crops.
6. In the sixth phase, which Denzin and To be sure, reducing any complex and
Lincoln call "p os texperimental," the multifaceted discipline to a singular burning
boundaries of qualitative inquiry are ex- question oversimplifi.es. But what is gained
panded to include creative nonfiction, are clarity and focus about what distin-
autobiographical ethnography, poetic guishes one lineage of inquiry from another.
representations, and multimedia pre- It is precisely that clarity and focus I shall
sentations. strive for in identifying the burning ques-
tions that distinguish major lineages of
They clearly expect qualitative inquiry to qualitative inquiry. In doing so, I shall dis-
continue developing in new directions for please those who prefer to separate para-
they call the future the "seventh moment" digms from philosophies from theoretical
or perhaps this will be the moment of rest, orientations from design strategies. For ex-
when qualitative researchers cease debating ample, social constructivism may be viewed
their differences and celebrate the marvel- as a paradigm, ethnography may be consid-
ous varie ty of their creations. ered a research strategy, and symbolic
interactionism may be examined as a theo-
Foundational Questions retical framework. However, distinctions
between paradigmatic, strategic, and theo-
This chapter, in contrast with the work of retical dimensions within any particular
qualitative theorists and historians cited approach are both arguable and somewhat
above, distinguishes theoretical perspectives arbitrary. Therefore, I have circumvented
by their foundational questions. A founda- those distinctions by focusing on and distin-
tional or burning question, like the mythical guishing foundational questions as the basis
burning bush of Moses, blazes with heat for understanding and contrasting long-
(controversy) and light (wisdom) but is not standing and emergent qualitative inquiry
consumed (is never fully answered). Disci- approaches.
Theoretical Traditions Anthropologists have traditionally stud-

and Orientations ied nonliterate cultures in remote settings,
what were often thought of as "primitive" or
"exotic" cultures. As a result, anthropology
Ethnography and ethnographers became intertwined
with Western colonialism, sometimes resist-
ing imperialism in efforts to sustam native
Foundatio^ql question: cultures and sometimes as handmaidens to
W h a t is the CM l+U y-e. of +his gfowp conquering empires as their findings were
oj' people? used to overcome resistance to change and
manage subjugated peoples.
Ethnography, the primary method of an- Modern anthropologists apply ethno-
thropology, is the earliest distinct tradition graphic methods to the study of contem-
of qualitative inquiry. The notion of culture porary society and social problems, for ex-
is central to ethnography Ethnos is the Greek ample, technological diffusion, globaliza-
word for "a people" or cultural group. The tion, environmental degradation, poverty,
study of ethnos then, or ethnography, is "de- the gap between rich and poor, and societal
voted to describing ways of life of human- breakdown (Scudder 1999); education (Spind-
kind . . . , a social scientific description of a ler and Hammond 2000); addiction (Agar
people and the cultural basis of their 1986; Agar and Reisinger 1999); child labor
peoplehood" (Vidich and Lyman 2000:38). (Kenny 1999); intercultural understanding
Ethnographic inquiry takes as its central and in schools (Jervis 1999); and international
guiding assumption that any human group border conflicts (Hart 1999), to give but a
of people interacting together for a period of few of many examples. The importance of
time will evolve a culture. Culture is that col- understanding culture, especially in relation
lection of behavior patterns and beliefs that to change efforts of ali kinds, is the corner-
constitutes "standards for deciding what is, stone of "applied ethnography" as it has
standards for deciding what can be, stan- emerged in modern society (Chambers
dards for deciding how one feels about it, 2000). This can be seen in the ongoing re-
standards for deciding what to do about it, ports of members of the Society for Applied
and standards for deciding how to go about Anthropology since its founding in 1941.
doing it" (Goodenough 1971:21-22). The pri- Whyte (1984), for example, has collected a
mary method of ethnographers is partici- number of classic examples of ethnographic
pant observation in the tradition of anthro- fieldwork applied to problems of industrial
pology. This means intensive fieldwork in democracies.
which the investigator is immersed in the Since the 1980s, understanding culture
culture under study. While ethnographers has become central in organizational studies
share an interest in culture, there is debate (Morgan 1986, 1989; Pettigrew 1983) and in
about the nature of its essence (Douglass much organizational development work
2000) as well as several different styles of (Raia and Margulies 1985; Louis 1983), in-
ethnography, including the classic holistic cluding major efforts to change an organiza-
style of Benedict and Mead, the semiotic tion's culture (Schein 1985; Silverzweig and
style of Boas and Geertz, and the behaviorist Allen 1976). Organizational ethnography
style of the Whitings (Sanday 1983). has a distinguished history that can be
82 !f. CONCEPTUALISSUES IN QUALITATIVE INQUIRY
Cross-cultural perspective
traced back to the influential Hawthome tion (Fetterman 1984,1989) and applied edu-
electric plant study that began in 1927 cational research (Dobbert 1982). Programs
(Schwartzman 1993). Ethnography has also develop cultures, just as organizations do.
emerged as an approach to program evalua- The program's culture can be thought of as
Culture, Culture Everywhere: Sample of Media

E X H I B I T 3.2 Headlines
"Stopping the Culture of Violence"

(appied alternatively to gangs, families, neighborhoods, television shows, hockey games,
even politics)
"Inside the Culture of Sports"
(Nike culture, Little League culture, "soccer mom" culture)
"Learn the Culture of Day-Trading Stocks"
(virtual trading culture, online business culture)
"Girls and the Barbie Doll Culture"
(or boys and the G.l. Joe action figure culture, kids and the Stor Wars culture)
"Eat Right: Fast Food Culture"
("Golden Arches Culture Around the World")
"The Culture of Negative Political Campaigning: Why It Wins"
Mu5ic, dance, or art culture
Postmodern culture
Virtual culture (aka Internet or Web culture)
part of the program's treatment. As such, the thinking about ethnographic research
culture affects both program processes and (LeCompte and Schensul 1999), Living the
outcomes. Improving a program, then, may Ethnographic Life (Rose 1990), Selecting
include changing the program's culture. An Ethnographic Informants (Johnson 1990), and
ethnographic evaluation would both facili- how to write ethnographies (Atkinson 1992)
tate and assess such change. or write the methods section in ethno-
Ironically, perhaps, awareness of the im- graphic reports (Stewart 1998). The Ethnog-
portance of culture has found its way into rapher's Toolkit hasbeenpublished (Schensul
popular culture and mass media to such an and LeCompte 1999).
extent that the term shows up nearly ubiqui- While traditionally ethnographers have
tously as an implied explanation for ali used the methods of participant observation
kinds of social problems and phenomena, as and intensive fieldwork to study everything
shown in Exhibit 3.2. from small groups to nation-states, what it
Ethnographic methods continue to de- means to "partcipate" or be in the "field"
velop as new approaches emerge, for exam- or even be a "group" has changed with
ple, Doing Team Ethnography (Erickson and the World Wide Web and the emergence of
Stull 1998), and new issues surface, for ex- the virtual ethnographerstudying people
ample, Ethnographic Decision Tree Modeling connected through distributed electronic
(Gladwin 1989) or Writing the New Ethnogra- environments (Ruhleder 2000). Neverthe-
phy (Goodall 2000). Other ethnographic less, whether doing ethnography in virtual
methodologists continue to delve deeply space, a nonliterate community, a multina-
into classic issues such as paradigms for tional Corporation, or an inner-city school,
what makes the approach distinct is the mat- uations, the other became the program cli-
ter of interpreting and applying the findings ent, the student, the welfare recipient, the
from a cultural perspective (Wolcott 1980:59; patient, the alcoholic, the homeless person,
Chambers 2000:852). the victim, the perpetrator, or the recidivist.
In organizational studies, the other was the
worker, the manager, the leader, the fol-
Autoethnography and
lower, and/or the board of directors. The
Evocative Forms of Inquiry
others were observed, interviewed, de-
scribed, and their culture conceptualized,
analyzed, and interpreted. Capturing and
PouiAda-Hcmal question;
being true to the perspective of those stud-
How does my own e^cpenence of
ied, what came to be called the emic per-
tkis cw Itu counect witk and offe.?
spective, or the insider's perspective, was
insigkts about+kis culture, situa+ion,
contrasted with the ethnographer's per-
eveiAf> and/ot* way of life?
spective, the etic, or outsider's, view. The etic
viewpoint of the ethnographer implied
We turn now from the earliest qualitative some important degree of detachment or
tradition, ethnography, to the lates t and still "higher" levei of conceptual analysis and
emergent approach: autoethnography. Eth- abstraction. To the extent that ethnog-
nography and autoethnography might be raphers reported on their own experiences
thought of as bookends, or opposite ends of as participant observers, it was primarily
a qualitative continuum, that frame a large methodological reporting related to how
number of distinct qualitative approaches to they collected data and how, or the extent to
be reviewed in this chapter. By considering which, they maintained detachment. To "go
them one after the other throughout this native" was to lose perspective.
chapter, it is hoped you'11 get a sense of the In the new postcolonial and postinodern
range of issues that distinguish qualitative world at the beginning of the 21 st century,
approaches. the relationship between the observed and
Ethnography first emerged as a method the observer has been called into question at
for studying and understanding the other. It every levei. Postcolonial sensitivities raise
was fascination with "exotic otherness" that questions about imbalances of power,
attracted Europeans to study the peoples of wealth, and privilege between ethnog-
frica, Asia, the South Sea Islands, and the raphers and those they would study, includ-
Amricas. "The life world of the 'primitive' ing criticai political questions about how
was thought to be the window through findings will be used. Postmodern critiques
which the prehistoric past could be seen, de- and deconstruction of classic ethnographies
scribed, and understood" (Vidich and have raised fundamental questions about
Lyman 2000:46). In the United States, for ed- how the values and cultural background of
ucated, White, university-based Americans the observer affect what is observed while
the others were Blacks, American Indians, re- also raising doubts about the desirability, in-
cent immigrants, working-class families, deed, the possibility, of detachment. Then
and the inner-city poor (and for that matter, there is the basic question of how an ethnog-
anyone else not well educated, White, and rapher might study her or his own culture.
university based). In recent times, when eth- What if there is no other as the focus of study,
nography began to be used in program eval- but I want to study the culture of my own
BMaiiii^lictf Varieties of Autoethnog raphy: A Partial Lexicology
David Hayano (1979) is credited with originating the term autaethnographyto describe studies
by anthropoogists of their own cultures. In their extensive review, Ellis and Bochner (2000) focus
on studying one's own culture and oneself as part of that culture to understand and illuminate a
way of life. They cite a large number of phrases that have emerged both to support this emergent
frontierof qualitative inquiry and to confuse exactly whatitis. In the end, they conclude, "increas-
ingly, autoethnography has become the term of choice in describing studies and procedures that
connect the personal to the cultural" (p. 740).
Other terms include
Autobiographical ethnography Lived experience

Auto-observation Literary ethnography
Ethnographic poetics Narrative ethnography
Creative analytic practice ethnography Native ethnography
Criticai autobiography Narratives of the self
Ethnic autobiography New ethnography
Ethnographic memoir Personal ethnography
Ethnobiog raphy Personal experience narratives
Ethnographic autobiography Personal narratives
Ethnographic stories Postmodern ethnography
Evocative narratives Reflexive ethnography
Experimental ethnography Self-ethnog raphy
First-person accounts Self-stories
Indigenous ethnography Socioautobiog raphy
Interpretive biography Sociopoetics
group, my own community, my own organi- Autoethnography is an autobiographical

zation, and the way of life of people like me, genre of writing and research that displays
or people I regularly encounter, or my own multiple layers of consciousness, connecting
cultural experiences? the personal to the cultural. Back and forth
These developments have contributed to autoethnographers gaze, first through an
the emergence of autoethnographystudy- ethnographic wide-angle lens, focusing out-
ing one's own culture and oneself as part of ward on social and the cultural aspects of their
that cultureand its many variations. personal experience; then, they look mward,
Goodall (2000) calls this the " n e w ethnogra- exposing a vulnerable self that is moved by
phy": "creative narratives shaped out of a and may move through, refract, and resist cul-
writer's personal experiences within a cultural interpretations. As they zoom backward
ture and addressed to academic and public and forward, inward and outward, distinc-
audiences" (p. 9). Exhibit 3.3 offers a list of tions between the personal and cultural be-
many, but not ali, of the terms that have come blurred, sometimes beyond distinct
emerged to describe variations in this gen- recognition. Usually written in first-person
eral approach. Carolyn Ellis (Ellis and voice, autoethnographic texts appear in a vari-
Bochner 2000) describes it this way: ety of formsshort stories, poetry, fiction,
novis, photographic essays, personal essays, sional life. The point is that, for an ethnogra-
journals, fragmented and layered writing, and pher, any experienceat home or abroad, of
social science prose. In these texts, concrete ac- self or of otheroffers the potential to become
tion, dialogue, emotion, einbodiment, spiritu- fieldwork. . . . For me, my personal and my
ality, and self-consciousness are featured, ethnographic persona have become so inter-
appearing as relational and institutional sto- twined that it would be impossible to separate
ries affected by history, social structure, and them even if I wanted to do so. (p. 317)
culture, which themselves are dialectically re-
vealed through action, feeling, thought, and
Anthropologist Mary Catherine Bate-
language. (p. 739)
son's (2000) autoethnographic description
of teaching a seminar at Spelman College in
In autoethnography, then, you use your
Atlanta, Gergia, includes detailed attention
own experiences to gamer insights into the
to the personal challenge she experienced in
larger culture or subculture of which you are
trying to decide how to categorize students
a part. Great variability exists in the extent to
of different ages in contemporary American
which autoethnographers make themselves
universities, for example, by calling older
the focus of the analysis, how much they
participants "elders." Aaron Tumer (2000)
keep their role as social scientist in the fore-
of Brunel University in the United Kingdom
ground, the extent to which they use the sen-
has explored using one's own body as a
sitizing notion of culture, at least explicitly,
source of data in ethnography, what he calls
to guide their analysis, and how personal
"embodied ethnography."
the writing is. At the center, however, what
Such personal writing is controversial
distinguishes autoethnography from eth-
among qualitative theorists because of its
nography is self-awareness about and re-
"rampant subjectivism" (Crotty 1998:48).
porting of one's own experiences and intro-
Many social science academics object to the
spections as a primary data source. Ellis
way it blurs the lines between social science
describes this process as follows:
and literary writing. One sociologist told me
angrily that those who want to write creative
I start with my personal life. I pay attention to nonfiction or poetry should find their way to
my physical feelings, thoughts, and emotions. the English Department of the university
I use what I call systematic sociological intro- and leave sociology to sociologists. Richard-
spection and emotional recall to try to under- son (2000b), in contrast, sees the integration
stand an experience I've lived through. Then I of art, literature, and social science as pre-
write my experience as a story. By exploring a cisely the point, bringing together creative
particular life, I hope to understand a way of and criticai aspects of inquiry. She suggests
life. (Ellis and Bochner 2000:737) that what these various new approaches and
emphases share is that "they are produced
In writing about his experiences in a through creative analytic practices," which
"New Age ashram located in Pennsylva- leads her to call "this class of ethnographies
nia," Bruner (1996) confronted the intersec- creative analytic practice ethnography" (Rich-
tion of the ethnographic and the personal: ardson 2000b:929). While the ethnographic
aspect of this work is constructed on a foun-
What started out as part of my personal life dation of careful research and fieldwork
was soon transformed mto part of my profes- (p. 937), the creative element resides primar-
ily in the writing, which she emphasizes is, a credibleaccount of a cultural, so-
itself, "a method of inquiry, a way of finding cial, individual, or communal sense of
out about yourself and your topic" (p. 923). the "real"? (Richardson 2000a:254,
But how is one to judge the quality of such 2000b :937)
nontraditional social scientific approaches
that encourage personal and creative eth- These criteria open up the possibility of
nographic writing? Richardson (2000b) has new writing formats. Elliot Eisner (1996), a
responded to this challenge by asserting that former president of the American Educa-
creative analytic practice ethnography should tional Research Association, has argued that
be held to "high and difficult standards; a novel as a form of qualitative repor ting
mere novelty does not suffice" (p. 937). She could be a legitimate form for a doctoral dis-
offers five criteria of quality drawn from sertation in social science or education. In
both science and creative arts. that vein, he has suggested that in "the new
frontier in qualitative research methodol-
1. Substantive contribution: Does this piece ogy" an artistic qualitative social science
contribute to our understanding of so- contribution canbe assessed by the "number
cial life? Does the writer demonstrate a and quality of the questions that the work
deeply grounded (if embedded) social raises" as much as by any conclusions of-
scientific perspective? How has this per- fered (Eisner 1997:268). In this regard, emi-
spective informed the construction of nent evaluator Emie House (1991) reminds
the text? us that where evaluation reports are con-
cerned, the possibility of fiction is always a
2. Aesthetic merit: Does this piece suc- subtext: "Our evaluation repor t proved to be
ceeded aesthetically? Does the use of so readable many people became enrap-
creative analytic practices open up the tured by it. Some said it read like a novel.
text, invite interpretive responses? Is the Others said it was a novel" (p. 113).
text artistically shaped, satisfying, com- Poetry is another artistic genre that has
plex, and not boring? emerged in ethnographic reporting. Glesne
(1997) converted interview transcripts into
3. Reflexivity: How has the author's sub- poems because she found poetry better cap-
jectivity been both a producer and a tured and communicated what her inter-
product of this text? Is there adequate view with an 86-year-old professor in Puerto
self-awareness and self-exposure for the Rico opened up and revealed. Richardson
reader to make judgments about the (1998) has published a number of field-
point of view? work-based poems, reflecting his view that
poetry offers a language especially
4. Impact: Does this affect me? Emo- well-suited "for those special, strange, even
tionally? Intellectually? Does it generate mysterious moments when bits and pieces
new questions? Move me to write? suddenly coalesce . . . , when the ethnogra-
Move me to try new research practices? pher, away from home and in a strange cul-
Move me to action? ture, has a heightened sense of the frailty of
being human. In such a sense, poetry ap-
5. Expression of a reality: Does this text em- pears to be a way of communicating in-
body a fleshed out, einbodied sense of stances when we feel truth has shown its
lived experience? Does it seem a "true" face" (p. 451). Travisano (1998) included po-
dictions they experience. Ironically, many

aren't observant enough of the world around
them. The self-questioning autoethnography
demands is extremely difficult. So is confront-
|||S||lfi:i||l:]- =; i' ai !:-.;!::vii;;j-' m ^-HS-I! Y '.; ing things about yourself tliat are less than
flattering. Believe me, honest autoethnog-
! m^Tlfety %S|la iftffi ik >p> raphy exploration generates a lot of fears and
y p ^ ^ ^ k s m m !
M!!: !;': ';!!]'!'irj =!;! | doubtsand einotional pain. Just when you
-/'jfufiifc lîi'!!!. th - Wffflffijriliil liiiTil !tf i: think you can't stand the pain anymore, well,
that's when the real work has only begun.
lffi^ :; Then there's the vulnerability of revealing
^^jriliiiiiii:^:iij... . ; . . ' ;:. j yourself, not being able to take back what
you've written or having any control over
how readers interpret it. lt's hard not to feel
your life is being critiqued as well as your
etry in his "autobiography of an ethnic iden- work. I t c a n b e humiliating. And the ethical is-
tity" in which he explored his lived experi- sues. Just wait until you've written about fam-
ence of becoming Italian American. ily members and loved ones who are part of
These new frontiers of qualitative inquiry your story. (p. 738)
and reporting, combining art and science,
are also leading to the integration of multi- Part of the challenge in autoethno-
ple forms in a single work. Consider Den- graphic writing is finding and owning one's
zin's (2000b) "Rock Creek History/' which voice. In the last chapter, I contrasted the
he describes as "an experimental, mixed- third-person passive voice of traditional ac-
genre narrative, combining autoethnog- ademic writing with the first-person active
phy with other evocative writing forms, voice of qualitative inquiry. Autoethnog-
including narratives of the self. Using the raphy increases the importance of voice and
techniques of fiction, I tell a story about my- raises the stakes because an authentic voice
self and my experiences with nature, the sa- enhances the authenticity of the work, while
cred, and a small Montana River named an inauthentic voice undermines it. Voice re-
Rock Creek" (p. 71). He also calls his writing veals the author's identity (Ivanic 1998). The
a "performance-based project" that draws tone of voice may be expressive, reflective,
on multiple writing forms and traditions in- searching, academic, or criticai, as in what
cluding, in addition to those already noted, Church (1995) has called the "forbidden
"the ethnographic story, nature writing, lit- narratives" of "criticai autobiography" in
erary nonfiction, the personal memoir, and social science. In voice resides Richardson's
cultural criticism" (p. 79). (2000b) fifth criterion for judging quality
Ellis (Ellis and Bochner 2000) warns that cited earlier, what she called "expression of a
autoethnographic writing is hard to do: reality: Does this text embody a fleshed out,
embodied sense of lived experience? Does it
It's amazingly difficult. lt's certainly not seem a 'true'a credibleaccount of a cul-
something that most people can do well. Most tural, social, individual, or communai sense
social scientists don't write well enough carry of the 'real'?" (p. 937).
it off. Or they're not sufficiently introspective These issues are being raised in a number
about their feelings or motives, or the contra- of disciplinary genres. Historian Edmund
Morris won a Pulitzer Prize for his biogra- doubts, weaknesses, and uncertainties. But
phy of America's 26th president, Theodore once the story was told, the final chapter of
Roosevelt. Partly on this basis, he was cho- the book that contrasts alternative coming-
sen as the "official" biographer of former of-age/initiation paradigms (Exhibit 1.1 in
president Ronald Reagan. The resulting Chapter 1) emerged relatively painlessly.
work (E. Morris 2000) proved highly contro- I've included as Appendix 3.1 at the end of
versial because to tell the story of Reagan's this chapter an excerpt from the book as an
life as he felt it needed to be told, he created a example of autoethnographic writing.
fictional character based on himself and fab- Johnstone (2000) argues that "interest in
ricated encounters with Reagan at various the individual voice" within anthropology
points that led him to first-person reflections can be understood, at least in part, "within
as if he had actually witnessed and partici- the context of a larger shift toward a more
pated in these events and encounters. phenomenological approach to language"
Thus, a traditional and highly respected (p. 405). Autoethnography integrates eth-
historian introduced a form of quasi- nography with personal story, a specifically
autoethnographic literary fiction into a stan- autobiographical manifestation of the more
dard biography in order to have a point general "turn to biographical methods in so-
of view from which to recount his subject's cial science" that strive to "link macro and
life. micro leveis of analysis . . . [and] provide a
In my own major effort at autoeth- sophisticated stock of interpretive proce-
nographic inquiry (Patton 1999a), the strug- dures for relating the personal and the so-
gle to find an authentic voiceauthentic cial" (Chamberlayne, Bornat, and Wengraf
first to me, then to others who know me, and 2000:2-3). Art Bochner (Ellis and Bochner
finally to those who do not know me 2000) has reflected on what this means:
tumed what I thought would be a one-year
effort into seven years of often painful, dis-
couraging writing. And I was only writing What is the point of a storied life? Narrative
about a 10-day period, a Grand Canyon hike truth seeks to keep the past alive in the pres-
with my son in which we explored what it ent. Stories show us that the meanings and sig-
means to come of age, or be initiated into nificance of the past are incomplete, tenta tive,
adulthood, in modern society. My son and revisable according to contingencies of
started and graduated from college while I our present life circumstances, the present
was learning how to tell the story of what we from which we narrate. Doesn't this mean that
experienced together. To make the story the stories we tell always run the risk of dis-
work as a story and make scattered interac- torting the past? Of course, it does. After ali,
tions coherent, I had to rewrite conversa- stories rearrange, redescribe, invent, omit, and
tions that took place over several days into a revise. They can be wrong in numerous
single evening's dialogue, I had to reorder waystone, detail, substance, etc. Does this
the sequence of some conversations to en- attribute of storytelling threaten the project of
hance the plot line, and I had to learn to fol- personal narrative? Not at ali, because a story
low the novelist's mantra to "show don't is not a neutral attempt to mirror the facts of
tell," advice particularly difficult for those of one's life
us who make our living telling. More diffi- Life and narrative are inextricably con-
cult still was revealing my emotions, foibles, nected. Life both anticipates telling and draws
No, Mom, I didn't say that you're a bad mother.

I know false consciousness sounds negative, but
don't take it personal. That's just the deconstruction
chapter. In the next chapter I do reconstruction and
creative synthesis. I think you'll really resonate to
the synergistic portrayal of an archetypal maternal
figure in the postmodern era of conflicted roles. You
just have to get into the mindset of autoethnography.
Confronting a critic of autoethnography

meaning from it. Narrative is both about liv- What these questions have in common is
ing and part of it. (pp. 745-46) the presumption that there is a real world
with verifiable patterns that can be observed
By opening this chapter with the contrast and predictedthat reality exists and truth
between ethnography and autoethnog- is worth striving for. Reality can be elusive
raphy, we have moved from the beginnings and truth can be difficult to determine, but
of qualitative methods in anthropological describing reality and detennining truth are
fieldwork more than a century ago, where the appropriate goals of scientific inquiry.
the ethnographer was an outsider among Working from this perspective, researchers
exotically distinct nonliterate peoples, to and evaluators seek methods that yield cor-
the most recent manifestation of qualitative respondence with the "real world/' thus this
inquiry in the postmodern age of mass is some times called a correspondence per-
communications, where autoethnographers spective.
struggle to find a distinct voice by docu- Reality-oriented inquiry and the search
menting their own experiences in an increas- for truth have fallen on hard times in this
ingly all-encompassing and commercialized skeptical postmodern age when honoring
global culture. To further sharpen contrasts multiple perspectives and diverse points of
in qualitative approaches, the next two sec- view has gained ascendancy in reaction to
tions illuminate some of the philosophical the oppressive authoritarianism and dog-
underpinnings that have informed and matism that seemed so often to accompany
shaped qualitative methods, including eth- claims of having found "Truth." Yet, many
nography and autoethnography, by con- people, especially policymakers and those
trasting the foundational question of real- who commission evaluation research, find it
ity-oriented research and evaluation (post- difficult to accept the notion that ali explana-
positivist realism) with that of postmodern tions and points of view hold equal merit.
constructivism and social construction. Some people in programs seem to be helped
more than others. Some students seem to
learn more than others. Some claims of effec-
Truth and Reality-Oriented tiveness are more plausible and have more
Correspondence Theory: merit than others. To test a claim of effective-
Positivist, Realist, and Analytic ness by bringing data to bear on it, including
Induction Approaches qualitative data, is to be engaged in a form of
reality testmg that uses evidence to examine
T-QufAclaticmai CjuestioKvss assertions and corroborate claims. In this
Wka+'s êally going on in fea 1
section, we shall examine how to recognize
wo^ld? What ccm we establish with and engage in a reality-testing or reality-ori-
some decj*ee oj- ceH-aitrty? ented approach to qualitative inquiry. In so
Wkat doing, I shall minimize philosophical dis-
ciê plausible expla^cî^ms for course and focus primarily on the practical
veî implications of this orientation, but a brief
fiable pa+terns ? VvWs the foray into philosophical foundations is nec-
tûik msoj-ar os we get at it?
CQKV
essary to provide a context for practice.
"H
Philosophical inquiry into truth and real-
ow can we study o, pke.Kvomenon ity involves examining the nature of knowl-
so that out* f-indmgs coyyespond/
inso-fak1 as it's possible, to tke.
êal world?
edge itself, how it comes into being and is paradigm debates and routinely used incor-
transmitted through language. Positivism, rectly though persistently. Shadish (1995b)
following Auguste Comte, asserted that argues that one would be hard-pressed to
only verifiable claims based directly on ex- find any contemporary social scientist, phi-
perience could be considered genuine losopher, or evaluator who really adheres to
knowledge. Comte was especially inter- the tenets of logical positivism. Rather, "the
ested in distinguishing the empirically term has become the Iinguistic equivalent
based "positive knowledge" of experience of 'bad/ a rhetorical device aimed at depriv-
from theology and metaphysics, which ing one's opponent of credibility by name-
depended on fallible human reason and calling. This is particularly true m the
belief. Logical positivism, developed by the quantitative-qualitative debate where some
Vienna Circle in ustria and the Berlin qualitative theorists are fond of labeling ali
School in Germany in the early part of the quantitative opponents as logical positiv-
20th century, added to the emphasis on di- ists," a fundamental but common "error"
rect experience from positivism a logic-based (p. 64).
commitinent "to theory development using Logical empiricism and postpositivism,
a rigorous procedural language such as which take into account the criticisms
symbolic logic. Knowledge comes either against and weaknesses of rigid positivism,
from direct experience or indirectly from in- now inform much contemporary social sci-
ferences from experience through the proce- ence research, including reality-oriented
dural language" (Shadish 1995b:64). Logical qualitative inquiry. Logical empiricism, a
positivism subsequently came to be associ- more modera te version of logical positivism
ai ed with philosophical efforts to specify ba- (Schwandt 2001), seeks unity in science,
sic requirements for what could be consid- through both theory formulation and meth-
ered scientific knowledge, which mcluded odological inquiry, and asserts that there are
the search for universal laws through empir- no fundamental methodological differences
ical verification of logically deduced hy- between natural and social sciences.
potheses with key concepts and variables Postpositivism, as articulated by eminent
operationally defined and carefully formu- methodologist Donald T. Campbell in his
lated to permit replication and falsification. collected writings about and vision for an
Thus, real knowledge (as opposed to mere "experimenting society" (Campbell and
beliefs) was limited to what could be logi- Russo 1999), recognizes that discretionary
cally deduced from theory, operationally judgment is unavoidable in science, that
measured, and empirically replica ted. Such proving causality with certainty in explain-
severe, narrow, and rigorous requirements ing social phenomena is problema tic, that
turned out to severely limit what could pass knowledge is inherently embedded in his-
for knowledge and to demand more torically specific paradigms and is therefore
certainty than the complex world of social relative rather than absolute, and that ali
phenomena could yield. Though influential methods are imperfect, so multiple meth-
in the first half of the 20th century, logical ods, both quantitative and qualitative, are
positivism has been "almost urversally re- needed to generate and test theory, improve
jected" as a basis for social science inquiry understanding over time of how the world
(Campbell 1999a:132). The legacy of the opera tes, and support informed policy mak-
fleeting influence of logical positivism is ing and social program decision making.
that the term lives on as an epithet hurled in While modest in asserting what can be
Variety in Qualitative Inquiri/ J. 9 3
known with any certainty, postpositivists do

assert that it is possible, using empirical evi-
COtmUCTED.
dence, to distinguish between more and less fe^feft.4:l^1vN3lM3i flEAUTY
plausible claims, to test and choose between
" T A C T I . T K - R O R N F ^ m T VI ' ' T ' ; , ti&i V ^ I , ! VI"
rival hypotheses, and to distinguish be-
tween "belief and valid belief" (Campbell .STj J:f !! ii!!.! :i'i Ur n?:r! .'!:,-iV. !! h YJ;. ;.!'=! '
1999b:151, emphasis added). : ;!!.!.::> :' R ^ Y ; A - A H -
!''!;; ,; {y RRN !.VY ri .V I !.s" :!I i' i i. u V?

Given this brief philosophical and episte-
!Vi!! ril!^! rh.:s!"!!!T!' h1'!s' f i^Viy t.VV ,'
mological overview, what are the practical
iriXi'!!. M " (!m i ;: a::^;;!;-!;
implications for qualitative inquiry of oper-
ating within a reality-oriented perspective?
It ineans using the language and concepts of
mainstream science to design naturalistic "truth value" and plausibility of findings;
studies, inform data gathering in the field, credibility, impartiality, and independence
analyze results, and judge the quality of of judgment; confirmability, consistency,
qualitative findings. Thus, if you are a re- and dependability of data; and explainable
searcher or evaluator operating from a real- inconsistencies or instabilities (GAO
ity-oriented stance, you worry about valid- 1987:53). You may even generalize case study
ity, reliability, and objectivity (e.g., Perkyl findings, depending on the cases selected
1997). You realize that completely value-free and studied, to generate or test theory (Yin
inquiry is impossible, but you worry about 1989:44, 1999b), establish causality (Ragin
how your values and preconceptions may 1987, 2000), or inform program improve-
affect what you see, hear, and record in the ment and policy decisions from patterns es-
field, so you wrestle with your values, try to tablished and lessons leamed (GAO
make any biases explicit, take steps to miti- 1987:51). In short, you incorporate the lan-
ga te their influence through rigorous field guage and principies of 21st-century science
into naturalistic inquiry and qualitative
procedures, and discuss their possible influ-
analysis to convey a sense that you are dedi-
ence m reporting findings. You may estab-
cated to getting as close as possible to what is
lish an "audit trail" to verify the rigor of your
really going on in whatever setting you are
fieldwork and confirmability of the data col-
studying. Realizing that absolute objectivity
lected because you want to minimize bias,
of the pur positivist variety is impossible to
maximize accuracy, and report impartially
attain, you are prepared to admit and deal
believing that "inaccuracy and bias are un-
with imperfectioris in a phenomenologically
acceptable in any case study" (U.S. General
messy and methodologically imperfect
Accounting Office [GAO] 1987:51). In re-
world, but you still believe that objectivity is
porting, you einphasize the empirical find-
worth striving for. As Kirk and Miller (1986)
ingsgood, solid description and analy-
assert,
sisnot your own personal perspective or
voice, though you acknowledge that some
subjectivity and judgment may enter in. You Objectivity, though the term has been taken by
include triangulation of data sources and some to suggest a naive and inhumane ver-
analytical perspectives to increase the accu- sion of vulgar positivism, is the essential basis
racy and credibility of findings (Patton of ali good research. Without it, the only rea-
1999b). Your criteria for quality include the son the reader of the research might have for
accepting the conclusions of the investigator rather than logical positivists, further evi-
would be an authoritarian respect for the per- dence that "the weight of criticisms" against
son of the author. Objectivity is a simulta- logical positivism has "caused its internai
neous realization of as much reliability and collapse" (Schwandt 2001:150). Realism as a
validity as possible. Reliability is the degree to qualitative stance is clearly reality oriented,
which the finding is independent of accidental and much of the language quoted above re-
circumstances of the research, and validity is mains in the revised edition. They acknowl-
the degree to which the finding is interpreted edge that knowledge is socially and histori-
in a correct way. (p. 20) cally constructed, and they "affirm the
existence and importance of the subjective,
In the introduction to their widely used the phenomenological, the meaning-mak-
and influential sourcebook Qualitative Data ing at the center of life." Then they return to
Analysis, Miles and Huberman (1984) stated their core reality-oriented stance:
modestly: "We think of ourselves as logical
Our aim is to register and "transcend" these
positivists who recognize and try to atone
processes by buildmg theories to account for a
for the limitations of that approach. Soft-
real world that is both bounded and perceptu-
nosed logical positivists, maybe" (p. 19).
ally laden, and to test these theories in our var-
They went on to explain what this means
ious disciplines.
and, in so doing, provide a succinct sum-
Our tests do not use "covering laws" or the
mary of the reality-oriented approach to
deductive logic of classical positivism. Rather,
qualitative research:
our explanations flow from an account of how
differing structures produced the events we
We believe that social pkenomena exist not observed. We aim to account for events, rather
orily in the mind but also in the objective than simply to document their sequence. We
worldand that there are some lawful and look for an individual or a social process, a
reasonably stable relationships to be found mechanism, a structure at the core of events
among them Given our belief in social reg- that canbe captured to provide a causai descrip-
ularities, there is a corollary: Our task is to ex- tion of the forces at work.
press them as precisely as possible, attending Transcendental realism calls both for
to their range and generality and to the local causai explanation and for the evidence to
and historical contingencies under which they show that each entity or event is an instance of
occur. that explanation. So we need not only an ex-
So, unlike some schools within social phe- planatory structure but also a grasp of the par-
nomenology, we consider it important to ticular configuration at hand. That is one
evolve a set of valid and verifiable methods for reason why we have tilted toward more in-
capturing these social relationships and their ductive methods of study. (Miles and Huber-
causes. We want to interpret and explain these man 1994:4)
phenomena and have confidence that others,
using the same tools, would arrive at analo- Analytic induction offers a specific form of
gous conclusions. (Miles and Huberman inductive analysis that begins deductively,
1984:19-20) by formulating propositions or hypotheses,
and then examines a particular case in depth
Ten years later, in their revised and ex- to determine if the facts of the case support
panded qualitative sourcebook, Miles and the hypothesis. If it fits, another case is stud-
Huberman called themselves "realists" ied, and so forth, in the search for generaliza-
tions. If a case does not support the hypothe- such as logical positivism, postpositivism,
sis, that is, it is a negative case, the hypothesis logical empiricism, realism, transcendental
is revised. The aim is to explain a phenome- realism, and objectivisin are jargon-ish, have
non satisfactorily using qualitative, case- disputed definitions, and carry negative
based inquiry (Schwandt 2001; Vidich and connotations for many, so they come with
Lyman 2000:57-58; Ryan and Bernard 2000: lots of baggage. I have attempted to be de-
786-87; Gilgun 1995; Taylor and Bogdan scriptive about the reality-oriented, corre-
1984: 127-28). Chapter 8 on analysis dis- spondence theory perspective by focusing
cusses the analy tical strategies of analytic in- on its core, foundational questions as articu-
duction in more detail and provides exam- lated at the beginning of this section. While,
ples. as the next section will show, many qualita-
While analytic induction focuses on tive methodologists assert that qualitative
method, realism focuses first on philosophy. inquiry is inherently constructionist or phe-
Realist philosophy (Baert 1998:189-97; Put- nomenological in perspective, the reality-
nam 1987, 1990) has recently has been oriented perspective remains widespread,
adapted by Mark, Henry, and Julnes (2000) even dominant, in those arenas of research
and Pawson and Tilley (1997) as offering the practice where scientific credibility carries a
foundation for a reality-oriented approach premium. These arenas include many dis-
to evaluation research that includes qualita- sertation committees in traditional disci-
tive inquiry. plines where qualitative dissertations are
just beginning to be allowed, in summative
Realism presumes the existence of an externai
evaluation and policy studies where mere
world in which events and experiences are
"anecdotal" evidence is demeaned, and in
triggered by underlying (and often unob-
fields such as medicai research where dou-
servable) mechanisms and structures (Bhas-
ble-blind experimental studies remain the
kar, 1975). Commonsense realism also gives
gold standard. To emphasize this latter
standing to everyday experiences. It is
point, I close this review of reality-oriented
antiformalist in the sense of not expecting log-
qualitative inquiry with an excerpt from a
ical, formal solutions to vexing problems such
medicai journal in which health researchers
as the n ature of truth. And it places a priority
are defending qualitative research to an au-
on practice and the lessons drawn from prac-
dience known to be skep tical. Their ap-
tice. . . . As realists, we see no meaningful
proach is to associate qualitative research
epistemological difference between qualita-
closely with accepted and credible forms of
tive and quantitative methods. Instead we see
experimental research. Such a perspective
both as assisted sensemaking techniques that
epitomizes the reality-testing orientation:
have specific benefits and limitations. And as
commonsense realists, we believe that al-
What, then, does the qualitative researcher do
though there is a world out there to be made
once he or she accomplishes a careful and
sense of, the specific constructions and
trustworthy understanding of the language
construais that individuais make are criticai
and behavior of an individual human being?
and need to be considered. (Mark et al.
Here is where we rely on our positivist skills
2000:15-16)
and methods [OJnce we carefully examine
Throughout this section, I have used the and articulate that which we understand one
term reality-oriented qualitative inquiry to human being to be doing, we attempt to col-
describe this perspective because labels late the language and behavior of many hu-
man beings, so many that we might be able to

test the relationships, for example, between
fh; c c.;;, K M IVIT
setting andbehavior orbetween age and hope.
Included within the domains of qualitative !''i;: | ijV ;.Ti... ori '-iiviaviVi:;-
science or narrative research, then, are efforts i'i h n-i'! i)s !* jYVi ng y1 r-s:!.:; ;: !'::.I i.1 . ! i1
to generalize, to predict, and to relate initial ;.V!VF H/,!JJ=',:Y;;-.;= H:Z^KF.I/J,; !. A - A : I ' I ' , !
states to outcomes. These efforts require the !:' !i!'ii!!!!l ir!;!!!'i't"i!'""!;!!!::i;l!i!i Ntti ! if! iV-! j<: Tttti
same evidence-based activities that are used J-:!:!.'IWW !Y! :;=.I!\!:: IV-H-Y::':::: W ^ R L<I
in testing any hypotheses. (Charon, Greene, : tii-iir!!! I:!1!^!!:!^!!.:!' ftlM! Ji :!!:!!:];:! !';!:: iVi - 17i M ;1
and Adelman 1998:68) iVi ' . : I ! m n tf != i i * = a z i ' : ; I:=,-i - I;, ! , m l
ro tf:;" !>:;!:v!::;!::';!:!'! :::==h!;= -i*.ii

Social Construction ! " l l t e M T I J - ^ T ' ! IRLîii. 1 ;ST: /n
and Constructivism Y :R . H ; : : ' I .:= = ' S I , .
I IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII^ ÎIIIIIIIIIIIIIIIII NU IIIIIHIIII III LI II II NII I SIIIHIIIIIIIJ
T-oundational questions: 'Vi' ! !'!:!:V:!! .:!:;'!: ','- !:- ! :'i! n ;i.i V - r ' V " :
How kave +ke people in +his setHng ::!' it i*..".rt; rr-.LVJ AL-V"" i.- /.u';
constructed reality? Wkat are tkeir :I M" IR;:' :;:: I^N-V. v i :. ! ' I1! '.1 |',V .I i V i V ! I. I- N I 'I"
êpo^ted perceptions, "trutks/' ; . T I F T F L C . Y I . ' I:M ! I ' ! I ! T . V \ | ; I = ! : , 1 V L I . - , ! ' . '
explanations, beliefs, and world- /. ^ I I ! ' ' ! * IY.VI.UY. ^ F C L V ! i i I ! ! R ! i.I i . v i IVRÎRM.
view? What ace tke consequences / S n u . " .'fli-i, 1
of tkeir constructions for tkei^

bekaviors and for tkose witk wkom
tkey interact?
constructivists study the multiple realities
constructed by people and the implications
Constructivism begins with the premise of those constructions for their lives and in-
that the human world is different from the teractions with others. Shadish (1995b) re-
natural, physical world and therefore must minds us that social constructionism "refers
be studied differently (Guba and Lincoln to constructing knozvledge about reality, not
1990). Because human beings have evolved constructing reality itself" (p. 67). Construc-
the capacity to interpret and construct re- tionists commonly assume that humans "do
alityindeed, they cannot do otherwise not have direct access to a singular, stable,
the world of human perception is not real and fully knowable externai reality. Ali of
in an absolute sense, as the sun is real, but is our understandings are contextually em-
"made up" and shaped by cultural and lin- bedded, interpersonally forged, and neces-
guistic constructs. To say that the socially sarily limited" (Neimeyer 1993:1-2). Any no-
constructed world of humans is not physi- tion of "truth," then, becomes a matter of
cally real like the sun doesn't mean that it "consensus among informed and sophisti-
isn't perceived and experienced as real by cated constructors, not of correspondence
real people. W. I. Thomas, a distinguished with an objective reality." Likewise, the no-
sociologist and a founding symbolic inter- tion of an objective "fact" has no meaning
actionist, formulated what has become "except within some value framework." It
known as Thomas's theorem: What is defined. follows that "there cannot be an 'objective'
or perceived by people as real is real in its conse- assessment of any proposition" (Guba and
quences (Thomas and Thomas 1928:572). So Lincoln 1989:44). Social construction, or
EXHIBIT 3.4 Constructivism Versus Constructionism
Michael Crotty (1998) makes an important and useful distinction between constructivism and
constructionism, a distinction that illustrates how the process of social construction unfolds
among scholars. It remains to be seen whether this distinction will gain widespread use since the
two terms are so difficult to distinguish and easy to confuse.
"It would appear useful, then, to reserve the term constructivism for the epistemological con-
siderations focusing exclusively on 'the meaning-making activity of the individual mind' and to
use constructionism where the focus includes 'the collective generation [and transmission] of
meaning'" (p. 58}
"Whatever the terminology, the distinction itself is an important one. Constructivism taken in
this sense points out the unique experience of each of us. It suggests that each one's way of making
sense of the world is as valid and worthy of respect as any other, thereby tending to scotch any hint
of a criticai spirit. On the other hand, social constructionism emphasizes the hold our culture has
on us: it shapes the way in which we see things (even in the way in which we feei things!) and gives
us a quite definite view of the world" (p. 58).
constructivist philosophy, is built on the the- tained and reproduced through social life."
sis of ontological relativity, which holds that (pp. 54-55)
ali tenable statements about existence de-
pend on a worldview, and no worldview is Elsewhere Crotty uses the example of a tree:
uniquely determined by empirical or sense
data about the world. Hence, two people can What the "commorisense" view commends to
live in the same empirical world, even us is that the tree standing before us is a tree. It
though one's world is haunted by demons, has ali the ineaning we ascribe to a tree. It
and the other's, by subatomic particles. Ex- would be a tree, with that same meaning,
hibit 3.4 distinguishes the worldviews of whether anyone knew of its existence or not.
constructionism and constructivism, which We need to remind ourselves here that it is hu-
are often used interchangeably. man beings who have constructed it as a tree,
How all-encompassing is the construc- givenit the name, and attributed to it the asso-
tionist view? Michael Crotty (1998) asserts, ciations we make with trees. It may help if we
recall the extent to which those associations
It is not just our thoughts that are constructed differ even within the same overall culture.
for us. We have to reckon with the social con- "Tree" is likely to bear quite different connota-
struction of emotions. Moreover, constructions in a logging town, an artists' settlement
tionism embraces the whole gamut of mean- and a treeless slum. (p. 43)
ingful reality. Ali reality, as meaningful reality,
is socially constructed. There is no exception. How, then, does operating from a con-
. . . The chair may exist as a phenomenal object structionist perspective actually affect quali-
regardless of whether any consciousness is tative inquiry? Let's consider its impact on
aware of its existence. It exists as a chair, how- program evaluation. A constructionist eval-
ever, only if conscious beings construe it as a uator would expect that different stake-
chair. As a chair, it too "is constructed, sus- holders involved in a welfare program (e.g.,
staff, clients, families of clients, administra-
tors, funders) would have different experi- pacts rather than reaching singular conclu-
ences and perceptions of the program, ali of sions. The mdium of the report carried the
which deserve attention and ali of which are message that multiple voices needed to be
experienced as real. The constructionist heard and valued as a manifestation of di-
evaluator would attempt to capture these versity (Stockdill et ai. 1992). The findings
different perspectives through open-ended were used for both formative and sum-
interviews and observations, and then mative purposes, but the parents and many
would examine the implications of different of the staff were most interested in using the
perceptions (or multiple "realities") but evaluation processes to make themselves
would not pronounce which set of percep- heard by those in power. Being heard was an
tions was "right" or more "true" or more end in itself quite separate from use of find-
"real," as would a reality-oriented (postings.
positivist) evaluator. Constructivist evalua- Guba and Lincoln (1989) included among
tors could compare clients' perceptions and the primary assumptions of constructivism
social constructions with those of funders or the following, whether for evaluation or re-
program staff and could interpret the effects search more generally:
of differences on attainment of stated pro-
gram goals, but they would not value staff "Truth" is a matter of consensus among
perceptions as more real or meaningful. In informed and sophisticated construc-
constructivist evaluation, then, "the claims, tors, not of correspondence with objec-
concerns, and issues of stakeholders serve as tive reality.
organizational foci (the basis for determin- "Facts" have no meaning except within
ing what information is needed)" (Guba and some value framework, hence there can-
Lincoln 1989:50). not be an "objective" assessment of any
Indeed, if constructivist evaluators were proposition.
also operating from a social justice frame-
"Causes" and effects do not exist except
work, they might give added weight to the
by imputation....
perspectives of those with less power and
privilege in order to "give voice" to the dis- Phenomena can only be understood
enfranchised, the underprivileged, the poor, within the context in which they are
and others outside the mainstream (Weiss studied; findings from one context can-
and Greene 1992:145). In the evaluation of a not be generalized to another; neither
diversity project in a school district in Saint problems nor solutions can be general-
Paul, Minnesota, a major part of the design ized from one setting to another. . . .
included capturing and repor ting the expe- b Data derived from constructivist inquiry
riences of people of color. Providing a way have neither special status nor legitima-
for African American, Native American, tion; they represent simply another con-
Chicano-La tino, and Hmong parents to tell struction to be taken into account in the
their stories to mostly White, corporate move toward consensus. (pp. 44-45)
funders was an intentional part of the de-
sign, one approved by those same White cor- Guba and Lincoln (1990:148) summarize the
porate funders. The final report was written constructivist perspective as being ontologi-
as a "multivocal, multicultural" presenta- cally relativist, epistemologically subjectiv-
tion that presented different experiences ist, and methodologically hermeneutic and
with and perceptions of the program's im- dialectic. The thread throughout is the em-
Variety in Qualitative Inquiri/ J. 9 9
phasis on the socially constructed nature of sumption that scientists, rather than being
reality as distinguishing the study of human bound by preconceptions, were open-
beings from the study of other natural phe- minded, value free, and unencumbered by
nomena. inherited ideas. Kuhn applied to science the
The idea that social groups such as street kind of language normally used to describe
gangs or religious adherents construct their confrontatioris between opposing political
own realities has a long history in sociology, and ideological communities, especially
especially the sociology of knowledge (e.g., during revolutions. He argued (and showed
Berger and Luckmann 1967). It wasn't until with natural science examples) that commu-
this idea of socially constructed knowledge nities of scientists, like ideological or reli-
was applied to scientists that construc- gious communities, were organized by cer-
tionism became an influential methodologi- tain traditions that periodically came under
cal paradigm. No work has been more influ- strain when new problems arose that
ential in that regard than Thomas Kuhn's couldn't be explained by old beliefs. New
classic The Structure of Scientific Revolutions explanations and ideas would then compete
(1970). Before Kuhn, most people thought until the old ideas were discarded or re-
that science progressed through heroic indi- vised, sometimes sweepingly. But the com-
vidual discoveries that contributed to an ac- petition was not just intellectual. Power was
cumulating body of knowledge that got involved. The leaders of scientific communi-
closer and closer to the way the world really ties wielded power in support of their posi-
worked. In contrast, Kuhn argued that tions just as political leaders do. The assess-
tightly organized communities of specialists ment of Kuhn's contribution, three decades
were the central forces in scientific develop- after his work first appeared, by Berkeley
ment. Ideas that seemed to derive from bril- historian David Hollinger shows the impor-
liant individual scientific minds were actu- tance of his analysis: "The Structure of Scien-
ally shaped by and dependent on paradigms tific Revolutions presented the strongest case
of knowledge that were socially constructed ever made for the dependence of valid sci-
and enforced through group consensus. ence on distinctly constituted, historically
Rather than seeing scientific inquiry as pro- particular human communities" (Hollinger
gressing steadily toward truth about nature, 2000:23).
he suggested that science is best seen as a se- Scientists constitute a criticai case for so-
ries of power struggles between adherents cial constructionism. If scientific knowledge
of different scientific worldviews. is socially constructed and consensually val-
Kuhn emphasized the power of precon- idated, as opposed to consisting of empirical
ceived and socially constructed ideas to con- truths validated by nature, then surely ali
trol the observations of scientists. He in- knowledge is socially constructed. "Accord-
sisted that without the focusing effect of ingly, not only the social scientistbut equally
agreed-on constructs, investigators would the natural scientist has to deal with realities
notbe able actually to engage in research. A that, as meaningful realities, are socially
fully "open" mind would not be able to fo- constructed. They are on equal footing in
cus on the details necessary to engage in this respect" (Crotty 1998:55).
"normal" science, that is, testing specific Kuhn's analysis, though remaining con-
propositions derived from a theory or "sci- troversial and heavily critiqued (e.g., Fuller
entific paradigm." What made this contribu- 2000), became a cornerstone of the post-
tion so important was the widespread pre- modern skepticism about scientific truth.
100 LEI. CONCEPTUAL ISSUES IN QUALITATIVE INQUIRY
Postmodernism, Radavich (2001) asserts,

has become "the most prevalent mode of
M-IAIVI N L L H > T E R J M F V T
thinking in our time.... Postmodernist dis-
W FtJStftilQpii^MISM
course is precisely the discourse that denies
the possibility of ontological grounding" !:'] Y i i V i ! " ii ' i ! V i i,' i V i i h . - i 1 rfw.i. ;.S=r!. : i",*i .'|i
(p. 6). In other words, no truth or "true iW-inVi ! V id: i ,','r,a i-i"! nif;rv I;- -
meaning" about any aspect of existence is t<er|i!' M ^ f L V U l Y i . v l rAjp.' -vM firCT m -
possible, at least not in any absolute sense; it iT!i:.::;i:!: : i'!i.i'i ': m C ! : h . : - i . ! " i !.I ' :;J I rrp: s-!.;s IV
can only be constructed. To understand fcj ;,! iii.j;^ .vi i y-t v. VM* iAvqVn jVln
constructionism and its implications for C13.VC1 i:l:=:i S.'i S I U ! H.U !R>Y'.>'I'!, "I;' T I ! Y ,T] JJ
qualitative inquiry, a brief review of EWUJ!- i:i=:i l.U :':I ! trniv u l m ^ j ":T îri:^!!.!!"'
postmodernism may be helpful in that it has SM:'..'! iJ. . I: ' U!! k ! : ' ! s n - i ' l v.xv h }f\ i " .> i j i r i jy/i r
shaped contemporary intellectual discourse !

f M i l Y S : ^ ! i.': li'!:: 1
ftl ^ . ' f i ; V i P J l t f l t (kiy if-H"^!-
in both science and art. in bit ! ri= i' i't! !Vi.!riV! ! o:i'n :>-
Belief in science as generating truth was jjiii':!.'!!1 :r i'ri i'j|iJ:iV sn \v>i,! 5; fri,", ^ ':vm" !! n !"
one of the cornerstones of modernism inher- ! I i.VL II ii H i NVFAR 'i' ;T!." > . ' ! I;; 1 V ! . < ' Y L V I V ;
ited from the Enlightenment. Postmod- i . : r i . i . : R ' n J r i r i | ':'! n r i >} v n k !'ji M i ', <': W E s : j ; v j
ernism attacked this faith in science by ques- ,i,'i.!-=ViiiS"ji!1 S d ^ . ; ; - n :

:
; i i ' 0.
tioning its capacity to generate truth, in part

because, like ali human communications, it
is dependent on language, which is socially
constructed and, as such, distorts reality. knowledge is the norm, and a permanent
Postmodernism asserts that no language, pluralism of cultures is the only real truth
not even that of science, can provide a direct that humans must continually face" (Turner
window through which one can view reality. 1998:599). Constructionism, then, consistent
Language inevitably and inherently is built with postmodernism, is relativistic in
on the assumptions and worldview of the stance, meariing knowledge is viewed as rel-
social group that has constructed it and the ative to time and place, never absolute
culture of which it is a part. Thus, language across time and space, thus the reluctance to
does not and cannot fully capture or repre- generalize and the suspicion of generaliza-
sent reality, a posture called the "crisis of tions asserted by others.
representation" (Denzin and Lincoln 2000b; Power comes into the picture here be-
Turner 1998:598). Translated into Kuhn's cause, as views of reality are socially con-
framework, scientific language and constructed and culturally embedded, those
structs are paradigm based and dependent. views dominant at any time and place will
It follows from this that the continuity of serve the interests and perspectives of those
knowledge over time and across cultures is who exercise the most power in a particular
called into question. Modernism/s faith in culture. By exercising control over language,
science included the assumption that and therefore control over the very catego-
knowledge increased over time and that ries of reality that are opened to conscious-
such accumulation constituted continuous ness, those in power are served.
progress toward deeper and deeper truths. Scientific knowledge, then, is socially
"Postmodernists argue that because there is constructed like ali other knowledge sys-
not a truth that exists apart from the ideolog- tems and, as such, is relative to and contin-
ical interests of humans, discontinuity of gent on the methods and paradigms within
Variety in Qualitative Inquiry |J, 101
which it was generated. "Science, like any tions and positions, from the radical "abso-
knowledge system, is based on incorrigible lutely no reality ever" to a milder "let's
assumptions, is an abstraction fromphysical capture and honor different perspectives
reality, is in need of reification and stabiliza- about reality." These positions share an in-
tion through the processes of institu- terest m the subjective nature of human
tionalization and emotional investment, and perceptions and skepticism about the
is bent on systematically subjugating other possibility of objectivity. Reality-oriented re-
knowledge systems to assertits own reality" searchers, inkind, are skeptical of the subjec-
(Turner 1998:614). Some postmodernists tive knowledge of constructivism. How con-
and constructivists question the possibility tentious is the debate? One gets some sense
of ever finding and expressing true reality, of the gulf that can separate these views
even in the physical world, because lan- from an assessment of postmodernism and
guage creates a screen between human be- constructivism by Rutgers mathematician
ings and physical reality. "Vocabularies are Norman Levitt (1998) in an article titled
useful or useless, good or bad, helpful or "Why Professors Believe Weird Things":
misleadmg, sensitive or coarse, and so on; "Scientific evidencewhich is to say the
but they are not 'more objective' or 'less ob- only meaningful evidencecannot be neu-
jective' nor more or less 'scientific' " (Rorty tralized by 'subjective knowledge,' which is
1994:57). This is because discovermg the to say bullshit" (p. 34). He goes on to com-
"true nature of reality" is not the real pur- ment on constructivism as a particular mani-
pose of language; the purpose of language is festation of postmodernism: "a particular
to communicate the social construction of technique for getting drunk on one's own
the dominant members of the group using words" (p. 35).
the language. Thomas Schwandt (1997a) in his very
The postmodern perspective, and its useful dictionary of qualitative terms strikes
many variationsfor postmodernism is not a more conciliatory tone, recognizing that
a unitary perspective (e.g., Pillow 2000; the rhetoric of constructivism can sound
Constas 1998)has given rise to an empha- radical (and silly) if taken too literally:
sis on deconstruction, which means to take
Although some versions of constructivism do
apart the language of a text to expose its criti-
appear to deny reality, many (if not most, I sus-
cai assumptions and the ideological inter-
pect) qualitative inquirers have a common-
ests being served. Perspective and power oc-
sense realist ontology, that is, they take seri-
cur as hand in glove in postmodern
ously the existence of things, events, struc-
critiques. Social constructions are presumed
tures, people, meanings, and so forth in the
to serve someone's interests, usually those
environment as independent in some way
of the powerful. As Denzin (1991) has as-
from their experience with them. And they re-
serted with reference to deconstructing
gard society, institutions, feelings, intelli-
mass media messages, a criticai analysis
gence, poverty, disability, and so on as being
should "give a voice to the voiceless, as it de-
just as "real" as the toes on their feet and the
constructs those popular culture texts which
sun in the sky. (p. 134)
reproduce stereotypes about the powerless"
(p. 153). Thus, deconstruction constitutes a Further deconstructing the phrase "social
core analytical tool of constructivists. construction," one may find "inescapable
In deconstructing constructionism and connotations of manufacturmg," as if peo-
constructivism, one finds a range of assump- ple sat around and made things up. But
DUALISTAND MONIST
CONSTRUCTIONISM
iiHLCiil^Mii.iiH.uci iarj .
: UCMftAHi: THF:I:1;!C!3!T ^''V^:^:'! Distinguishing dualist from monist ap-
- W ^JMfflB^SlS^tiÎSfi-l : proaches to social constructionism takes the
deconstruction process through one final
. :ii U iilVlVi.i^.!J):! b. f f ^ j v i filter.
Si.!i':!"':! ' r i i l j i i i^fjjfc.lj
Dualist constructionism distinguishes be-
k* \ riTi lh! n! il-jil'^]!.!^!!^ tween actual states of affairs and perceptions,
. !.i:'j|:i r Vii!.'i: n';lt:-!i:. ItfN
interpretations, or reactions to those affairs
p:=!i U v k w i W , fere! iiiî-.Lî^L^ri^ligjH^r^ i i v i i i h y . ,
When Berger and Luckmann (1967) say that
the Sociology of Knowledge "must concern it-
' . 'iitflji':!ii!:i:vii|'iiiilj'!!'!'!Wl^W
self with whatever passes for 'knowledge' in a
ilfc LfeH !!:i'!V ri/i-iMij = !=
; s1^!^yr!/jr^fl
society" (p. 3), their putting knowledge in
quotation marks demonstrates a commitment
; !!':!-I:,:;.!I; ^ ' J J J T O F C L F R ^ I I ^ ^ P : 5 ^ ^ ^ ! 1 ! ^ ^ ^
to a dualist position. There is knowledge then
: iri- i::i:L'l;i:;:;j:j."j:i:j'j
there is "knowledge." The latter will be
: cfrei ft-! i:ii:J;
:
treated as knowledge by some social group,
i:!> JlTl+jiS^
but judgment can be made on the ultima te va-
!l!!!!!!!!!!!!!!!!!!!!!!!|j||||g| lidity of this group's claims, determining

whether "knowledge" really is knowledge
; 11]|| lliy^^filIltlIfHiHl!^^ i HlfHiflllilplIlIlH^
If the approach to the domain of inquiry is
'.' Slil&fci; j;i|||li!::|i;n|!;j!!||^
dualist this means that the analyst distin-
Illlllllllllllllllllllslliilllisiiiiiis guishes between the objective features of the
domain and members' representations of
those features. The dualist is prepared to judge
the adequacy of the members' representations
to say that people produce the world is not the (beliefs, interpretation). If the approach to the
same as saying that they are sopsists, that domain is monist, then there are only mem-
they are able to fashion the world according to bers' representations, the adequacy of which
their w h i m s . . . . [Ojne cannot ordinarily pro- cannot be raised as an issue; there are no objec-
duce an imaginary or nonsensical phenome- tive features in the domain upon which to base
non and expect to be taken seriously. The a judgment of the adequacy, for example, of a
mistake is to think of the process of production claim of racism. There are only representa-
as one that is free of constraints when in fact it tions of features, for example, the representa-
is a structure of constraints. (Watson and tion/claim of racism. (Heap 1995:54)
Goulet 1998:97)
We can conclude by emphasizing the ba-

Attending to the social construction of real- sic contributions of social construction and
ity, then, points us not only to what is con- constructivist perspectives to qualitative in-
structed but to how it is constructed and the quiry, namely, the emphasis on capturing
very question of what it means to say it is and honoring multiple perspectives; attend-
constructed. For an excellent review of these ing to the ways in which language as a social
issues in both the social and natural sciences, and cultural construction shapes, distorts,
see Hacking (2000). and struetures understandings; how meth-
Young lady, the court doesn't need to hear any

more about difficulties with validity & reliability in
qualitative research methods, discrepancies between
different postmodern epistemologies, or this "Great
Paradigm Debate." Please just tell us what you saw.
ods determine findings; and the importance cially the effects of inequitable power dy-
of thinking about the relationship between namicsand how that relationship affects
the investigator and the investigated, espe- what is found.
Phenomenology What these various phenomenological

and phenomenographic approaches share
in common is a focus on exploring how hu-
man beings make sense of experience and
Troundational question: transform experience into consciousness,
Wkat is +ke meaning,, strwc+wê, both individually and as shared meaning.
and essence oj-tke lived experi- This requires methodologically, carefully,
ence oj- +kis pkenomenon j-or this and thoroughly capturing and describing
p e r s o n or g r o w p oj- people?
how people experience some phenomenon
how they perceive it, describe it, feel about
it, judge it, remember it, make sense of it,
"Phenomenology asks for the very nature and talk about it with others. To gather such
of a phenomenon, for that which makes a data, one must undertake in-depth inter-
some-'thing' what it isand without which views with people who have directly experi-
it could not be what it is" (Van Manen enced the phenomenon of interest; that is,
1990:10). The initial clarity of this definition they have "lived experience" as opposed to
can fade rapidly because the term phenomen- secondhand experience.
ology has become so popular and has been so
widely embraced that its meaning has be- Phenomenology aims at gaining a deeper un-
come confused and diluted. It can refer to a derstanding of the nature or meaning of our
philosophy (Husserl 1967), an inquiry para- everyday experiences....
digm (Lincoln 1990), an interpretive theory Anything that presents itself to conscious-
(Denzin and Lincoln 2000b:14)/ a social sci- ness is potentially of interest to phenomenol-
ence analytical perspective or orientation ogy, whether the object is real or iinagined,
(Harper 2000:727; Schutz 1967, 1970), a ma- empirically measurable or subjectively felt.
jor qualitative tradition (Creswell 1998), or a Consciousness is the only access human be-
research methods framework (Moustakas ings have to the world. Or rather, it is by vir tue
1994). Varying forms complicate the picture of being conscious that we are already related
even more; transcendental, existential, and to the world. Thus ali we can ever know must
hermeneutic phenomenology offer different present itself to consciousness. Whatever falls
nuances of focusthe essential meanings of outside of consciousness therefore falls out-
individual experience, the social construc- side the bounds of our possible lived experi-
tion of group reality, and the language and ence. . . . A person cannot reflect on lived
structure of communication, respectively experience while living through the experi-
(Schwandt 2001:191-94). Phenomenologi- ence. For example, if one tries to reflect on
cal traditions in sociology and psychology one's anger while being angry, one finds that
vary in imit of analysis, group or individual the anger has already changed or dissipa ted.
(Creswell 1998:53). Adding further confu- Thus, phenomenological reflection is not in~
sion to the mix, the term phenornenography trospective but retrospective. Reflection on lived
was coined by Ulrich Sonnemann (1954) to experience is always recollective; it is reflec-
emphasize "a descriptive recording of im- tion on experience that is already passed or
mediate subjective experience as reported" lived through. (Van Manen 1990:9-10).
(p. 344). (For an annotated bibliography of
phenomenographic research, see Bruce and The phenomenon that is the focus of in-
Gerber 1997.) quiry may be an emotionloneliness, jeal-
Hi, l'm a graduae student,

and Vm learning about a
new type of research that
focuses on the "lived ex-
perience" of different life
events. 1 just have a few
questions..
Phenomenological abduction
ousy, anger. The phenomenon may be a rela- influences have been Merleau-Ponty (1962),
tionship, a marriage, or a job. The phenome- Whitehead (1958), Giorgi (1971), and Zaner
non may be a program, an organization, or a (1970). More recently, phenomenology has
culture. become an important influence in certain
Phenomenology as a philosophical tradi- approaches to psychotherapy (Moustakas
tion was first used in the development of a 1988,1995).
rigorous science by the German philosopher By phenomenology Husserl (1913) meant
Edmund H. Husserl (1859-1938). The work the study of how people describe things and
of Alfred Schutz (1899-1959) was an impor- experience them through their senses. His
tant influence in applying and establishing most basic philosophical assumption was
phenomenology as a major social science that we can only know what we experience by at-
perspective (Schutz 1977). Other important tending to perceptions and meanings that
awaken our conscious awareness. Initially, the phenomenon as directly as possible for
ali our understanding comes from sensory ourselves. This leads to the importance of
experience of phenomena, but that experi- participant observation and in-depth inter-
ence must be described, explicated, and in- viewing. In either case, in reporting phe-
terpreted. Yet, descriptions of experience nomenological findings, "the essence or na-
and interpretations are so intertwmed that ture of an experience has been adequately
they often become one. Interpretation is es- described in language if the description re-
sential to an understanding of experience awakens or shows us the lived quality and
and the experience includes the interpreta- significance of the experience in a fuller and
tion. Thus, phenomenologists focus on how deeper manner" (Van Manen 1990:10).
we put together the phenomena we experi- There is one final dimension that differen-
ence in such a way as to make sense of the tiates a phenomenological approach: the as-
world and, in so doing, develop a sumption that there is an essence or essences to
worldview. There is no separate (or objec- shared experience. These essences are the core
tive) reality for people. There is only what meanings mutually understood through a
they know their experience is and means. phenomenon commonly experienced. The
The subjective experience incorporates the experiences of different people are brack-
objective thing and becomes a person's real- eted, analyzed, and compared to identify the
ity, thus the focus on meaning making as the essences of the phenomenon, for example,
essence of human experience. the essence of loneliness, the essence of be-
ing a mother, or the essence of being a partic-
From a phenomenological point of view, we ipant in a particular program. The assump-
are less interested in the factual status of par- tion of essence, like the ethnographer's
ticular rnstances: whether some thing hap- assumption that culture exists and is impor-
pened, how often it tends to happen, or how tant, becomes the defining characteristic of a
the occurrence of an experience is related to purely phenomenological study. "Phenom-
the prevalence of other conditions or events. enological research is the study of essences" (Van
For example, phenomenology does not ask, Manen 1990:10). Phenomenologists are
"How do these children leam this particular
material?" but it asks, "What is the nature or rigorous in their analysis of the experience, so
essence of the experience of learning (so that I that basic elements of the experience that are
can now better understand what this particu- common to members of a specific society, or ali
lar learning experience is like for these chil- human beings, can be identified. This last
dren)?" (Van Manen 1990:10) point is essential to understanding the philo-
sophical basis of phenomenology, yet it is of-
There are two implications of this per- ten misunderstood. On the other hand, each
spective that are often confused in discuss- person has a unique set of experiences which
ing qualitative methods. The first implica- are treated as truth and which determine that
tion is that what is important to know is individual^ behavior. In this sense, truth (and
what people experience and how they inter- associate behavior) is totally unique to each in-
pret the world. This is the subject matter, dividual. Some researchers are misled to think
the focus, of phenomenological inquiry. that they are using a phenomenological per-
The second implication is methodological. spective when they study four teachers and
The only way for us to really know what describe their four unique views. A phenom-
another person experiences is to experience enologist assumes a commonality in those hu-
man experiences and must use rigorously the lieved to be essential in invs tigations of hu-
method of bracketing to search for those com- man experience. The root meaning of heuristic
monalities. Results obtained from a phenom- comes from the Greek word heuriskein, mean-
enological study can then be related to and ing to discover or to find. It refers to a process
integrated with those of other phenomen- of internai search through which one discov-
ologists studying the same experience, or phe- ers the nature and meaning of experience and
nomenon. (Eichelberger 1989:6) develops methods and procedures for further
investigation and analysis. The self of the re-
In short, conducting a study with a
searcher is present throughout the process
phenomenological focus (i.e., getting at the
and, while understanding the phenomenon
essence of the experience of some phenome-
with increasing depth, the researcher also ex-
non) is different from using phenomenology
periences growing self-awareness and self-
to philosophically justify the methods of
knowledge. Heuristic processes incorporate
qualitative inquiry as legitimate m social sci-
creative self-processes and self discoveries.
ence research. Both contributions are impor-
(Moustakas 1990b:9)
tant. But a phenomenological study (as op-
posed to a phenomenological perspective) is
one that focuses on descriptions of what
There are two focusing or narrowing ele-
people experience and how it is that they ex-
ments of heuristic inquiry within the larger
perience what they experience. One can em-
framework of phenomenology. First, the re-
ploy a general phenomenological perspec-
searcher must have personal experience
tive to elucidate the importance of using
with and intense interest in the phenomenon
methods that capture people's experience of
under study. Second, others (co researchers)
the world without conducting a phenom-
who are part of the study must share an in-
enological study that focuses on the essence
tensity of experience with the phenomenon.
of shared experience (at least that is my ex-
Heuristics is not inquiry into casual experi-
perience and interpretation of the phenome-
ence. Heuristic inquiry focuses on intense
non of phenomenology).
human experiences, intense from the point
of view of the investigator and coresearch-
Heuristic Inquiry ers. It is the combination of personal experi-
ence and intensity that yields an under-
]~pimdationqi question; standing of the essence of the phenomenon.
What is of this "Heuristics is concerned with meanings, not
pkenomenon cmd +h< essential measurements; with essence, not appear-
e^pefience ance; with quality, not quantity; with ex-
of o+h e.rs who also perience, not behavior" (Douglass and
experience +his pkenomenou Moustakas 1985:42).
m+ensely? The reports of heuristic researchers are
Heuristics is a form of phenomenological filled with the discoveries, personal in-
inquiry that brings to the fore the personal sights, and reflections of the researchers.
experience and insights of the researcher. Discovery comes from being wide open to
the thing itself, a recognition that one must
"Heuristic" research came into my life when I relinquish control and be tumbled about
was searching for a word that would mean- with the newness and drama of a searching
ingfully encompass the processes that I be- focus, "asking questions about phenomena
that disturb and challenge" (Douglass and understanding is achieved by indwelling.

Moustakas 1985:47). (Polanyi 1967:160)
The uniqueness of heuristic inquiry is the
extent to which it legitimizes and places at The rigor of heuristic inquiry comes from
the fore these personal experiences, reflec- systematic observation of and dialogues
tions, and insights of the researcher. The re- with self and others, as well as depth inter-
searcher, then, comes to understand the es- viewing of coresearchers. This mode of in-
sence of the phenomenon through shared quiry "affirms the possibility that one can
reflection and inquiry with coresearchers as live deeply and passionately in the moment,
they also intensively experience and reflect be fully immersed in mysteries and mira-
on the phenomenon in question. A sense of cles, and still be engaged in meaningful re-
connectedness develops between researcher search experience" (Craig 1978:20).
and research participants in their mutual ef-
forts to elucidate the nature, meaning, and The power of heuristic inquiry lies in its poten-
essence of a significant human experience. tial for disclosing truth. Through exhaustive
The fundamental methods text on heuris- self-search, dialogues with others, and cre-
tic inquiry is by the primary developer of ative depictions of experience, a comprehen-
this approach, Clark Moustakas (1990b). His sive knowledge is generated, beginning as a
classic works in this tradition include stud- series of subjective and developing into a sys-
ies of loneliness (1961, 1972, 1975) and hu- tematic and definitive exposition. (Douglass
manistic therapy (1995). Other examples are and Moustakas 1985:40)
Bernthal (1990), Clark (1988), Hawka (1986),
Weidman (1985), Katz (1987), Cheyne (1988),
Heuristic inquiry is derived from but dif-
Marino (1985), and Craig (1978). Heuristic
ferent from phenomenology in four major
inquiry has strong roots in humanistic psy-
ways (Douglass and Moustakas 1985):
chology (Maslow 1956, 1966; Rogers 1961,
1969, 1977) and Polanyi's (1962) emphasis
1. Heuristics emphasizes connectedness
on personal knowledge, indwelling, and the
and relationship, while phenomenology
tacitdimension (1967). "Tacit knowing oper-
encourages more detachment in analyz-
ates behind the scenes, giving birth to the
ing an experience.
hunches and vague, formless insights that
characterize heuristic discovery" (Doug-
2. Heuristics leads to "depictions of essen-
lass and Moustakas 1985:49). Polanyi ex-
tial meanings and portrayal of the in-
plained tacit knowing as the inner essence of
trigue and personal significance that
human understanding, what we know but
imbue the search to know," while phe-
can't articulate.
nomenology emphasizes definitive der
scriptions of the structures of experi-
Tacit knowing now appears as an act of in- ence.
dwelling by which we gain access to a new
meaning. When exercising a skill we literally 3. Heuristics concludes with a "creative
dwell in the innumerable muscular acts which synthesis" that includes the research-
contribute to its purpose, a purpose which er^ intuition and tacit understand ings,
constitutes their joint meaning. Therefore, while phenomenology presents a distil-
since ali understanding is tacit knowledge, ali lation of the structures of experience.
4. "Whereas phenomenology loses the Qualitative Heuristics:

persons in the process of descriptive A German Alternative Tradition
analysis, in heuristics the research par-
ticipants remain visible in the examina- Since no authority exists to monitor and
tion of the data and continue to be por- sort out nomenclature, conflicts in usage oc-
trayed as whole persons. Phenom- cur, contributing to confusion and the im-
enology ends with the essence of experi- portance of reaffirming the admonition to al-
ence; heuristics retains the essence of the ways define one's terms. Heuristic inquiry,
person in experience" (p. 43). la Clark Moustakas and discussed in the pre-
vious section, has a nomenclature rival in
Systematic steps in theheuristic inquiry pro- "qualitative heuristics," an approach devel-
cess lead to the "definitive exposition" of ex- oped at the University of Hamburg, Ger-
periential essence: immersion, incubation, many, which aims to "bring back the quali-
illumination, explication, and creative syn- ties of systematic exploration and discovery
thesis (Moustakas 1990a). into psychological and sociological re-
What is important about heuristics for my search" (Kleining and Witt 2000:1). It is
purpose here, that is, describing variety in based on four rules.
qualitative inquiry, is that heuristic research
epitomizes the phenomenological emphasis Rule 1. The research person should be
on meanings and knowing through per- open to new concepts and change his or
sonal experience; it exemplifies and places at her preconceptions if the data are not in
the fore the way in which the researcher is agreement with them.
the primary instrument in qualitative in-
quiry; and it challenges in the extreme tradi-
Rule 2. The topic of research is preliminary
tional scientific concerns about researcher
and may change during the research
objectivity and detachment, as in autoeth-
process. It is only fully known after be-
nography (described earlier in this chapter).
ing successfully explored.
In essence, it personalizes inquiry and puts
the experience (and voice) of the inquirer
front and center throughout. Rule 3. Data should be collected under the
paradigm of maximum structural varia-
tion of perspectives. Variation of the
If I am investigating the meaning of delight, sample and of research methods avoids
then delight hovers nearby and follows me one-sidedness of representation of the
around. It takes me fully into its confidence topic; variation of questions avoids just
and I take it into mine. Delight becomes a Iin- one answer. If researchers assume that a
gering presence; for awhile, there is only de- variable may influence the data they
light. It opens me to the world in a joyous way should implement variations. Structural
and takes me into a richness, playfulness and variations mean sampling of positions
childlikeness that move freely and effortlessly. in reference to the topic, i.e., when
I'm ready to see, feel, touch or hear whatever studying an emotion, the collection of
opens me to a fuller knowledge and under- data past and present, before and after
standing of the experience of delight. (Mous- its occurrence, in different situations,
takas 1990b:ll) from different respondents, if possible
from different times and cultures, by dif- unfamiliar sense, has no following and is
ferent methods, etc. tartly reminded that "it isn't in the dictionary"
although down to the time of the first lexi-
Rule 4. The analysis is directed toward dis- cographer (Heaven forgive him!) no author
covery of similarities. It locates similari- ever had used a word that was in the dictio-
ties, accordance, analogies or hom- nary. In the golden prime and high noon of
ologies within these most diverse and English speech; when from the lips of the great
varied data. It tries to overcome differ- Elizabethans fell words .that ma de their own
ences. The rule follows Simmel's fa- meaning and carried it in their very sound;
mous chapter on method saying that when a Shakespeare and a Bacon were possi-
"out of complex phenomena the ho- ble, and the language now rapidly perishing
mogenous will be extracted . . . and the at one and slowly renewed at the other was in
dissimilar paralyzed." (Kleining and vigorous growth and hardy preservation
Witt 2000: online) sweeter than honey and stronger than a lion
the lexicographer was a person unknown,
the dictionary a creation which his Creator
This approach emphasizes "introspec-
had not created him to create. (p. 110)
tion" as a criticai part of the analytical pro-
cess, an element also central to "heuristic in-
quiry" in the tradition of humanistic
Ethnomethodology
psychology. However, neither heuristic in-
quiry as articulated by Moustakas (1990b)
nor this German alternative labeled "quali- Fo unda+ional question:
tative heuristics" can be derived directly -How do people mal<e sense of+k eir
from the common dictionary definition of everyday activities so as to bekave
heuristics, defined as techniques to assist In socJally accep+able ways?
learning or techniques for exploratory prob-
lem solvingthough neither approach con-
flicts explicitly with the dictionary defini- Where heuristic inquiry focuses on issues
tion. Those who lament such variations in of intense personal interest, ethnometh-
meanings, denotations, and connotations odology focuses on the ordinary, the routine,
may find some comfort in Ambrose Bierce's the details of everyday life. Harold
([1906] 1999) DevWs Dictionary definition of Garfinkel (1967) invented the term. While
lexicographer: working with the Yale cross-cultural files,
Garfinkel came across such labels as "eth-
A pestilent fellow who, under the pretense of nobotany," "ethnophysiology," and "ethno-
recording some particular stage in the devel- physics." At the time he was studying jurors.
opment of a language, does what he can to ar- He decided that the deliberation methods
rest its growth, stiffen its flexibility and of the jurors, or for that matter of any
mechanize its methods. For your lexicogra- group, constituted an "ethnomethodol-
pher, having written his dictionary, comes to ogy" wherein ethno refers to the "availability
be considered "as one having authority," to a member of common-sense knowledge
whereas his function is only to make a record, of his society as common-sense knowledge
not to give a l a w . . . . Recognizing the truth that of the 'whatever' " (Turner 1974:16). For the
language must grow by innovation if it grow jurors this was their ordinary, everyday un-
at ali, makes new words and uses the old in an derstanding of what it meant to deliberate as
a juror. Such an understanding made jury ing something out of the ordinary. A very
duty possible. simple and well-known such experiment is
Ethnomethodology studies the social or- turning to face the other people on an eleva-
der "by combining a phenomenological sen- tor instead of facing the doors. When they
sibility with a paramount concern for every- conduct such qualitative experiments, "the
day social practice" (Gubrium and Holstein researchers are interested in what the sub-
2000:490). Wallace and Wolf (1980) defined jects do and what they look to in order to
ethnomethodology as follows: "If we trans- give the situation an appearance of order, or
lated the 'ethno' part of the term as 'mem- to 'make sense' of the situation" (Wallace
bers' (of a group) or 'folk' or 'people/ then and Wolf 1980:278). Garfinkel (1967) offered
the term's meaning can be stated as: mem- a number of such experiments (see espe-
bers' methods of making sense of their social cially pp. 38,42,47, 79, and 85).
world" (p. 263). Ethnomethodology gets at Ethnomethodologists also have special
the norms, understandings, and assump- interests in observing naturally occurring
tions that are taken for granted by people in experiments where people are thrust into
a setting because they are so deeply under- new or unexpected situations that require
stood that people don't even think about them to make sense of what is happening,
why they do what they do. It studies "the or- "situations in which meaning is problem-
dinary methods that ordinary people use to atic" (Wallace and Wolf 1980:280). Such situ-
realize their ordinary actions" (Coulon ations include intake into a program, immi-
1995:2). Rooted in phenomenology, ethno- gration clearance centers, the first few weeks
methodology has been particularly impor- in a new school or job, and major transition
tant in sociology. points or criticai incidents in the lives of peo-
ple, programs, and organizations.
Ethnomethodology is, as the name suggests, a
In some respects, ethnomethodologists
study of methods. It asks not why, but how. It
attempt to make explicit what might be
asks how people get things donehow they
called the group's "tacit knowledge," to ex-
transform situations or how they persevere,
tend Polanyi's (1967) idea of tacit knowl-
situation "unchanged," step by step, and mo-
edge from the individual to the group. Heu-
ment to moment. As its name also suggests, it
ristic inquiry reveals tacit knowledge
is interested in ordinary methods, the meth-
through introspection and intersubjective
ods of the people rather than their theorists.
inquiry with coresearchers. Ethnometh-
(Watson and Goulet 1998:97)
odologists get at a group's tacit knowledge
Ethnomethodologists elucidate what a by forcing it to the surface through disrupt-
complete stranger would have to learn to being violations of ordinary experience, since
come a routinely functioning member of a ordinary routines are what keep tacit knowl-
group, a program, or a culture. To do this, edge at an unconscious, tacit levei.
ethnomethodologists conduct depth inter- In short, ethnomethodologists "bracket
views and undertake participant observa- or suspend their own belief in reality to
tion. They stray from the nonmanipulative study the reality of everyday life" (Taylor
and unobtrusive strategies of most qualita- and Bogdan 1984:11). Elucidating the
tive inquiry in employing "ethnometh- taken-for-granted realities of everyday life
odological experiments." During these ex- in a program or organization can becoine a
periments, the researcher "violates the force for understanding, change, and estab-
scene" and disrupts ordinary activity by do- lishing a new reality based on the kind of ev-
eryday environment desired by people in interpret the world. Only through close con-
the setting being studied. The findings of tact and direct interaction with people in
an ethnomethodological evaluation study open-minded, naturalistic inquiry and in-
would create a programmatic self-aware- ductive analysis could the symbolic
ness that would, in principie at least, facili- interactionist come to understand the sym-
tate program change and improvement. bolic world of the people being studied.
Blumer was also one of the first to use group
Symbolic Interaction discussion and interview methods with key
informants. He considered a carefully se-
Poundationa questions lected group of naturally acute observers
and weH-informed people to be a real "panei
W K Q + common set of symbols
of experts" about a setting or situation, ex-
and understandings \\cxs emefged
perts who would take the researcher inside
+o give me.av\'iv\) to people's
the phenomenon of interest, for example,
interactions?
drug use. As we shall see in the chapter on
Symbolic interaction is a social-psycho- interviewing, group interviews and focus
logical approach most closely associated groups have now become highly valued and
with George Herbert Mead (1934) and Her- widely used qualitative methods.
bert Blumer (1969). It is a perspective that Labeling theorythe proposition that
places great emphasis on the importance of what people are called has major conse-
meaning and interpretation as essential hu- quences for social interactionhas been a
man processes in reaction against behavior- primary focus of inquiry in symbolic inter-
ism and mechanical stimulus-response psy- action. For example, using a sample of 46
chology. People create shared meanings participants in a 12-step group, Debtors
through their interactions, and those mean- Anonymous, Hayes (2000) studied how
ings become their reality. Blumer articulated people who are unable to manage their fi-
three major premises as fundamental to nances responsibly come to feel shame. In
symbolic interactionism: program evaluation, labeling theory can be
applied to such terms as dropouts and at-risk
1. Human beings act toward things on the you th because language matters to staff and
basis of the meanings that the things participants and can affect how they ap-
have for them. proach attaining desired outcomes (Hopson
2000; Patton 2000).
2. The meaning of things arises out of the Though this theoretical perspective
social interaction one has with one's fel- emerged in the 1930s, symbolic interac-
lows. tionists are showing that they can keep up
with the times, for example, by applying
3. The meanings of things are handled in their perspective to "cybersex" on the
and modified through an interpretative Internet. Waskul, Douglass, and Edgley
process used by the person in dealing (2000) have suggested that the new technol-
with the things he or she encounters. ogies of computer-mediated communica-
tion
These premises led Blumer to qualitative
inquiry as the only real way of understand- allow us to examine the nature of human inter-
ing how people perceive, understand, and action in a uniquely disembodied environ-
ment that potentially transforms the nature of tific study of a space-time event like a solar
self, body, and situation. Sexfundamentally eclipse or rat behavior," Walker Percy
a bodily activityprovides an ideal situation (1990:150) has explained, "is that as soon as
for examrng these kinds of potential trans- one scratches the surface of the familiar and
formations. In the disembodied context of comes face to face with the nature of lan-
on-line interaction both bodies and selves are guage," one also finds oneself face to face
fluid symbolic constructs emergent in com- with the nature and essence of being human.
munication and are defined by sociocultural This is so because semiotics, in working to
standards. Situations such as these are sugges- "unite logical analysis with the explanatory
tive of issues related to contemporary trans- enterprise of science" (p. 243), has hit upon
gressions of the empirical shell of the body, the fruitful insight that humans are distinc-
potentially reshaping body-to-self-to-social- tively sign-using and symbol-generating an-
world relationships. (p. 375) imais. Thus, semiotics offers a framework
for "analyzing talk and text" (Silverman
2000:826) or studying "organizational sym-
For our purposes, the importance of symbolism" (Jones 1996). The foundational
bolic interactionism to qualitative inquiry is question of semiotics is: How do signs
its distinct emphasis on the importance of (words, symbols) carry and convey meaning
symbols and the interpretative processes in particular contexts?
that undergird interactions as fundamental
to understanding human behavior. For pro-
gram evaluation, organizational develop- Hermeneutics
ment, and other applied research, the study
of the original meaning and influence of Houndafioât question:
symbols and shared meanings can shed Wkat condi+ions u n d e r
light on what is most important to people, wki^k Q Kuman a c t took p t a c e
what will be most resistant to ehange, and or a produci voas pyoduae.^
what will be most necessary to change if the m a k e i+ p o s s i b l e t o i n t e r p r e t i+s
program or organization is to move in new meanings?
directions. The subject matter and methods
of symbol interactionism also emphasize the In this brief (or not-so-brief, depending on
importance of paying attention to how par- your perspective) excursion through the vari-
ticular interactions give rise to symbolic un- ety of qualitative inquiry, we depart now
derstandings when one is engaged in chang- from phenomenology and its derivative
ing symbols as part of a program im- approaches: heuristic research, ethnometh-
provement or organizational development odology, and symbolic interactionism. Her-
process. meneutics is yet a different theoretical ap-
A related theoretical tradition informing proach that can inform qualitative inquiry
some qualitative inquiry is semiotics, a blend and also help put ali the other theoretical ori-
of linguistics and social science, which fo- entations in this chapter in perspective in
cuss on the analysis of signs by studying that it reminds us that what something
the rules or forms of language as well as the means depends on the cultural context in
relationship between language and human which it was originally created as well as the
behavior (Manning 1987). "The importance cultural context within which it is subse-
of a study of language, as opposed to a scien- quently interpreted. This is a reminder that
each of the theoretical perspectives pre- cism worked from a conception of knowl-
senteei in this chapter emerged from a partic- edge as correct representation of an inde-
ular context to address specific concerns at pendent reality and was (is) almost exclu-
that time. As we adopt and adapt those per- sively interested in the issue of establishing
spectives to current inquiries, we do so in a the validity of scientific knowledge claims"
different historical, scholarly, and cultural (Schwandt 2000:196). In other words, her-
context. meneutics challenged the assertion that an
Hermeneutic philosophy, first developed interpretation can ever be absolutely correct
by Frederich Schleiermacher (1768-1834) or true. It must remain only and always an
and applied to human science research by interpretation. The meaning of a text, then, is
Wilhelm Dilthey (1833-1911) and other Ger- negotiated among a community of interpret-
man philosophers, focuses on the problem ers, and to the extent that some agreement is
of interpretation. Hermeneutics provides a reached about meaning at a particular time
theoretical framework for interpretive un- and place, that meaning can only be based
derstanding, or meaning, with special atten- on consensual community validation. Texts,
tion to context and original purpose. The then, must be "situated" within some liter-
term hermeneutics derives from the Greek acy context (Barton, Hamilton, and Ivanic
word hermeneuein, meaning to understand 1999).
or interpret. Kvale (1987) has suggested,
There is an obvious link between hermeneuein The attempts to develop a logic of validation
and the god Hermes. Hermes is the within the hermeneutical tradition are rele-
fleet-footed divine messenger (he has wings vant for clarifying the validity of interpreta-
on his feet!). As a messenger, he is the bearer of tion in the qualitative research interview.
knowledge and understanding. His task is to The interpretation of meaning is character-
explain to humans the decisions of the gods. ized by a hermeneutical circle, or spiral. The un-
Whether hermeneuein derives from Hermes or derstanding of a text takes place through a
the other way round is not certain. (Crotty process where the meaning of the separate
1998:88) parts is determined by the global meaning of
text. In principie, such a hermeneutical expli-
In modern usage, hermeneutics offers a cation of the text is an infinite process while it
perspective for interpreting legends, stories, ends in practice when a sensible meaning, a
and other texts, especially biblical and legal coherent understanding, free of inner contra-
texts. To make sense of and interpret a text, it dictions has been reached. (p. 62)
is important to know what the author
wanted to communicate, to understand in- Kneller (1984) has offered four principies
tended meanings, and to place documents in for hermeneutic inquiry and analysis that
a historical and cultural context (Palmer can be applied beyond the interpretation of
1969). Following that principie, hermeneu- legends, literature, and historical docu-
tics itself must be understood as part of a ments:
19th- and 20th-century "broad movement
away from an empiricist, logical atomistic, 1. Understanding a human act or product,
designative, representational account of and hence ali learning, is like interpret-
meaning and knowledge Logical empiri- ing a text.
2. Ali interpretation occurs within a tradi- review of the historical development of her-
tion. meneutics and its influence on qualitative
theory, "Our debt to the hermeneutic tradi-
3. Interpretation involves opening myself tion is large" (p. 111).
to a text (or its analogue) and question-
ing it. Narratology or Narrative Analysis
4. I must interpret a text in the light of my Foundational questions:

situation (p. 68). What does this narrative or story
reveal about the person and wo rld
Hermeneutic researchers use qualitative -from which it carne? How can th is
methods to establish context and meaning narrative be interpreted so that it
for what people do. Hermeneutists "are provides an understanding of cxv\<^
much clearer about the fact that they are con- illuminates the life and culture that
structing the 'reality' on the basis of their in- created it?
terpre tations of data with the help of the par-
ticipants who provided the data in the study Hermeneutics originated in the study of
. . . If other researchers had different back- written texts. Narratology, or narrative anal-
grounds, used different methods, or had dif- ysis, extends the idea of text to include
ferent purposes, they would likely develop in-depth interview transcripts, life history
different types of reactions, focus on differ- narratives, historical memoirs, and creative
ent aspects of the setting, and develop some- nonfiction. The hermeneutical perspective,
what different scenarios" (Eichelberger with its emphasis on interpretation and
1989:9). For concrete examples of hermeneu- context, informs narrative studies, as do
tic investigations in psychology, see Packer interpretivist social science, literary nonfic-
and Addison (1989). tion, and literary criticism. Narrative studies
Thus, one must know about the re- are also influenced by phenomenology's
searcher as well as the researched to place emphasis on understanding lived experi-
any qualitative study in a proper, hermeneu- ence and perceptions of experience. "Todo-
tic context. Hermeneutic theory argues that rov coined the term narratology in 1969 in an
one can only interpret the meaning of some- effort to elevate the form 'to the status of an
thing from some perspective, a certain object of knowledge for a new science'"
standpoint, a praxis, or a situational context, (Riessman 1993:1).
whether one is reporting on one's own find- Personal narratives, family stories, sui-
ings or reporting the perspectives of people cide notes, graffiti, literary nonfiction, and
being studied (and thus reporting their life histories reveal cultural and social pat-
standpoint or perspective). These ideas have terns through the lens of individual experi-
become commonplace in much contempo- ences. Rhetoric of ali kinds can be fodder for
rary social science and are now fundamen- narrative analysis, for example, the rhetoric
tal, even basic, in qualitative inquiry, but of politicians or teachers (Graham 1993).
such was not always the case. Two centuries The "biographical turn in social science"
of philosophical dialogue provide our cur- (Chamberlayne et al. 2000) or the "narrative
rent foundation for understanding the cen- turn" in qualitative inquiry (Bochner 2001)
trality of interpretivism m qualitative re- honors people's stories as data that can
search. As Crotty (1998) concluded after his stand on their own as pure description of ex-
perience, worthy as narrative documentary Ali great literature, I think, lures those who ex-
of experience (the core of phenomenology) or perience it away from the shores of literal truth
analyzed for connections between the psy- and out into uncharted waters where meaning
chological, sociological, cultural, political, is more ambiguous.. . .
and dramatic dimensions of human experi- Ultimately, I erased the boundary between
ence. Robert Coles, Harvard professor of the realm of text which purports to give only
psychiatry and medicai humanities (his title the facts and that of the metaphor-laden story
offers interesting narratological fodder), has which dares (as Sartre once put it) to lie in or-
written The Call ofStories (1989) as a basis for der to tell the truth. But I did so haltingly, and
teaching, learning, and moral reflection. Mi- not in a single confident stroke of understand-
chael White and David Epston in Narrative ing. Indeed, my insight carne only gradually,
Means to Therapeutic Ends (1990) look at the after confronting a form of writing that aims to
power of stories in the lives of individuais straddle the boundary between actual and vir-
and families and the connection between tual worlds, one foot firmly planted in each.
storytelling and therapy. They suggest that These works are hybrids of textual species,
people have adjustment difficulties because essays/stories written in a literary style but
the story of their life, as created by them- shelved (curiously) m the nonfiction section
selves or others, does not match their lived of the library. (pp. 61-62)
experience. They propose that therapists can
help their patients by guiding them in re- Here we have an example of personal nar-
writing their life stories. rative in the form of the narrative re-
The idea of "story," of personal narrative, searcher^ report of his joumey into cross-
intersects with our earlier look at autoeth- genre exploration of the nature of textual in-
nography in which the researcher's story be- terpretation. Later he uses narrative as a
comes part of the inquiry into a cultural phe- method for exploring what it means to be a
nomenon of interest. The language of story professional educational researcher, explor-
carries a connotation different from that of ing the narratives researchers construct
case study. For example, in program evalua- about themselves and implications of those
tions, people may be invited to share their narratives for their relationships with non-
stories instead of being asked to participate researchers (Barone 2000:201-28).
in case studies. The central idea of narrative Tierney (2000), in contrast, examines his-
analysis is that stories and narratives offer torical biographies and testimonios to ex-
especially translucent windows into cul- plore interpretive challenges in using life
tural and social meanings. histories in the postmodern age. His narra-
Much of the methodological focus in nar- tive analysis looks at the intersection of the
rative studies concerns the nature of inter- interpreted purpose of a text, the con-
pretation, as in Norman Denzin's seminal structed and interpreted "truth" of a text,
qualitative works Interpretive Biography and the persona of the author in text creation,
(1989a), Interpretive Interactionism (1989b), ali of which are called into interpretive ques-
and Interpretive Ethnography (1997b). Inter- tion in the postmodern age.
pretation of narrative poses the problem of Tedlock (2000) examines different genres
how to analyze "talk and text" (Silverman of ethnography as constituting varying
2000). Tom Barone (2000) has entered into lit- forms of narrative. She distinguishes life
erary nonfiction to hone his interpretive aes- histories and memoirs from "narrative eth-
thetic: nography," a hybrid form that was created
Today would you read

us a story that works on
many leveis?
Budding narratologist
in and attempts to portray accurately the bi- tion" because it unsettled "the boundaries
ographies of people in the culture studied that had been central to the notion of a self
but also to include ethnographers' own ex- studying an other" and replaced it with an
periences in their texts. She assesses this as a "ethnographic interchange" between self and
"sea change in ethnographic representa- other within a single text (pp. 460-61).
Narrative analysis has also now emerged oped out of sociology. Heuristic inquiry is
as a specific approach to studying organiza- grounded in humanistic psychology. A dif-
tions. As such, it takes at least four forms: ferent psychology-based perspective is eco-
logical psychology, which represents a differ-
1. organizational research that is written in ent tradition and theoretical orientation
storylike fashion (tales from the field); because it makes different assumptions
about what is important to understand
2. organizational research that collects or-
about the human experience (Jacob 1987).
ganizational stories (tales of the field);
Robert Barker (1968) and Herbert Wright
3. organizational research that conceptual- (1967) of the University of Kansas devel-
izes organizational life as story making oped ecological psychology drawing
and organizational theory as story read- heavily on natural history field studies.
ing (nterpretative approaches); and They see individuais and the environment
4. a disciplinary reflection that takes the as interdependent (Barker and Wright 1955;
form of literary critique (Czarniawska Barker et al. 1978; Schoggen 1978). They be-
1998:13-14). gin with pure, detailed descriptions of an in-
dividual in an environment. They observe
Stories are at the center of narrative analy- (as spectators, not participant observers)
sis, whether they be stories of teaching "streams of behavior" that are subsequent-
(Preskill and Jacobvitz 2000), stories of and ly analyzed in terms of presumed goal-
by students (Barone 2000:119-31), stories of directed actions. "Coders draw upon their
participants in programs (Kushner 2000), ordinary knowledge and perceptions to in-
stories of fieldwork (Van Maanen 1988), sto- fer the goals that actors intend to achieve,
ries of relationships (Bochner, Ellis, and marking off segments of narrative descrip-
Tillman-Healy 1997), or stories of illness tions into segments leading toward specific
(Frank 1995,2000). How to interpret stories goals" (Jacob 1988:17). The ecological meta-
and, more specifically, the texts that tell the phor can also inform psychological clinicai
stories, is at the heart of narrative analysis. research by seeking "to understand the pa-
tienfs concern within the context of his or
her life worldthe patienfs personal, fam-
Ecological Psychology
ily, community, and ecological stories"
(Miller and Crabtree 2000:617).
T~oundationql question: The unit of analysis in ecological psychol-
Wkat is tke rela+ionskip ogy is primarily the individual, but Barker
between kuman bekaviou and and Schoggen (1973) have also applied this
tke environment? approach in delineating Qualities of Commu-
nity Life. What makes this approach of po-
tential interest for program evaluation and
Several theoretical perspectives that in- organizational or community development
form qualitative inquiry are associated with is the focus on goal-directed behavior.
particular disciplines. For example, herme-
neutics is derived from linguistics and phi- They assume that there are subjective aspects
losophy. Ethnography is the primary to behavior which they examine in terms of
method of anthropology, while ethnometh- the goals of human behavior. They also as-
odology and symbolic interaction devel- sume that there is a subjective aspect to the en-
viionments which they usually discuss in

terms of a person's emotional reactions to the
j j r i ; ; ! w y EcLcnar
environment. For example, they might be con-
cerned whether a boy does an activity unwill- IWftyClS-Eiyli 1 !! 1 'm!.' i"h.t-riij-i i"iiY
ingly or unhappily. (Jacob 1988:17) :'JHTLI/!: N."I.I M'iii::IRI!iJ:Vi:':i HFO,'[fi -PJIIS:;V.VI
fj't!:Mj! iVCVd:. it ij: :f!WLvî.'.Pn."

Ecological psychologists also focus on de- PI.D tO yni'--; ! SiiriM!! i ;.hj-rf bii-iV
lineating the central features of behavior set- jtl-rj'r]ri;!y1'hrdfuf:i?:iVii':tiitioqyw !!'i!i!,'
tings, the particular constellations of places, iT!.l!.'' jyEEfeS. iHilPV mlsViT" ..r!..':i."i):!i'ii'!i;;!
things, and times that constitute a definitive !'"!:> :' airAiWi :; !V! C 'utfj.'m' i:!:i"n:!;'
environment. Such an approach can help iHi"i'!irjj-i,' LU-''!.Vv FIRIS :'i'!i,;!i!' il-!!i!;='!:i"
make explicit what variety of environments I.HW:1 Mik it* fft";!.:: irn-' j-iVt-y"?.! r-
program participants or organization mem- ! il l&afi IS .1 bi!V :!!'!!' !*!! !:i 'nY iij.1 i"n i/iij i r i
: iu,' IVi j |, fi.<i' iilsiriV! h? n!*,!i.l !! i; k i.!.'; i ,Y! .;! r
bers experience.
While ecological psychologists begin ! iVau! '' !-::;J y -I-J J i j; p. LS1? ;: hm
,"p .feirt.aT .'hM-j.1 nMh ,h rVi jiVj='? li i'i! i'1'.'s
with detailed, qualitative descriptionsbased
d r;.^!:':I .MIJMîftlli?] h?:!- '' Jll:' h.:iM:i:!.';
on observations in natural environments,
' tmtiH .:ii".:r V:!- M-to-iUa.'' ^ri-liiv!!;!1
their coding schemes and analysis proce-
' !! h: n : |:i!,:i! JT.YIH" n ' >
" -:i*i ri! i.
dures are quantitative. Segments of goal-
ns i.!"vJi!i":r Lw i:'i:.IVvi.XViy &\).h!:!iji:i
directed behavior and characteristics of be- i' !'M!' Im!!i' -ii'i ii!'''ii i! dfiSir-jJ
:i ! ! v^y-î--
havior settings are coded numerically and ififa MMi!' hoWi ^ Nl!i.S.-ii n't ;' iV-^':
analyzed statistically. This illustrates a point ^prfh.JiW r 1 iVv-jiCi"!.! i.'i .'ilMij:' ViiVi! !l. !!:'
to which we shall return later: One can go s!J.!i! i!!fcfd'Yli!i'iiiri'i:i-T''.
from the thick description of qualitative data :T!;ii' tit n^&itf /irwni ;; rn,^ v r !i, ::Vi!r
to quantitative analysis, but not vice versa. ,'i M:; i-rii1!:'!!A'RTJ v.H t' ru!! i.V, ! : I"I \!v. ! i1 ii wA! ! T
One cannot generate thick description and AEW i::=I!T'..i,S!,i:! ia b*'Mr- l^Vi ::'
qualitative narrative from original quantita- ^ M N TI!:' J ! : I 5 : N ! ' : N I ' Y IFCNL
tive data. !" JIL'!?:!: \ I;!:!; i ;: !'J:T i.= i. !i.< ,ii,'ilj' AUi'
!:'I ! I'-
The focus in ecological psychology on the ^'idVOii :ii!'i!!!!1Y ii-.i :ii!i ,'i' .nH-.!!!'!ii!'!!.!!
relationship between human behavior and U.TdMI :1! * y r f p , l Jpi'!'.!:' !!!!>>"!!"
the environment provides a good transition 1111111111111
to the next perspective, systems theory,
which is much more comprehensive and m-
terdisciplinary in examining the context for Parallel to the historical philosophical
human actions in programs, organizations, and methodological paradigms debate be-
and communities. tween positivists and constructivists, there
has been another and corresponding para-
A Systems Perspective digms debate about mechanistic/linear con-
and Systems Theory structions of the world versus organic/sys-
tems constructions. This debate has been
particularly intense among classic organiza-
Tôundational question: tional theorists (Burns and Stalker 1972;
"How and why does this sys+em Azumi and Hage 1972; Lincoln 1985;
a s a whole fwnction a s it does? Gharajedaghi 1985; Morgan 1986, 1989). It
includes concern about definitions of closed
systems versus open systems, and the im- into independent parts as discrete entities of
plications of such boundary definitions for inquiry because the effects of the behavior of
research, theory and practice in understand- the parts on the whole depend on what is
ing programs, organizations, entire soci- happening to the other parts. The parts are
eties, and even the whole world (Wallerstein so interconnected and interdependent that
1980). any simple cause-effect analysis distorts
It is important to note at the outset that more than it illuminates. Changes in one
the term systems has many and varied mean- part lead to changes among ali parts and the
ings. In the digital age, systems analysis of- system itself. Nor can one simply add the
ten means looking at the interface between parts in some linear fashion and get a useful
hardware and software, or the connectivity sense of the whole.
of various networks. The idea of "systems Gharajedaghi and Ackoff (1985:23) are
thinking" was popularized as the crucial quite insistent that a system as a whole can-
"fifth discipline" of organizational leaming not be understood by analysis of separate
in Peter Senge's (1990) best-selling book. A parts. They argue that "the essential proper-
number of management consultants have ties of a system are lost when it is taken
made systems thinking and analysis the cen- apart; for example, a disassembled automo-
terpiece of their organizational develop- bile does not transport and a disassembled
ment work (e.g., Ackoff 1987, 1999a, 1999b; person does not live." Furthermore, the
Kim 1993, 1994, 1999; Anderson and John- function and meaning of the parts are lost
son 1997). Indeed, over the past 30 years, when separated from the whole. Instead of
since publication of Ludwig Von Berta- taking things apart, they insist that a sys-
lanffy's classic General System Theory (1976), tems approach requires "synthetic think-
a vast literature has developed about sys- ing":
tems theory and applied systems research
(e.g., Checkland 1999). Some of it is highly
quantitative and involves complex coin- Synthetic thinking is required to explain system
puter applications and simulations. Given behavior. It differs significantly from analysis.
this broad and multifaceted context, my In the first step of analysis the thing to be ex-
purpose is quite modest. I want to call to the plained is taken apart: in synthetic thinking it
reader's attention three points: (1) A systems is taken to be a part of a larger whole. In the
perspective is becoming increasingly impor- second step of analysis, the contained parts
tant in dealing with and imderstanding are explained: in synthetic thinking, the con-
real-world complexities, viewing things as taining whole is explained. In the final step of
whole entities embedded in context and still analysis, knowledge of the parts is aggregated
larger wholes; (2) some approaches to sys- into knowledge of the whole: in synthetic
tems research lead directly to and depend thinking understanding of the containing
heavily on qualitative inquiry; and (3) a sys- whole is disaggregated to explain the parts. It
tems orientation can be very helpful in fram- does so by revealing their role or fitnction in
ing questions and, later, making sense out of that whole. Synthetic thinking reveals func-
qualitative data. tion rather than structure: it reveals why a sys-
Holistic thinking is central to a systems tem works the way it does, but not how it does
perspective. A system is a whole that is both so. Analysis and synthesis are compleinen-
greater than and different from its parts. In- tary: neither replaces the other. Systems think-
deed, a system cannot validly be divided ing incorporates both.
Because the effects of the behavior of the have become very important in family re-
parts of a system are interdependent, it can be search and therapy (Schultz 1984; Mont-
shown that if each part taken separately is gomery and Fewer 1988; Rosenblatt 1985;
made to perform as efficiently as possible, the Miller and Winstead-Fry 1982; Hoffman
system as a whole will not function as effec- 1981). A systems approach has also become
tively as possible. For example, if we select one of the central orientations to interna-
from ali the automobiles available thebestcar- tional development efforts in recent years.
buretor, the best distributor, and so on for each Specifically, the farming systems approach
part required for an automobile, and then try to development (Farming Systems Support
to assemble them, we will not even obtain an Project [FSSP] 1986) illustrates some unique
automobile, let alone the best one, because the ways of engaging in qualitative inquiry to
parts zoill not fit together. The performance of a support development, intervention, and
system is not the sum of the independent ef- evaluation from a systems perspective. The
fects of its parts; it is the product of their inter- farming systems approach to evaluation
actions. Therefore, effective management of a and research is worth examining in detail
system requires managing the interactions of because it has developed as a theory-based
its parts, not the actions of its parts taken sepa- yet practical solution to agricultural devel-
rately. (Gharajedaghi and Ackoff 1985:23-24) opment problems.
In the first three decades following World
War II, much international development
This kind of systems thinking has pro-
was conceived as direct technology transfer
found implica tions for program evaluation
from more developed to less developed
and policy analysis where the parts are often
countries. Scientists and change agents
evaluated in terms of strengths, weaknesses,
made technology transfer recommenda-
and impacts with little regard for how the
tions within their disciplinary areas of spe-
parts are embedded in and interdependent
cialization, for example, crops, livestock,
with the whole program or policy (Patton
water, and so on. This approach to develop-
1999c). For example, Benko and Sarvimaki
ment epitomized a mechanistic orientation.
(2000) applied systems theory as a frame-
In reaction to the dismal failures of the
work for patient-focused evaluation in nurs-
mechanistic, specialized technology transfer
ing and other health care areas. Such a
approach to development, a farming sys-
framework, they found, allowed complex
tems approach emerged (Shaner, Philipp,
features of processes in health care to appear
and Schmehl 1982b). Several elements are
by conducting simultaneous analyses of re-
central to a farming systems perspective, el-
latioriships on different leveis and with dif-
ements that lead directly to qualitative
ferent methods. This contrasts with the
methods of research.
mostly one-level, reductionist designs that
have usually been employed in nursing and
1. Farming systems research and develop-
health care research. Their "systemic
ment (FSRD) is a team effort (Shaner,
model" offered insights into system dy-
Philipp, and Schmehl 1982a).
namics in both "downward" and "upward"
directionsand the interconnections of 2. FSRD is interdisciplinary. The team con-
these systems dynamics in affecting patient sists of representatives from a mix of
care and outcomes. both agricultural and social science
In addition to their influence in organiza- disciplines (Cernea and Guggenheim
tional development, systems approaches 1985).
3. FSRD takes place in the field, on real A farming systems approach includes
farms, not at a university or goverrunent both qualitative and quantitative forms of
experiment station (Simmons 1985). inquiry. It includes direct observations, in-
4. FSRD is collaborativescientists and formal interviews, naturalistic fieldwork,
farmers work together on agricultural and inductive analysis, ali within a systems
productivity within the goals, values, framework. Well over 100 such projects in
and situation of participating farmers FSRD have been undertaken worldwide
(Galt and Mathema 1987). (FSSP 1987). There maybe no larger-scale ex-
ample of efforts to integra te naturalistic in-
5. FSRD is comprehensive, including atten- quiry, quantitative methods, and a systems
tion to ali farm family members; ali perspective through interdisciplinary eval-
farming operations, both crops and live- uation and research teamwork for the pur-
stock; ali labor sources; ali income pose of promoting long-term social and eco-
sources; and ali other factors that affect nomic developments.
small farm development (Harwood
FSRD is just one example of a systems ap-
1979).
proach to intervention, research, and evalu-
6. FSRD is inductive and exploratory, begin- ation. What this and other systems ap-
ning by open-ended inquiry into the proaches illustrate is that the complex world
nature of the farming system from the of human beings cannot be fully captured
perspective of those in the system and understood by simply adding up care-
(Holtzman 1986). fully measured and fully analyzed parts. At
7. FSRD begins with qualitative description. the system levei (the whole program, the
The first team task is fieldwork to quali- whole farm, the whole family, the whole or-
tatively describe the system (Sands ganiza tion, the whole community), there is a
1986). qualitative difference in the kind of thinking
that is required to make sense of what is hap-
8. FSRD is sensitive to context, placing the pening. Qualitative inquiry facilitates that
farming system in the larger agro- qualitative difference in understanding hu-
ecological, cultural, political, economic, man or "purposeful systems" (Ackoff and
and policy environments of which it is a Emery 1982).
part (Shaner et al. 1982a). A final story will reinforce this point, the
9. FSRD is interactive, dynamic, and process fable of the nine blind people and the ele-
oriented. The interdisciplinary team be- phant, which I used in the second chapter to
gins with inductive exploration, then illustrate the importance of context, and
moves to trying out system changes, ob- which I repeat here because it illustrates so
serving the effects, and adapting to well the real challenge of systems thinking.
emergent findings. The work is ongoing Besides, good stories have layers of mean-
and developmental (FSSP 1986). ing, and this one has phenomenological,
hermeneutic, and even ethnographic impli-
10. FSRD is situationalh/ responsive and adap- ca tions, which the reader may want to reflect
tive. There are many variations in FSRD on, but TIl simply reintroduce it as a systems
projects depending on priority prob- tale. Ironically, it is of ten offered as an exam-
lems, available resources, team member ple of systems thinking, but is, in its usual
preferences, and situation-specific pos- Western telling, actually quite linear and
sibilities (Sands 1986; FSSP 1987). mechanical.
As the story goes, nine blind people en- groups, programs. Chos or complexity
counter an elephant. One touches the ear theorists and researchers are primarily theo-
and proclaims that an elephant is like a fan. retical physicists, meteorologista, biologists,
Another touches the trunk and says the ele- and other natural scientists. Chos research
phant most surely resembles a snake. The has developed as a highly quantitative spe-
third feels the elephanfs massive side and cialty requiring supercomputer calculations
insists that it is like a wall. Yet, a fourth, feel- (Cambei 1992). But the assumptions that un-
ing a solidly planted leg, counters that it dergird chos theory pose challenges to so-
more resembles a tree trunk. The fifth grabs cial science research at the most fundamen-
hold of the tail and experiences the elephant tal leveis of basic conceptualization.
as a rope. And so it goes, each blindly touch- Complexity theory is already being
ing only a part and generalizing inappropri- viewed as a new paradigm of natural science
ately to the whole. The usual moral of the (Nadei and Stein 1995; Murali 1995; Hall
story is that only by putting ali the parts to- 1993; Holte 1993; Waldrop 1992; Gleick 1987;
gether in right relation to each can one get a Cronbach 1988). At least at the levei of meta-
complete and whole picture of the elephant. phor, chos and complexity notions are be-
Yet, from a systems perspective, such a ing used to inf orm approaches to economics
picture yields little real understanding of the (Ormerod 2001), anthropology (Agar 1999),
elephant. To understand the elephant, it organizational development (Eoyang 1997;
must be seen and understood in its natural Allison 2000), and leadership (Wheatley
ecosystem, whether in frica or Asia, as one 1992). The concepts of system and complex-
element in a complex system of flora and ity are often closely related. For example, the
fauna. Only m viewing the movement of a self-organization of systems, as premised by
herd of elephants across a real terrain, over complexity theory, implies the maintenance
time and across seasons, in interaction with of a certain levei of organization or the im-
plants, trees, and other animais will one be- provement of the systems (Rhee 2000). As
gin to understand the evolution and nature social scientists begin to understand its as-
of elephants and the system of which ele- sumptions, complexity theory about nonlin-
phants are a part. That understanding can ear dynamics may become a new paradigm
never come at a zoo. for approaching human complexities. In the
Thus, are we reminded of the challenge meantime, theory and research about non-
and importanceof bringing a systems linear dynamics (complexity) raise ques-
perspective into qualitative inquiry. tions about how we bring order to what we
observea fundamental epistemological
problem for ali forms of inquiry, including
Chos and Complexity Theory: qualitative inquiry. In Exhibit 3.5 (p. 126), I
Nonlinear Dynamics offer some teasers from Gleick's (1987) pop-
ularization of chos theory to suggest impli-
T-pu^daticmal question: cations for qualitative inquiry.
What is the- undeîymg order, if At this point, complexity theory offers,
cmy, oj- disorderly phenomena? perhaps more than anything else, a new set
of metaphors for thinking about what we
What are disorderly phenomena? The observe, how we observe, and what we
weather, waterfalls, fluids in motion, volca- know as a result of our observations. Chos
noes, galaxiesand human beings, human theory challenges our need for order and
prediction, even as it offers new ways to ful- Michael Agar, a distinguished anthropol-
fill those needs. While much chos research ogist, used complexity theory, especially the
is highly mathematical, making sense of re- work of John Holland (1995, 1998), to inter-
sults seems to depend heavily on meta- pret fieldwork findings in his study of a her-
phors. Here is an intersection with qualita- oin epidemic among suburban youth in Bal-
tive inquiry that holds particular promise timore County, Maryland. He concluded:
because much work in qualitative analysis,
organiza tional development, and programs Complexity [theory] served, at least at the
includes resort to metaphor (Patton 2000; metaphorical levei, to better define a research
Ronai 1999; Brady 1998). Indeed, Gleick problemexplaining heroin trendsand it
(1987) offers a metaphor to explain the very helps articulate why traditional social re-
nature of inquiry into chos: "It's like walk- search has not answered this most basic ques-
ing through a maze whose walls rearrange tion of drug research: How and why do trends
themselves with every step you take" (p. 24). occur? It also points at the kind of data we
This metaphor fits a great deal of field- need to obtain and organize to do just that,
work in real-world settings, but the implica- however difficult that data might be to obtain.
tions can be so threatening to our need for Furthermore, complexity handles some cur-
order that we ignore the rearranging walls rent anthropological research issueslike the
and describe the maze with a single, static inclusion of the researcher, broadening histor-
diagram. If nothing else, the history and ical and political context, and the issue of pre-
emergent ideas of chos theory may give us dictionas part of its central themes. With
the comfort and courage to describe nonlin- characteristics like holism, emergence, and
ear dynamics (chos) when we find it, with- feedback that map onto anthropological as-
out imposing false order to fulfill the pre- sump tions more so than any previous formal
sumed traditional purpose of analysis. models, complexity is clearly worth a closer
Chos theory challenges us to deal with un- look. (Agar 1999:119)
predictability and indeterminism in human
behavior (Cziko 1989)and therefore in the The metaphors of chos, complexity, and
interventions (programs) we devise to alter nonlinear dynamics open up new possibili-
human behavior as well as the unpredict- ties for doing fieldwork in and understand-
ability and indeterminism of the methods ing those settings that feel like walking
we use to study and evaluate those interven- through a maze whose walls rearrange
tions. themselves with every step you take.
Grounded Theory
T he grounded theory approach is the most influential paradigm for quali-

tative research in the social sciences today.
Norman K. Denzin (1997a:18)

gies of theory development in contrast to theory

Tôundational question: generated by logical deduction from a priori
Wka+ tkeory emerges from assumptions.
systematic comparative analysis
and is grounded in fieldwork so In contrasting grounded theory with
as to explîn wkat kas been and logico-deductive theory and discussing and
is observed? assessing their rela tive merits in ability to fit
and work (predict, explain, and be relevant),
we have taken the position that the adequacy
Now we tum from the fluidity of chos of a theory for sociology today cannot be di-
to the solidity of the ground, specifically, vorced from the process by which it is gener-
grounded theory. Most of the theoretical ated. Thus one canon for judging the use-
perspectives examined thus far focus on a fulness of a theory is how it was generated
particular aspect of human experience: Eth- and we suggest that it is likely to be a better
nography focuses on culture, ethnometh- theory to the degree that it has been induc-
odology on everyday life, symbolic inter- tively developed froin social r e s e a r c h Gen-
actionism on symbolic meanings in be- erating a theory from data means that most
havior, semiotics on signs, hermeneutics on hypotheses and concepts not only come from
interpretations, and phenomenology on the data, but are systematically worked out in
lived experience. Their theoretical frame- relation to the data during the course of the re-
works direct us to particular aspects of hu- search. Generating a theory involves a process of
man experience as especially deserving of research. (Glaser and Strauss 1967:5-6)
attention in our attempt to make sense of the
social world. In contrast, grounded theory This theory-method linkage is of great
focuses on the process of generating theory concem in many of the orientations exam-
rather than a particular theoretical content. ined in this chapter. The idea of a the-
It emphasizes steps and procedures for con- ory-method linkage means that how you
necting induction and deduction through study the world determines what you leam
the constant comparative method, compar- about the world. Grounded theory depends
ing research sites, doing theoretical sam- on methods that take the researcher into and
pling, and testing emergent concepts with close to the real world so that the results and
additional fieldwork. findings are grounded in the empirical
Concern for theory development is often world. Herbert Blumer (1978) has offered a
quite marked in the literature on qualitative metaphor for explaining what it means to
methods. The writings of Glaser (1978, generate grounded theory by being im-
2000), Strauss and Corbin (1998), Denzin mersed in the empirical world:
(1978b), Lofland and Lofland (1984), Blumer
(1969), Whyte (1984), and Becker (1970), to The empirical social world consists of on-go-
name but a few well-known qualitative ing group life and one has to get close to this
methodologists, take as a major focus the life to know what is going on in it. The meta-
task of theory construction and verification. phor that I like is that of lifting the veils that
What distinguishes the discussion of theory obscure or hide what is going on. The task of
in much of the literature on qualitative scientific study is to lift the veils that cover the
methods is the emphasis on inductive strate- area of group life that one purposes to study.
Complexity (Chos) Theory Precepts and

EXHIBIT 3.5 Qualitative Inquiry Implications
Chos Precepts and Assumptions Implications for Qualitative

(Gltick 1987) Inquiry an Human Systems
1. "Nonlinearity means that the act of 1. The entry of the researcher into a setting
piaying the game has a way of changing may do more than create problems of
the rules" (p. 24). validity and reactivity. The researcher's
entry may make it a different setting
altogether-and forever.
2. A butterfly in Beijing flapping its wings 2. Small, minute events can make criticai
may affect the weather in New York differences. Qualitative importance is not
next month or next year. "The butterfly dependent on quantitative magnitude.
effect" has a technical na me: Sensitive For want of a n a i l . . . , the war was lost
dependence on initial conditions (p. 23).
3. A determinstic system can produce much 3. Much qualitative analysis attempts to
more than just periodic behavior. There bring orderfrom chos, identifying
can be "wild disorder" among "islands of patterns in the noise of human complexity.
structure." "A complex system can gve Chos theory suggests we need to learn to
rise to turbulence and coherence at the observe, describe, and value disorder and
same time," each of which is important turbulence without forcing patterns onto
(p. 56). genuine, meaningful chos.
4. "Simple systems can do complicated 4. What presumptions do we bring to field-

things" (p. 167). work and analysis about simplicityand
complexity? These are not neutral terms.
5. "A healthy body is a chaotic one; when 5. How do we observe and describe
you reach an equiibrium in biology you dynamic, constantly changing phenomena
are dead" (p. 298). without imposing a static structure by the
very boundaries we create in seeking to
define and understand?
6. "On the collective scale and on the 6. Chos theory's meanings and implications
personal scale, the ideas of chos for qualitative inquiry in human settings
advance in different ways and for remain to be developed.
different reasons" (p. 316).
The veils are not lifted by substituting, in odology that do not encourage or allow this
whatever degree, preformed images for betray the cardinal principie of respecting the
first-hand knowledge. The veils are lifted by nature of one's empirical world [T]he merit
getting close to the area and by digging deep of naturalistic study is that it respects and
in it through careful study. Schemes of meth- stays close to the empirical domain. (p. 38)
Ali of the approaches to theory and re- Let me be clear. Grounded theory is a general
search in this chapter use qualitative meth- method. It can be used on any data or combi-
ods to stay grounded in the empirical world. nation of data. It was developed partially by
Yet, they vary considerably in their concep- me with quantitative data. It is expensive and
tualizations of what is important to ask and somewhat hard to obtain quantitative data, es-
consider in elucidating and understanding pecially m comparison to qualitative data.
the empirical world. While the phrase Qualitative data are inexpensive to collect,
"grounded theory" is often used as a general very rich in meaning and observation, and
reference to inductive, qualitative analysis, very rewarding to collect and analyze. So, by
as an identifxable approach to qualitative in- default, due to ease and growing use,
quiry it consists of quite specific methods grounded theory is being linked to qualitative
and systematic procedures (Glaser 2000, data and is seen as a qualitative method, using
2001). In their book on techniques and proce- symbolic interaction, by many. Qualitative
dures for developing grounded theory, grounded theory accounts for the global
Strauss and Corbin (1998:13) emphasized spread of its use.
that analysis is the interplay between re- I can only caution the reader not to confuse
searchers and data, so what grounded the- this empirical use and the spread of its use
ory offers as a framework is a set of "coding with the fact that it is a general method. In
procedures" to "help provide some stan- some quarters of research, grounded theory is
dardization and rigor" to the analytical pro- considered qualitative, symbolic interaction
cess. Grounded theory is meant to "build research. It is a kind of takeover that makes
theory rather than test theory." It strives to routine qualitative research sound good by
"provide researchers with analytical tools positive stigma. Only highly trained grounded
for handling masses of raw data." It seeks to theory researchers can see the difference and
help qualitative analysts "consider alter- the confusion. Much of it revolves around the
native meanings of phenomenon." It em- notion of emergence versus forcing and the
phasizes being "systematic and creative si- lack of use of ali the grounded theory method-
multaneously." Finally, it elucidates "the ological steps. Any kind of data can be con-
concepts that are the building blocks of the- stantly compared. However, it is prudent for
ory." Glaser (1993) and Strauss and Corbin researchers to go with qualitative grounded
(1997) have collected together in edited vol- theory when that is where the resources are to
umes a range of grounded theory exemplars do it and when that is where researchers can
that include several studies of health (life af- reap career and personal rewards. (p. 7)
ter heart attacks, emphysema, chronic renal
failure, chronically ill men, tuberculosis,
Grounded theory has opened the door to
Alzheimer's disease), organizational head-
qualitative inquiry in many traditional aca-
hunting, abusive relationships, women
demic social science and education depart-
alone in public places, selfhood in women,
ments, especially as a basis for doctoral dis-
prison time, and characteristics of contem-
sertations, in part, I believe, because of its
porary Japanese society.
overt emphasis on the importance of and
While grounded theory has become specific procedures for generating theory. In
widely thought of as an approach specific to addition, I suspect its popularity may owe
qualitative inquiry, Glaser (2000) does not much to the fact that it unabashedly admon-
limit it in that way: ishes the researcher to strive for "objectiv-
ity." As discussed earlier in this chapter, the She believes that the guidelines for
postmodern attack on objectivity has found grounded theory offered by Strauss and
its way into qualitative inquiry through Corbin (1990, 1998) "structure objectivist
constructivism, hermeneutic interpretivism, grounded theorists' work. These guidelines
and the emphasis on subjective experience are didactic and prescriptive rather than
in phenomenology. Emergent autoethno- emergent and interactive" (Charmaz 2000:
graphic and heuristic approaches to qualita- 524). In contrast, she believes that in a
tive inquiry place even greater emphasis on constructivist grounded theory, "causality is
the researcher's personal and subjective ex- suggestive, incomplete, and indeterminate.
perience. Those social scientists and aca- . . . It looks at how 'variables' are grounded
demics who find some value in the methods given meaning and played out in subjects'
of qualitative inquiry, namely, in-depth in- lives. . . . Their meanings and actions take
terviewing and observation, but who es- priority over researchers' analytic interests
chew the philosophical undeipinnings of and methodological technology" (p. 524). To
constructivism and interpretivism can find illustrate a constructivist approach to
comfort in the attention paid to objectivity in grounded theory, she presents to the reader
grounded theory. the kinds of questions she would ask to
study a topic such as pain:
It is important to maintain a balance between
the qualities of objectivity and sensitivity I start by viewing the topic of pain subjectively
when doing analysis. Objectivity enables the as a feeling, an experience that may take a vari-
researcher to have confidence that his or her ety of forms. Then I ask these questions: What
findings are a reasonable, impartial represen- makes pain, pain? (That is, what is essential to
tation of a problem under investigation, the phenomenon as defined by those who ex-
whereas sensitivity enables creativity and the perience it?) What defining properties or char-
discovery of new theory from data, (Strauss acteristics do ill people attribute to it? When
and Corbin 1998:53) do they do so? How does the person expe-
rience this pain, and what, if any thing, does he
or she do about it? My questions aim to get at
At the same time, the language of meaning, not at truth. As a result, a con-
"grounded theory" has found its way into structivist grounded theory may remain at a
the constructivist literature. Charmaz (2000) more intuitive, impressionistic levei than an
compares "objectivist" (reality-oriented) and objectivist approach. (Charmaz 2000:526)
constructivist approaches to grounded the-
ory and, though she finds examples of both,
Beyond drawing on the inductive and
believes that the majority of grounded theo-
layered emphases in grounded theory la
rists are objectivist in orientation.
Strauss and Corbin, it is hard to see how
what Charmaz describes is different from
Objectivist grounded theory accepts the basic phenomenological inquiry. As a mat-
positivistic assumption of an externai world ter of philosophical distinctness, then,
that can be described, analyzed, explained, grounded theory is best understood as fun-
and predicted: truth, but with a small t . . . It damentally realist and objectivist in orienta-
assumes that different observers will discover tion, emphasizing disciplined and proce-
this world and describe it in similar ways. dural ways of getting the researcher 's biases
(p. 524) out of the way but adding healthy doses of
creativity to the analytic process. We shall what conceptual framework will direct
consider the analytic procedures of fieldwork and the interpretation of findings.
grounded theory in more detail in the chap- For example, one can undertake a study
ter on analyzing qualitative data. As a theo- from a feminist perspective, a Marxist per-
retical framework, I have included it in this spective, a capitalist perspective, or a Freud-
chapter because of its emphasis on generat- ian perspective, among others. In these in-
ing theory as the primary purpose of quali- stances, the ideological orientation or
tative social science and its overt embrace of perspective of the researcher determines the
objectivity as a research stance. focus of inquiry.
A feminist perspective presumes the im-
portance of gender in human relationships
Orientational Qualitative and societal processes and orients the study
Inquiry: Feminist Inquiry, in that direction (Guerrero 1999b; Ribbens
Criticai Theory, and Queer and Edwards 1998; Maguire 1996; Reinharz
Theory as Examples 1992; Glennon 1983; Smith 1979). Principies
of feminist inquiry (Guerrero 1999a:15-22;
One of the strengths of qualitative meth- Thompson 1992) can include
ods is the inductive, naturalistic inquiry
strategy of approaching a setting without o a sense of connectedness and equality
predetermined hypotheses. Rather, under- between researcher and researched;
standing and theory emerge from fieldwork n explicitly acknowledging and valuing
experiences and are grounded in the data. "women's ways of knowingj' including
The problem is how to approach the field integrating reason, emotion, intuition,
with an open mind. Phenomenology in- experience, and analytic tfiought;
cludes recommended procedures for be-
n participatory processei that support
coming clear about and taking into account
consciousness-raisiiig and researcher re-
biases and predispositions during both
flexivity; and /
fieldwork and analysis so as to get at the true
essence of the phenomenon under study. going beyond knowledge generation,
Hermeneutics takes the position that noth- beyond "knowledge for its own sake," to
ing can be interpreted free of some perspec- engage in using knowledge for change,
tive, so the first priority is to capture the per- especially "knowledge about women
spective and elucidate the context of the that will coritribute to women's libera-
people being studied. The researcher's own tion and einancipation" (Guerrero 1999a:
perspective must also be made explicit, as 16-17).
must any other tradition or perspective
brought to bear when interpreting mean- How does the lens of gender shape and
ings. affect our understandings and actions?
Orientational qualitative inquiry goes Philosopher Elizabeth Minnich has m-
one step farther. Orientational qualitative vestigated the ways in which conceptual
inquiry eschews any pretense of open- approaches to classifying human beings,
mindedness in the search for grounded or embedded historically, culturally, and po-
emergent theory. Orientational qualitative litically, continue to shape our thinking
inquiry begins with an explicit theoretical or through the very language and categories
ideological perspective that determines available to us. Her book on the subject,
Transforming Knowledge (1990, forthcoming), inquiry challenges the phenomenological

speaks precisely and insightfully to the ori- notion that one can cleanse oneself of such
entation of feminist inquiry. fundamental language-based conceptions
when doing fieldwork and data analysis.
The root problem reappears in different guises Moreover, feminist inquiry provides not
in ali fields and throughout the dominant tra- only conceptual and analytical direction but
dition. It is, simply, that while the majority of also methodological orientation in empha-
humankind was excluded from education and sizing participatory, collaborative, change-
the making of what has been called knowl- oriented, and empoweringforms of inquiry.
edge, the dominant few not only defined them- A quite different theoretical framing for
selves as the inclusive kind of human but also as the inquiry would be a Freudian orientation that
norrn and the ideal. A few privileged men de- assumes that individual behavior must be
fined themselves as constituting mankind/ understood as a manifestation of the strug-
humankind and simultaneously saw them- gle between id, ego, and superego as influ-
selves as akin to what mankind/humankind enced by very early childhood relationships
ought to be in fundamental ways that distin- and sexual experiences that have left their
guished them from ali others. Thus, at the mark on the unconscious. Orientations can
same time they removed women and non- be combined, as in a feminist psychoana-
privileged men within their culture and other lytical framework (Eichenbaum and Orbach
cultures from "mankind," they justified that 1983).
exclusion on the grounds that the excluded Racism and ethnicity can be another de-
were by nature and culture "lesser" people (if fining lensor orientationfor qualitative
they even thought of the others as having "cul- inquiry in research and evaluation (Ladson-
tures"). Their notion of who was properly hu- Billings 2000; Stanfield 1999; Patton 1999d),
man was both exclusive and hierarchical with as can inclusiveness (Mertens 1998, 1999).
regard to those they took to be properly sub- "Queer theory," an orienta tio nal approach
ject to themwomen in ali roles; men who focused on sexual orientation, "took social
worked with their hands; male servants and constructionist insights and added a post-
slaves; women and men from many other cul- structuralist critique of the unified, autono-
tures. mous self," so a lesbian, gay, bisexual, and
Thus, they created root definitions of what
transgender orientation informs inquiry as
it means to be human that, with the concepts
"a deconstructive enterprise, taking apart
and theories that flowed from and reinforced
the view of a self defined by something at its
those definitions, made it difficult to think
core, be it sexual desire, race, gender, nation,
well about, or in the mode of, anyone other
or class" (Gamson 2000:348).
than themselves, just as they made it difficult One of the most influential orientational
to think honestly about the defining few. frameworks is "criticai theory," which fo-
(Minnich 1990:37-38) cuses on how injustice and subjugation
shape people's experiences and understand-
ings of the world.
The concepts and conceptual frameworks
we use, whether unconsciously as a matter A criticai social theory is concerned in particu-
of tradition and training or intentionally as a lar with issues of power and justice and the
matter of choice, carry embedded messages ways that the economy, matters of race, class,
about what and who is important. Feminist and gender, ideologies, discourses, education,
religion and other social institutions and cul- term orientational to describe such studies
tural dynamics interact to construct a social because they are oriented in a particular di-
system. . . . Inquiry that aspires to the name rection or framed from a specific perspec-
criticai mustbe connected to an atteinpt to con- tive. Orientational is a more neutral term
front the injustice of a particular society.... than ideologically based inquiry
Research thus becomes a transformative en- The extent to which any particular study
deavor unembarrassed by the label political is orientational is a matter of degree.
and unafraid to consummate a relationship Ethnographic studies can be viewed as
with emancipatory consciousness. (Kincheloe orientational to the extent that they presume
and McLaren 2000:281,291) the centrality of culture in explaining hu-
man experience. "Criticai ethnography"
Thus, what gives criticai theory its name (Thomas 1993) combines the focus on cul-
what makes it criticaiis that it seeks not ture with the commitinent to use findings
just to study and understand society but for change. Symbolic interactionism is
rather to critique and change society. Influ- orientational in focusing on the importance
enced by Marxism, informed by the pre- of the meanings that emerge as people de-
sumption of the centrality of class conflict in fine situations through interpersonal inter-
understanding community and societal actions. Orientational qualitative inquiry is
structures (Crotty 1998; Heydebrand 1983; a legitimate and important approach to the-
Carchedi 1983), and updated in the radical oretical or ideological elaboration, confir-
struggles of the 1960s, criticai theory pro- mation, and elucidation. What is required is
vides a frameworkboth philosophy and that the researcher be very clear about the
methodsfor approaching research and theoretical framework being used and the
evaluation as fundamentally and explicitly implications of that perspective on study
political, and as change-oriented forms of focus, data collection, fieldwork, and anal-
engagement. Fonte (2001) offers an example ysis.
of criticai theory applied to public policy.
Fonte applies the perspective of Marxist in-
Variety in Qualitative
tellectual Antonio Gramsci to contemporary
American politics, considering how domi- Inquiry: Different Answers
nant and subordinate groups based on race to Core Questions
and gender struggle over power in ways
that make every aspect of life political. Exhibit 3.6 summarizes the theoretical
Within any of these theoretical or ideolog- and philosophical perspectives presented in
ical orientations one can undertake qualita- this chapter. This is not an exhaustive list of
tive inquiry, but the focus of inquiry is deter- theoretical possibilities, but it does include
mined by the framework within which one the most common conceptual and philo-
is operating and findings are interpreted sophical frameworksand it certainly doc-
and given meaning from the perspective of uments the variety of perspectives that can
that preordinate theory. Such qualitative in- inform qualitative inquiry.
quiry, therefore, aims to describe and ex- No consensus exists about how to classify
plain specific manifestations of already-pre- the varieties of qualitative research. As
sumed general patterns. Such inquiry is noted in the opening of this chapter, but
aimed at confirmation and elucidation worth repeating as a review of variety in
rather than discovery. I have chosen the qualitative inquiry, Crotty (1998:5) elabo-
EXHIBIT 3.6 Variety in Qualitative Inquiry: Theoretical Traditions
Perspective Disciplinary Roots Central Questions
1. Ethnography Anthropology What is the culture of this group of people?

2. Autoethnography Literary arts How does my own experience of this culture
connect with and offer insights about this
culture, situation, event, and/or way of life?
3. Reality testing: Philosophy, social What's really going on in the real worid?
Positivist and realist sciences, and What can we establish with some degree of
approaches evaluation certainty? What are plausible explanations for
verifiable patterns? What's the truth insofar
as we can get at it? How can we study a
phenomenon so that our findings correspond,
as much as possible, to the real world?
4. Constructionl5m/ Socoogy How have the people in this setting
constructivism constructed reality? What are their reported
perceptions, "truths," explanations, beliefs, and
worldview? What are the consequences
of their constructions for their behaviors and
for those with whom they interact?
5. Phenomenology Philosophy What is the meaning, structure, and essence of
the lived experience of this phenomenon
for this person or group of people?
6. Heuristic inquiry Humanistic What is my experience of this phenomenon and
psychology the essential experience of others who also
experience this phenomenon intensely?
7. Ethnomethodoiogy Sociology How do people make sense of their everyday
activities so as to behave in socially
acceptable ways?
8. Symbolic interaction Social psychology What comrnon set of symbols and understand-
ings has emerged to give meaning to people's
interactions?
rated five major theoretical perspectives as tions of qualitative inquiry, but a different
the foundations of social research: positiv- five: biography, phenomenology, grounded
ism (and postpositivism), interpretivism theory, ethnography, and case study. Jacob
(which includes phenomenology, herme- (1987) chose yet a different five for a qual-
neutics, and symbolic interactionism), criti- itative taxonomy: ecological psychology,
cai inquiry, feminism, and postmodernism holistic ethnography, ethnography of com-
(to which he adds an "etc." to suggest the munication, cognitive anthropology, and
open-ended nature of such a classification). symbolic interactionism. Schwandt (2000)
Creswell (1998) also settled on five tradi- highlighted "three epistemological stances
Perspective Disciplinary Roots CentroI Questions
9. Semiotics Linguistics How do signs (words, symbols) carry and

convey meanng in particular contexts?
10. Hermeneutics Linguistics, philosophy, What are the conditions under which a
iiterary crticism, human act took place or a product was
theoiogy produced that makes it possible to interpret
its meanings?
11. Narratology/ Social sciences What does this narrative or story reveal about
narrative analysis (interpretive): the person and world from which it carne?
Literary crticism, How can this narrative be interpreted to
Iiterary nonfiction understand and illuminate the life and culture
that created it?
12. Ecoiogical psychology Ecology, psychology How do individuais attempt to accomplish
their goals through specific behaviors in
specific environments?
13. Systems theory Interdisciplinary How and why does this system as a whole
function as it does?
14. Chos theory: Theoretical physics, What is the underlying order, if any, of
Nonlinear dynamics natural sciences disorderly phenomenon?
15. Grounded theory Social sciences, What theory emerges from systematic
methodology comparative analysis and is grounded in
fieldwork so as to explain what has been
and is observed?
16. Orientational: ideologies: Poltica!, How is X perspective manifest in this
Feminist inquiry, cultural, and economic phenomenon?
criticai theory, queer
theory, among others
for qualitative inquiry": interpretivism, her- branches showing different "qualitative

meneutics, and social constructivism. strategies." Tesch (1990) identified 27 variet-
Denzin and Lincoln (2000a) organized their ies. Having examined some of the various
reviewof qualitative variety around seven attempts to classify qualitative approaches,
historical periods and seven "para- Miles and Huberman (1994) concluded, "As
digms/ theories": positivist/postpositivist, comprehensive and clarifying as these cata-
constructivist, feminist, ethnic, Marxist, cul- logs and taxonomies may be, they tum out
tural studies, and queer theory. Wolcott to be basically incommensurate, both in the
(1992) created a family tree of 20 distinct way different qualitative strands are defined
and m the criteria used to distinguish them. a How do we know what we know? (episte-
The mind boggles in trying to get from one mological debates about the possibility
to another" (p. 5). and desirability of objectivity, subjectiv-
Adding to this complexity is the practice ity, causality, validity, generalizability)
of combining some perspectives. For exam- a How should we study the world? (method-
ple, one can do a heuristic feminist (orien- ological debates about what kinds of
tational) study, that is, undertake a heuristic data and design to emphasize for what
inquiry from a feminist perspective. Or do purposes and with what consequences)
"criticai ethnography" (Thomas 1993),
What is worth knowing? (philosophical
combing elements of criticai theory and eth-
debates about what matters and why)
nography. Bentz and Shapiro (1998) have of-
fered what they call "mindful inquiry" as a o What questions should we ask? (disciplin-
synthesis of phenomenology, hermeneutics, ary and interdisciplinary debates about
criticai theory, and Buddhism. From phe- the importance of various burning ques-
nomenology they take the focus on experi- tions, inquiry traditions, and areas of in-
ence and consciousness. From hermeneutics quiry)
they take the focus on texts, on the process of How do we personally engage in inquiry?
understanding, and on letting new mean- (praxis debates about interjecting per-
ings emerge from the research process. From sonal experiences and values into the
criticai theory they direct attention to the so- inquiry, including issues of voice and
cial and historical context of both the re- political action)
searcher and the research topic, including
attention to domination, injustice, and op- The same program, organization, or com-
pression. From Buddhism they take the fo- munity studied by researchers from differ-
cus onbecoming aware of one's own "addic- ent perspectives will lead to quite different
tions" and attachments and on practicing studies even though they might ali under-
compassion. In positing this synthesis, they take observations, interviews, and docu-
aim to place the researcher, rather than re- ment analysis. Nor would it necessarily be
search techniques, at the center of the re- possible to synthesize the descriptions and
search process. This adds something of a re- findings of such different studies even
flexive, autoethnographic orientation as though they took place in the same setting.
another foundation of mindful inquiry be- When researchers operate from different
cause the mindful inquirer uses awareness frameworks, their results will not be readily
of personal, social, and historical context, interpretable by or meaningful to each other.
and personal ways of knowing, to shape the While the frameworks provide guidance
research. and a basis for interaction among research-
The variety of qualitative frameworks is ers operating within the same fr ame work,
distinguished by answers to six core ques- the different theoretical frameworks consti-
tions (one for each day of the week plus a tute barriers that impede interaction across
day left over to integrate your answers): and among different perspectives. In effect,
each theoretical framework is a minipara-
What do we believe about the nature ofreal- digm with its own internai logic and as-
ity? (ontological debates concerning the sumptions.
possibility of a singular, verifiable reality This means one cannot reasonably ask
and truth vs. the inevitability of socially which theoretical framework is "right," best,
constructed multiple realities) or most useful. It depends on what one
wants to do and which assumptions one Finally, a caution would seem in order
shares. Gareth Morgan (1983) stated the about the danger of reifying the theoretical
problem quite succinctly after presenting a distinctions offered in this chapter. Take a
variety of research perspectives: look again at Exhibit 3.1, my reply to a letter
from a graduate student desperate to figure
There was the question as to how the reader
out what category of inquiry she fit into. The
could come to some conclusion regarding the
boundaries between perspectives remain
contrary nature, significance, and claims of
fuzzy. Adherents within each perspective
the different perspectives. . . . I realized that
canbe found arguing about what is essential
there was a major problem h e r e . . . . There is a
to that perspective. Tom Schwandt, who has
fallacy in the idea that the proposi tions of a
studied these distinctions as much as any-
system can be proved, disproved, or evalu-
one and is the lexicographer of the Dictio-
ated on the basis of axioms within that system.
nary of Qualitative Inquiry (2001), offers this
. . . This means that it is not possible to judge
reflection on theoretical distinctions:
the validity or contribution of different re-
search perspectives in terms of the ground as- It seems to be a uniquely American tendency
sumptions of any one set of perspectives, since to categorize and label complicated theoretical
the process is self-justifying. Hence the at- perspectives as either this or that. Such label-
tempts in much social science debate to judge ing is dangerous, for it blinds us to enduring
the utility of different research strategies in issues, shared concerns, and pomts of tension
terms of universal criteria based on the impor- thatcut across the landscape of the movement,
tance of generalizability, predictability and issues that each inquirer must come to terms
control, explanation of variance, meaningful with in developing an identity as a social in-
understanding, or whatever are inevitably quirer. In wrestling with the ways in which
flawe: These criteria inevitably favor research these philosophies forestructure our efforts to
strategies consistent with the assumptions that understand what it means to " d o " qualitative
generatesuch criteria as meaningfulgitidelines for inquiry, what we face is not a choice of which
the evaluation of research. It is simply inade- labelinterpretivist, constructivist, herme-
quate to attempt to justify a particular style of neuticist, or something elsebest suits us.
research in terms of assumptions that give rise Rather, we are confronted with choices about
to that style of research Different research how each of us wants to live the life of a social
perspectives make different kinds of knowl- inquirer. (Schwandt 2000:205)
edge claims, and the criteria as to what counts
as significant knowledge vary from one to an-
other. (pp. 14-15)
li. P r a g m a t i s m
In other words, readers must make their
own decisions about the relative value of Having documented the variety of theoreti-
any given perspective. Each has strengths. cal perspectives that inform qualitative in-
Each has limitations. There is no universal quiry, we now leave the world of theory and
standard that can be applied to choose enter the world of practice and pragmatism.
among these different frameworks. Quite Not ali questions are theory based. Indeed,
the contrary, the diversity itself is a good in- the quite concrete and practical questions of
dicator of the complexity of human phe- people working to make the world a better
nomena and the challenges involved in con- place (and wondering if what they're doing
ducting research. is working) can be addressed without plac-
Sample Internet E-mail Discussion Groups (listservs)

E X H I B I T 3.7 and Sites Relevant to Qualitative Inquiry and Theory
1. [email protected]: Use of ethnographic research methods in education; to

subscribe, send this message to [email protected]: join ethnography-in-education firstname
lastname
2. [email protected]: Ethnomethodology/conversation analysis; to subscribe, send this message to
[email protected]: join ethno ourname
3. http://www.tgsa.edu/online/cybrary/phenom.html: Phenomenology
4. http://www.ped.gu.se/biorn/phgraph/home.html: Phenomenography
5. www.groundedtheory.com/vidserie5l.html: Grounded theory Web site
6. [email protected]: Q Methodology discussion list on this broad approach to the study
of subjectivity; to subscribe, send this message to [email protected]: subscribe Q-METHOD
ourname; for help contact [email protected]
7. [email protected]: Biographicai Methods for the Social Sciences; to subscribe, send
this message to [email protected]: join BIOG-METHODS
LASTNAME
8. [email protected]: A discussion of narrative in everyday life; to subscribe, send this
message to [email protected],nz: subscribe psych-narrative
9. www.chass.utoronto.ca/epc/srb/cyber/cyber.html: Cyber Semiotic Institute
10. www.shop.affinia.com/ppsesystem5theory/5tore1/: Systems theory site
11. [email protected]: H-Net/Oral History Association Discussion List on Oral History; to
subscribe, send this message to [email protected] subscribe H-ORALHIST
LASTNAME AFFILIATION
NOTE: Thanks to Judith Preissle, Aderhold Distinguished Professor, Social Foundations of Education, University
of Gergia, for list subscription details. These sites and subscription detaJs may change, and this list is not ex-
haustive.This list is meantto be suggestive of the qualitative analysis resources available through the Internet.
See Chapter 1, Exhibit 1.5; Chapter 4, Exhbit 4.9; and Chapter 8, Exhibit 8.3, for additional qualitative re-
sources through the Internet.
ing the study in one of the theoretical frame- students writing dissertations and academic
works in this chapter. While these intellec- scholars will necessarily be concerned with
tual, philosophical, and theoretical tradi- theoretical frameworks and theory genera-
tions have greatly influenced the debate tion, there is a very practical side to qualita-
about the value and legitimacy of qualitative methods that simply involves asking
tive inquiry, it is not necessary, in m y opin- open-ended questions of people and observ-
ion, to swear vows of allegiance to any sin- ing matters of interest in real-world settings
gle epistemological perspective to use in order to solve problems, improve pro-
qualitative methods. grams, or develop policies. In short, in
Indeed, I would go farther (at the risk of real-world practice, methods can b e sepa-
being heretical) and suggest that one need rated jfrom the epistemology out of which
not even be concerned about theory. While they have emerged.
One can use statistics in straightforward able ways to f ind out what is happening in
ways without doing a philosophical litera- programs and other human settings.
ture review of logical empiricism or realism. The next chapter explores some of the
One can make an interpretation without ways in which qualitative inquiry can con-
studying hermeneutics. And one can contribute to practical knowledge and prag-
duct open-ended interviews or make obser- matic understandings. To help make that
vations without reading treatises on phe- transition, this chapter ends with a practical,
nomenology. The methods of qualitative cautionary tale from Halcolm.
inquiry now stand on their own as reason-
X k e . y V p p I e of \Souv-
After Halcolm had completed explaining to a scholarly assembly the many dif-
fering perspectives one could use in looking at the world, he was hungry. While he
answered questions and continued the discussion, he sent a listener to inquire if
the midday meai was ready. The messenger did not return, so Halcolm sent a sec-
ond messenger. The second messenger did not return. So Halcolm wenthimself.
He found the two messengers, the chef, and three visiting scholars engaged in
heated debate. Ignoring the debate, Halcolm asked, "Is the midday meai ready?"
The first and eldest visiting scholar responded, "I have been explaining to these
young men that the state of the food is not the only issue in determining readiness.
A meai is not just food. The meai must include those who would par take of the
food, so the meai is not ready until everything is in order and those who would eat
are assembled."
The second visiting scholar said, "I dared to taste the meai. From the perspec-
tive of a gourmet chef, this meai will never be ready. It is hopeless; let us return to
the city."
The third visiting scholar said, "Readiness is a state of mmd, not a physical
state. Since the food has no mind, the food cannot be ready. Only people can be
ready."
The chef added, "The midday meai is at midday every day. At midday the meai
is ready. Why ask if the meai is ready? It is midday. This is the meai. Therefore the
midday meai is ready."
With that, they ali began talking at once making ever finer points, drawing ever
narrower distinctions.
Halcolm, meanwhile, sat down and ate his midday meai.
A student asked why he had not joined the debate to clarify these important is-
sues. Halcolm took another bite and replied, "The apple of your eye won't satisfy
the emptiness in your stomach. There is a time to talk about the nature of eat-
ingand there is a time to eat."
From Halcolm's Guidefor Gourmands
APPENDIX 3.1
S S S
Example of Autoethnographic Writing
Introduction. This excerpt is from the first chapter of Grand Canyon Celebration:
A Father-Son Journey ofDiscovery (Patton 1999a). The excerpt combines inquiry
into a cultural phenomenon of interest with personal reflection on and experi-
ence of that phenomenon, in this case, male coming of age in modern society.
My son Brandon and I were joined by Malcolm, a friend and our Grand Can-
yon guide.
Vishnu Metamorphism
To see the enormity of the Grand Canyon you have to be orbiting the Earth.
To feel it, you have to descend within. To learn from it, you have only to stay
awhile and be present. At least thafs what Malcolm had claimed when he
first urged me to hike with him from Apache Point to Eives Chasm years ear-
lier. And learn I had, about bloody blisters, debilitating thirst, and the impor-
tance of moving quickly when a rock ledge gives way a thousand feet above
the canyon floor, especially if you're standing on it at the time. Modest
learnings. But they left an impression. As did the depth and beauty of the
Canyon.
Malcolm had been bringing questions about his life to the Canyon for
years. And getting answers. I had gotten no answers on that first trip. But
that, Malcolm explained, was because I had brought no questions. Fair
enough. I had come for the hike and a chance to walk among the oldest ex-
posed rocks on the Earth's surface.
But I did get an idea. Standing atop Mount Huethawali and staring across
the Colorado Ri ver at Holy Grail Temple, I imagined someday hiking with
my son, then just entering toddlerhood, and initiating him into manhood
there amidst buttes named King Arthur Castle, Guinevere Castle, and
Excalibur, and gorges named Merlin Abyss and Modred Abyss. Malcolm
called it a vision, which beguilingly transformed a passing notion into a
quest, like framing a telephone doodle and calling it art. What better place for
grandiosity than the Grand Canyon?
The gilt frame, however, didn't quite make it back with me to Minnesota. I
realized that I lacked a few of the basic necessities for conducting an initia-
tion. Tribal elders, for example. Hard to come by if you don't have a tribe. As
are other essentials, like tradition, a sacred place, ritual, terrifying gods to ap-
pease, wisdom to pass on, and life-threatening tests for the initiate to pass
(preferably ones that the initiator has successfully survived). From what I re-
Variety in Qualitative Inquiry |J.
called of anthropology, strong gender identity would also be a prerequisite.

That, however, might be conjured up. I had felt a vague sense of something
while gazing toward Lancelot Point. Malcolm suggested that the Canyon was
putting me in touch with my masculine collective unconscious. After ten days
in the Canyon such things could be said without soundmg absurd. Like eating
freeze-dried food. It can taste gourmet scrumptious after a hard day hiking, but
cooked at home, it's ghastly. So I found that my Canyon initiation vision didn't
reconstitute well mixed with urban fluoridated water.
But it also didn't evaporate.
Malcolm now smiles and says he never doubted. I, on the other hand, still
find myself amazed that we actually did return with Brandon for an initiation
experience. And, berng a social science researcher, I kept field notes. Not, I
should add, because I had any premonition that they might reveal something
important about a humanist approach to coming of age in contemporary soci-
ety. I did it for family history and, I concede, out of habit. I had spent too many
years in sociological observation to turn off that part of myself just because I
had brought my eldest son into one of the most magnificent landscapes on
earth after many years of anticipation. I considered leaving my scientific side
behind. I even tried .Justbe a father, I told myself. Justbe in the Canyon. Be pres-
ent with Brandon. Don't analyze it while it's happening. Stay with the experi-
ence.
Or were those Malcolm's admonitions? Certainly, some part of me was in-
trigued by Malcolm/s belief that he got answers from the Canyon. And, unlike
our first hike years earlier, this time I found I had come with a question, though
I wasn't fully aware of it until our second night.
We were campe d within the inner gorge, just short of the Colorado River,
where White Creek flows out of Muav Canyon into Shinumo Creek. My aching
body craved rest after two hard days hiking, but Brandon's after-dinner ques-
tions about how different cultures define manhood had left me tossing and
turning. He slept near enough for me to hear his slow, evenbreathing. As I stud-
ied him, he rolled from his back onto his side, pulling his knees up fetus-like, al-
most, but not quite, transforming his gangling, 18-year-old frame into a picture
of innocence. He looked like the question he had asked over dinner.
He had begun with a mocking tone: "So, this is my initiation. When do I find
out about the manhood thing? I'm sure you two have come prepared with im-
portant insights. Might as well get on with it. I, your humble initiate, am ali
ears."
Our subsequent anthropological discussion about how different societies
define manhood was rooted in cultural relativism as solidly as the large cotton-
wood that sheltered our campsite. The discussion had been serious, intense
and surprisingly lacking in satisfaction. Not for Brandon. For me.
As I gazed at Brandon sleeping, voices argued with each other in my head.
What does a modern father tell his son about being a man? Some voices, re-
corded in my memory long ago, rasped repeatedly like a worn needle stuck in
scratched grooves from the waxen days of graduate school. Others, more re-
cently entered, played intermittently through the scratches. The messages from
different eras competed to be heard, rising to a discordant crescendo, like being
caught in a small gym between opposing fans and their blaring pep bands at a
championship basketball gameexhilarating only if you know which side to
cheer for.
Such imagery being incongruent with my peaceful environs, though I enjoy
both debate and athletic competition, I redirected my inner musings to the
steady gurgle of nearby rapids and the chirping melodies of the canyon night. I
quietly got up to strollback and forth along the creek, pondering what I wanted
to pass on to Brandon about the nature of manhood. I paused in the shadow of
ancient rock and listened as Shinumo's rapids asserted the constant flow of the
present. I tried out possibilities on a disinterested moon: metaphors of male in-
candescence and female florescence.
What was left to tell Brandon that he hadn't already heard from me ad nau-
seam? I could affirm that the moon is disinterested, that the Canyon is rock, and
that life offers many pathways for being a man and developing as a person,
none of them certain. I could offer perspective and Canyon-inspired meta-
phors. . . .
This trip, this "initiation," felt like a last chance. When, if ever again, would I
have Brandon's undivided attention? Or at least some part of it? I was not quite
so delusional as to believe I could attain the impossibly high standard of "undi-
vided attention" with a member in good standing of the generation that grew
up on channel-surfing, But I did have ten uninterrupted days and nights with
Brandon. No outside influences. No competition from television, telephones,
friends or work.
Ten whole days with my son in the Grand Canyon. Ten days before he left homefor
college and the rest ofhis adult life. Ten final days. A last chance.
I returned to where Brandon slept and, gazing at him, considered whether it
much mattered what I had to saywords, after ali, being only words. But
words matter in my world, as do answers. Thought matters. And so I thought
some more until, under the influence of that elixir unique to the small hours
when the body is exhausted and the internai dialogue worn down, I experi-
enced at last a euphoria of analogical clarity. It came as I tumed and peered into
the dark gorge through which we had descended. That very afternoon, we had
traversed the Canyon's Great Unconformity, in one step passing through a gap
of 250 million years across a space that had once been filled with massive
mountains. Recalling that moment took me through what felt like a par aliei un-
conformity, insignificant by standards of Canyon time, but huge when mea-
sured on the modest scale of human evolution. Canyon metaphor offered so-
ciological insight. Malcolm would later say the Canyon had answered my
question.
The Canyon's Great Unconformity had once been filled with towering Pre-
cambrian formations of Bass Limestone, Hakatai Shale, Shinumo Quartzite,
Dox Sandstone, and Cardenas Basalts 800 million to one-and-a-quarter billion
years old. They had been turned sideways and thrust up higher than the Rock-
ies by monumental tectonic movements. During this churning, twisting and
thrusting, even more ancient rocks were exposed in places: hardened magma of
Zoroaster Granites and the oldest rock in the Grand Canyon, the metamor-
phosed lava-black Vishnu Schist, 1.7 billion years old. O ver millions of years
these mountains were eroded until the space they once occupied was filled
with sandstone deposited by encroaching seas.
When we arrived at the Great Unconformity, we joked about what it meant
to arrive some place that isn't there. As we hiked on within the depths of the in-
ner canyon, we marveled at the dramatic transition from sand and gravei to
sculptured stone, its significance gradually penetrating with the cold feel of the
marble-like rock. Now, inspired by the memory of that geologic gap, I contem-
plated the chasm that exists between modern society and ancient times. Many
experience the gap as a painful loss. Lately, contemporary male elders have
been trying to fill in the gap, build a bridge back or at least make a connection.
They hope a return to ancient initiation rites will help close the gap. I had been
attracted by that possibility myself, but Brandon's reactions during our hike
down said it wouldn't work, at least not for modern young people who have
tasted choice, experienced the power of intellect, learned to value individuality
and abhor control. The Great Unconformity impressed on me the gap between
past and present when societal customs have been eroded to the point of van-
ishing. Our ancestral past will necessarily and inevitably remain a foundation,
like the ancient Vishnu Schist, formed by 75,000 pounds per square inch of tec-
tonic pressure and named for the Hindu god, the Preserver. The Tapeats forma-
tion now rises atop that preserved foundation, but is neither part of it nor con-
tinuous in time.
I imagined a contemporary coming of age journey that recognizes ancient
foundatons of human experience, but is separate and distinct in accordance
with modern discontinuities and the great unconformity of human potential in
our timesa coming of age process that does not require the societal equivalent
of 75,000 pounds per square inch of pressure to assure conformity. Indeed, a
coming of age process that does not even have conformity as its goal. That
would be the greatest unconformity.
In elucidating the role of traditional initiation for Brandon, Malcolm, my
longtime friend and Canyon guide, also an anthropologist and family thera-
pist, had explained that initiation rites functioned to psychologically separate
sons from parental domination in tribal societies with extended families where
generations would live together in a confined village space. But in modern soci-
ety, just the opposite is the case. Our children are separated from us by daycare,
schools, music, television, peer groups and easy geographic mobility. The chal-
lenge of contemporary times is not to provide for the physical and psychologi-
cal separation of children from parents. Society has evolved multiple mecha-
nisms for detachment. Parents and children today are subjected to
unprecedented centrifugai forces. The challenge now is to bond.
I thought I had come to the Grand Canyon for a ritual of initiationrecog-

nizing and celebrating Brandon's manhood. But as we had descended into the
inner Canyon, the focus shifted for me. There, in the moonlight, I admitted why.
He was leaving home and going off to college. We needed no ceremony to rec-
ognize his independence. It was not in doubt. Nor was his manhood. What I
craved, that ancient rituais could not provide and had not been designed to ar-
range, was connection.
Abruptly, propelled by the force of illusory insight, I turned again away from
the rapids toward Brandon and sleep. A piercing pain in my leg stopped me. I
had connected with a Prickly Pear cactus. Examining the offending thom, I
heard my voice say: "Reality-check." Suddenly self-conscious, I looked
around, then laughed out loud at the ridiculous figure I presented: pacing the
canyon floor dressed only in the ephemeral threads of an emerging sociological
paradigm shift.
SOURCE: Patton (1999a:21-27). Copyright 1999 by Michael Quinn Patton.

Particularly Appropriate
Qualitative Applications
; p p ^ e k v H c e s k i p in T - V a g m a + i s m
A young carp enter, at the beginning of his career, carne to Halcolm in distress.
He had studied diligently to master carpentry. At the completion of his appren-
ticeship, the master carpenters said that his technical competence and skill were
unmatched for one so young.
Halcolm knew ali this, for word of the young man's mastery had reached
even the great one. Yet, Halcolm could also see that the young carpenter was in
great distress. "What troubles you?" Halcolm asked gently.
"My parents, my townspeople, my master teachers have been most generous.
Upon completion of my apprenticeship, they joined together to give me a fine
set of tools. Ihave been trained by the best. I am told that my skills arewhat can
I say without being immodest?my skills are adequate." The young man
paused, his distress obvious and growing even as he spoke.
"Then what is the problem?" asked Halcolm. The young man looked down,
embarrassed in the presence of the great one. It was a long time before he spoke,
and then only in a whisper. "I have no thing to build."
"Ah, I see," said Halcolm.
"No one will give me any orders," continued the young man.
13. 143
170LEI.CONCEPTUAL ISSUES IN QUALITATIVE INQUIRY
"Let me make some inquiries," offered Halcolm. "Return in a week and IT1 tell
you what I have learned."
The seven days were as an agonizing eternity for the young man. At last it was
time to find out what Halcolm had learned through his discrete interviews with a
few knowledgeable and well-connected people.
"I have confirmed ali that you told me," Halcolm began. "Your skill is much re-
spected. Your tools are the finest, given with much affection. Your competence is
not in doubt. And yet, you have nothing to build."
The young man waited for Halcolm to continue.
"During your apprenticeship you did the latticework on the new cathedral.
The craftsmanship you displayed is admire d by ali. You designe d and constructed
the intricate woodwork of the new town hall directrixanother work of great art
admire d even by your masters. You carved and installed the elabora te wine racks
in the guardian's estate. In ali these efforts you have distinguished yourself and
pleased those for whom you built."
The young man was pleased but perplexed as he heard Halcolm affirm the
quality of his work. Indeed, hearing the affirmations deepened his distress atfind-
ing himself now with nothing to build. Halcolm continued.
"You now have nothing to build because the townspeople believe your artistry
and craftsmanship are far superior to their simple needs. They need simple chairs,
tables, and doors. You work on cathedrals, town halls, and estates. You have de-
signed objects of great beauty and complexity. You have not designed and built
objects of great simplicity and practicality. To do the latter only looks easier, but
takes no less skill.
"Build me a simple, functional, and practical bookcase at reasonable cost, and
let us see what the townspeople think. Apply your skills to the everyday needs of
the people and you shall not lack for work."
Halcolm expected the young man to be delighted at the prospect of a solution to
his problemand regular employment. Instead, he saw the distress deepen into
despair.
"I do not know how to build simple, practical, and functional things," lamented
the young man. "I have never applied my skills and my tools to such things."
"Then your apprenticeship is not over," said Halcolm.
And they went down to Halcolm's workshop where the young man began to
learn anew.
From Halcolm's Applied Arts and Sciences

Particular!}/ Appropriate Qualitative Applications j. 145
J. Practical Purposes and that this review, while including a great vari-
Concrete Questions ety of applications, is far from exhaustive.
My purpose is to expand the horizons of
Qualitative methods are first and foremost what is possible and appropriate for both
research methods. They are ways of finding practitioners and decision makers. Because
out whatpeople do, know, think, and feelby the opportunities for qualitative inquiry are
observing, interviewing, and analyzing so vast, the examples offered here can be no
documents. The last chapter reviewed how more than teasers, merely hinting at the
qualitative methods contribute to generat- enormous array of qualitative applications
ing and confirming social science theory that are possible.
This chapter reviews how qualitative meth- Moreover, I have not examined in this
ods can contribute to useful evaluation, prac- chapter how the varying theoretical and
tical problem solving, real-world decision paradigmatic approaches discussed in the
making, action research, policy analysis, previous chapter might affect inquiry into
and organizational or community develop- any of these practical issues. For example,
ment. This chapter offers examples of how one might examine program quality, the first
qualitative methods can help answer con- topic below, phenomenologically, ethno-
crete questions, support development, and graphically, or heuristicallyto cite but
improve programs. Ali of these are ways of three possibilities. Or one might simply con-
contributing to what is sometimes called duct interviews and gather observa tion data
"action science" (Argyris, Putnam, and to answer concrete program and organiza-
Smith 1985). tional questions without working explicitly
Qualitative methods are not appropriate with a particular theoretical, paradigmatic,
for every inquiry situation. The aim of this or philosophical perspective. Well-trained
chapter is to illustrate when it may be partic- and thoughtful interviewers can get mean-
ularly appropriate to use qualitative meth- ingful answers to practical questions with-
ods. Certain purposes, questions, problems, out making a paradigmatic or philosophical
and situations are more consonant with pledge of allegiance. Pragmatic and utilitar-
qualitative methods than others. This chap- ian frameworks can guide qualitative in-
ter samples some of the research and evalua- quiry on their practical and applied under-
tion questions for which qualitative inquiry pinnings without having to be attached to or
strategies are especially appropriate and derived from a theoretical tradition. Prag-
powerful. The actual and potential applica- matism, then, is the foundational orienta-
tions of qualitative methods are so diverse tion of this chapter.
A Focus on Quality
^ ^ o m e , give us a taste of your quality.
William Shakespeare, Hamlet, Act II, scene ii
Quality care. Quality education. Quality agement. Continuous quality improvement.

parenting. Quality time. Total quality man- Quality control. Quality assurance. Malcolm
Baldridge National Quality Award. Quality quality movementW. Edwards Deming

is the watchword of our times. People in and Joseph M. Juranwere preaching qual-
"knowle dge-intensive societies . . . prefer ity in manufacturing before World War II. In
'better' to 'more'" (Cleveland 1989:157). the 1930s, for example, Juran was applying
More requires quantitative dimensions; concepts of empowered worker teams and
better evokes qualitative criteria. continuous quality improvement to reduce
At the most fundamental levei, the de- defects at Western Electric's Hawthorne
bates about abortion, on the one hand, and Works in Chicago (Deutsch 1998; Juran
death with dignity and physician-assisted 1951). Deming and his disciples have long
suicide, on the other, concern, in part, what viewed quality from the customer's per-
is meant by "quality of life" (Humphrey spective, defining quality as meeting or ex-
1991). Kenneth E. Boulding (1985), one of the ceeding customer expectations. More than
most prominent intellectuals of the modern 20 years ago, Philip B. Crosby (1979) wrote a
era, devoted a book to the grand and grandi- best-selling book on "the art of making qual-
ose topic of human betterment. In that book, ity certain." Some of his assertions have be-
he defined development as "the learning of come classic:
quality" (Chapter 8). He struggled, ulti-
ma tely in vain, to define "quality of life." He o The first struggle, and it is never over, is
found the ide a beyond determina tive expli- to overcome the "conventional wisdom"
cation and certainly beyond numerical mea- regarding quality (p. 7).
surement. It is a subject particularly well
o The cost of quality is the expense of do-
suited for in-depth, holistic qualitative in-
ing things wrong (p. 11).
quiry.
Concern for quality surrounds us in the H Quality is ballet, not hockey (p. 13).
postmodern age. Quality has become the H The problem of quality management is
primary marketing theme of our time, for not what people don't know about. The
example, "Quality Is Job One" (Ford Motor problem is what they think they do know
advertising slogan). Customers demand (p. 13).
quality. This may stem, at least in part, from
H Quality management is a systematic way
the fact that in the busy lives we now lead, at
of guaranteeing that organized activities
least in postindustrial society, we simply
happen the way they are planned. . . .
don't have time for things to break down.
Quality management is needed because
We don't have time to wait for repairs. We
nothing is simple anymore, if indeed it
can't afford the lost productivity of not hav-
ever was (p. 19).
ing things work (either products or pro-
grams). We have taken to heart the admoni-
tion that, in the long rim, it is cheaper do it Efforts to implement these and other
right the first timewhatever "it" or "right" principies swelled crescendo-like into a
may refer to. It is within this larger societal national movement as management con-
context that we shall examine what qualita- sultants everywhere sang of total quality
tive inquiry brings to the challenge of study- management and continuous quality im-
ing and evaluating quality. provement. The music began in the corpo-
The current mania spotlighting quality rate sector, but by the early 1990s the "cult of
can give the impression that this is a rela- total quality" had permeated deeply into the
tively recent concern, but the founders of the government and nonprofit sectors (Walters
1992). The Malcolm Baldridge National school's documented history of high-quality

Quality Awards became, and remain, the education.
pinnacle of recognition that the mountain- Understanding what people value and
top of quality has been reached. the meanings they attach to experiences,
Nor was concern about quality limited to from their own personal and cultural per-
management books. Robert Pirsig's classic spectives, are major inquiry arenas for quali-
Zen and the Art of Motorcycle Maintenance tative inquiry. This is especially true when
(1984) was an investigation into quality and making judgments about quality, or valuing,
excellence, themes he revisited in Lila (1991) which brings us to evaluation and quality
ashe explored the "metaphysics of quality/' assurance.
What the Metaphysics of Quality adds to Quality Assurance

James' pragmatism and his radical empiricism is and Program Evaluation
the idea that the primai reality from which
subjects and objects spring is value. By doing
Program evaluation and quality assur-
so it seems to unite pragmatism and radical
ance have developed as separate functions
empiricism into a single fabric. Value, the
with distinct purposes, largely separate lit-
pragmatic test of truth, is also the primary em-
eratures, different practitioners, and vary-
pirical experience. The metaphysics of quality
ing jargon. Each began quite narrowly, but
says pure experience is value. Experience
each has broadened its scope, purposes,
which is not valued is not experienced. The
methods, and applications to the pomt
two are the same. This is where value fits.
where there is a great deal of overlap and,
Value is not at the tail-end of a series of super-
most important, both functions can now be
ficial scientific deductions that puts it some-
built on a single, comprehensive program
where in a mysterious undetermined loca tion
information system.
in the cortex of the brain. Value is at the very
front of the empirical procession. (Pirsig
Program evaluation traces its modem be-
1991:365) ginnings to the educational testing work of
Thorndike and colleagues in the early 1900s.
Program evaluation was originally focused
An important policy question of our time on measuring attainment of goals and objec-
is whether educational quality can be tives, that is, finding out if a program
achieved through state-imposed standards "works," that is, if it's effective. This came to
and mandates. One example of the differ- be called summative evaluation, which origi-
ence between theory and practice in this re- nally relied heavily on experimental designs
gard is captured astutely in a case study by and quantitative measurement of outcomes.
Goodson and Foote (2001) that describes In recent years, program improvement (for-
what happened when a highly successful, mative) evaluation has become at least as
innovative, and creative alternative to tradi- important and pervasive as summative
tional education was confronted by the de- evaluation (Patton 1997a).
mands of contemporary standardized ac- Quality assurance (QA) in the United
countability. They chronicle the resistance of States had its official birth with the passage
a particular school, the Durant School, to the of the Community Mental Health Act
imposition of state standards and mandated Amendments of 1975 (Public Law 94-63).
tests in an effort, ironically, to maintain the This law required federally funded commu-
Important methods of QA include clinicai

case investigations and peer reviews. Ali
mM:m T A
cases that fail to meet certain standards are
PFRWIVf rriNC- :;;'v
reviewed in depth and detail. For example,
"Tii ! ;'"i!Vi:! ir i:!;M:=i;!; iT!.! lH,iT,rilji . i1! i patients who remain hospitalized beyond
! ..-ii i:ij-i frriiiVi-W sii !j/jftiJi Ys- ;i l l ^ j f ;ii|-T! . an accepted or expected period may trigger
j. i^^riii:'!5!!!.!!, "liri.1!1! ; a review. An original difference between
:. , iT:jivi|i MÎV:! | : ; ' r i t ^ f ; . i' i! li !;'i!'i; jHrtftj . :
;
program evaluation and QA was that QA fo-
: . . : r Y i ! ' i ; i . !:h!i iiii h : i ; i ! ! ^ ' j ^ f e i i f l I. !V! .!<h:i i.j;
: ;
:
:
; !
: cused on individuais on a case-by-case basis,
M ftfM -M. i i - j f o l i i ^ ^: i . ! V y . V r l . ; while program evaluation focused on the
âWd, }d|l;i;vi:i]:iii:!Jfcri:^^îrifii'!!|!:< overall ! program. The traditional concems
with unique individual cases and quality in
ftalAY&rttl11:!.^^ Wtf^^ : QA systems continue to be quite consonant
!:iT 'rn-!Yii' :ti"l'r! :!iJi:iIM-PI!-'! r u
!
: \ with qualitative methods. Moreover, QA ef-
H*!'. !:;i:i- ji;!V: m & i . forts have now moved beyond health care to
V :-,' Vji^l rh!i-i:;!: ii!i the full spectrum of human service pro-
grams and government services (Human
rnriptu-ktA$ r ; j ; ' { f j & ^ t . :,v; Services Research Institute 1984).
jinap-nsT!!! ^ f g f l , S ^ ^ f l i : In-depth reviews of the quality of care for
participants m programs can draw heavily
on clinicai case files only if the files contain
approprate and valid information. When
nity mental health centers to have QA efforts files are to be used for research and evalua-
with utilization and peer review systems. Its tion purposes, clinicians need special train-
purpose was to assure funders, including ining and support in how to gather and report
surers and consumers, thatestablished stan- highly descriptive qualitative data in clinicai
dards of care were being provided. QA sys- case files (Cox 1982). Because there can be
tems involve data collection and evaluation great variation in the quality of clinicai case
procedures to document and support the records, particularly the descriptive quality,
promise made by health and mental health a program of QA must include evaluation of
care providers to funding sources, including and attention to the quality of qualitative
third-party insurance carriers and consum- data available for QA purposes.
ers, "that certain standards of excellence are Moreover, it is useful to distinguish qual-
being met. It usually involves measuring the ity control from quality enhancement. Qual-
quality of care given to individual clients in ity control efforts identify and measure
order to improve the appropriateness, ade- minimum acceptable results, for example,
quacy, and effectiveness of care" (Lalonde minimum competency testing in schools or
1982:352-53). As QA systems have devel- maximum acceptable waiting times before
oped, special emphasis has been placed on seeing a physician in an emergency room.
detecting problems, correcting deficiencies, Quality enhancement, in contrast, focuses
reducing errors, and protecting individual on excellence, that is, leveis of attainment
patients. In addition, QA aims to control well beyond minimums. Quality control re-
costs of health care by preventing over- quires clear, specific, standardized, and
utilization of services and overbilling by measurable leveis of acceptable results. Ex-
providers. cellence, however, often involves individu-
B38njlsl''iW Comparing Program Evaluation and Quality Assurance
Progrom Evaluation Quality Assuronce
1. Focus on program processes and 1. Focus on individual processes and

outcomes outcomes
2. Aggregate data 2. Individual clinicai cases
3. Goals-based judgment 3. Professional-based judgment
4. intended for decision makers 4. Intended for clinicai staff
alization and professional judgment that control, but attention has shifted to concern
cannot and should not be standardized. Ex- for quality enhancement. Program evalua-
cellence is manifest in quality responses to tion began with an emphasis on summative
special cases or especially challenging cir- judgments about whether a program was ef-
cumstances. Thus, while quality control re- fective or not but has shifted to improving
lies on standardized statistical measures, program effectiveness. In their shared con-
comparisons, and benchmarks, quality en- cern for gathering useful information to sup-
hancement relies more on nuances of judg- port program improvement, program eval-
ment that are often bst captured qualita- uation and QA now overlap and find
tively through case studies and cross-case common ground. Accountability demands
comparisons. can be well served, in part, by evidence that
Traditionally, given their different ori- programs are improving.
gins, program evaluation and QA have had Both accountability and program im-
different emphases. These differences are provement require comprehensive program
summarized in Exhibit 4.1. information systems. We've learned that
The distinctions between QA and pro- such systems should be designed with the
gram evaluation have lost much of their im- direct involvement of intended users; that
portance as both functions have expanded. information systems should be focused on
Program evaluation has come to pay much criticai success factors (not data on every
more attention to program processes, imple- variable a software expert can dream up);
mentation issues, and qualitative data. QA that systems should be streamlined with
has come to pay much more attention to out- utility in mind; and that program improve-
comes, aggregate data, and cumulative in- ment systems benefit from both qualitative
formation over time. What has driven the and quantitative information, both case data
developments in both program evaluation and aggregate data. Indeed, harking back to
and QAand what now makes them more the opening discussion about total quality
similar than differentis concern about pro- management and continuous quality im-
gram improvement and gathering really provement, the systems that support such
useful information. Both functions had their efforts began with a heavy emphasis on sta-
origins in demands for accountability. QA tistical process control and "objective" indi-
began with a heavy emphasis on quality cators of performance, but have come in-
creasingly to value qualitative perspectives on focuses on what spelling means to the stu-
quality. It turns out that one of the particular dent. How is spelling integrated into the
strengths of qualitative inquiry, perhaps student's approach to writing? How does
commonsensically, is illuminating the na- the student think about spelling, approach
ture and meaning of quality in particular spelling, feel about spelling? The answer to
contexts. This takes on added significance such questions requires description of indi-
since "quality of life has become a com- vidual students' perspectives and situations
monly used concept and is showing grow- such that the meaning of the experience for
ing significance in economic and political the students is elucidated.
terms . . . [and] has two aspects, psychologi- The same distinction holds with regard to
cal and environmental, [yet] some research- programs that emphasize deinstitution-
ers have totally neglected the perception of alizationfor example, community mental
the people" (Turksever and Atalik 2001:163). health programs, community corrections,
There are many aspects of program oper- and community-based programs for the el-
ations, includmg implementation activities derly. It is possible to count the number of
and client outcomes, that can be measured in people placed in the community. It is possi-
terms of relative quantity. It makes sense to ble even to measure on standardized scales
count the number of people who enter a pro- certain attributes of their lives and liveli-
gram, the number who leave the program, hoods. It is possible to have them subjec-
and the number who receive or report some tively rate various aspects and dimensions
concrete benefit from the program. How- of quality of life. However, to fully grasp the
ever, many attributes of programs do not meaning of a change in life for particular
lend themselves to counting. Even the quan- persons it is necessary to develop a descrip-
titative scaling of quality attributes is an in- tion of life quality that integrates interde-
adequate way of capturing either program pendent dimensions of quality into a whole
quality or the effect of a program on the that is placed in context: What is their daily
quality of life experienced by participants life like? Who do they interact with? How do
during and after the program. they perceive their lives? How do they make
For example, school outcomes can be sense of what they experience? What do they
looked at both in terms of quantity of change say about the path they are on? How do they
and quality of change. Quantity of change talk about their quality of life? What do they
may involve the number of books read; a compare themselves to when deciding how
score on a standardized achievement test; well they're doing? These are areas of quali-
the number of words spelled correctly; and tative inquiry that support quality enhance-
the number of interactions with other stu- ment efforts and insights.
dents, the teacher, or people of a different Quality has to do with nuance, with de-
race. Each of these outcomes has a corre- tail, and with the subtle and unique things
sponding quality dimension that requires that make a difference between the points on
description rather than scaling. Thus, to find a standardized scale. In-depth quality de-
out what it means to a student to have read a scriptions can illuminate what the lives and
certain number of books is an issue of qual- perspectives of two different people are like,
ity. How those books affected the student one of whom responded on a scale of 5
personally and intellectually is a question of pomts that an experience was "highly satis-
quality. In contrast to counting the correct factory," while the other responded that it
number of words spelled, the quality issue was an "extremely satisfying" experience.
This is not a question of interval versus ordi- ability-driven evaluation. The accountabil-
nal scaling, but one of meanings. What do ity movement is not so much about achiev-
programs mean to participants? What is the ing quality (the previous section) as it is
quality of their experiences? Answers to about demonstrating responsible use of
such questions require detailed, in-depth, public funds to achieve politically desired
and holistic descriptions that represent peo- results. The U.S. Government Performance
ple in their own terms and that get close and Results Act (GPRA) of 1994 mandates
enough to the situation being studied to un- outcomes reporting by government agen-
derstand firsthand the nuances of quality. cies. Philanthropic foundations are de-
The failure to find statistically significant manding outcomes evaluation (Porter and
differences in comparing people on some Kramer 1999) as are health care systems
outcome measure does not mean that there (Morse, Penrod, and Hupcey 2000). Indeed,
are no important differences among those in every arena of actionhealth, education,
people on those outcomes. The differences criminal justice, employment, mtemational
may simply be qualitative rather than quan- developmentemphasis has shifted from
titative. A carpenter is reported to have ex- providing services to attaining priority out-
plained this point to William James. The car- comes. A good example of this emphasis is
penter, having worked for many different the widely used United Way (1996) manual
people, observed, "There is very little differ- Measuring Program Outcomes:
ence between one man and another; but
what little there is, is very important." Those In growing numbers, service providers, gov-
differences are differences of quality. ernments, other funders, and the public are
calling for clearer evidence that the resources
they expanded actually produce benefits for
Evaluation Applications people. Consumers of services and volunteers
who provide services want to know that pro-
Outcomes Evaluation grams to which they devote their time really
make a difference. That is, they want better ac-
For programs engaged in healing, transforma- countability for the use of resources. One clear
tion, and prevention, the best source and form and compelling answer to the question of
of information are client stories. It is through "Why measure outcomes?" is: To see if pro-
these stories that we discover how program grams really make a difference in the lives of
staff interact with clients, with other service people. (p. 4)
providers, and with family and friends of their
clients to contribute to outcomes, and how the However, reading this manual one would
clients, themselves, grow and change in rethink that the only way to docurnent out-
sponse to program inputs and other forces and comes attainment is with numbers. The fo-
factors in their lives. There is a ricliness here cus is entirely on numerical indicators of
that numbers alone cannot capture. It is only outcomes and statistics in accomplishments.
for a story not worth telling, due to its inherent Percentage increases in desired outcomes
simplicity, that numbers will suffice. (Kibel (e.g., higher achievement test scores) and
1999:13) percentage decreases in undesirable out-
comes (e.g., reductions in rates of child
Outcomes evaluation has become a cen- abuse and neglect) are important to provide
tral focus, if not the central focus, of account- concrete evidence of overall patterns of ef-
fectiveness. What such statistics cannot do, behind the numbers, capture unintended
however, is show the human faces behind impacts and ripple effects, and illuminate
the numbers. This is important to provide dimensions of desired outcomes that are dif-
criticai context when interpreting statistical ficult to quantify (e.g., what it means for
outcomes as well as to make sure that the someone to become "self-sufficient"). Such
numbers can be understood as representing qualitative data can add significantly to sta-
meaningful changes in the lives of real peo- tistical reporting to create a more compre-
ple. hensive accountability system.
In an adult literacy program, the test re- Detailed case studies can be even more
sults showed an average increase of 2.7 important when evaluating outcomes at-
grade leveis over a three-month period. The tainment for program improvement (as op-
people in this sample included posed to externai accountability reporting).
To simply know that a targeted indicator has
a Puerto Rican man who was learning to been met (or not met) provides little infor-
read English so that he could help his ma tion for program improvement. Getting
young daughter with schoolwork; into case details better illuminates what
worked and didn't work along the journey
an 87-year-old African American grand-
to outcomesthe kind of understanding a
mother who, having worked hard
program needs to undertake improvement
throughout her life to make sure that her
initiatives.
children and grandchildren completed
Exhibit 4.2 (p. 155) presents highlights of
school, was now attending to her own
a case study from an employment training
education so that she could read the
program. In addition to illuminating what
Bible herself; and
the outcome of a "job placement" actually
a manager in a local corporation who meant to a particular participant, the case
years earlier had lied on her job applica- documents attainment of hard-to-measure
tion about having a high school diploma outcomes such as "understanding the
and was now studying at night to attain a American workplace culture" and "speak-
general equivalency diploma (GED). ing up for oneself" that can be criticai to
long-term job success for an emigrant like
the woman in the story.
In judging the effectiveness of this pro-
We'11 return to the theme of documenting
gram and making decisions about its future,
outcomes through stories near the end of
it can be as important to understand the sto-
this chapter in the Capturing and Communi-
ries behind the numbers as to have the statis-
cating Stories section. The next section, how-
tics themselves. One can justifiably criticize
ever, looks at the special capacity of qualita-
the past reporting practices of many human
tive inquiry to document individualized
service agencies for having been limited to
outcomes in programs that especially value
successful anecdotes with no accountability
individualization.
reporting on overall patterns of the effective-
ness. However, swinging the pendulum to
the other extreme of only reporting aggre- Evaluating Individualized
gate statistics presents its own problems and Outcomes
limitations. Numbers are subject to selection
and distortion no less than anecdotes. Individualization means matching pro-
Well-crafted case studies can tell the stories gram services and treatments to the needs of
Particularhj Appropriate Qualitative Applications lj. 153
o o r c o t f e MARFINS AND rhi;; o u m j T j

IMPACT 0LUJ5
:
: " :!;. ;; i!! {:!: !.! ! i':i i! i!i !!!:: $ i: ! :i ' i ! :>:! !. =-T , .::'i,-.1 i -r - I 'II ! I 7: U ... |.:: !'! = ! ' T 1 = " ! VI KTTi
;V ! !-: :*i! \! .vikw;o!K;!!:!i !.'::!=::i-:;\MV"" I:,"VI:'ALI! i' 'I I" I 'I J !.VI I
il^lVirrY! !>:-=.: Co:M j. : 'ir* s-11 - -> : .".!' 1;V. !' :-!:!;V'V Vi
i ;; r ,vni- s V;:i Y! WttQ !! ; m i " ! ! I I - - . ' ! ! ! . - I . ' . , 5 ; V : : I V - : ' " ; : ^
! V R I > T I
; I: !". " : ! V| :VL I V ,=" I I.VI . 1 F LV-I; J I'-:I . F , I; V (I U i i vi i" i .'"! 11" i. i!' i! !i i'<" i!! '"'" " ; !':'-:Vm n^ :,:., -
! 1 |- 1
.;i !::;. !: !"! f.\ :=.! ,!i ,tv ! i!- KQ '! i:: !i O .,:." ' : ! ,"" -. ' ' I ! I ' . ' ! J / . ' I . ' i I'O!:."'' N I > . " H ' V
jr;vi:;! n''lirn m u ' I.; k\ n\i h" l5 hW""
! v:;.". :.v!' -:::!/' !' = - >J!^ :ivivj;vinl'1'
' VR !:!::" j,V:i r-: vi!,:. !
- : Vi II ! iti- [il Ti >'V\
! ' T I ' ! ' ! ! ! ' . I ' L " P "
! I I ' I ! '
!!<, I I : !'C:!
!::!.-. vS!; l i':." "" 11! :>!.i' /'I ='!':l li1 i/!'';!}!!' i'i!,.!'!"!i i-1ri i"vi " "ii"i! ]. i".;i:- ':i!!'.'i.;r!."!vi'i i'i'i
1 1 !
=" s.'!:: M N V Y ; : : -= - . : : ::V I; .'VIIY

5
: !:: - " - ,'i i',i ii.',f !".'.j i.'Vi; y\IVI"i.i;ri'!V|'
is::.-.: ;n;:,;v;.::';:r !';:ir\ ,:vT ::v.:ri\rfn'*?-KTG!!.!: |ri."!,I ki: Vilu!r-i iiYli,;Yiji jnri<'rirrijIV:!.
.V ! | |.;i |*.s.*i":i IVi I;- :;;: ir;! vi" 'I ::.;/T!:V! ; ;'r!C:;.!:;!! -'*i:ri/ii hivij * 1 !::.';n::.!;:;
: 5 ! : : !;VV! r:1.:r"
i ii- :ft!j:I !h o v !'I:T;:::V vfn I i >'" ^ \ I" W W- - l.'!!/'M'.'.! V, !ii.!,il'h!i; I'! '!'i1"!.' i|11''11' 1
5 â:-a:-a:=='s--:i !"!:::i i.Yi1:!'; ! ri1.11':.i." M. !' 'p >'i''0!i: I.

i!i ! ' !'" ! i ' ' !! ! "''T" i.'ir:r .Vi i i=yli iVi ll i: :;V;:
! 5 : Vi|i" rVr.i !i'!i1 1 .'!i.'. !|,-:>ri1 iV111'1 /1!l'1!1'!'i iMi'''"
1
:; :; I V : " : ! I:S I. 'H.: .:.I:

=
S ! !:I.-:" V;-. I^VI:::; !';:-I:J: >. \ v ' -
: 1
M
lIl IMMIMMMMIMMMMMMMMMMMMMBSSSSSSSSSiSSiSSSii
m
irei!" i'=:=.'"hi-iVii;Vj;^"ri:" |':V !;:< h i i i - I I I I Y S : . ' ; I : I Ii:. :'i1'i.vii.irv, U.1 L| ! !.Vi i'l 'Oii lM>r.i
:
! rlij'
O !TR."P-FI :i : . : V : V I : " * I T V I T I > >i .;::= vi-LH/iVI-:! i'i| !::i'CO =:!.:-> i: :>'i I ' I: 'i n.vi i' s.-i I.V.Vi , !< l l. l!Vj i!'!! ",1 il' 1 < i -
II!!!=!.:; LI" ! T V ; ! ! I N V I " I = U !!:.= S wwqvtt ,::;I::'.1 .V,1!j=- .r:-i il 'liH-i IVili' J-iv!''V'
X.- Liivii,:: I/,":! :>:'.!:!.! S T T O !=:I TI 1 IVTI
=;.i iUm !"i i:y-.'ri A V I ; ; " i; I . ' , I i vi iri:i..! i!i i 1.1:1;!. i !:i;i ] i!
!
i5:: !::-!::=!:!!!=:;! jn VZ wvwr =! == :=: t-!:! ;; . = :i w ;;v -i i rj Ir'. 111,1.VI.V/11 Ij - i i il f L! !! i'! li

; - .nsiilkipy V-; ik =: -ri s1 i; = '. V rp ' i - / 1 l-!V I I ! , ' 11! i! m l li' TI I" iVi l!
!!:!!! !.!.!:!.';,!V!|;!. =!Vj!==D;m:!.!i.i,:ii!='i:vi ! IVII::' !':: iV/i -i. I'I I-VIAIIVI:','! Ailmi!'
!
.VIi ! i:i:i'.:; === :=:Ii,VI:; m ^ ^ | ! i i : : h::,! ir =,=;=,rdSi. l:.:.!,.-! vi ; l':!i'JiVIlJ.|'g

.VI - ! ! . . . ! :'.:I !'I I t f a ';!, -: r i ' R V - ; I : v t : ! : " 1 h ? r i ^ w - I.-;I: , I ; I / I V I ! : : = : > = 13:= N : , R ; : . ! - , I R IF , ! IL ,', ! ;I-
i"i ! I" i !i iii\ ^ -I-s^ l J D U=! =!i ! != =" :; ! :V:.VI ! V ! I,- Tu't. 1=1=: Vi u.rrtiirthrriV -- --1 ! V i^Mh:'
m i.v::ii:vii.;ii:i!Vi:i,:i- :':'.ii.i\.:: Vi'!i:!':! s"ri-ni;:! i.'= 11.1. !=: : !:i ;:-':;. hu i" ivi i n -n .tivv r
' }, i" 11< n r:;
IVI : I ':V.' i-ii I. ! i i i i I : = | : : O! >\\FFTL I^'!: !! !: I!"''!" .W.t .'ii:.:;!":; u : : : :
iVi:0 li.! s/iViV^V:!:,- !! = :! IVVI.: ^ ^S!in'rîh>.
" !" 1p;.i 1; :"i I.i! I. L. 11 ! ti D A 11M 1'I ^ TI I -! i=

: =a:-=-:I IJ!
.I j=
: .!, !v=:i. .31.-;:! rv fvi =.I.!=: I I m
! \I ! - i -u n :-I I-.I I
i=-i !.T! =!
:
: ! 1" iî =:=i ^'ilT! ] ! :=. H1 :; ===
! ! ir, r ri/i dp
!!!.: :;:'.;..! I rldi :: ! i' ! 1= Vd "1
ir rvrinsiiTi:i:I :i"II-T:.
iiii m.!:; |yo!:! ;; ::i:!'
!.'! =.:""!.;: rl: -; I! =!
M:-iv. Vi i=' = ! II li :'. I!:r i-;;:-!.
iis: ;=.! wpert*
!!n i h:-:!. E !.;:> vr :!= ;=;:.!
(continued)
=
: !.::!: i 1 birrl !'!:! IviVj." ::=!: !: I !'!'! ii ;"i
. " I. j :j r ' - : !-:->> :.:
'!="-=5 = = i.j- \> == :! =
' :: !=|
: !: ! !' : i: ! !>1 !' ::.-r i. li i
; ! : : M M N T !.W=! M ;:;:= llllllllllllJIIIIIIIIIJSlIIJIII
i:. I! , =:":":- .Vi:' ii! i = ===
.=.! i.=! = :.: v:ii!::-s \-r I"-.! Ldlsn.
V:'-:,- !' " ij i'-: " Ij iV ."i:-":" ;!' =.! i r n:.'"
.âc jv!:0= i: i'k=
' !!::! j . ! = 1 i! =ii-
'rttnrl l::, ;,!>. !ih,il !!!:!.-; i" :j
j- i- -j i i. j:-,.-:i. ! ; i :,!;. i !.:!=:;! !ii=y !i=vi
rim ""i: lri li" . ==
! : :;!:=: i!!!'"!!' ::.;!: .=.: -. s
. !v ! il:- >
: S :: :-. !!"!:! v ! :! -V ! I :' : : .-'I
\Y S j":1; I !': ;:! "^y . i ' i ! ! ::.=!' ':*!; " " ! vi.
Aud ! i:! !:'i V-0 ftN" =:!ii =

!:V::..I -.! =J::.=! :!,--::-.-!: = :.: I :I v i ! :I
lj ;.i; A : I ! " !rr j ::i" i" i 'v I" i i-:ri":i' i !i ! h:= |i' 'i I j " 1
'flil-Hul "V:!I !;! =! 11'::!;!'. = =.=" :! :' j.'!:.j.i"- .i..
1S
!! "IE*:*E:"5 ':S:V 'i:: i h - :i <:> . :
M I:-:; I IVJ: I l-:j;i:;i|; ' ij l" ii '
::!':! ::!. !!!:::: ar i-i .:.! U ;! '.i' i,' \
i' i :i'i:'i.':i"! . V .1 i i ' 1 : ! ' il "-I
"o-.! &arfr \>k ':=:=! = !: Ij: l ; I i,'.

.JHJ! ;:Ij.: i.:. !,ii::: 11: :==:-!.=!i Iln i":i ='!. ii '!' '.'li" 1.1

! " r :! : ri" :! .i Y ;i ; i :! -:j :: i"i:'"i:i * IY i ' <" r r r : r ' ! !
"Ov ;i-!!. r. .;!:!. i i.Y jji;! i:|B lirh.y; VHj;; >y-, K i- 'h':- ! :'
I J I T I 'MSV;! :! =: .L""i="I! !v. " , :-. I I '.!= :I V : H JW I " ISI":
individual clients. Successful social and ed- staff are justifiably reluctant to generate
ucational programs adapt their interven- standardized criteria and scales against
tions to the needs and circumstances of spe- which ali clients are compared. They argue
cific individuais and families (Schorr that their evaluation needs are for documen-
1988:257). Flexibility, adaptability, and indi- tation of the unique outcomes of individual
vidualization can be important to the effec- clients rather than for measures of outcomes
tiveness of educational and human service standardized across ali clients.
programs. Highly individualized programs There are numerous examples of indi-
operate under the assumption that out- vidualized programs or treatments. Open
comes will be different for different clients. education, for example, is partly a model
Not only will outcomes vary along specific of educational processes that assumes that
common dimensions, but outcomes will be the outcomes of education for each child
qualitatively different and will involve qual- are unique. Open and experiential ap-
itatively different dimensions for different proaches to education offer diverse activi-
clients. Under such conditions, program ties to achieve diverse and individualized
Particularhj Appropriate Qualitative Applications lj. 155
Behind the Numbers of an Employment Program:

The Story of Li
Outcome Statistics
Completed WORK program, stayed in job placement one year.
Highest wage before WORK program, $8.25 without benefits; wage following WORK
program completion, $11.75 with benefits, more than a 42% increase.
Graduated from technical school with a 3.66 GPA (out of 4.0)
TABE test (math and language skills): reading, 7th grade equivaient; language,
4th grade; spelling, 10.6; math computation, 12.9highest score possible, and applied
math, 9.9. Average increase: 5.4 grade leveis.
Participation data: attended 89 classes at WORK, missed 6 and was late to 1.
Outcome Story
Li entered the employment program called deserted her. She found a cleaning job at a
"WORK" two years after she arrived in the bakery and iater at a hotel. Li had always be-
United States from Vietn m. As a recentimmi- lieved that through education she would get
grant, Li faced language and cultural barriers ahead, so she again took up the study of ac-
at school, at work, and in her day-to-day living. counting, begun in Vietnam, by enrolling in a
She was originaUy from Saigon City, where she local technical coliege four months after arriv-
tookthe nationaltestto study atthe university ing in the United States. After two years in the
but failed both times she took it. She gained United States, she found a data entry job for a
entrance to the Vietnmese Technical Coliege, retail business where she received $8.25 per
where she completed a degree in payroll and hour without benefits. She ost that job when
human resources when she was 23. She soon she had to stay home for a week to care for her
went to work for a Vietnamese company as an very sick daughter. She then went on welfare.
accountant. Li first heard about the employment pro-
Li married a man from North Vietnam de- gram, WORK, from some friends at the techni-
spite her family's opposition. Li and her hus- cal coliege. She entered and concentrated on
band were not accepted in either of their par- improving her English, technical skills, and as-
ents' homes, even though tradition calls for sertiveness with support from program staff.
the married couple to live in the home of the WORK provided tuition assistance, cost-of-
husband's parents. As a result, the couple living support, and bus passes. She also re-
rented a small room where they struggled to ceived tutoring in accounting (1.5 hours per
improve their lives. Shortly after the birth of week) from the program's accountant. The
their daughter, the couple discovered thatthey program secured for her a work experience
might be eligible to go to the United States be- placement ata local bankata rate of $7.25 per
cause the U.S. government was granting visas hour. This work environment proved very
to former officers of the Vietnamese army. Af- stressful because her language skills were in-
ter successfuliy negotiating the difficultappli- adequate and she was teased and harassed by
cation and emigration process, they arrived in other workers.
Minnesota where they knew no one. Accord- The WORK program supported her leaving
ing to Li, her husband began treating her very the bank job so that she could more intensively
badly. He immediately got a job but would not study business English, refine her workplace
give her any financial support Within a year he communications skills, and enhance her ac-
(continued)
m#4nmm*M contmued
counting software skills. The program case re- found it important to learn interviewing
cords show that at times she expressed feel- skills, since the interview is not common in
ings of hopelessness. Her staff worker Vietnam. She emphasized that WORK played
counseied her to focus on ali the barriers that a criticai role in helping her find a profes-
she had overcome by moving to this country sional job in "American corporate culture,"
and her hopes for a positive future for her explaining, "I compare myself with my friends
daughter. The program supported her return to at the Tech College and even now they don't
the technical college where she graduated have a job. If I didn't have WORK, I probably
with dual diplomas n accounting and data en- wouldn't have a job either." The program staff
try. Shortly thereafter, she began interviewing emphasized the challenges Li faced in gain-
for jobs. She went through more than seven ining confidence and learning how to speak up
terviews with different companies before she for herself, issues that the staff continued to
received a final job offerof an accounting posi- work on with her after her job placement.
tion at a retail firm. She said that she had ex- Li believes that WORK played a vital role in
pected more help with placement from the supporting her to overcome the challenges in
program than she got. According to WORK her life. She said, "I can talk to staff about any
staff, there was some confusion about their problem-about my job, money, and my
role in finding a placement for participants. At daughter. I talk to them to figure things out
any rate, Li persevered and felt good about the and solve problems. Even now that I have a
result: "Even though I failed many interviews job, I call and get help from staff." She re-
and I thought I might never get a job, ali the counted getting help dealing with a fellow
staff encouraged me to keep trying. They worker who was making life difficult for her.
talked to me about my good qualities. They Program staff talked her through the process
were ali positive, not negative. In the end, they of discussing the situation with her supervi-
really helped me get a good job even though I sor and getting help to resolve the situation.
didn't understand the mitsof their role at the Li said, "Even though I wasn't successful at
time." home with my husband, I feel like WORK is
While her language and math tests showed my family now. It makes people feel safe here.
great improvement (from elementary-ievel re- The staff encourages us to go forward. If
sults to high school-level results over the there is a problem, they help us solve it. The
two-year period in WORK), Li says that what important thing is that we treat each other
was most important was what she learned at like a family. I iearned so many things here.
WORK about how to be professional in the Things I can't get n school. I learned inter-
workplace and how "to mesh with American view skills, workplace skills, empowerment
culture." Li specifically mentioned that she skills. I learned English that is more effective
outcomes. Moreover, the outcomes of hav- a field trip, follow ed by dictating stories to
ing engaged in a single activity may be quite the teachers and volunteers about that field
different for different students. For example, trip, and then learning to read their stories.
a group of primary school students may take For some students such a process may in-
on a professional levei for communication. Li feels that her primary challenge is the
WORK helps us not be afraid. They gave me a fact that she is a "foreigner." She says, "An
computer and some cost-of-living support, and American learns one thing and I have to learn
they paid fora pronunciation class. They heped double. I'm slower than another American
me so much." worker, because it's ali new for me."
A friend of Li's interviewed for this case According to Li, her "life has changed a lot
study reported that WORK was "especiaily because of WORK." She explained that before
helpful to Li in making the adjustmentfrom her she received WORK support, it was hard to
culture to this cuture. It was a touchstone for imagine herseif in a professional position in
her. It helped her in her adapting. It was vital." an office environment in this country. Li
Li now works full time. She wakes at 5:30 a.m., noted, "I was so excited to get an office job. It
catches the bus at 6:30, arrives at work at is the luckiest thing in my life, and the biggest
7:30, and begins work at 8:00. She works until challenge." According to Li, one resut of her
4:30 p.m. and immediately takes the bus,arriv- effort with WORK is that she feels she can talk
ing home at 5:30. Several times a week, Li and interact with anyone at work or in the
swims after work at a local community pool. community.She's moresecurein dealing with
Each evening she prepares dinner for her others and has more confidence in herseif. in
daughterand helps her with her homework. Af- addition, Li is abie to manage her own finan-
ter her daughter goes to bed, Li frequently ciai situation and make confident decisions
works on her computer to improve her skills, or about raising her daughter. Her interactions
does other self-study. with her daughter have improved and she says
Li says that she is still working on communi- thatsheis no longerashamed of herdivorce.
cating better in the workplace. Her supervisor Li's future includes short-term and long-
said in an interview:"Weencouraged hertobe- term goals. In the short term, she would like
come more assertive. We wanted her to know to save money to buy the software for Pro-
that it'5 okay to stand up for herseif. It's okay to fessional Payroli Accounting, complete the
be more assertive. It's the American way. She's training, and move nto payroli accounting.
learning." Her job supervisor continued: "WORK She doesn't see herseif staying at her cur-
has been here a couple of times to sit and talk rent company for more than several years.
with myself and the personnel manager when Eventually, she would like to go back to
we were having difficulty with Li and one school to become a CPA and increase her in-
coworker. We wanted their input on what to do. come to support herseif and her daughter.
So we ali kind of worked together to try and One day, Li hopes to travei to Vietnam to visit
solve the problem. They worked really well with her mother.
us and are continuing to do so."
volve learning about the mechanics of lan- cess may be learning how to spell certain
guage: sentence structure, parts of speech, words. For other students the important out-
and verb conjugation, for example. For other come may be having generated an idea from
students the major outcome of such a pro- a particular experience. For yet other stu-
dents the important outcome may have been have. However, in programs that empha-
something that was learned in the exercise size individualization of treatment and out-
or experience itself, such as knowledge eomes, program staff may argue, quite justi-
about the firehouse or the farm that was vis- fiably, that independence has a different
ited. Other students may become more artic- meaning for different people under differ-
ulate as a result of the dietation exercise. Still ent life condi tions. Thus, for example, for
other students may have learned to read one person independence may have to do
better as a result of the reading part of the ex- with a changing family dynamic and
ercise. The criticai point is that a common ac- changed relationships with parents. For an-
tivity for ali students can result in drasti- other person independence may have to do
cally different outeomes for different with nonfamily relationshipsthat is, inter-
students depending on how they approach actions with persons of the opposite sex, so-
the experience, what their unique needs cial activities, and friendships. For still other
were, and which part of the activity they clients the dominant motif in independence
found most stimulating. Educators involved may have to do with employment and eco-
in individualized approaches, then, need nomic factors. For still others it has to do
evaluation methods thatpermit documenta- with learning to live alone. While clients in
tion of a variety of outeomes, and they resist each case may experience a similar psycho-
measuring the success of complex, individu- therapeutic intervention process, the mean-
alized learning experiences by any limited ing of the outeomes for their personal lives
set of standardized outcome measures (e.g., will be quite different. What program staff
improved reading scores, better spelling, or wants to document under such conditions is
more knowledge about some specific sub- the unique meaning of the outeomes for
ject). Qualitative case studies offer a method each client. Staff members need descriptive
for capturing and reporting individualized information about what a client's life was
outeomes. like on entering treatment, the client's re-
A similar case can be made with regard to sponse to treatment, and what the clienfs
the individualization of leadership develop- life was like following treatment. They also
ment, criminal justice, community mental want to report documented outeomes
health, job training, welfare, and health pro- within the context of a clienfs life for "suc-
grams. Take, for example, the goal of in- cessful programs see the child in the context ofthe
creased independence among a group of cli- family and the family in the context of its sur-
ents receiving treatment in a community roundings" (Schorr 1988:257). Such descrip-
mental health center. It is possible to con- tive information results in a set of individual
struct a test or checklist that can be adminis- case studies. By combining these case histo-
tered to a large group of people measuring ries, it is possible to construct an overview of
their relative degrees of independence. In- the patterns of outeomes for a particular
deed, such tests exist. They typically involve treatment facility or modality.
checking off what kind of activities a person The more a program moves beyond train-
takes responsibility for, such as personal hy- ing in standard basic competencies to more
giene, transportation, initiatives in social in- individualized development, the more qual-
teraction, food preparation, and so on. In itative case studies willbe needed to capture
many programs, measuring such criteria in the range of outeomes attained. A leader-
a standardized fashion provides the infor- ship program that focuses on basic concepts
mation that program staff would like to of planning, budgeting, and communica-
tions skills may be able to measure outcomes means that actively involving people in the
with a standardized instrument. But a lead- development process is an end in itself, not
ership program that engages in helping par- just a means to some more concrete end; the
ticipants think in systems terms about how process is the point rather than simply the
to find leverage points and intervention means of arriving at some other point. The
strategies to transform their own organiza- journey, not the destina tion, is what matters.
tions will need case studies of the actual For example, a planning process for a com-
transforma tion efforts undertaken by partic- munity or organization may be carried out
ipants, for their individual endeavors are with a heavy emphasis on participation and
likely to vary significantly. One may be the involvement such that building relation-
director of a small community-based non- ships and mutual understandings along the
profit organization. Another may be a midway is at least as important as the focus of
dle-level government manager. Still another the actual plan produced. The process, in
may be part of a large national organization. such a case, becomes the outcome. That is,
"Transforma tion" will mean very different producing a plan (the apparent intended
things in these different settings. Under such outcome) actually becomes a means to
circumstances, qualitative case study meth- building community (the real desired out-
ods and design strategies canbe particularly come).
useful for evaluation of individualized par- In contrast, other interventions and pro-
ticipant outcomes and organization-level grams play down process. The emphasis is
impacts. on results and outcomes. Even in these
cases, however, some process is undertaken
Process Studies to achieve results and understanding the
A focus on process involves looking at process-outcomes relationship necessitates
how something happens rather than or in ad- documenting and understanding processes.
dition to examining outputs and outcomes.
Evaluations vary in their emphasis on pro- Process evaluations study process. Qualitative
cess in part because programs vary in their inquiry is highly appropriate for studying
attention to process. Some therapy ap- process because (1) depicting process re-
proaches in psychology are highly process quires detailed descriptions of how people
oriented in that they focus on the relation- engage with each other, (2) the experience of
ship between the client and therapist, how process typically varies for different people
the client is approaching issues, how the cli- so their experiences need to be captured in
ent feels about the process, and the nature of their own words, (3) process is fluid and dy-
the interactions that occur during therapy, namic so it can't be fairly summarized on a
rather than focusing only or primarily on be- single rating sale at one point in time, and
havioral outcomes. Groups, programs, even (4) participants' perceptions are a key pro-
entire organizations may be characterized as cess consideration.
highly "process oriented" if how members Process evaluations aim at elucidating
and participants feel about what is happen- and understanding the internai dynamics of
ing is given as much attention as the results how a program, organization, or relation-
achieved. There are styles of community and ship operates. Process studies focus on the
organizational development that operate on following kinds of questions: What are the
the premise "What we do is no more impor- things people experience that make this pro-
tant than how we do it." This statement gram what it is? How are clients brought
into the program, and how do they move A process study is especially approprate
through the program once they are partici- when the following kinds of statements are
pants? How is what people do related to made about some intervention, relationship,
what they're trying to (or actually do) ac- organization, or program:
complish? What are the strengths and weak-
nesses of the program from the perspective We take people through a developmental pro-
of participants and staff? What is the nature cess made up of a series of steps or phases.
of staff-client interactions?
The nature of our process is what makes us
A process evaluation requires sensitivity
unique.
to both qualitative and quantitative changes
in programs throughout their development, We are a very process-oriented place.
which typically means monitoring and de-
scribing the details of the program's imple- We need to spend more time processing
mentation. Process evaluations not only whafs going on.
look at formal activities and anticipated out- I'm having trouble getting a handle on the pro-
comes, but they also investigate informal cess.
patterns and unanticipated interactions. A
variety of perspectives may be sought from What is the process? Is it the same for every-
people with dissimilar relationships to the one? Is the process working for people?
program, that is, inside and outside sources.
Process data permit judgments about the A good example of what can emerge from
extent to which the program or organization a process study comes from an evaluation of
is operating the way it is supposed to be op- the efforts of outreach workers at a prenatal
erating, revealing areas in which relation- clinic in a low-income neighborhood. The
ships can be improved as well as highlight- outreach workers were going door to door
ing strengths of the program that should be identifying women, especially teenagers, in
preserved. Process descriptions are also use- need of prenatal care in order to get them
ful in permitting people not intimately in- into the community prenatal clinic. Instead
volved in a programfor example, externai of primarily doing recruiting, however, the
funders, public officials, and externai agen- process evaluation found that the outreach
ciesto understand how a program oper- workers were spending a great deal of time
ates. This permits such externai persons to responding to immediate problems they
make more intelhgent decisions about the were encountering, for example, need for rat
program. Formative evaluations aimed at control, need for EngHsh as a second lan-
program improvement often rely heavily on guage classes, and protection from neglect,
process data. Finally, process evaluations are abuse, or violence (Philliber 1989). The ac-
particularly useful for dissemination and tual interactions that resulted from the
replication of model interventions where a door-to-door contacts turned out to be sig-
program has served as a demonstration pro- nificantiy different from the way the
ject or is considered to be a model worthy of door-to-door process was designed and con-
replication at other sites. By describing and ceptualized. These findings, which emerged
understanding the details and dynamics of from interviews and observations, had im-
program processes, it is possible to isolate portant implications for staff recruitment
criticai elements that have contributed to and training, and for how much time needed
program successes and failures. to be allocated to cover a neighborhood.
Implementation Evaluation evaluating outcomes. Where outcomes are

evaluated without knowledge of implemen-
A prominent theme running through the tation, the results seldom provide a direction
preceding sections is that qualitative meth- for action because the decision maker lacks
ods are particularly useful for capturing information about what produced the ob-
differences among people and programs. served outcomes (or lack of outcomes). Pure
Evaluating mdividualized outcomes, devel- pre-post outcomes evaluation is the "empty
oping unique case studies of people and pro- box" approach to evaluation.
grams, and documenting the local diversity One important way of studying program
within national or statewide programs implementation is to gather detailed, de-
these are evaluation research issues for scriptive information about what the pro-
which qualitative strategies are particularly gram is doing. Implementation evaluations
appropriate. This section looks more closely answer the following kinds of questions:
at the appropriateness of qualitative meth- What do clients in the program experience?
ods for evaluating program implementa- What services are provided to clients? What
tion. does staff do? What is it like to be m the pro-
It is important to know the extent to gram? How is the program organized? As
which a program is effective after it is fully these questions indicate, implementation
implemented, but to answer that question it evaluation includes attention to inputs, ac-
is important to learn the extent to which the tivities, processes, and structures.
program was actually implemented. In his Implementation evaluations tell decision
seminal study Social Program Implementation, makers what is going on in the program,
Walter Williams (1976) concluded, "The lack how the program has developed, and how
of concern for implementation is currently and why programs deviate from initial
the crucial impediment to improving com- plans and expectations. Such deviations are
plex operating programs, policy analysis, quite common and natural, as demon-
and experimenta tion in social policy areas" strated in the findings of RAND's classic
(p. 267). "Change Agent Study" of 293 federal pro-
In Utilization-Focused Evaluation (Patton grams supporting educational change
1997a), I suggested that if one had to choose (McLaughlin 1976). That study found that
between implementation information and national programs are implemented incre-
outcomes information because of limited mentally by adapting to local conditions, or-
evaluation resources, there are many in- ganizational dynamics, and programmatic
stances in which implementation informa- uncertainties.
tion would be of greater value. A decision
maker can use implementation information Where implementation was successful, and
to make sure that a policy is being put into where significant change in participant atti-
opera tion according to designor to test the tudes, skills, and behavior occurred, imple-
very feasibility of the policy. Unless one mentation was characterized by a process of
knows that a program is operating accord- mutual adaptation in which project goals and
ing to design, there may be little reason to ex- methods were modified to suit the needs and
pect it to produce the desired outcomes. Fur- interests of the local staff and in which the staff
thermore, until the program is implemented changed to meet the requirements of the pro-
and a "treatment" is believed to be in opera- ject. This finding was true even for highly
tion, there may be little reason even to bother technological and initially well-specified pro-
jects; unless adaptations were rnade in the The Follow Through data analysis
original plans or technologies, implementa- showed greater within-group variation than
tion tended to be superficial or symbolic, and between-group variation; that is, the 22
significant change in participants did not oc- models failed to show treatment effects as
cur. (McLaughlin 1976:169) such. Most effects were null, some were neg-
ative, but "of ali our findings, the most per-
If a process of ongoing adaptation to local vasive, consistent, and suggestive is proba-
conditions characterizes program imple- bly this: The effectiveness of each Follow
mentation, then the methods used to study Through model depended more on local circum-
implementation should correspondingly be stances than on the nature ofthe model" (Ander-
open-ended, discovery oriented, and capa- son 1977:13). The evaluators, however,
ble of describing developmental processes failed to study the local circumstances that
and program changes. Qualitative methods affected variations in program implementa-
are ideally suited to the task of describing tion and outcomes. "Little remains in the ex-
such program implementation. isting Follow Through evaluation that spe-
Failure to monitor and describe the cifically addresses the problem of how well,
nature of implementation, case by case, pro- and by what process, program models are
gram by program, can render useless stan- implemented" (Elmore 1976:119).
dardized, quantitative measures of program
The study of these important program
outcomes. The national evaluation of Follow
implementation questions requires case
Through was a prime example of this point.
data rich with the details of program content
Follow Through was a planned variation
and context. Because it is impossible to an-
"experiment" in compensatory education
ticipate in advance how programs will
featuring 22 different models of education to
adapt to local conditions, needs, and inter-
be tested in 158 school districts on 70,00 chil-
ests, it is impossible to anticipate what stan-
dren throughout the nation. The evalua-
dardized quantities could be used to capture
tion alone employed 3,000 people to collect
the essence of each program's implementa-
data on program effectiveness. The
tion. Under these evaluation conditions, a
multimillion-dollar evaluation focused al-
strategy of naturalistic inquiry is particu-
most entirely on standardized outcomes
larly appropriate. For a more extensive dis-
aimed at making possible comparisons of
cussion of evaluating program implementa-
the effectiveness of the 22 models. It was as-
tion, see King, Morris, and Fitz-Gibbon
sumed in the evaluation plan that models
(1987) and Patton (1997a: Chapter 9).
could be and would be implemented in
some systematic, uniform fashion. Eugene
Tucker (1977) of the U.S. Office of Education, Logic Models and
however, has poignantly described the error Theories of Action
of this assumption:
It is safe to say that evaluators did not know A logic model or theory of action depicts,
what was implemented in the various sites. usually in graphic form, the connections be-
Without knowing what was implemented it is tween program inputs, activities and pro-
virtually impossible to select valid effective- cesses (implementation), outputs, immedi-
ness measures. . . . Hindsight is a marvelous ate outcomes, and long-term impacts. For
teacher and in large scale experimentations an example, the classic educational model of
expensive one. (pp. 11-12) the popular DARE (Drug Abuse Resistance
Education) program in schools followed the

following simple logic model: (1) Recruit rNE \M\C M . ! CHAUNiat
and train select police officers (inputs) to
teach children the dangers of drug use; (2) " M ii!?:-!-.)!!-.'!- W J + L V C T I i'n:i":=::=:i'i-:i)-': ! ME"
have the police, in uniform, teach children in . î^ !;!!;'!^ !:*!'! ntf'rf"tCC!
special classes in school (implementation); LHI=.II'!;!:!!-!.!! s"ii i i I . i ! .Vi i V i : : ! " i ! A ? . : : $::-:R? T
and, as a result, the children will (3) find the ; liVri'!! !.^! î-i...: ' \ !i::: i !=
j IV! !! rc - i fr~
teaching credible (process evaluation) and i !1 !: ! !l=!i'!:i>.!.!i C L T I !
-ctMij,: K V < : v : s * c
(4) learn facts about drugs (cognitive out- = HtHn! jiH;!"' li 1i n t !:-i strt. d: =.<. ! '=.
come), which will (5) convince them not to I s*i-5|f=ia "tf SI'-!! a mn
j mmh i'i.!.' i! tiiv' rs ! -Vr -
use drugs (attitude change outcome), which iLVi-if rn ii;'iVi'M!riLvE i l ! M k ftV!."'i :ij-i!!;:v
will (6) result in students not using drugs iM M ii r !:i.'i! i' , r ' i ; V ! i ' ,'i Vift.":.! k . l . ' ! 1 ; IJ! : 1.-S-." i
(behavior change outcome), which will ulti- : tWWiMU" !"SuîiU- 1!:''!:Y.
mately show up in community indicators

showing less drug use (impact). At least that
was the model, or theory. In practice, evalua-
tions of DARE consistently showed that the gram staff, administrators, and evaluation
theory didn't work in practice. researchers spent a day together working on
Attention to program theory has become a logic model for a program, Road to Recov-
a major focus of evaluation research (see ery, that transported cncer patients to treat-
Rogers et al. 2000), and with that attention ment. As a result of thorough contextual
has come some confusion about terminol- analysis and interviews with key infor-
ogy. I distinguish a logic model from a the- mants, including patients and volunteer
ory of change. The only criterion for a logic drivers, the program was reconceptualized
model is that it be, well, logical, that is, that it as a treatment compliance strategy aimed at
portrays a reasonable, defensible, and se- cncer control rather than just a transpor ta-
quential order from inputs through activi- tion program. The reason for transporting
ties to outputs, outeomes, and impacts. A patients was to ensure complete and consis-
theory of change or theory of action, in tent treatment. This reconceptualization had
contrast, bears the burden of specifying and significant implications for how the pro-
explaining assumed, hypothesized, or gram was implemented, the outeomes that
tested causai linkages. Logic models are de- were measured, and the importance of the
scriptive. Theory of change and theory of ac- program in relation to American Cncer So-
tion models are explanaiory and predictive. ciety priorities.
The connotative difference between a "the- Organizational theorist Chris Argyris
ory of change" and "theory of action," (1982) introduced what has become a elassie
though the phrases are often used inter- distinetion between "espoused theories"
changeably, is that theory of change is more and "theories-in-use." The espoused theory
research based and scholarly in orientation, is what people say they do; it's the official
whereas a theory of action is practitioner de- version of how the program or organization
rived and practice based. operates. The theory-in-use is what really
The results of a logic modeling or the- happens. Interviewing supervisory or man-
ory-of-action exercise can make a dramatic agerial staff and administrators, and analyz-
difference in how people understand a pro- ing official documents, reveals the espoused
gram. A group of national and local pro- theory. Interviewing participants and front-
line staff, and directly observing the pro- mon need for a period of time for evaluators
gram, reveals the theory-in-use. The result- to work with program staff, administrators,
ing analysis can include comparing the funders, and participants on clarifying goals
stated ideais (espoused theory) with real pri- and strategiesmaking them realistic,
orities (theory-in-use) to help ali concerned meaningful, agreed on, and evaluable.
understand the reasons for and implications Evaluability assessments often include in-
of discrepancies. terviews and focus groups with diverse pro-
This ideal-actual comparison can support gram constituencies to determine how much
organizational development to improve ef- consensus there is among various stake-
fectiveness. It also helps move toward a rea- holders about a program's goals and inter-
sonably realistic depiction of lhe program vention strategies and to identify where dif-
that can be put to a summative te st, that is, ferences lie. Based on this kind of contextual
one can study the extent to which the model analysis, an evaluator can work with pri-
or treatment actually accomplishes the hy- mary intended users of the evaluation to
pothesized and desired outcomes and im- plan a strategy for goals clarification and
pacts. But such a study can take place only logic model development.
when the model has been described in realis- Studies of evaluability assessments (Rog
tic terms. Qualitative inquiry is especially 1985; Smith 1989) have found that the pro-
appropriate for achieving that description. cess of qualitative inquiry often becomes a
forma tive evaluation as program staff mem-
Evaluability Assessments bers learn about the strengths and weak-
nesses of their program conceptualizations.
Evaluability assessments (Wholey 1979, The evaluability assessments become im-
1994; Smith 1989) are conducted through in- provement-oriented experiences that lead to
terviews, document analysis, and observa- significant program changes rather than just
tions to determine whether a program is a planning exercise preparing for summa-
sufficiently well conceptualized and consis- tive evaluation.
tently implemented to undertake a formal
and rigorous evaluation, especially a
summative evaluation aimed at determm- Comparing Programs:
ing overall effectiveness. Clarifying the pro- Focus on Diversity
gram logic model or theory of action (previ-
ous section) is also an important purpose of
many evaluability assessments. In essence, We have frequently encountered the idea that
an evaluability assessment involves making a program is a fixed, unchanging object, ob-
sure that the program treatment or model is servable at various times and places. A com-
clearly identifiable and logical; that out- mon administra tive fiction, especially in
comes are clear, specific, and evaluable; and Washington, is that because some money asso-
that implementation strategies are reason- ciated with an administrative label (e.g., Head
ably and logically related to expected out- Star t) has been spent at several places and o ver
comes. a period of time, entities spending the money
One reason evaluability assessment has are comparable from time to time and from
become an important preevaluation tool is place to place. Such assumptions can easily
that by helping programs get ready for eval- lead to evaluation-research disasters. Pro-
uation, the process acknowledges the com- grams differ from place to place because
places differ. (Edwards, Guttentag, and Snap- What these data did not reveal was that
per 1975:142) some of the programs with large student-
staff ratios made extensive use of volun-
Individualizing services to clients has teers. These regularly participating addi-
been one major theme of social action and tional volunteer staff made the real stu-
educational programs in recent years. An- dent-adult ratios much smaller. The global
other closely related theme has been the im- and uniform reporting of the data, how-
portance of adapting programs to local ever, did not allow for that nuance to be
community needs and circumstances as dis- recorded.
cussed earlier in the Implementation Evalu- A good example of the diversity that can
ation section of this chapter. While some ba- emerge from attention to the qualitative dif-
sic framework of how programs should ferences among programs is Sharon Fei-
function may originate in Washington, D.C., man's (1977) classic study of national teacher
Ottawa, Brussels, or Canberra or some state center programs. Although funded as a sin-
or provincial capital, it is clear that program gle national program with common core
implementation at the local levei seldom foi- goals and the shared label "teacher centers,"
lows exactly the proposed design. When an Feiman found that three quite different types
evaluation requires gathering data from sev- of center programs had emerged, what she
eral local sites, quantitative measures may called "behavioral" centers, "humanistic"
be appropriate for comparing local pro- centers, and "developmental" centers. Dif-
grams along standardized dimensions, but ferences among the three types are summa-
qualitative descriptions are necessary to rized in Exhibit 4.3.
capture the unique diversities and contrasts Feiman^ analysis highlights the ways in
that inevitably emerge as local programs which different teacher centers were trying
adapt to local needs and circumstances. Lo- to accomplish different outcomes through
cal sites that are part of national or even in- distinct approaches to teacher center pro-
ternational programs show corisiderable gramming. Uniform, quantitative measures
variation in implementation and outcomes. applied across ali programs might capture
These variations are not such that they can some of these criticai differences, and such
be fully captured and measured along stan- measures have the advantage of facilitating
dardized scales; they are differences in direct comparisons. However, qualitative
kind differences in content, process, goals, descriptions permit documentation of
implementation, politics, context, outcomes, deeper and unanticipated program differ-
and program quality. To understand these ences, idiosyncrasies, and uniquenesses. If
differences, a holistic picture of each unique decision makers want to understand varia-
site is needed. tions in program implementation and out-
Using only standardized measures to comes, qualitative case studies of local pro-
compare programs can seriously distort grams can provide such detailed informa-
what is actually occurring in diverse sites. tion, as with the Camegie Foundation's
Consider data from a national educational excellent Portraits of High Schools (Perrone
program that measured staff-student ratios 1985). Thus, qualitative data are necessary to
across the country. A few programs had stu- give a complete evaluation picture of local
dent-staff ratios as high as 75:1 according to variations within national programs, a pic-
the uniform measures used; other programs ture that is necessarily incomplete so long as
had student-staff ratios as small as 15:1. the only data available are aggregated and
I4HIIMMI Types of Teacher Centers
Prmary Process far

Types of Centers Working With Teachers Primary Outcomes afthe Process
Behavioral centers Curriculum specialists directly Teachers adopt comprehensive curriculum

and formally nstruct teachers. systems and methods.
Humanistic centers Informal, undirected teacher Teachers feel supported and important: pick
exploration; "teachers select up concrete and practical ideas and materi-
their own treatments." ais for immediate use in classes.
Developmental centers Advisers establish warm, Teachers think in new ways about what
nterpersonal, and trusting they do and why they do it, developing new
relationships with teachers insights and fundamental skilis and capabil-
over time. ities.
SOURCE: Feiman (1977).
standardized statistics about these diverse moved toward healthier lifestyle practices
programs. during the 1980s, essentially wiping out the
control aspect of the intended control group.
Prevention Evaluation Time-series designs examine drops in indi-
cators of interest, for example, teenage sui-
There may be nothing more difficult to cide or alcoholism, but, again, there are seri-
evaluate than preventionthe nonoccur- ous problems determining the causes of
rence of some problem. Yet, there is no more changes in indicators.
important direction for the long-terin solu- Qualitative data can add an important di-
tion of health and social problems than pre- mension to prevention evaluations by find-
vention efforts (Sociometrics 1989). The ing out the extent to which desired attitude
usual designs for evaluating prevention pro- and behavior changes linked to prevention
grams use experimental and control groups actually occur. Interviews with teenagers
or time-series designs. In an experimental about their decisions regarding sexual activ-
design, a sample targeted for prevention is ity will reveal if teenagers are using preven-
compared with one that is not. There are of- tion practices and ideas. This helps get at
ten ethical problems with such designs (e.g., more than whether the desired outcome was
withholding needed services from the con- achieved. It helps illuminate how those in
trol group), and there can be problems in the targeted group think about and under-
control, that is, the control group may be stand what is being attempted.
subject to some other mtervention. A An excellent example of prevention-ori-
10-year study of heart disease prevention ented qualitative inquiry is Agar's (1999)
found no differences between treatment and study of heroin use among suburban youth
control groups because the whole society in Baltimore County, Maryland. His field-
work looked at how young people begin to tinue living m their homes, sometimes years
experiment with heroin and the stories that beyond what would otherwise have been
are generated and disseminated based on possible. While statistical data could docu-
initial experimentation. Those early stories ment comparative death rates, hospitaliza-
tend to be positive: "Heroin puts one into a tion rates, and costs, only direct interviews
blissful, relaxe d, dreamy state; the stresses, and observations could reveal quality-of-life
strains, and worries of everyday life fali differenceswhat it meant to these elderly
away." Over time, positive stories are fol- people to stay in their own homes. I remem-
lowed by negative stories as some youth be- ber in particular the story of one very frail
come addicted and a few overdose. "The im- woman in her 80s who took the interviewer
age of the addict, as opposed to the to into her bathroom, opened her medicine
experimenter, was uniformly and strongly cabinet, pointed to a bottle of capsules and
negative among the youth we mterviewed, said, "lf they come to take me away, Fll ex-
mostly based on their own observations or cuse myself for a moment, come in here, and
stories they had heard." As a result of his take these pills. I'm never leaving here.
in-depth work on how heroin addiction Never." This scenario was ali clearly worked
spreads through the youth community, out in her mind and routinely practiced
Agar (1999) learned how to be more effective mentally. Understanding prevention in-
in prevention education: cludes understanding what people think
and do as a result of prevention efforts.
I learned in the heroin lecture to emphasize the
dangers of sliding into physical dependency
Documenting Development
true addictionand what post-addicted
life was like. The theme contrasts with the nor-
Over Time and Investigating
mal approach, which conveys to youth that System Changes
addiction and death may occur even with one
experiment. Since youth rely on stories to The final practical application in this sec-
evaluate drug effects and since so many sto- tion focuses on the utility and appropriate-
ries contradict the normal premise, drug edu- ness of using qualitative methods to follow
cation loses its credibility. (pp. 115-16) and document development changes. This
returns us to the value of qualitative inquiry
His fieldwork also included looking at sup- for process studies, discussed earlier in this
ply and demand of heroin, and the larger chapter, for development is best understood
system of which drug use was apart, ali us- as a process. Organizational development,
ing qualitative methods to enhance efforts to community development, human develop-
prevent heroin use. ment, leadership development, professional
We evaluated a prevention program developmentthese are process-oriented
aimed at helping elderly people remain liv- approaches to facilitating change. Pre- and
ing in their homes, that is, preventing posttests do not do justice to dynamic devel-
institutionalization. In some cases, the delay opment processes. Pre- and postmeasures
in institutionalization was only six months imply a kind of linear, ever upward,
to a year. Through volunteer visits, home less-to-more image of growth and develop-
nursing care, and meals on wheels pro- ment. In reality, development usually occurs
grams, however, elderly people could con- in fits and starts, some upward or forward
progress, then backsliding or consolidation. nace. Just monitoring the temperature

Pre- and posttesting tell you where you gauge won't solve the problem. So it is with
started and where you ended up, but not management information systems. Blips or
what happened along the way. Quantitative changes in indicators are a signal that field-
measures can parsimoniously capture snap- work is needed to find out what's really go-
shots of pre- and poststates, even some n- ing on where the action is.
terim steps, but qualitative methods are
more appropriate for capturing evolution-
From Evaluation Issues
ary and transforrnational developmental
dynamics. For example, I worked with a to Evaluation Models
new and innovative employment training
program that was constantly reorganizing Thus far in this chapter, we have re-
staff and participant teams, realigning viewed the appropriateness of qualitative
courses, trying new mixes of activities, and inquiry for dealing with fairly common
restructuring as growth occurred. We found evaluation challenges: quality assurance
that even quarterly reflective practice ses- and enhancement, documenting outcomes
sions were insufficient to capture ali the through case studies, capturing individual-
changes occurring. To study this highly dy- ized outcomes, process studies and imple-
namic program required ongoing fieldwork menta tion evaluation, clarifying a pro-
framed against a chos theory metaphor (see gram^ theory or logic model by linking
Chapter 3) because the program develop- processes and implementation to outcomes,
ment really was like walking through a evaluability assessments, comparing di-
maze whose walls rearrange themselves verse programs, evaluating prevention pro-
with every step you takeand it was a ma- grams, documenting development over
jor challenge to develop a data collection time, and investigating sudden changes in
process that could capture those rearrange- management information system indica-
ments. tors. This is by no means an exhaustive list of
Even routine monitoring using a stan- evaluation applications, but each in some
dardized management informa tion system way illustrates a practice-oriented application
can provide only an overview of up-and- of qualitative methods, the focus of this
down patterns. A break in a trend line or chapter, in the sense that qualitative findings
sudden blips in a time series can indicate are used to improve programs, deal with
that some qualitative difference may have real problems, or support concrete decision
occurred. Statistical data from monitoring making rather than contribute to social sci-
and information systems canbe used to trig- ence theory or generate knowledge as an
ger more in-depth qualitative studies that end in itself. The next section reviews some
focus on finding out what the changes in the of the major models of evaluation research
statistical indicators mean. Consider this that are most closely associated with qualita-
analogy from the northern climes: The ther- tive methods: goal-free evaluation, respon-
mometer on a fumace thermostat shows the sive evaluation, illuminative evaluation,
temperature throughout winter; if the tem- the transaction model, the connoisseurship
perature starts falling quickly in a house, or art criticisrn approach, and utilization-
someone had better have a look at the fur- focused evaluation.
!=J. Evaluation Models
H olopis kuntal baris: Indonesian phrase uttered to gain extra strength

when carrying heavy objects.
Howard Rheingold (1988:49)
Conducting an evaluation can be a heavy opher-evaluator Michael Scriven (1972b)

load. Evaluation models help with the first proposed the idea of goal-free evalua-
heavy lifting. Models provide frameworks tion. Essentially, goal-free evaluation means
17 like the metal frame on a backpack that gives doing fieldwork and gathering data on a
support and shape to the load on a hiker's broad array of actual effects or outcomes,
back. Models offer evaluators structure and then comparing the observed outcomes
support. They structure certain method- with the actual needs of program partici-
ological decisions, offer guidance about the pants. The evaluator makes a delibera te at-
appropriate steps to follow in design, pro- tempt to avoid ali rhetoric rela ted to pro-
vide direction in ways of dealing with stake- gram goals; no discussion about goals is
holders (Alkin 1997), and identify the im- held with staff; no program brochures or
portant issues to consider in undertaking a proposals are read; only the program's ob-
study. Models provide frameworks rather servable outcomes and documentable ef-
than recipes, helping evaluators and evalua- fects are studied in relation to participant
tion users identify and distinguish among needs. There are four primary reasons for
alternative approaches. For example, the doing goal-free evaluation:
classic model of evaluation is goals-based
evaluation, that is, measuring the extent to 1. to avoid the risk of narrowly studying
which a program or intervention has at- stated program objectives and thereby
tained clear and specific objectives. One al- missing important unanticipated out-
ternative to goals-based evaluation, consid- comes;
ered below, is "goal-free" evaluation. The
2. to remove the negative connotations at-
models briefly presented here have in com-
tached to the discovery of unanticipated
mon that they have been closely associated
effects: "The whole language of 'side-
with or heavily rely on qualitative methods.
effect' or 'secondary effect' or even 'un-
anticipated effect' tended to be a put-
Goal-Free Evaluation down of what might well be the crucial
achievement, especially in terms of new
In the classic model of evaluation, the fo- priorities" (Scriven 1972b:l-2);
cus is on the intended services and outcomes
3. to eliminate the perceptual biases intro-
of a programits goals. However, an evalu-
duced into an evaluation by knowledge
ator can turn up some very interesting re-
of goals; and
sults by undertaking fieldwork in a program
without knowing the goals of the program, or 4. to maintain evaluator independence by
at least without designing the study with avoiding dependence on goalsa staff
goal attainment as the primary f ocus. Philos- or administrative creation and, often,
fictionthat can limit the evaluator's quiry is especially compatible with goal-
range and freedom of inquiry. free evaluation because it requires capturing
directly the actual experiences of program
In Scriven's (1972b) own words: participants in their own terms. Moreover,
and in particular, goal-free evaluation re-
It seemed to me, in short, that consideration quires the evaluator to suspend judgment
and evaluation of goals was an unnecessary about what the program is trying to do and
but also a possibly contaminating s t e p The to focus instead on finding out what is actu-
less the externai evaluator hears about the ally occurring in and as a result of the pro-
goals of the project, the less tunnel-vision will gram. The evaluator can thus be open to
develop, the more attention will be paid to whatever data emerge from the phenomena
looking for actual effects (rather than checking of the program itself and participants' expe-f
on alleged effects). (p. 2) riences of the program. But, as Ernie House
(1991) has reported, somewhat tongue-
Goal-free evaluation, in its search for "ac- in-cheek, actually conducting a goal-free
tual effects," employs an inductive and ho- evaluation can be challenging for ali con-
listic strategy aimed at countermg the logi- cerned because during on-site fieldwork
cal-deductive limitations inherent in the staff keep dropping hints about program
usual goals-based approach to evaluation, goals:
Goal-free evaluation was a radical departure
from virtually ali traditional evaluation Many are indignant that you do not want to
thinking and practice. For example, promi- know their objectives and incensed at the idea
nent evaluation researcher Peter Rossi of your looking at what they are domg rather
(Rossi and Williams 1972) asserted that "a than what they are professmg. As they chauf-
social welfare program (or for that matter feur you around m their car, some will blurt
any program) which does not have clearly out the goals as if accidentally, with sly apolo-
specified goals cannot be evaluated without gies for their indiscretion. Others will write
specifying some measurable goals. This them on restroom wallsanonymously.
statement is obvious enough to be a truism" (p-112)
(p. 18). Carol Weiss (1972) emphasized the
centrality of goals in evaluation when she It is important to note that goal-free eval-
stated, "The goals must be clear so that the uations can employ both quantitative and
evaluator knows what to look for. . . . Thus qualitative methods. Moreover, Scriven
begins the long, often painful process of get- (1972b) has proposed that goal-free evalua-
ting people to state goals in terms that are tions might be conducted in parallel with
clear, specific, and measurable" (pp. 24-26). goals-based evaluations, but with separate
In contrast to the predominant goals- evaluators using each approach to maxi-
based approach to evaluation, goal-free mize the strengths and minimize the weak-
evaluation opens up the option of gathering nesses of each approach. (For a more de-
data directly on program effects and effec- tailed discussion of goal-free evaluation,
tiveness without being constrained by a nar- and critiques of this idea, see Alkin 1972;
row focus on stated goals. Qualitative in- Patton 1997a.)
Transaction Models: Responsive and Illuminative Evaluation
R esponsive evaluation enables an alert reprter to capture the deep-felt

opinions of those most affected by the program, a product of grass-
roots populism.
Ernest House (1991:113)
Robert Stake's (1975) "responsive ap- This sensitivity allows the evaluator to
proach to evaluation" places particular em- collect data and report findings with those
phasis on the importance of personalizing differing perspectives clearly in mind, but
and humanizing the evaluation process. Be- with special attention to those whose per-
ing responsive requires having face-to-face spectives are less often heard. Subsequently,
contact with people in the program and Guba and Lincoln (1989) added an explic-
learning Krsthand about diverse stake- itly constructivist perspective to responsive
holders' perspectives, experiences, and con- evaluation in proposing "responsive con-
cerns. structivist evaluation" as the "fourth gener-
ation" of evaluation. The first generation
focused on measurement; the second on de-
Responsive evaluation is an alternative, an old
scription; the third on judgment; and the
alternative, based on what people do natu-
fourth generation of evaluation focuses on
rally to evaluate things, they observe and re-
issues-derived, values-based perspectives.
act. . . . To do a responsive evaluation, the
(See Chapter 3 for a discussion of construc-
evaluator conceives of a plan of observations
tivism.)
and negotiations. He arranges for various per-
Responsive evaluation includes the fol-
sons to observe the program, and with their
lowing primary emphases:
help prepares brief narrative portrayals, prod-
uct displays, graphs, etc. He finds out what is
1. identification of issues and concerns
of value to his audiences, and gathers expres-
based on direct, face-to-face contact with
sions of worth from various individuais
people in and around the program;
whose points of view differ. Of course, he
checks the quality of his records: he gets pro- 2. use of program documents to further
gram personnel to react to the accuracy of his identify important issues;
portrayals; and audience members to react to
the relevance of his findings. He does most of 3. direct, personal observations of pro-
this informallyiterating and keeping a re- gram activities before formally design-
cord of action and reaction. (Stake 1975:14) ing the evaluation to increase the evalu-
ator^ understanding of what is im-
portant in the program, and what can/
Guba and Lincoln (1981) have integrated should be evaluated;
naturalistic inquiry and responsive evalua-
tion into an overall framework for improv- 4. designing the evaluation based on is-
ing the usefulness of evaluation results. The sues that emerged in the preceding three
openness of naturalistic inquiry permits the steps, with the design to include contin-
evaluator to be especially sensitive to the dif- uing direct qualitative observations in
fering perspectives of various stakeholders. the naturalistic program setting;
5. reporting information in direct, per- tasks and academic experiences are most af-
sonal contact through themes and por- fected. It aims to discover and document what
trayals that are easily understandable it is like to be participating in the scheme,
and rich with description; and whether as teacher or pupil, and, m addition,
to discern and discuss the innovation's most
6. matching information reports and re-
significant features, recurrmg, concomitant,
porting formats to specific audiences
and criticai processes. In short, it seeks to ad-
with different reports and different for-
dress and to illuminate a complex array of
mats for different audiences.
questions. (Parlett and Hamilton 1976:144)
Responsive evaluation is a form of what
evaluation theorist Ernie House (1978) has Transaction evaluation approaches are
called the "transaction model" of evaluation based on the same assumptions that under-
in that it "concentrates on the educa tional gird qualitative research: the importance of
(or program) processes themselves. . . . It understanding people and programs in con-
uses various informal methods of investiga- text; a commitment to study naturally occur-
tion and has been drawn increasingly to the ring phenomena without introducing exter-
case study as the major methodology . . . , nai controls or manipulation; and the
[derived from] a subjectivist epistemology assumption that understanding emerges
[that] tends to be naturalistic" (p. 5). It treats most meaningfully from an inductive analy-
each case as unique and places prime em- sis of open-ended, detailed, descriptive data
phasis on perception and knowing as a gathered through direct interactions and
transactional process between researcher and transactions with the program and its partic-
research participant, thus the label for this ipants.
model.
Connoisseurship Studies
One can study perceptions only by studymg
particular transactions in which the percep-
While responsive evaluation places the
tions can be observed. Ali parties of the situa-
program's stakeholders at the center of the
tion enter into the transaction as "active
evaluation process, connoisseurship evalua-
participants," and do not appear as separa te
tion places the evaluator's perceptions and
already-existing entities. . . . [The evaluator]
expertise at the center of the evaluation pro-
affects and is affected by the situation, thus he
cess. The researcher as connoisseur or expert
is part of the transaction. (House 1978:9)
uses qualitative methods to study a program
Another variation on the transaction or organization, but does so from a particu-
model is "illuminative evaluation," origi- lar perspective drawing heavily on his or her
nally developed as an educa tional evalua- own judgments about what constitutes ex-
tion approach that emphasized context and cellence, thus the term connoisseur. This is a
interpretation. practical and personalistic version of what I
called "orientational qualitative inquiry" in
The aims of illummative evaluation are to the last chapter and what Eisner (1985:184)
study the innovative program: how it oper- has called "prefigured," that is, the terms
ates; how it is influenced by the various school and focus of the "educational criticism" are
si tuations in which it is applied; what those di- determined in advance by the evaluator and
rectly concerned regard as its advantages and those seeking the evaluation. Prefiguring,
disadvantages; and how students' intellectual however, does not mean that the observer
cannot be open to a new, emergent focus literary and the factual complement each
during fieldwork. other" (Eisner 1985:182).
This means that evaluative criticism or
A critic might be invited to a school or class-
connoisseurship is also highly interpretive
room without a prefgured focus and after sev-
and makes value judgments about the mer-
eral days or weeks perceive an aspect of the
its of what has been described and inter-
school or classroom that is of considerable sig-
preted, using criteria that are appropriate to
nificance but which could not have been antic-
the situation based on the expertise of the
ipated. For example, one of my students
evaluator (the connoisseur) and the agree-
received permission from a secondary school
nents struck with those who are a party to
English teacher to observe and to write an ed-
the study. In that regard, design and data
ucational criticism about her class. What
collection decisions are explicitly political
emerged during my student's observation
and subjective. While there is a qualitative
was the extraordinary way in which the
research foundation to this kind of evalu-
teacher used satire in her teaching. It was not
ative connoisseurship, the method also "re-
the case that the teacher was herself teaching
quires no small degree of artistry" (Eisner
satire; it was that she was satirical in her teach-
1985:187). The factual aspects of this ap-
ing. Such a process or an approach could
proach to evaluation communicate knowl-
hardly have been prefgured. The point here is
edge. The artistic aspects convey not only
that the focus of criticism can be either prefg-
knowledge of facts "but also knowledge of
ured as a part of the research bargain between
feeling. Art can be said to be that activity
the critic and the teacher, or it can be emergent
concerned with the creation of images of
or it can be both. (Eisner 1985:184-85)
feeling. The situations, people, and objects
The art criticism model as a "con- we encounter are never without affect"
noisseurship" approach has been most fully (Eisner 1988:17). The connoisseur, then, is
articulated by Elliot W. Eisner (1985). The explicitly and purposefully a qualitative re-
imagery of evaluators as "connoisseurs" searcher and an artistic critic of the phenom-
making criticai appraisals of programs is enon being studied.
analogous to the traditional way in which
literary and artistic connoisseurs and critics Utilization-Focused Evaluation
work, Eisner having had considerable expe-
rience as an art educator. Utilization-focused evaluation (Patton
The connoisseur approach has many ele- 1997a) offers an evaluative process, strategy,
ments that relate to naturalistic inquiry. Cen- and framework for making decisions about
tral among these are direct observation and the content, focus, and methods of an evalu-
immersion in the setting under study. Edu- ation. Utilization-focused evaluation begins
cational criticism or connoisseurship is, ac- with identification and organization of spe-
cording to Eisner, first and foremost descrip- cific, relevant decision makers and informa-
tive, both factually and artistically. The tion users (not vague, passive audiences)
factual component reports direct observa- who will use the information that the evalu-
tions and interviews. "The artistic aspect of ation produces. A focus on intended use by in-
description is literary and metaphorical; intended users undergirds and informs every
deed, it can evenbe poetic in places In or- design decision in the evaluation. The evalu-
der to optimize communication, the poten- ator works with these intended users (often
tial of language is exploited so that the an evaluation task force representing several
constituencies, e.g., program staff, clients, best techniques. Researchers and decision
funders, administrators, board members, makers operate within quite narrow meth-
and community representatives) to focus odological paradigms about what consti-
relevant evaluation questions. From these tutes valid and reliable data, rigorous and
questions flow the appropriate research scientific design, and personal or imper-
methods and data analysis techniques. sonal research methods. As I noted in dis-
Utilization-focused evaluation plans for cussing methodological paradigms (Chap-
use before data are ever collected. The ques- ter 2), most social scientists routinely apply
tion that underlies the ongoing interactions those methods in which they have been
between evaluators and intended users is, trained with little sensitivity to the biases in-
"What difference will this study make?" The troduced by a particular data collection
evaluator asks, "What would you do if you scheme. Social and behavioral scientists
had answers to the questions you're ask- experts in the ultimate subjectivity and arbi-
ing?" In answering the evaluation questions trariness of ali human perceptionare often
of specific intended evaluation users, utili- least aware of their own sociomethod-
zation-focused evaluation does not pre- ological biases and how these biases affect
clude the use of any of the full variety of their view of the social program world. Yet,
methodological options available. Qualita- to be sure, social scientists are not the only
tive inquiry strategies may emerge as appro- participants in the evaluation process oper-
priate in a particular utilization-focused ating on the basis of selective perception; de-
evaluation as a result of defining the infor- cision makers and program staff also hold
mation needs of the specific intended evalu- conditioned views about research and meth-
ation needs. ods. One of the tasks to be accoinplished
Creative, practical evaluators need a full during the interactions between evaluators
repertoire of methods to use in studying a and intended evaluation users is to mutu-
variety of issues. This repertoire should in- ally explore design and data biases so that
clude but not be limited to qualitative meth- the evaluation generates information that is
ods. By offering intended users method- useful and believable to ali concerned.
ological options, utilization-focused evalu- Utilization-focused evaluation was de-
ators collaborate in making criticai design veloped from a study of the factors that
and data collection decisions so as to in- seemed to explain variations in the actual
crease the intended users' understanding use of evaluations. That study used qualita-
and buy-in thereby facilitating increased tive methods to study the uses of 20 federal
commitment to use fmdings. The evalua- health evaluations. Our study team inter-
tor^ responsibility is to interact with deci- viewed evaluators, funders, and program
sion makers about the strengths, weak- managers to find out how evaluation find-
nesses, and relative merits of various meth- ings were used (Patton 1997a). Many others
ods so that mutually agreed-on, informed have since confirmed and elaborated the
methods decisions can be made. The evalua- major elements of utilization-focused evalu-
tor may well challenge entrenched method- ation, again using qualitative methods
ological biases while remaining ultimately (Alkin, Daillak, and White 1979; King and
respectful of the importance of users getting Pechman 1982; Campbell 1983; Holley and
something they will believe m and use. Arboleda-Florez 1988; Ferguson 1989). The
Measurement and methods decisions are utilization-focused approach was further re-
not simply a matter of expertly selecting the fined through experience and practice and
closely associated with and rely heavily on

qualitative methods. The next sections in
: IIH-S MI IÎ OF this chapter review interactive applications
\ !=Alt I"1 -Xii i:: Ir .RH D G 1'i.iVM ! . . .
of qualitative methodspractical and prag-
; ; >..." ;
matic forms of inquiry in which the re-
j 'V'Ti .!!=:!'!:; ttaP! !:f!! fpr8 W&W ^ I-
searcher is especially sensitive to the per-
spectives of others and interacts closely
'Si&siiS ,: iWi i! i-:"; with them in designing and/or implement-
ing the study. Such interactive approaches
: -:r^raJamiyfflW * ^ [ ~ that are especially congruent with qualita-
tive methods include efforts to personalize
and humanize research and evaluation,
working with stakeholders to harmonize
program and evaluation values, action
I. S!:i T I : I== ; i l h i l j i i i : ; ! ; : ! ! : ! ) ; ' . :.!:'!.
! learning and reflective practice, apprecia-
= : :=T! \
: I I I I ^ ^ ^ ^ J - J Y ^ ^ H V ' ! I : : : L : ! L ' 1 $ : - ^ J L L L L - C : :;j tive inquiry, facilitating collaboration with
; ; :=::!!!': " ji;n:i:;jrfii^VV! coresearchers through participatory re-
search, supporting democratic dialogue
nill: lUniTITli HitJi Eili E i Ti: iTI :t Tti H H IIIinnHllTiHlHlHHl^
and de libera tion, and supporting democ-
[ . :
V v : : ; f ! ; = . | ; T ! i:".'i:Ji i 1:
tf, n ^ racy through process use.
; .' jrrn?:;:r ;'!:; i'j i
; ' ; TFNJ S?FI ^ IIRR R R
: Mrhhyrjn;:=: !=:i;|;!. j.-n:;.:;iîflfF^î^.; Personalizing and

' l & H f t l t l i =:;";*!"!l:-!.^:!"::-^;^^
Humanizing Evaluation
One reason for using qualitative methods

in evaluation and organizational develop-
ment processes is that such strategies maybe
by studying how well-known and effective perceived and experienced by program staff
evaluators actually consult with clients and and participants as personal in nature, what
conduct themselves in doing evaluation re- Michael Cernea (1991) has called "putting
search. Utilization-focused evaluation, then, people first." Programs that are based on hu-
is a process for creatively and flexibly inter- manistic concerns and principies often resist
acting with intended evaluation users about any kind of quantification because of the
their information needs and alterna tive perception that numbers and standardized
methodological options, taking into account categorization are cold, hard, and imper-
the decision context in which an evaluation sonal. The issue here is not whether such ob-
is undertaken. jections are reasonable. The point is that
such objectification feels real to those who
hold such views. In programs where staff,
S Interactive and
funders, and/or participants feel this way,
Participatory Applications evaluations that rely on quantitative mea-
Thus far, this chapter has been describing surement may be rejected out of hand.
various evaluation issues that are especially The personal nature of qualitative inquiry
appropriate for qualitative inquiry and derives from its openness, the evaluator's
some specific evaluation models that are close contact with the program, and the pro-
cedures of observation and in-depth inter- tative methods may be perceived as more
viewing, particularly the latter, that commu- humanistic and personal simply by avoid-
nicate respect to respondents by making ing numbers.
their ideas and opinions (stated in their own Personalizing and humanizing evalua-
terms) the important data source for the tion are particularly important for educa-
evaluation. Kushner's book Personalizing tion, therapy, and development efforts that
Evaluation (2000) epitomizes this emphasis are based on humanistic values (Patton
especially in advocating that the perspective 1990). Humanistic values that undergird
of participants be given primacy: both qualitative inquiry and humanistic ap-
proaches to intervention and change include
I will be arguing for evaluators approaching the core principies listed in Exhibit 4.4.
programs through the experience of individu- Where people in a program, organization, or
ais rather than through the rhetoric of pro- community hold these kinds of values, qual-
gram sponsors and managers. I want to itative inquiry is likely to feel particularly ap-
emphasize what we can learn about programs propriate.
from Lucy and Ann. This does not mean ig-
noring the rights of program managers and
Harmonizing Program
sponsors with access to evaluation. There is no
case for using evaluation against any stake-
and Evaluation Values
holder group; though there is a case for assert- The suggestion that one reason for using
ing a compensatory principie in favor of those qualitative methods is that such strategies
who start out with relatively lower leveis of may be perceived by program staff and pro-
access to evaluation. I don't think there is a se- gram clients as more personal in nature
rious risk of evaluators losing touch with their opens up a whole range of potential philo-
contractual obligations to report on programs sophical, political, and value orienta tions
and to support program management and im- that can influence methods decisions. The
provement; I don't think there is a danger that argument here is that it can be appropriate
evaluators will ever lose their preoccupation and desirable to include among the criteria
with program effects. There is always a risk, for making methods decisions the value ori-
however, that evaluators lose contact with entations of the intended users of the study.
people; and a danger that in our concern to re- One example of a framework for supporting
port on programs and their effects we lose harmony between a program philosophy
sight of the pluralism of programs. So my ar- and evaluation approach is presented in Ex-
guments will robustly assert the need to ad- hibit 4.5. This framework illustrates how the
dress "the person" in the program. (pp. 9-10) decision to use qualitative methods in evalu-
ation can flow from the values of the people
Qualitative methods may also be per- who will use the evaluation information.
ceived as more personal because of their in- Understanding, relevance, interest, and
ductive nature. This means that, again, use are ali increased when evaluators and
rather than imposing on people or a pro- users share values about methods. The final
gram some predetermined model or hy- design of an evaluation depends on calcu-
potheses, the results unfold in a way that lated trade-offs and weighing options, in-
takes into account idiosyncrasies, unique- cluding political/philosophic/value con-
ness, and complex dynainics. Finally, quali- siderations. The design also depends on
Common Principies Undergirding Qualitative Inquiry

EXHIBIT 4.4 and Humanistic Values
1. Each person or community is unique.

2. Each person or community deserves respect.
3. Equity, fairness, and mutual respect should be foundations of human interactions.
4. Change processes (and research) should be negotiated, agreed to, and mutually understoodnot im-
posed, forced, or required.
5. One expresses respect for and concern about others by learning about them, their perspective, and
their world-and by being personally involved.
6. Change processes shouid be person centered, attentive to the effects on real people as individuais
with their own unique needs and interests.
7. Emotion, feeling, and affect are natural, healthy dimensions of human experience.
8. The change agent, therapist, or researcher is nonjudgmental, accepting, and supportive in respecting
others' right to make their own decisions and live as they choose. The point is empowerment of oth-
ers, not contrai or judgment.
9. People and communities should be understood in context and holistically.
10. The process (how things are done) is as important as the outcome (what is achieved).
11. Action and responsibility are shared; unilateral action ts avoided.

12. Information should be openly shared and honestly communicated as a matter of mutual respect and
in support of openness as a value.
opportunity, resources, time constraints, nesses in relation to this complex constella-

and commitment. What is to be avoided is tion of both values and technical factors in
the routine selection of a design without an effort to make methods and values rela-
consideration of its strengths and weak- tively harmonious.
Developmental Applications: Action Research, Action Learning,

R e f l e c t i v e Practice, and L e a r n i n g O r g a n i z a t i o n s
V ou leam something new every day. Actually, you learn something old
every day. Just because you just learned it, doesn't mean it's new.
Other people already knew it. Columbus is a good example.
George Carlin (1997:135)
A variety of organizational, program, emerged that involve inquiry within orgarri-

and staff development approaches have zations aimed at learning, improvement,
FCDEF
Matching Program Philosophy and Evaluation

Approach: An lliustration
What are the values of the school? And what are the values of the evaluation? Throughout
the life of the school, three strong themes have emerged: (a) personalized curriculum, (b) the
experiential nature of learning, and (c) the holistic nature of learning. The structures of the
school organization, the formal goals of the school, emphases in classrooms, and investments
of time and money in staff development activities have ali placed strong value on these aspects
of learning. These are not mutually exclusive, but are intricately intertwined. Their meaning for
the school and for the evaluation of the school are described in the following excerpts from the
Marcy Open School evaluation document.
Marcy Open School Philosophy Evaluation of Marcy Open School
1. Personalized curriculum 1. Personalized evaluation

Curriculum will vary for each child as teaching The determination of the success of the school
extends from the interests, needs, and abilities will vary depending upon the values and
of each child or group of children. The school perspectives of interested people. This evaluation
personnei seek to be aware of each chd as an will present a statement of what was made
individual, and of potential learning activities available by the school and of what was
and materiais. The individual chd, the teacher, accomplished by children in the school. Decisions
and the parents make decisions on the as to the success of the school and as to the
curriculum. validity of those activities must be left to the
individual reading the evaluation report,
according to his or her own perspective.
2. Experiential nature of learning 2. Experiential nature of evaluation

The school seeks to have the children experience This evaluation will attempt to provide an
language rather than only learning to read, to opportunity for the reader to experience the
experience computation rather than only to learn school and its children. It will provide not only
math, to be in and to learn from the community charts and statistics, but photographs, drawings,
rather than only to learn about social studies. and works of children and adults. Even so, it is
Participants in the school believe that experience acknowledged that, at best, any such report can
is the best transmitter of knowledge. Further- only be an imperfect representation of the school
more, the child is expected to interact with his and its processes. Furthermore, this report is not
[or her] environment-to have an effect upon it presented as a final document representing the
in the process of experiencing itto change it aecompiishments of the school, It s, instead,
or to recognize ways in which he [or she] seeks presented as a report-in-process, to be reacted
to move toward change. to and sent back for new descriptions, new data,
about which its readers may be concerned.
Marcy Open School Phiiosophy Evaluation of Marcy Open School
3. Holistic nature of learninq 3. Holistic evidence for evaluation

Much emphasis has been placed on the Three of the schoofs goais for children have
interrelatedness of learning. Organizationa! been chosen for special attention in this
structures, activities, and materiais are evaluation. They include a range of process,
considered in terms of their multidimensionality content, and context typicai of the goal
of goals. Goa! statements and staff development statement as a whole. The evidence presented
activities give conscious attention to children's attempts to observe a natural order of events
feelings about themselves and their world, how as they happen in the school with a minimum
those relate to the child's relationships with of distortion through departmentalization.
others and how those relate to the child's Both objective figures and subjectiv judgments
interest and ability in learning. The staff seeks are included and are considered valid. The
activities that allow the child to experience the provisioning of the school and the activities
relationship between language, computation, and and products of the children are viewed,
other knowledge rather than compartmentaizing insofar as it's possible, in terms of their
them into separate content areas. mutidimen- sionality of effect.
SOURCE: Olson 1974.
and development (e.g., Argyris and Schon ing the program. In an ongoing "reflective
1978; Senge 1990; Watkins and Marsick 1993; practice group," the teachers shared their
Aubrey and Cohen 1995; Torres, Preskill, cases and interpreted the results. Based on
and Piontek 1996). These efforts are called these reflections and thematic cross-case
various things: "action learning" (Pedler analysis, staff members revised their em-
1991; McNamara 1996), "team learning" phasis on goal setting as the priority first
(Jarvis 2000), "reflective practice" (Schon step in the program. The interviews showed
1983,1987; Tremmel 1993), "action research" that new immigrants lacked sufficient expe-
(Whyte 1989; Gore and Zeichner 1995; rience with and knowledge of American ed-
Stringer 1996), internai evaluation (Sonnich- ucational and employment opportunities to
sen 2000; Love 1991), or organizational de- set meaningful and realistic goals. As a re-
velopment (Patton 1999c). These problem- sult, students typically acquiesced to coun-
solving and learning-oriented processes selor suggestions just to get the mandated
often use qualitative inquiry and case study goal-setting f orms completed, but they carne
approaches to help a group of people reflect out of the orientation without any substan-
on ways of improving what they are doing tial commitment to or ownership of what
or understand it in new ways. were supposedly their own goals. Here we
For example, the teaching staff of an eve- have an example of a short-term, rapid tum-
ning basic skills adult education program around qualitative inquiry for reflective
undertook an action learning inquiry into practice or action learning aimed at program
the experiences of students who had re- improvement. By participating in the pro-
cently immigrated to the United States. Each cess, staff members also deepened their sen-
staff person interviewed three students and sitivity to the perspectives and needs of new
wrote short case studies based on the stu- immigrant students, thereby developing
dents' reports about their experiences enter- professionally.
I'm at that point where mixed No probiem.

findings would be reassuring so We'll just use mixed methods.
we don't have to act.
Inaction research
The leaming that occurs as a result of succeed if they are carefully designed and
these processes is twofold: (1) The inquiry implemented systemically to deal with or-
can yield specific insights and findings that ganic, cultural, and power complexities. In
can change practice, and (2) those who par- qualitative terms, this means action learning
ticipate in the inquiry learn to think more inquiry that is holistic, grounded, and sensi-
systematically about what they are doing tive to context.
and their own relationship to those with Harvey and Denton (1999) examined the
whom they work, what Bawden and twin and interrelated themes of "organiza-
Packham (1998) have called "systemic tional learning" and the "learning organiza-
praxis." In many cases, the specific findings tion" in the business sector. The qualitative
are secondary to the more general learnings research underpinning their study was con-
that result from being involved in the pro- ducted over a three-year period (1994-1997)
cess, what I have called "process use" as and involved detailed examina tion of orga-
opposed to findings use (Patton 1997a). Pro- nizational learning aspira tions and practices
cess use is greatly enhanced in "develop- within the British operations of five major
mental evaluation" (Patton 1994; 1997a) in manufacturing companies. Sixty-six inter-
which the purpose of the evaluation is ongo- viewees were classified into three groups:
ing learning, internai improvement, and strategy, human resources, and research and
program development rather than generat- development (R&D). They identified a set of
ing reports and summative judgments for six antecedents that together explain the rise
externai audiences or accountability. to prominence of emphasis on organiza-
A lot of attention in recent years has been tional learning: (1) the shift in the rela tive
paid to action learning as a way of helping impor tance of factors of production away
people in organizations cope with change. from capital toward labor, particularly intel-
Mwaluko and Ryan (2000) offer a case study lectual labor; (2) the ever more rapid pace of
showing how action learning programs can change in the business environment; (3)
widespread acceptance of knowledge as Sorensen, Yaeger, and Nicoll (2000:4) in in-

a prime source of competitive advantage; troducing a special issue of the journal OD
(4) the greater demands being place d on Practitioner devoted to AI.
ali businesses by customers; (5) increasing What interests us here is that AI is
dissatisfaction, among managers and em- grounded in qualitative understandings
ployees, with the traditional, command- and prescribes a particular process of quali-
and-control management paradigm; and (6) tative inquiry within an organization that in-
the intensely competitive nature of global cludes a dialogue process among partici-
business. Their analysis of qualitative inter- pants based on their interviewing each
views featured the interplay of thoughts and other. They ask each other questions that
feelings between management practitioners "elicit the creative and life-giving events ex-
and organizational theorists in getting in- perienced in the workplace" (Watkins and
side the idea of organizational learning. Cooperrider 2000:9):
While they focused on business applica-
tions, the idea of being a learning organiza- 1. Looking at your entire experience with
tion has also become a prominent theme in the organization, remember a time
the nonprofit, government, and philan- when you felt most alive, most fulfilled,
thropic sectors (Sonnichsen 1993; Preskill or most excited about your involvement
and Torres 1999; Preskill and Preskill 1997). in the organization....
Qualitative inquiry as practiced through the
2. Let's talk for a moment about some
lenses of action learning and reflective prac-
things you value deeply; specifically, the
tice can be one of the foundations of a learn-
things you value about yourself, about
ing organization.
the nature of your work, and about this
organization
Appreciative Inquiry
3. What do you experience as the core fac-
tors that give life to this organization?
Appreciative inquiry (AI) has emerged as
Give some examples of how you experi-
a popular organizational development ap-
ence those factors.
proach that emphasizes building on an orga-
nization^ assets rather than focusing on 4. What three wishes would you make to
problems, or even problem solving. Con- heighten the vitality and health of this
ceived and described in the work of David organization? (p. 9)
Cooperrider and his colleagues at Case
Western Reserve's School of Organization These questions aim at generating spe-
Behavior (Watkins and Cooperrider 2000), cific examples, stories, and metaphors about
AI is being offered by its advocates as "a positive aspects of organizational life. Par-
worldview, a paradigm of thought and un- ticipants in the process analyze the results in
derstanding that holds organizations to be groups looking for the themes and topics
affirmative systems created by humankind that can become the foundation for positive
as solutions to problems. It is a theory, a organizational development going forward.
mind-set, and an approach to analysis that
leads to organizational learning and creativ- For example, if the original data suggest that
ity" (Watkins and Cooperrider 2000:6). As COMMITMENT is an important factor in
such, "AI reflects the core values of OD [or- many of the stories about the best of times in
ganizational development]/' stated editors the organization, then the workgroup might
choose to ask more questions from others in In the film Awakenings, based on Dr. Oli-
the workplace about their experiences with ver Sacks's real-life experiences in a New
commitment. This second round of interviews York institution for mentally ill people
produces informa tion about four to six topics (Sacks 1973), Dr. Sacks (played by Robin Wil-
that become the basis for building "possibility liams as the character Dr. Malcolm Sayer)
propositions" that describe how the organiza- engages the assistance of nurses, orderlies,
tion will be in the future. Each topic or theme janitorial staff, and even patients in joint ef-
can be fashioned into a future statement. And forts to discover what might reach certain
these statements become an integral part of patients suffering from a rare form of catato-
the vision for the organization. Often this pro- nia. The film powerfully displays a form of
cess is completed with a future search confer- collaborative research in which trained re-
ence that uses the appreciative inquiry data as searchers and nonresearchers undertake an
a basis for imaging a positive and creative fu- inquiry together. One orderly tries music. A
ture for the organization. (Watkins and nurse tries reading to patients. A volunteer
Cooperrder 2000:10) tries card games. Together they figure out
what works.
AI has been criticized for being unbal- In an African village, the women were
anced and uncritical in its emphasis (critics skeptical of public health workers' admoni-
say oueremphasis) on accentuating the positions to use well water rather than surface
tive. It may even, ironically, discourage in- water for drinking during the rainy season.
quiry by discouraging constructive criticism Going to the well was more work. Instead of
(Golembiewski 2000). Whether it endures as simply trying to convince the women of the
a viable and popular approach to organiza- wisdom of this public health advice, the ex-
tional development remains to be seen, but tension educators created an experiment in
its questioning strategies could be incorpo- which half the village used well water and
rated into more balanced approaches. AI in- half used surface water. They kept track of
tegrates inquiry and action within a particu- illnesses with matchsticks. At the end of
lar developmental framework that guides three months, the villagers could see for
analysis and processes of group interaction. themselves that there were many more
The qualitative questioning and thematic matchsticks in the surface water group. By
analysis processes constitute a form of inter- participating in the study rather than just re-
vention by the very nature of the questions ceiving results secondhand, the findings be-
asked and the assets-oriented framework came more meaningful to them and more
used to guide analysis. In this way, inquiry useful.
and action are completely integrated. Other
Early in my career, I was commissioned
forms of participatory inquiry also seek inte-
by a provincial deputy minister in Canada to
gration of inquiry and action.
undertake an evaluation in a school division
he considered medocre. I asked what he
wanted the evaluation to focus on. "I don't
Participatory Research
care what the focus is," he replied. "I just
and Evaluation: Valuing
want to get people engaged in some way.
and Facilitating Collaboration Educa tion has no life there. Parents aren't in-
volved. Teachers are just putting in time.
Let's start with three quite different ex- Administrators aren't leading. Kids are
amples of participatory qualitative inquiry. bored. I'm hoping that having them partici-
pate in an evaluation will stir things up and history. John Elliott (1976) worked with
get people involved again." classroom teachers as coresearchers doing
When conducting research in a collabora- action research "developing hypotheses
tive mode, professionals and nonpro- about classrooms from teachers' practical
fessionals become coresearchers. Humanis- constructs." Bill Hull (1978) worked with
tic research and heuristic inquiry (Douglass teachers in a reflective, research-oriented
and Moustakas 1985) value collaboration, as process to study children's thinking. The
does "cooperative inquiry" (Heron 1996). Boston Woinen's Teachers' Group (1986)
Participatory action research (Wadsworth was organized as a research collaborative for
1993a, 1993b; King and Lonnquist 1994a, studying the effects of teaching on teachers.
1994b) encourages joint collaboration within Eleanor Duckworth's (1978) classic evalua-
a mutually acceptable ethical framework to tion of an African primary science program
understand and/or solve organizational or was collaborative in approach.
community problems. Feminist methods are Genuinely collaborative approaches to
participatory in that "the researcher invites research and evaluation require power shar-
members of the setting to join her in creating ing. One of the negative connotations often
the study" (Reiriharz 1992:184). "Empower- associated with evaluation is that it is some-
ment evaluation" aims to foster "self-deter- thing done to people. Participatory evalua-
mination" among those who participate in tion, in contrast, involves working with peo-
the inquiry process (Fetterman 2000a; ple. Instead of being research subjects, the
Fetterman, Kaftarian, and Wandersman people in the research setting become
1996). This can involve forming "empower- "co-investigators." The process is facilitated
ment partnerships" between researchers by the researcher, but is controlled by the
and participants (Weiss and Greene 1992) people in the program or community. They
and teaching participants to do research undertake a formal, reflective process for
themselves (Wadsworth 1984). In-depth in- their own development and empowerment.
terviewing and description-oriented obser- Participatory evaluation has been used
vations are especially useful methods for with great success as part of intemational
supporting collaborative inquiry because and community development efforts by a
the methods are accessible to and under- number of nongovernmental organizations
standable by people without much technical (NGOs) and private voluntary organiza-
expertise. tions (PVOs) in the Third World (e.g., Aubel
Interest in participatory research has ex- 1993). A collaborative group called Private
ploded in recent years, especially as an ele- Agencies Collaborating Together (PACT)
ment of larger community change efforts published the excellent users guide Partici-
(Stoecker 1999). The principal researcher patory Evaluation (1986) as well as the more
trains the coresearchers to observe, inter- general Evaluation Sourcebook (Pietro 1983).
view, reflect, and / or keep careful records or The guide includes techniques for actively
diaries. Those involved come together peri- involving nonliterate people as active par-
odically to share in the data analysis process. ticipants in evaluating the development ef-
The purpose of such shared inquiry is typi- forts they experience, often using qualita-
cally to elucidate and improve the nature of tive methods.
practice in some arena of action. The processes of participation and collab-
Qualitative, collaborative research efforts oration have an impact on participants and
in educational settings have a distinguished collaborators quite beyond whatever find-
ings or report they may produce by working themselves not so important as what is
together. In the process of participating in re- learned from the discussion and from the pro-
search, participants are exposed to and have cess of reaching consensus on what questions
the opportunity to learn the logic of evi- should be used to evaluate group perfor-
dence-based inquiry and the discipline of mance and capacity, and on what answers best
evidentiary reasoning. Skills are acquired in describe their group's present status. (p. 272)
problem identification, criteria specifica-
tion, and data collection, analysis, and inter- It was not a group's specific questions or
pretation. Through acquisition of inquiry answers that Uphoff found most affected the
skills and ways of thinking, a collaborative groups he observed. It was the process of
inquiry process can have an impact beyond reaching consensus about questions and en-
the findings generated from a particular gaging with each other in the meaning of the
study. answers turned up. The process of participa-
Moreover, people who participate in cre- tory self-evaluation, in and of itself, pro-
ating something tend to feel more owner- vided useful learning experiences for partic-
ship of what they have created and make ipants.
more use of it. Active participants in re- Viewing participatory inquiry as a means
search and evaluation, therefore, are more of creating an organizational culture com-
likely to feel ownership not only of their find- mitted to ongoing learning, as discussed in
ings but also of the inquiry process itself. the previous section, has become an impor-
Properly, sensitively, and authentically tant theme m recent literature linking pro-
done, it becomes their process. Participants gram evaluation to "learning organizations"
and collaborators can be community mein- (e.g., King 1995; Leeuw, Rist, and Son-
bers, villagers, organizational workers, pro- nichsen 1993). "The goal of a participatory
gram staff, and/or program participants evaluator is eventually to put him- or herself
(e.g., chents, students, farmers). Some times out of work when the research capacity of
administrators, funders, and others also the organization is self-sustaining" (King
participate, but the usual connotation is that 1995:89). Indeed, the "self-evaluating orga-
the primary participants are "lower down" nization" (Wildavsky 1985) constitutes an
in the hierarchy. Participatoiy evaluation is important direction in the institution-
bottom-up. The trick is to make sure that alization of evaluation logic and processes.
participation is genuine and authentic, not I advise caution and care in using the la-
just token or rhetorical, especially in bel "participatory inquiry," "empowerment
participative evaluation, where differing evaluation," or "collaborative research," for
political and stakeholder agendas often these terms mean different things to differ-
compete (Fricke and Gill 1989). ent peopleand serve different purposes.
Norman Uphoff (1991) has published "A Some use these terms interchangeably or as
Field Guide for Participatory Self-Evalua- mutually reinforcing concepts. Wadsworth
tion" aimed at grassroots community devel- (1993a) distinguished "research on people,
opment projects. After reviewing a number for people or with people" (p. 1). Levin (1993)
of such efforts, he concluded: distinguished three purposes for collabora-
tive research: (1) the pragmatic purpose of
If the process of self-evaluation is carried out increasing use of findings by those involved
regularly and openly, with ali group members as emphasized by, for example, Cousins and
participating, the answers they arrive at are in Earl (1992); (2) the philosophical or method-
Principies of Fully Participatory and Genuinely

EXHIBIT 4.6 Coliaborative Inquiry
The inquiry process involves participants in learning inquiry logic and skills, for example, the na-
ture of evidence, establishing priorities, focusing questions, interpreting data, data-based deci-
sion making, and connecting processes to outcomes.
Participants in the process own the inquiry. They are invoived authentically in making majorfo-
cus and design decisions. They draw and apply conclusions. Participation is real, not token.
Participants work together as a group and the inquiry facilitator supports group cohesion and
collective inquiry.
Ali aspects of the inquiry, from research focus to data analysis, are undertaken in ways that are
understandable and meaningful to participants.
The researcherorevaluatoracts as a facilitator, collaborator.and learning resource; participants
are coequa!.
The inquiry facilitator recognizes and values participants' perspectives and expertise and works
to help participants recognize and value their own and each other's expertise.
Statusand power differences between the inquiry facilitator and participants are minimized, as
much as possible, practical, and authentic, without patronizing or game playing.
ological purpose of grounding data in par- setting being studied as co-inquirers, at least
ticipants' perspectives; and (3) the political to some important extent, though the de-
purpose of mobilizing for social action, for gree and nature of involvement vary widely.
example, empowerment evaluation or what My purpose here has been to point out that
is sometimes called "emancipatory" re- these participatory approaches often em-
search (Cousins and Earl 1995:10). A fourth ploy quahtative methods because those
purpose, identified here, is teaching inquiry methods are understandable, teachable, and
logic and skills. Since no definitive defini- usable by people without extensive research
tions exist for "participatory" and "coliabo- training.
rative evaluation," these phrases must be
defined and given meaning in each setting
where they're used. Exhibit 4.6 presents pri- Supporting Democratic
mary principies for fully participatory and Dialogue and Deliberation
genuinely coliaborative inquiry. This list can
be a starting point for working with partici- Most of this chapter has examined rela-
pants in a research or evaluation setting to tively small-scale applications of quali-
decide what principies they want to adopt tative methods in evaluating programs,
for their own process. developing organizations, supporting plan-
Regardless of the terminologypartici- ning processes and needs assessments, and
patory, coliaborative, cooperative, or em- providing insights into communities. This
powermentthese approaches share a com- section considers a much larger agenda, that
mitment to involving the people in the of strengthening democracy. House and
Howe (2000) have articulated three require- sequence that the evaluator should seek to
ments for evaluation done in a way that sup- represent the full range of interests in the
ports democracy: inchision, dialogue, and de- course of designing an evaluation. In that
liberation. They worry about the power that way, an evaluator can support an informed
derives from access to evaluation and the citizenry, the sine qua non of strong democ-
implications for society if only the powerful racy, by acting as information broker be-
have such access. tween groups who want and need knowl-
edge of each other. The democratic
We believe that the background conditions for evaluator must make the methods and tech-
evaluation should be explicitly democratic so niques of evaluation accessible to nonspe-
that evaluation is tied to larger society by dem- cialists, that is, the general citizenry. Mac-
ocratic principies argued, debated, and ac- Donald^ democratic evaluator seeks to
cepted by the evaluation community. Evalua- survey a range of interests by assuring confi-
tion is too important to society to bepurchased dentiality to sources, engaging in negotia-
by the highest bidder or appropriated by the tion between mterest groups, and making
most powerful interest. Evaluators should evaluation findings widely accessible. The
be self-conscious and deliberate about such guiding ethic is the public's right to know.
matters.... Saville Kushner (2000) has carried for-
If we look beyond the conduct of individ- ward, deepened, and updated MacDonald's
ual studies by individual evaluators, we can democratic evaluation model. He sees eval-
see the outlines of evaluation as an influential uation as a form of personal expression and
societal institution, one that can be vital to the political action with a special obligation to
realization of democratic societies. Amid the be critics of those in power. He uses qualita-
claims and counterclaims of the mass media, tive methods to place at the center of evalua-
amid public relations and advertising, amid tion the experiences of people in programs.
the legions of those in our society who repre- The experiences and perceptions of the peo-
sent particular interests for pay, evaluation ple in programs, the supposed beneficiaries,
can be an institution that stands apart, reliable are where, for Kushner, we will find the in-
in the accuracy and integrity of its claims. But tersection of Politics (big PPolicy) and pol-
it needs a set of explicit democratic principies itics (small ppeople). He uses qualitative
to guide its practices and test its intuitions. case studies to capture the perspectives of
(House and Howe 2000:4) real peoplechildren and teachers and par-
entsand the realities of their lives in pro-
Qualitative inquiry figures into this dem- gram settings as they experience those reali-
ocratic approach to evaluation because, as ties. He feels a special obligation to focus on,
discussed in the section on participatory re- capture, report, and therefore honor the
search, qualitative methods are especially views of marginalized peoples. He calls this
accessible to and understandable by non- "personalizing evaluation," but the larger
researchers, and because case studies can be agenda is strengthening democracy. Con-
an excellent resource for supporting inclu- sider these reflections on the need for evalu-
sion and dialogue. In Europe, the demo- ators and evaluations to address questions
cratic evaluation model of Barry MacDonald of social justice and the democratic contract:
(1987) illustrates these emphases. He argued
that "the democratic evaluator" recognizes Where each social and educational program
and supports value pluralism with the con- can be seen as a reaffirmation of the broad so-
ciai contract (that is, a re-confirmation of the of findings for enhanced decision making
bases of power, authority, social structure, and program improvement and, therefore,
etc.), each program evaluation is an opportu- making sure that findings reflected the di-
nity to review its assumptions and conse- verse perspectives of multiple stakeholders,
quences. This is commonly what we do at including the less powerful and participants
some levei or another. Ali programs expose de- in programs (instead of just staff, adminis-
mocracy and itsfailings; each program evaluation tra tors, and funde rs). While this thrust re-
is an assessment ofthe effectiveness ofdemocracy mains important, a parallel and reinforcing
in tackling issues in the distribution ofwealth and use of evaluation focuses on helping people
power and social goods. Within the terms of the learn to think and reason evaluatively,, and how
evaluation agreement, taking this levei of rendering such help can contribute to
analysis into some account, that is, renewing strengthening democracy over the long
part of the social contract, is to act more au- term. I turn now to elaborate that contribu-
thentically; to set aside the opportunity is to tion.
act more mauthentically, that is, to accept the
fictions. (Kushner 2000:32-33, emphasis added)
Supporting Democracy
Through Process Use: Helping
While MacDonald, Kushner, and House
the Citizenry Weigh Evidence
and Howe make explicit linkages between
evaluation and democracy, a number of and Think Evaluatively
other evaluation approaches imply such
linkages by emphasizing various degrees Let me begin by offering some context for
and types of stakeholder participation and reflecting on the role of knowledge creation
involvement and, correspondingly, evalua- in relation to democracy In the autumn of
tor responsiveness. For reviews of the vari- 2000,1 had the opportunity to participate in
ety of such approaches and distinctions a seminar sponsored by the Italian Evalua-
among them, see Cousins and Earl (1995) tion Society. While in Rome I visited the F-
and Alkin (1997). The work of Mertens rum, wandered among the ruins of ancient
(1998, 1999) on "inclusive evaluation" and Rome, and spent some meditative time in
the "empowerment evaluation" model of the remains of the Senate seeking inspiration
Fetterman et al. (1996) offer additional ex- about what I might say regarding evalua-
amples of evaluation approaches that em- tion's potential contributions to democracy,
phasize qualitative inquiry and support the theme of the European Evaluation Soci-
democratic principies, social justice, and ex- ety conference in Lausanne, Switzerland,
plicitly political foundations of evaluation where I was headed after Rome. Nothing
in support of those whose stakes tend to be came to me in the Frum, at least nothing
underrepresented in policy discussions be- about evaluation. I couldn't get past vivid
cause they are marginalized economically, images of Caesar's death in that place as
socially, and politically. portrayed by Shakespeare. After leaving the
Taken together, these writings on evalua- Frum, I walked across to the Coliseum,
tion^ role in supporting democratic pro- where gladiators did battle before emperors
cesses reflect a significant shift in the nature and citizens of Rome. There, standing in the
of evaluation^ real and potential contribu- platform area reserved for the senators, I got
tion to strengthening democracy. A decade a distinct image. I imagined an evaluator
ago, the emphasis was ali on increasing use presenting important policy findings to the
citizens of Rome in the Coliseum. At the end racy. Having experienced totalitarianism,
of the report, the emperor would invite the then having fled it, she devoted much of her
crowd to render a thumbs-up or thumbs- life to studying it and its opposite, democ-
down on the evaluation. Thumbs- up would racy. She believed that thinking thought-
mean a laurel wreath, coin of the realm, and fully in public deliberations and acting
an all-expenses-paid trip to present at the democratically were intertwined. Totalitar-
annual meeting of the Imperial Evaluation ian- ism is built on and sustained by deceit
Society. Thumbs-down would mean the and thought control. To resist efforts by the
lions got fed. I left the Coliseum quickly powerful to deceive and control thinking,
thinking how fortunate I was to be engaged Arendt believed that people needed to prac-
in evaluation at a time when the stakes, tice thinking. Toward that end she devel-
though high, are not quite so high as my Col- oped "eight exercises m political thought"
iseum vision and when, instead, in the beau- (Arendt 1968). She wrote that "experience in
tiful city of Lausanne, in the year 2000, an in- thinking... canbe won, like ali experience in
ternational community of professionals met doing something, only through practice,
together to spin visions of strengthening de- through exercises" (p. 4). From this point of
mocracy by the ways in which we engage in view, might we consider every evaluation an
evaluation and applied research. opportunity for those involved to practice
So what is the connection between quali- thinking? This would mean that every eval-
tative inquiry and democracy? uation is an opportunity to strengthen de-
Start with the premise that a healthy and mocracy by teaching people how to think
strong democracy depends on an informed evaluatively. In this regard, we might aspire
citizenry. A central contribution of policy re- to have policy research, action research, par-
search and evaluation, then, is to help ensure ticipatory research, and collaborative evalu-
an informed electorate by disseminating ation do what Arendt hoped her exercises in
findings as well as to help the citizenry political thought would do, namely, give us
weigh evidence and think evaluatively. This "experience in how to think." Her exercises
involves thinking processes that must be do not prescribe "what to think or which
learned. It is not enough to have trustworthy truths to hold"; rather, they focus on the act
and accurate informa tion (the informed part and process of thinking. For example, she
of the informed citizenry). People must also thought it important to help people think
know how to use information, that is, to conceptually, to "discover the real origins of
weigh evidence, consider inevitable contra- original concepts in order to distili from
dictions and inconsistencies, articulate val- them anew their original spirit which has so
ues, interpret findings, deal with complex- sadly evapora ted from the very keywords of
ity, and examine assumptions, to note but political languagesuch as freedom and
a few of the things meant by "thinking justice, authority and reason, responsibility
evaluatively." Moreover, in-depth demo- and virtue, power and gloryleaving be-
cratic thinking includes political sophistica- hind empty shells" (Arendt 1968:14-15).
tion about the origins and implications of Might we add to her conceptual agenda for
the categories, constructs, and concepts that examination and public dialogue such terms
shape what we experience as information as outcomes and performance indicators, inter-
and "knowledge" (Minnich forthcoming). pretation and judgment, and beneficiary and
Philosopher Hannah Arendt was espe- stakeholder, among many evaluative possi-
cially attuned to this foundation of democ- bilities?
Helping people learn to think evalu- viewing a variety of people and under-
atively by participating in real evaluation standing in-depth case examples are espe-
exercises is what I've come to call "process cially effective ways of enhancing nonre-
use" (Patton 1997a, 1998). I have defined searcher involvement in evaluation and
process use as relating to and being indi- research to help them increase their capacity
cated by individual changes in thinking and to think about evidence and draw appropri-
behaving that occur among those involved ate conclusions from data.
in evaluation as a result of the learning that Helping people learn to think in these
occurs during the evaluation process. (Changes ways can have a more enduring impact from
in program or organizational procedures a study than use of specific findings gener-
and culture may also be manifestations of ated in that same study. Findings have a very
process impacts, but that is not our focus short half-lifeto use a physical science
here.) This means an evaluation can have metaphor. They deteriorate very quickly as
dual tracks of impact in strengthening de- the world changes rapidly. Specific findings
mocracy: (1) a more informed electorate typically have a small window of relevance.
through use of findings and (2) a more In contrast, learning to think and act
thoughtful and deliberative citizenry evaluatively can have an ongoing impact.
though helping people learn to think and en- The experience of being involved in an eval-
gage each other evaluatively. uation, then, for those stakeholders actually
One way of thinking about process use is involved can have a lasting impact on how
to recognize that evaluation constitutes a they think, on their openness to reality test-
cultural perspective of sorts. When we ening, on how they view the things they do,
gage other people in the evaluation process, and on their capacity to engage thoughtfully
we are providing them with a cross-cultural in democratic processes.
experience. This culture of evaluation, Democratic evaluations debunk the myth
which we as evaluators take for granted in that methods and measurement decisions
our own way of thinking, is quite alien to are purely technical. Nonresearchers then
many of the people with whom we work at become savvier about both the technical and
program leveis. Examples of the values of nontechnical dimensions of evaluation.
evaluation include clarity, specificity, and Moreover, we know that use is enhanced
focusing; being systematic and making as- when practitioners, decision makers, and
sumptions explicit; operationalizing pro- other users fully understand the strengths
gram concepts, ideas, and goals; distin- and weaknesses of evaluation data and that
guishing inputs and processes from out- such understanding is increased by being in-
eomes; valuing empirical evidence; and volved in making methods decisions. We
separating statements of fact from interpre- know that use is enhanced when intended
tations and judgments. These values consti- users participate in making sure that as
tute ways of thinking that are not natural to trade-offs are considered, as they nevitably
people and, indeed, quite alien to many. are because of limited resources and time,
When we take people through a process of the path chosen is informed by relevance.
participatory research or evaluationat We know that use is enhanced when users
least in any kind of stakeholder involvement buy into the design and find it credible and
or collaborative process, they are in fact valid within the scope of its intended pur-
learning how to think in these ways. Inter- poses as determined by them. And we know
that when evaluation findings are pre- blandishments, I for one will have none of it.
s ente d, the substance is less likely to be un- (p-1)
dercut by debates about methods if users
have been in volve d in those decisions prior For others who will have none of it, one
to data collection (Patton 1997a). way to address the issue of methodological
At its roots, participatory evaluations are quality in democratic evaluations is to
informed by a fundamental confidence in reframe the policy analysfs function from
the wisdom of an informed citizenry and a an emphasis on generating expert judg-
willingness to engage ordinary citizens re- ments to an emphasis on supporting in-
spectfully in ali aspects of evaluation, informed dialogue, including methodological
cludingmethodological discussions and de- dialogue. The traditional expert-based sta-
cisions. This point is worth emphasizing tus of researchers, scholars, and evaluators
because somenot ali, to be sure, but has fueled the notion that we provide scien-
someresistance to participatory evalua- tifically based answers and judgments to
tion derives from the status associated with policymakers while, by our independence,
research expertise and an elitist or patroniz- we ensure accountability to the general pub-
ing attitude toward nonresearchers (they lic. Playing such a role depends on a knowl-
are, after ali, "subjects"). Egon Guba (1978) edge paradigm in which correct answers
has described in powerful language this ar- and independent judgments can be con-
chetype: ceived of existing. However, postinod-
ernism, deconstruction, criticai theory, femi-
nist theory, empowerment evaluation, and
It is my experience that evaluators sometimes constructivism, among other perspectives,
adopt a very supercilious attitude with respect share skepticism about the traditional
to their clients; their presumptuousness and truth-oriented knowledge paradigm. They
arrogance are sometimes overwheLming. We offer, in contrast, an emphasis on interest-
treat the client as a "child-like" person who acknowledged interpretations articulated
needs to be taken in hand; as an ignoramus and discussed within an explicit context (po-
who cannot possibly understand the tactics litical, social, historical, economic, and cul-
and strategies that we will bring to bear; as tural). Constructivist orienta tions to qualita-
someone who doesn't appreciate the ques- tive inquiry have played a criticai role in the
tions he ought to ask until we tell himand emergence of dialogical forms of inquiry
what we tell him often reflects our own biases and analysis. Participatory methods have
and interests rather than the problems with increased the access of nonresearchers to
which the client is actually beset. The phrase both research findings and processes. In
"Ugly American" has emerged in interna- combination, constructivist, dialogical, and
tional settings to describe the person who en- participatory approaches offer a vision of re-
ters into a new culture, immediately knows search and evaluation that can support de-
what is wrong with it, and proceeds to foist his liberative democracy in the postmodern
own solutions onto the locais. In some ways I knowledge age. Such a grandiose, even
have come to think of evaluators as "Ugly bombastic, vision derives from recognition
Americans." And if what we are looking for that in this emergent knowledge age, re-
are ways to manipula te clients so that they will searchers have larger responsibilities than
fali in with our wishes and cease to resist our just publishing in academic journals.
lL Special Applications
f you come to a fork in the road, take it.
Yogi Berra
These final sections examine some special questionnaire. Indeed, administration of

applications and situations where qualita- such instruments may produce artificial re-
tive strategies are especially appropriate: the sults because respondents are affected by
need for unobtrusive measures; state-of-the- the process, what measurement specialists
art considerations; adding depth, detail, and call "reactivity/' a source of invalidity in
meaning to quantitative analyses; rapid re- measurement. Educational researcher Edna
connaissance; capturing and communicat- Shapiro (1973), to her surprse, found this to
ing stories; legislative auditing and monitor- be precisely the case in her study of innova-
ing; futuring applications; and breaking tive Follow Through classrooms. She found
routines. that standardized tests biased evaluation re-
sults by imposing an obtrusive and con-
The Need for trolled stimulus in an environment where
spontaneity, creativity, and freedom of ex-
Unobtrusive Measures
pression were valued and encouraged.
Shapiro found that the results of the test
Qualitative strategies can be particularly
measured response to a stimulus (the test)
appropriate where the administration of
that was essentially alien to the experience
standardized instruments, assigning people
of the children. Because the classrooms she
to comparison groups, and/or the collection
studied relied substantially less on paper-
of quantitative data would affect program
and-pencil skills than traditional schools,
operations by being overly intrusive. Exam-
and because student progress was moni-
ples of unobtrusive measures include ob-
tored daily on a personal basis without the
serving and measuring wear and tear on
use of written examinations, student out-
carpets placed in front of different museum
comes in these classrooms could notbe "ob-
exhibits to evaluate visitor interest in differ-
jectively" measured by the sudden introduc-
ent exhibits; observing people in a public
tion of standardized tests.
place, as a participant in whatever is going
on, without taking notes; and using docu-
ments or reports prepared for other pur- I assumed that the internalized effects of dif-
poses (e.g., clinicai case notes) to generate re- ferent kinds of school experience could be ob-
search case data in situations where no served and inferred only from responses in
additional human subject protection per- test situations, and that the observation of
mission is required because the data are rou- teaching and learning iri the classroom should
tinely collected and findings will be re- be considered auxiliary information, useful
ported only in the aggregate. chiefly to doeument the differences in the chil-
Observations of program activities and dren^ group learning experiences. . . . The
informal interviews with participants can findings of this study, with the marked dispar-
often be carried out in a less obtrusive fash- ity between classroom responses and test re-
ion than having everyone complete a test or sponses, have led me to reevaluate this
rationale. This requires reconsidera tion of the outcomes, no acceptable, valid, and reliable
role of classroom data, individual test situa- measures exist. The extent to which one be-
tion data, and the relation between them. . . . lieves that particular instruments, such as
The individual's responses in the test situation personality tests, are useful, valid, and reli-
have conventionally been considered the pri- able can be a matter of debate and judgment.
mary means to truth about psychological Moreover, for desired program outcomes
functioning. Test behavior, whether consid- where measures have not been developed
ered as a sign or sample of underlying func- and tested, it can be more appropriate to
tion, is treated as a pure measure. Yet, the test gather descriptive information about what
situation is a unique interpersonal context in happens as a result of program activities
which what is permitted and encouraged, ac- than to use some scale that has the merit of
ceptable and unacceptable, is carefully de- being quantitative but whose validity and
fined, explicitly and implicitly. Responses to reliability are suspect.
tests are therefore made under very special circum- Creativity is a prime example. While
slances. The variables that influence the outcome there are some instruments that purport to
are different from those which operate in the class- measure creativity, the applicability of those
room. (Shapiro 1973:532-34, emphasis added) instruments in diverse situations is at least
open to question. Thus, a program that aims
In their imagina tive classic, Unobtrusive to support students or participants in being
Measures, Webb et ai. (1966) discussed at more creative might do better to document
length the problems of "reactive measure- in detail the activities, products, behaviors,
ment effects." A basic theme of their work feelings, and actual creations of participants
was that research participants' awareness instead of administering some standardized
that they are part of a study (as they com- instrument of marginal or dubious rele-
plete questionnaires or take tests) might dis- vance. Qualitative documenta tion can be in-
tort and confound the study's findings. spected and judged by interested evaluation
Their documenta tion of the sources and na- users to make their own interpretations of
ture of reactivity problems in scholarly so- the extent to which creativity was exhibited
cial science research makes it highly likely by the products produced.
that such problems are magnified in evalu- Even such hallowed concepts as self-es-
ation research (see Holley and Arboleda- teem are open to considerable controversy
Florez 1988). While qualitative methods are
when it comes to specifying measurement
also subject to certain reactivity problems (to
criteria. In addition, for people whose
be discussed in later chapters), the less for-
self-esteem is already quite high, instru-
mal and less obtrusive nature of some quali-
ments that measure self-esteem will not be
tative strategies can reduce or even elimina te
very sensitive to incrementai changes that
distorting reactivity.
may be important to the people involved.
For staff development or leadership training
State-of-the-Art programs that include enhanced self-esteem
Considerations: Lack of Proven as an outcome goal, it may be more useful to
Quantitative Instrumentation do case studies to document changes experi-
enced by participants rather than rely on a
Another reason for using qualitative standardized measurement scale of prob-
methods is that for particular phenomena or lematic relevance and sensitivity.
The same point can be made with regard curred at a time when farnily violence and
to controversy surrounding even long- child sexual abuse were just emerging into
standing measureinent instruments. The societal consciousness and as a focus of
use of standardized achievement tests to scholarly inquiry. Exploratory work of this
measure student learning is a prime exam- kind is the way that new fields of inquiry are
ple. Strong arguments have been made at- developed, especially in the policy arena.
tacking the relevance of universal, standard-
ized achievement tests for the evaluation of Confirmatory and
particular local programs (Perrone 1977). Elucidating Research: Adding
The way in which norm-referenced, stan- Depth, Detail, and Meaning
dardized achievement tests are constructed to Quantitative Analyses
reduces their relevance and validity for par-
ticular local programs, especially those that At the opposite end of the continuum
serve populations where scores are likely to from exploratory research is the use of quali-
cluster at the lower or higher extremes of the tative methods to add depth and detail to
normal curve. For such programs, more ac- completed studies that used quantitative
curate evaluation results can be produced data where the statistical results indicate
through documentation of actual student global patterns generalizable across settings
portfolios, that is, developing case histories or populations. For example, when a
of what students can do and have done over large-scale survey has revealed certain
time rather than relying on their responses marked and significant patterns of re-
to a standardized instrument administered sponses, it is often helpful to fill out the
under artificial conditions at a moment in meaning of those patterns through in-depth
time (Carrni 1975,1979; Buxton 1982). study using qualitative methods. The quan-
A related state-of-the-art consideration is titative data identify areas of focus; the qual-
explora tory research. In new fields of study itative data give substance to those areas of
where little work has been done, few defini- focus. Consider: What did people really
tive hypotheses exist and little is known mean when they marked that answer on the
about the nature of the phenomenon, quali- questionnaire? What elaborations can re-
tative inquiry is a reasonable beginning spondents provide to clarify responses?
point for research. Excellent examples of How do the various dimensions of analysis
such exploratory research are Angela fit together as a whole from the perspective
Browne's (1987) study When Battered Women of respondents? Follow-up interviews with
Kill; a qualitative study of female child sex- a subsample of respondents can provide
ual offenders in a Minnesota treatment pro- meaningful additional detail to help make
gram (Mathews, Matthews, and Speltz sense out of and interpret survey results.
1989); follow-up interviews documenting Qualitative data can put flesh on the bons
the effects of reunification on sexually abu- of quantitative results, bringing the results
sive families (Matthews, Raymaker, and to life through in-depth case elaboration.
Speltz 1991); Jane Gilgun's (1991) work on Moreover, while the role of qualitative re-
the resilience of and intergenerational trans- search in exploratory inquiry is relatively
mission of child sexual abuse; and related well understood, the confirmatory and elu-
frontline, sinall- scale studies of farnily sex- cidating roles of qualitative data are less well
ual abuse (Patton 1991). These studies oc- appreciated. Adding depth and detail to sta-
tistical findings is one aspect of confirmation spend less time planning and more time out
and elucidation. Within major traditions of in the trenches talking to people and watch-
theory-oriented qualitative inquiry (Chap- ing what's going on.
ter 3), qualitative methods are also the meth- In crisis epidemiological work, as in the
ods of choice in extending and deepening outbreak of highly contagious diseases (e.g.,
the theoretical propositions and under- the Ebola virus in frica) or the emergence
standings that have emerged from previous of AIDS, rapid reconnaissance teams made
field studies. In short, qualitative inquiry is up of medicai personnel, public health re-
not just for exploratory purposes. searchers, and social scientists are deployed
to investigate the crisis and determine im-
Rapid Reconnaissance mediate interventions and longer-term ac-
tions needed. The film And the Band Played
Sometimes information is needed On, about the early incidences of the HIV vi-
quickly. Indeed, this is increasingly the case rus before it had been identified and named,
in our rapidly changing world. There may shows the intensity of this kind of inter-
be no time to search the literature, develop view-based, snowball-sampling (getting
hypotheses, select probabilistic samples new contacts from each person inter-
based on definitive population enumera- viewed), and field-based crisis reconnais-
tions, and develop, pilot test, and administer sance inquiry.
new instruments. One major advantage of In the late 1980s, cooperative extension
qualitative methods is that you can get into services in the United States adopted a new
the field quickly. approach to programming based on rapid
Experimental, deductiv, hypothesis- response to emergent issues (Patton 1987b,
testing strategies can require a lot of 1988a; Dalgaard et al. 1988). This meant do-
front-end work. You've got to be quite cer- ing ongoing "environmental scanning,"
tain about design and instrumentation before which included content analysis of newspa-
data collection because once the study is un- pers and periodicals, conducting focus
der way, changes in design and measure- groups with emergingnew client groups, in-
ment undermine both internai and externai terviewing key informants well placed to
validity. Naturalistic inquiry, in contrast, identify cutting-edge issues, and making
permits the researcher to enter the field with systematic observations of what is happen-
relatively little advance conceptualization, ing in counties and regions throughout the
allowing the inquirer to be open to whatever United States. Issues teams worked together
becomes salient to pursue. The design is using both quantitative and qualitative in-
emergent and flexible. The questions unfold formation to identify trends, scan the envi-
as the researcher pursues what makes sense. ronment, and formulate new programs
Rapid reconnaissance or "rapid assess- based on emergent issues. These teams
ment" (Beebe 2001) connotes doing field- sometimes undertake rapid reconnaissance
work quickly, as does "quick ethnography" fieldwork to get detailed, descriptive infor-
(Handwerker 2001). In our highly dynamic mation about a developing situation, for ex-
world, it's important to stay close to the ac- ample, an influx or out-migration of new
tion. Best-selling author Tom Peters (1987) people in an area; the impact of sudden eco-
called his field-based, close-to-the-action nomic changes in a county; the sudden ap-
management approach "thriving on chos." pearance of a crop disease or pest; or a rapid
He included the admonition to managers to increase in some problem such as teenage
pregnancies, crack cocaine-addicted new- nomena and assess quickly developing situ-
borns, homelessness, or elderly persons in ations in a world of rapid change.
need of long-term care. Action research
teams can do rapid reconnaissance on these Capturing and
issues quickly, going where the action is, Communicating Stories
talking to people and observing what is hap-
pening. To keep my head straight I reserve certain
The farming systems approach to interna- words for specialized meaningscase study
tional development was developed using and ethnography are two such labels. I write
rapid reconnaissance teams (Shaner, Phil- stories, not case studies, although readers of
ipp, and Schmehl 1982a, 1982b). Interdisci- my stories may mistakenly call them ethno-
plinary teams conduct fieldwork and in- graphic research or case studies. Were I doing
formal interviews to construct an initial, ethnography or ethnology, which I never do, I
exploratory, and qualitative portrayal of the would have a much heavier burden. I would
farming system in a particular agroeco- have to address questions of validity, of theory
logical area. Through the fieldwork, which contribution, of completeness, of generality,
may last from one to three weeks, the teams of replicability.... A story documents a given
are able to identify system characteristics, milieu in an attempt to communicate the gen-
farmers' needs, extension possibilities, and eral spirit of things. The story need not test
new applied agricultural research priori- theory; need not be complete; and need not be
ties. The Caribbean Agricultural Extension robust in either time or depth. (Denny 1978:1)
Project, for example, used 10-day rapid re-
connaissance studies in each of 10 different In The Springboard, Stephen Denning
islands to assess needs and develop inter- (2001) explams "how storytelling ignites ac-
ventions for the extension services in those tion in knowledge-era organizations." He
countries (Alkin and Patton 1987; Alkin et al. teaches storytelling as a powerful and for-
1989; Patton 1988b). mal discipline for organizational change
In the farming systems literature, these and knowledge management. What he calls
rapid reconnaissance surveys are often "springboard" stories are those that com-
called sondeos after the Spanish word mean- municate new or envisioned strategies,
ing "sounding." A sondeo is a qualitative structures, identities, goals, and values to
"sounding out" of what is happening. Infor- employees, partners, and even customers.
mal interviews and observations are done He argues that storytelling has the power to
on farms and in homes to document and un- transform individuais and organizations.
derstand variations within some defined He offers as an example his frustrated ef-
geographical area. Once interventions be- forts, as director of knowledge management
gin, either research or extension interven- of the World Bank, to convince colleagues to
tions, the sondeos may be periodically re- share information throughout the organiza-
peated to monitor system changes and tion. His reports and graphs were not prov-
development, as well as to evaluate the ining effective in making his case, so he told
terventions. the staff a story. In 1995, a health care worker
The point here is that the very nature of in Zambia was searching for a treatment for
qualitative inquiry makes it possible to get malaria. He found what he needed at the
into the field quickly to study emerging phe- Web site of the Centers for Disease Control.
The story communicated what his memos Davies 1996) that involves several steps.
had not: the potential life-and-death impor- First, key program stakeholders and partici-
tance of having criticai knowledge available pants (e.g., farmers in an extension pro-
and easily accessible to any World Bank gram) come to an agreement on which "do-
worker in any remote place in the world. mains of change" to monitor with stories.
In another management book, Managing Second, monthly stories of change written
By Storying Around, David Armstrong (1992) by farmers and field staff are collected.
turns the noun story into a verb, storying, to Third, volunteer reviewers and evaluators
emphasize the direct and active impact of using agreed-on criteria select the "most sig-
constructing stories to influence organiza- nificant stories" during regional and state-
tional values and culture. Shaw, Brown, and wide committee meetings. Last, at the end of
Bromiley (1998) reported how capturing the year a document is produced containing
and using "strategic stories" helped the ali the "winning" stories. This document
multinational 3M company improve their forms the basis for a round-table discussion
business planning both interna lly, for those with "key influentials" and funders of the
involved, and externally, for those to whom project, who then also select the most signifi-
strategic results were reported: "A good cant stories according to their views. "This
story (and a good strategic plan) defines re- approach goes beyond merely capturing
lationships, a sequence of events, cause and and documenting client stories; each story is
effect, and a priority among itemsand those accompanied by the storyteller's interpreta-
elements are likey to be remembered as a complex tion, and after review the stories are also ac-
whole" (p. 42). companied by the reviewer 's interpretation.
Qualitative inquiry can be used to dis- One of the ideas behind the process is that it
cover, capture, present, and preserve the sto- promotes a slow but extensive dialog up and
ries of organizations, programs, communi- down the project hierarchy each month"
ties, and families. Barry Kibel (1999) (Dart 2000). See Exhibit 4.7 for an example of
developed a process for capturing the a "most significant change" story. This story
"success stories" of clients in programs also illustrates a qualitative approach to out-
and aggregating them in a method he comes documentation discussed earlier in
called "results mapping." His approach in- this chapter.
volves an arduous and rigorous coding pro- Cognitive scientists have found that sto-
cess that can be challenging to manage, but ries are more memorable and better support
the fundamental idea is that "for programs learning and understanding than nonstory
engaged in healing, transformation, and narratives (Shaw et al. 1998:42). Language
prevention, the best source and form of in- scholar Richard Mitchell (1979) has ob-
formation are client stories" (Kibel 1999:13). served, "Our knowledge is made up of the
Story collecting can be integrated into on- stories we can tell, stories that must be told
going program evaluation, monitoring, and in the language that we know.... Where we
development (organizational learning) pro- can tell no story, we have no knowledge"
cesses. For example, the Institute of Land (p- 34).
and Food Resources at the University of As I noted in the last chapter in discus-
Melbourne in Australia developed a story- sing narrative analysis, which focuses on
based change-monitoring approach called stories as a particular form of qualitative in-
"most significant changes" (Dart et al. 2000; quiry, the language of "storytelling" is less
Particularly Appropriate Qualitative Applications L3. 197
Example of a "Most Signficant Change" Story
Title: 'TH Not Be Miking Cows When l'm 55'

Name of person recordng story: Mark Saddington, dairy farmer
Region: Gippsland
Date of narration: Round 2August 21, 1998
Who was involved: Farmer and family
When did it happen: 1998
What happened? We did the pilot Dairy Business Focus, Program in March, and for the first
time, my wife carne along. We were abfe to look at our farm as a business, not just as a farm.
As a consequence of doing the program, we did a few sums and made a few decisions. We
worked out that we can afford to have her on the farm, and she has ieft her job at the bank.
We will generate enough tncome on the farm to make it more profitable for her to be here. The
kids will benefit from seeing her a lot more, and they won't be in day care. So far this year, this
has made the calving so much easier, we have a joint input, and it has been such a turn around
in my lifestyle. It has been so good.
We actually went to the accountantyesterdaytogetsomefinancial adviceon how weshould be
investing off-farm. He was amazed that what we are doing is treating the farm as a business.
I said: "Now everything that we earn on this farm is going to be put away so that I am not milking
cows when I am 55 years old!"
We have gota debt-reduction program runningforthenext 12 months, but after that the money
will be channeled to off-farm investment. I want to retire young enough to enjoy what we have
been working towards for the last 20 or 30 years. My boss is 77 and is still working on the farm. If I
am that fit when I am his age,! want to be touring around the world.
It has opened up our ves. We are now lookng at off-farm investment, as capita! investment
on-farm is not that great. We are not going to invest in new machinery but are going to invest in
contractors to do any work we can't do. There is no point buying new machinery, as it depreciates.
Instead, we wil! buy shares and invest off the farm. This proves that you can farm on 120 cows, you
don'thavetogetbig, andyoudon't have to milka lot of cows. Itjustdepends whatyou do with your
money. If only we could educate the younger farmers to think ahead instead buying the largest SS
Commodore or the latest dual cab. I followed the same track for a few years until we sat down and
worked out where we were going and where we could be. We made a few mistakes in the past, but
the past is the past.
Feedback from the statewide committee:

9
This story generated lots of discussion. But is it really about profitability or quality of life or
changes in farm practice?
The general consensus was that there needed to be more detail in the story for it to be clearly
about profitability.
B
t is a really powerful story that shows considerable change.
(continued)
E X H I B I T 4.7 Continued
Feedback from the round-tablc mtina:

The story showed strong evidence of attitudinal change, leading to self-improvement and
goal setting. These people will be high achievers and reap the rewards. They will be good role
models for others who desire similar rewards.
This approach is okay, but it isn't necessarily a prescription for others.
It has some good messages, but it hasn't got ali the answers.
This is a very good example of achieving the goal of the DBF Program: i.e., getting strategic
thinking/planning followed by farmer action.
liked this story as it highlights the diversity in personal goals and ways to get there.
SOURCE: Dart et al. (2000:8-9). Used with permission.
intimidating to nonresearchers than the to go back to the drawing board to search

language of "case studies" or "ethnogra- still deeper for the breakthrough that would
phy," words that sound heavy and aca- lead to a computer that could truly think.
demic. It makes quite a difference when talk- They made revisions and adaptations based
ing to people in a community to say, "We'd on ali their prior failures until finally they
like to hear and record your stories" versus felt more hopeful than ever that they had
"We'd like to do a case study of you." succeeded. The computer program passed
One of my favorite stories was told by an- ali the preliminary tests. They became in-
thropologist Gregory Bateson (1978), the creasingly excited. The moment arrived for
story of attempts to create a computer that the penultimate test. They asked the com-
could think like human beings. Two scien- puter: "Can you think like a human being?"
tists became particularly enthralled with The computer processed the question and
this idea and devoted their lives to the quest. on the screen appeared the answer: "That
Each time they thought they had succeeded, question reminds me of a story." They knew
their efforts failed criticai tests and they had then they had succeeded.
Legislative Auditing and Monitoring
W e've funded nine of these so-called regional transportation

coordinating boards around the state and we have no real idea who's
involved, how well they're workmg, or even what they're really doing. We
need an mdependent perspective on what the hell's going on. Think you can
find out?
Phone inquiry from a state legislator

On occasion some legislative body or generally, legislative staff members who are
board that has mandated and appropriated particularly interested in the program can be
funds to a new program wants information expected to read such case histories with
about the extent to which the program is op- some care. From a political point of view,
erating in accordance with legislative intent. programs are more likely to be in trouble or
Legislative intent may involve achieving cause trouble for legislators because they
certain outcomes or may focus on whether failed to follow legislative intent in imple-
some mandated delivery specifications are mentation rather than because they failed to
being followed. Sometimes the precise na- achieve desired outcomes. In this case, the
ture of the legislated delivery system is only purpose of legislative monitoring or audit-
vaguely articulated. For example, suchman- ing is to become the eyes and ears of the
dates as "deinstitutionalization," "decen- legislature or board. This means providing
tralization," "services integration," and program descriptions that are sufficiently
"community-based programming" involve detailed and evocative that the legislator or
varied conceptualizations of legislative in- legislative staff can read such descriptions
tent that do not lend themselves easily to and have a good idea of what that program
quantitative specification. Indeed, for the is really like. Having such descriptions en-
evaluator to unilaterally establish some ables legislators to decide whether their own
quantitative measure of deinstitutional- interpretations of legislative intent are being
ization that provides a global, numerical met. The observation of a parent education
summary of the nature of program opera- program reported in the first chapter is an
tions may hide more than it reveals. example of fieldwork done for the purpose
To monitor the complexities of program of monitoring legislative intent. There are
implementation in the delivery of govern- excellent program evaluation units within a
ment services, it can be particularly helpful number of state legislative audit commis-
to decision makers to have detailed case de- sions that use fieldwork to do policy re-
scriptions of how programs are operating search and evaluation for legislators. When
and what they're accomplishing. Such legis- done well, such fieldwork goes beyond sim-
lative monitoring would include descrip- ple compliance audits by using qualitative
tions of program facilities, decision making, methods to get at program processes, imple-
outreach efforts, staff selection procedures, mentation details, and variations in impacts.
the nature of services offered, and descrip- Detailed case histories may also be of con-
tions from clients about the nature and re- siderable service to the programs being
sults of their experiences. An exemplar of monitored because a case history permits
such an effort is an in-depth study of "trans- them to tell their own story in some detail.
portation partnerships" throughout Minne- Thus, where they have deviated from legis-
sota that included case studies in each relative intent, such case histories would be
gion and an extensive cross-case analysis of expected to include information from pro-
patterns and variations (DeCramer 1997). gram administra tors and staff about con-
Busy legislators cannot be expected to straints under which the program operates
read in detail a large number of such histo- and the decisions staff has made that give
ries. Rather, legislators or funders are likely the program its character. At the same time,
to be particularly interested in the case histo- the collection of such case histories through
ries of those programs that are within their site visits and program monitoring need not
own jurisdiction or legislative district. More neglect the possibility for including more
global statements about statewide patterns cases. Thus, qualitative methods used for
in programs, or even nationwide patterns. It legislative monitoring allow one to docu-
is quite possible through content analysis to ment common patterns across programs as
identify major patterns of program opera- well as unique developments within specific
tions and outcomes for a number of separate programs.
Futuring Applications: Anticipatory Research

and Prospective Policy Analysis
am often surprised, but I am never taken by surprise.
Robert E. Lee, U.S. Civil War general
Lee's statement captures, in essence, eas called Project Future. Part of the devel-
what it means to anticipate the future. Our opment effort involved teams of community
rapidly changing world has increased inter- members interviewing people in their own
est in and need for futures studies. Such communities about their visions, expecta-
work has moved beyond the supermarket tions, hopes, and fears for the future. The
tabloids to become a focus of scholarly in- community teams then analyzed the results
quiry (e.g., Helmer 1983). Much futures and used them to construct alternative fu-
work involves statistical forecasting and ture scenarios for their community. The
computer simulations. There are, however, community next reviewed these scenarios,
also qualitative inquiry futuring research changed them through discussion, and se-
strategies. le cted a future to begin creating. (This is also
One important futuring tool is scenario a form of participatory research discussed
construction (Edmunds 1978; Godet 1987; earlier in this chapter.) Ethnographic futures
Fitzsimmons 1989). Scenarios are narrative research can be part of a community devel-
portrayals of future systems. A scenario can opment process (Domaingue 1989; Textor
be constructed for an organization, a com- 1980).
munity, a farming system, a society, or any The Evaluation Unit of the U.S. General
other unit of interest. Useful scenarios are Accounting Office (GAO) developed meth-
highly descriptxve. One technique for writ- ods for "prospective studies" to help policy-
ing scenarios is to base the scenario on imag- makers anticipate the implications and con-
ined future fieldwork. The scenario is writ- sequences of proposed laws. Prospective
ten as if it were a qualitative study of that studies can include interviewing key knowl-
future system. As such, it would include in- edgeables in a field to solicit the latest and
terview results, observational findings, and best thinking about a proposal, sometimes
detailed descriptions of the phenomenon of f eeding back the findings for a second round
interest. of interviews (a qualitative Delphi tech-
Qualitative methods can also be used to nique). Prospective methods can also in-
gather the data for scenario development. clude doing a synthesis of existing knowl-
The Minnesota Extension Service undertook edge to pull together a research base that
a community development effort in rural ar- will help inform policy making. The GAO
Somebody sent some

researchers here think-
ing they could get some
interviews from people
interested in the long-
term future. But I get off
work in just 15 minutes.
Are you interested?
handbook Prospective Methods (1989) includes futuring research efforts. Constructing fu-
attention to qualitative and quantitative syn- tures scenarios can also be an effective way
thesis techniques, with particular attention to contextualize evaluation findings and rec-
to the problem of drawing together diverse ommendations, to help decision makers
data sources, including case studies. Indeed, think about the varying future conditions
the Prospective Methods guidebook presents that could affect implementation of alterna-
much of the material in a case study format. tive recommendations (Patton 1988d, 1997a:
Rapid reconnaissance fieldwork can also 328-29).
be used for anticipatory or futuring re- In summary, while most evaluation work
search. Being able to get into the field involves looking at the effectiveness of past
quickly to get a sense of emerging develop- efforts in order to improve the future effec-
ments can be criticai to futures-oriented tiveness of interventions, a futuring per-
needs assessment techniques and forward- spective involves anticipatory research and
looking planning processes. The content forward thinking in order to affect current
analysis techniques of qualitative inquiry, actions toward creating desirable futures.
especially media analysis (Naisbitt and Qualitative inquiry can play a role in both
Aburdene 1990; Merriam and Makower studies of the past and anticipatory research
1988; Naisbitt 1982), are central to many on the future.
Breaking the Routine: find that the collection of qualitative data

Generating New Insights has become a routine and that new insights
can be gained through even the temporary
Being futures oriented includes being use of some quantitative measures. This
open and flexible. Both individuais and or- suggestion for periodically changing meth-
ganizations can easily become trapped in ods derives from a concern with enhancing
routine ways of doing things, including rou- the use of evaluation research. Given the
tine ways of thinking about and conducting ease with which human beings and social
evaluations or engaging in research. Thus, I systems settle into inertia and routine, eval-
end this chapter with one final rationale for uators who want their results to make a dif-
using qualitative methods: breaking the rou- ference need to find creative ways to get peo-
tine or "making the familiar strange" ple to deal with the empirical world.
(Erickson 1973:10). Programs and organiza- Exploring methodological variations may
tions that have established ongoing evalua- be one such approach.
tion systems or management information It is also worth noting that evaluators can
approaches may have become lulled into a settle into routines and inertia. Evaluators
routine of producing statistical tables that who have been using the same methods
are no longer studied with any care. Inertia over and over may have lost the cutting
and boredom can seriously reduce the use- edge of their own creativity. Utilization-
fulness of program evaluation results. After focused evaluators need to have at their dis-
program staff or other decision makers have posal a large repertoire of possible data
seen the same statistical results used in the collection techniques and approaches. Eval-
same kinds of statistical tables year after uators can be more helpful to programs if
year, those results can begin to have a numb- they themselves are staying alert to the
ing effect. Even though the implications of many possibilities available for looking at
those results may vary somewhat from year the world. Indeed, a change in methods may
to year, the very format used to report the do as much or more to reenergize the evalua-
data can reduce the impact of the reports. tor as it does to renew the program evalua-
Mao Tse-tung commented on the tention process.
dency of human beings to settle into numb-
ing routines when he said that a revolution is
! Summary: A Vision of the
needed in any system every 20 years. Revo-
Utility of Qualitative Methods
lutions in the collection of evaluation data
may be needed much more often. One such As I was completing this book, my local cof-
revolution may be to introduce a totally new fee shop went out of business. It had been
approach to evaluation simply for the pur- open only two years, having been part of a
pose of attracting renewed attention to the surge of new coffee shops that sprang up
evaluation process. At the time, changing throughout the United States as Americans
the method may produce new insights or at fell in love with cappuccino. This particular
least force people to deal with the old in- coffee shop was unique because it was also
sights in a new way. a biker hangout. That is, it catered to aficio-
Of course, collection of qualitative data nados of large Harley-Davidson motorcy-
can also become routine. Programs based on cles affectionately called "Hawgs." The
a humanistic philosophy and/or programs owners, Scott and Connie, a young hus-
with an emphasis on individualization may band-and-wife team, were also bikers. Be-
cause the coffee shop was only a block from tive methods, especially ethnography, "be-
my office, I had become a regular, despite cause it provides a window onto the ways
my lacking the appropriate leather attire consumers interact with products in their
and loud steel machine. One morning, Con- everyday lives" (p. 377). Designers, in devel-
nie mentioned that she had decided to have oping innovative products and services,
laser surgery to correct her nearsighted vi- must be concerned with satisfying the needs
sion. I left for a trip wishing her a positive of users of their products. Qualitative meth-
outcome from the surgery. When I returned ods, including extensive use of videotape to
two weeks later the coffee shop was closed capture people using products and services,
and I found a barely legible, handwritten offer designers insights into the cultural and
note on the door: environmental contexts within which real
consumers use real products. Exhibit 4.9 of-
Closed indefinitelyvision problem fers examples of Intemet-based resources
that can help you monitor new develop-
I made inquiries at a nearby gasoline sta- ments in qualitative applications.
tion, but ali I learned was that the shop had The emphasis in this chapter on practical
closed very suddenly without notice. Some and useful applications stands in contrast to
three weeks later, I happened to see Scott the philosophical and theoretical focus of
riding his motorcycle on the street and the previous chapter. Taken together, these
waved him over to ask how Connie was do- two chapters demonstrate the importance of
ing. He said she was fine. What about the qualitative inquiry to both social science the-
sign on the coffee shop door? That had noth- ory and practice-oriented inquiry, especially
ing to do with Connie's surgery, he ex- in evaluation and organizational develop-
plained. "I just couldn't vision myself serv- ment.
ing coffee the rest of my life." Practical applications of qualitative meth-
This chapter has been aimed at helping ods emerge from the power of observation,
you decide if you can envision yourself in- openness to what the world has to teach, and
terviewing people, doing fieldwork, con- inductive analysis to make sense out of the
structing case studies, and otherwise using world's lessons. While there are elegant
qualitative methods in practical applica- philosophical rationales and theoretical un-
tions. Exhibit 4.8 lists the applications of- derpinnings to qualitative inquiry, the prac-
fered in this chapter, situations and ques- tical applications come down to a few very
tions for which qualitative methods are basic and simple ideas: Pay attention, listen
particularly appropriate. These are by no and watch, be open, think about what you
means exhaustive of the possibilities for ap- hear and see, document systeinatically
plying qualitative approaches, but they sug- (memory is selective and unreliable), and
gest the wide range of arenas in which quali- apply what you learn.
tative methods can be and are being used. The story of the invention of modern run-
New applications of qualitative methods ning shoes illustrates these principies and
continue to emerge as people in a variety of provides a helpful metaphor to close this
endeavors discover the value of in-depth, chapter. The design of sneakers varied little
open-ended inquiry into people's perspec- until the 1960s when competitive runners
tives and experiences. For example, Wasson began to turn to lighter-weight shoes. Re-
(2000) has reported that members of the deducing the weight of shoes clearly improved
sign profession have begun using qualita- performance, but problems of traction re-
Qualitative Inquiry Applications: Summary Checkst of

Particularly Appropriate Uses of Qualitative Methods
Studies Focusing on Quality

Understanding and illuminating quality
Quality assurance
Evaluation Applications
Outcomes evaluation
Evaluating individualized outcomes
Process studies
Implementation evaluation
Logic models and theories of action
Evaluability assessments
Comparing programs: Focus on diversity
Prevention evaluation
Documenting development over time and investigating system changes
Evaluation Models
Goal-free evaluation
Transaction models: Responsive and illuminative evaluation
Connoisseurship studies
Utilization-focused evaluation
Interactive and Participatory Applications
Personalizing and humanizing evaluation
Harmonizing program and evaluation values
Developmental applications: Action research, action learning, reflective practice, and
learning organizations
Appreciative inquiry
Participatory research and evaluation: Valuing and facilitating collaboration
Supporting democratic dialogue and deliberation
Supporting democracy through process use: Helping the citizenry weigh evidence and think
evaluatively
Special Applications
Unobtrusive measures
State-of-the-art considerations: Lack of proven quantitative instrumentation
Confirmatory and elucidating research: Adding depth, detail, and meaning to quantitative
analyses
Rapid reconnaissance
Capturing and communicating stories
Legislative monitoring and auditing
Futuring applications: Anticipatory research and prospective policy analysis
Breaking the routine: Generating new insights
mained. A running coach, Bill Bowerman, One morning while he was making waf-
went into the sneaker business in 1962. He fles, he had an idea. He heated a piece of rub-
paid close attention to the interest in ber in the waffle iron to produce the first
lighter-weight shoes and the problems of waffle-shaped sole pattern that became the
traction. world standard for running shoes (Panati
Sample Internet E-mail Discussion Groups (listservs) and

EXH1BIT 4.9 Sites Relevant to Qualitative Applications and Practice
1. EVALTALK@bama,ua.edu: American Evaluation Association (AEA) Discussion List; to

subscribe, send this message to [email protected]: subscribe evaltalk ourname
AEA home page with links to evaluation organizations, training programs, and
Internet resources: www.evai.org
2. [email protected]: Action Research Mailing List; to subscribe, send this message to

[email protected]: subscribe ARLIST-L
LASTNAME
3. [email protected]: Action research methodology network; to subscribe, send this

message to [email protected]: subscribe ARMNET-L firstname lastname
4. Organizational development Web site: www.mnodn.org
5. Organizational Development Network: www.ODNetwork.org
6. Evaluation Center at Western Michigan University: www.wmich.edu/evalctr/
7. World Bank Evaluation Unit Web site: www.wbln0018.worldbank.org/wbies/wbievalu.nsf
8. Empowerment Evaluation:
www.stanord.edu/~davidf/EmpowermentWorkshopCSAP/sld001.htm
9. [email protected]: International Visual Sociology Association; to subscribe,

send this message to [email protected]: subscribe ivsa ourname
NOTE: These sites and subscription details may change, noris this list exhaustive.This list ismcant to besug-
gestive of the qualitative analysis resources available through the internet. For other Internet resources, see
Chapter 1, Exhibit 1.5; Chapter 3, Exhibit 3.7; and Chapter 8, Exhibt 8.3.
1987:298-99). Subsequently, engineers and fieldwork and highly skilled interviewing,

computers would be used to design and test the essence of qualitative inquiry is paying
the best waffle patterns for different athletic attention, being open to what the world has
purposes. But the initial discovery came to show us, and thinking about what it
from Bowerman's paying attention, being means. C. Wright Mills (1961) quoted Nobel
open, making connections, drawing on per- Prize winning physicist Percy Bridgman in
sonal experience, getting a feel for what was this regard:
possible, exploration, documenting his ini-
tial results, and applying what he learned. There is no scientific method as such, but the
While later chapters will present in detail vital feature of the scientists procedures has
techniques for increasing the validity and re- been merely to do his utmost with his mind, no
liability of qualitative data through rigorous holds barred. (p. 58)
PART 2
Qualitative Designs
and Data Collection
Always be suspicious of data collection that goes according to plan.

Research subjects have also been known to be people.
The evaluator 's scientific observation is some person's real-life experience.
Respect for the latter must precede respect for the former.
Total trust and complete skepticism are twin losers in the field. Ali things in
moderation, especially trust and skepticism.
Evaluators are presumed guilty until proven innocent.
Make sure when you yield to temptation in the field that it appears to have
something to do with what you are studying.
A fieldworker should be able to sweep the floor, carry out the garbage, carry in
the laundry, cook for large groups, go without food and sleep, read and write by
candlelight, see in the dark, see in the light, cooper ate without offending, sup-
press sarcastic remarks, smile to expressboth pain and hurt, experiencebothpain
and hurt, spend time alone, respond to orders, take sides, stay neutral, take risks,
avoid harm, be confused, seem confused, care terribly, become attached to noth-
ing The nine-to-five set need not apply.
Always carry extra batteries and getaway money.
From Halcolm's Fieldivork Lazus
!J. 207
Designing Qualitative Studies
~Uhe F i ^ s t S v a l u a t i o n
The young people gathered around Halcolm. "Tell us again, Teacher of Many
Things, about the first evaluation."
"The first evaluation," he began, "was conducted a long, long time ago, in an-
cient Babylon when Nebuchadnezzar was king. Nebuchadnezzar had just con-
quered Jerusalem in the third year of the reign of Jehoiakim, King of Judah. Now
Nebuchadnezzar was a shrewd ruler. He decided to bring carefully selected
children of Israel into the palace for special training so that they might be more
easily integrated into Chaldean culture. This special program was the forerun-
ner of the compensatory education programs that would become so popular in
the 20 th century. The three-year program was royally funded with special allo-
cations and scholarships provided by Nebuchadnezzar. The ancient text from
the Great Book records that
the king spake unto Ashpenaz the master of his eunuchs that he shoud bring cer-
tain of the children of Israel, and of the king's seed, and of the princes; children in
whom was no blemish, but well-favored and skillful in ali wisdom, and cunning in
knowledge, and understanding science, and such as had ability m them to stand in
the king's palace, and whom they might teach the learning and the tongue of the
Chaldeans.
And the king appointed them a daily provision of the king's meat, and of the
wine which he drank; so nourishing them three years, that at the end thereof they
might stand before the king. (Daniel 1:3-5)
!5. 209
210 [fj. QUALITATIVE DESIGNS AND DATA COLLECTION
"Now this program had scarcely been established when the program director,
Ashpenaz, who happened also to be prince of the eunuchs, found himself faced
with a student rebellion led by a radical named Daniel, who decided for religious
reasons that he would not consume the king's meat and wine. This created a seri-
ous problem for the director. If Daniel and his coconspirators did not eat their dor-
mitory food, they might fare poorly in the program and endanger not only future
program funding but also the program director's head! The Great Book says:
But Daniel purposed in his heart that he would not defile himself with the portion of
the king's meat, nor with the wine which he drank; therefore he requested of the
prince of the eunuchs that he might not defile himself.
And the prince of the eunuchs said unto Daniel, I fear my lord the king, who hath
appointed your meat and your drink; for why should he see your faces worse liking
than the children which are of your sort? Then shall ye make me endanger my head to
the king. (Daniel 1:8,10)
"At this point, Daniel proposed history's first educational experiment and pro-
gram evaluation. He and three friends (Hananiah, Mishael, and Azariah) would
be placed on a strict vegetarian diet for ten days, while other students continued
on the king's rich diet of meat and wine. At the end of ten days the program direc-
tor would inspect the treatment group for any signs of physical deterioration and
judge the productivity of Daniel's alternative diet plan. Daniel proposed the ex-
periment thusly:
Prove thy servants, I beseech thee, ten days; and let them give us pulse to eat, and wa-
ter to drink. Then let our countenances be looked upon before thee, and the counte-
nance of the children that eat of the portion of the king's meat: and as thou seest, deal
with thy servants.
Sohe consented to them in this matter, and proved them ten days. (Daniel 1:12-14)
"During the ten days of waiting Ashpenaz had a terrible time. He couldn't
sleep, he had no appetite, and he had trouble working because he was preoccu-
pied worrying about how the evaluation would turn out. He had a lot at stake. Be-
sides, in those days they hadn't quite worked out the proper division of labor so he
had to play the roles of both program director and evaluator. You s e e . . . . "
The young listeners interrupted Halcolm. They sensed that he was about to
launch into a sermon on the origins of the division of labor when they still wanted
to hear the end of the story about the origins of evaluation. "How did it turn out?"
they asked. "Did Daniel end up looking better or worse from the new diet? Did
Ashpenaz lose his head?"
"Patience, patience," Halcolm pleaded. "Ashpenaz had no reason to worry.
The results were quite amazing. The Great Book says that
at the end of ten days their countenances appeared fairer and fatter in flesh than ali the
children which did eat the portion of the king's meat.
Designing Qualitative Studies !fj. 211
Thus, Melzar took away the portion of their meat, and the wine that they should
drink; and gave them pulse.
As for these four children, God gave them knowledge and skill in ali learning and
wisdom; and Daniel had understanding in ali visions and dreams. Now at the end of
the days that the king had said he should bring them in, then the prince of the eunuchs
brought them in before Nebuchadnezzar. And in ali matters of wisdom and under-
standing, that the king inquired of them, he found them ten times better than ali the
magicians and astrologers that were in ali his realm. (Daniel 1:15-18,20)
"And that, my children, is the story of the first evaluation. Those were the good
old days when evaluations really got used. Made quite a difference to Ashpenaz
and Daniel. Now off with youand see if you can do as well."
From Halcolm's Evaluation Histories
A Meta-Evaluation other things, aside from a change in diet,

either group was involved in that might
have explained the outcomes observed.
A meta-evaluation is an evaluation of an
evaluation. A great deal can be learned 5. Possibility of interaction effects between
about evaluation designs by conducting a the diet and the students/ belief system
meta-evaluation of history's first program (i.e., potential Hawthorne and halo ef-
evaluation. Let us imagine a panei of experts fects).
conducting a rigorous critique of this evalu-
ation of Babylon's compensatory education 6. Outcome criteria vague: Just what is
program for Israeli students: "countenance"?
7. Outcome measurement poorly opera-

1. Small sample size (N = 4). tionalized and nonstandardized.
2. Selectivity bias because recruitment into 8. Single observer with deep personal in-
the program was done by "creaming," volvement in the program introduces
that is, only the best prospects among possibility of selective perception and
the children of Israel were brought into bias in the observations.
the program.
9. Validity and reliability data are not re-
3. Sampling bias because students were ported for the instruments used to mea-
self-selected into the treatment group sure the final, summative outcome ("he
(diet of pulse and water). found them ten times better than ali the
magicians and astrologers").
4. Failure to clearly specify and control the
nature of the treatment, thus allowing 10. Possible reactive effects from the stu-
for the possibility of treatment contami- dents' knowledge that they were being
nation because we don't know what evaluated.
Despite ali of these threats to internai va- The information he needed concerned the
lidity, not to mention externai validity, the consequences of that specific change and
information generated by the evaluations only that specific change. He showed no in-
appears to have been used. The 10-day for- terest in generalizing the results beyond
ma tive evaluation was used to make a major those four students, and he showed no inter-
decision about the program, namely, to est in convmcing others that the measures he
change the diet for Daniel and his friends. made were valid and reliable. Only he and
The end-of-program summative evaluation Daniel had to trust the measures used, and
conducted by the king was used to judge the so data collection (observation of counte-
program a success. (Daniel did place first in nance) was done in such a way as to be
his class.) Indeed, it would be difficult to meaningful and credible to the primary in-
find a more exemplary model for the uses of tended evaluation users, namely, Ashpenaz
evaluation in making educational policy de- and Daniel. If any bias existed in his obser-
cisions than this "first evaluation" con- vations, given what he had at stake, the bias
ducted under the auspices of Nebuchad- would have operated against a demonstra-
nezzar so many years ago. This case study is tion of positive outcomes rather than in fa-
an exemplar of evaluation research having vor of such outcomes.
an immediate, decisive, and lasting impact While there are hints of whimsy in the
on an educational program. Modern evalua- suggestion that this first evaluation was ex-
tion researchers, flailing away in seemmgly emplary, I do not mean to be completely fa-
futile efforts to affect contemporary govern- cetious. I am serious in suggesting that the
mental decisions, can be forgiven a certain Babylonian example is an exemplar of utili-
nostalgia for the "good old days" in Babylon zation-focused evaluation. It contains and il-
when evaluation really made a difference. lustrates ali the factors modem evaluation
But should the results have been used? researchers have verified as criticai from
Given the apparent weakness of the evalua- studies of utilization (Patton 1997a). The de-
tion design, was it appropriate to make a cision makers who were to use findings gen-
major program decision on the basis of data erated by the evaluation were clearly identi-
generated by such a seemingly weak re- fied and deeply involved in every stage of
search design? the evaluation process. The evaluation ques-
I would argue that not only was use im- tion was carefully focused on needed infor-
pressive in this case, it was also appropriate mation that could be used in the making of a
because the research design was exemplary. specific decision. The evaluation methods
Yes, exemplary, because the study was set up and design were appropriately matched to
in such a way as to provide precisely the in- the evaluation question. The results were
formation needed by the program director understandable, credible, and relevant.
to make the decision he needed to make. Feedback was immediate and utilization
Certainly, it is a poor research design to was decisive. Few modern evaluations can
study the relationship between nutrition meet the high standards for evaluation set
and educational achievement. It is even a by Ashpenaz and Daniel more than 3,000
poor design to decide if ali students should years ago.
be placed on a vegetarian diet. But those This chapter discusses some ways in
were not the issues. The question the direc- which research designs can be appropri-
tor faced was whether to place four specific ately matched to evaluation questions in an
students on a special diet at their request. attempt to emulate the exemplary match be-
Observing countenance
tween evaluation problem and research de- alternative purposes along a continuum
sign achieved in the Babylonian evaluation. from theory to action:
As with previous chapters, I shall emphasize
the importance of being both strategic and 1. Basic research: To contribute to funda-
practical in creating evaluation designs. Be- mental knowledge and theory
ing strategic begins with being clear about 2. Applied research: To illuminate a societal
the purpose of the intended research or eval- concern
uation.
3. Summative evaluation: To determine pro-
gram effectiveness
l Clarity About Purpose: 4. Formative evaluation: To improve a pro-
A Typology gram
Purpose is the controlling force in research. 5. Action research: To solve a specific prob-
Decisions about design, measurement, lem
analysis, and reporting ali flow from pur-
pose. Therefore, the first step in a research Basic and applied researchers publish in
process is getting clear about purpose. The scholarly journals, where their audience is
centrality of purpose in making methods other researchers who will judge their con-
decisions becomes evident from examining tributions using disciplinary standards of
rigor, validity, and theoretical import. In Did it accomplish his objectives? Should it
contrast, evaluators and action researchers be continued? Could the outcomes he ob-
publish reports for specific stakeholders served be attributed to the program? This is
who will use the results to make decisions, the kind of research we have come to call
improve programs, and solve problems. summative evaluationsumming up judg-
Standards for judging quality vary ments about a program to make a major de-
among these five different types of research. cision about its value, whether it should be
Expectations and audiences are different. continued, and whether the demonstrated
Reporting and dissemination approaches model can or should be generalized to and
are different. Because of these differences, replicated for other participants or in other
the researcher mustbe clear at the beginning places.
about which purpose has priority. No single Now imagine that researchers from the
study can serve ali of these different pur- University of Babylon wanted to study the
poses and audiences equally well. With clar- diet as a manifestation of culture in order to
ity about purpose and primary audience, the develop a theory about the role of diet in
researcher can go on to make specific design, transmitting culture. Their sample, their
data-gathering, and analysis decisions to data collection, their questions, the duration
meet the priority purpose and address the of fieldwork, and their presentation of re-
intended audience. sults would ali be quite different from the
In the Babylonian example, the purpose formative evaluation undertaken by Ash-
was simply to find out if a vegetarian diet penaz and Daniel. The university study
would negatively affect the healthy appear- would have taken much longer than 10 days
ances (countenances) of four participants and might have yielded empirical general-
not zvhy their countenances appeared izations and contributions to theory, yet
healthy or not (a causai question), but would not have helped Ashpenaz make his
luhether the dietary change would affect simple decision. On the other hand, we
countenance (a descriptive question). The might surmise that University of Babylon
design, therefore, was appropriately simple scholars would have scoffed at an evalua-
to yield descriptive data for the purpose of tion done in 10 days, even a formative one.
making a minor program adjustment. No Different purposes. Different criteria for
contribution to general knowledge. No test- judging the research contribution. Different
ing or development of theory. No general- methods. Different audiences. Different
izations. No scholarly publication. No elab- kinds of research.
orate report on methods. Just find out what These are examples of how purpose can
would happen to inform a single decision vary. In the next section, I shall present a
about a possible program change. The par- more formal framework for distinguishing
ticipants in the program were involved in these five different research purposes and
the study; indeed, the idea of putting the diet examine in more depth the implications of
to an empirical test originated with Daniel. varying purposes for making design deci-
In short, we have a very nice example of sim- sions. Previous chapters have presented the
ple formative evaluation. nature and strategies of qualitative inquiry,
The king's examination of program par- philosophical and theoretical foundations,
ticipants at the end of three years was quite and practical applications. In effect, the
different. We might infer that the king was reader has been presented with a large ar-
judging the overall value of the program. ray of options, alternatives, and variations.
How does one sort it ali out to decide what work to generate new theories or test exist-
to do in a specific study? The answer is to get ing theories. Doctoral students are typically
clear about purpose. The framework that expected to make theoretical contributions
follows is meant to facilitate achieving this in their dissertations. Theories encapsulate
necessary clarity about purpose while also the knowledge of a discipline.
illustrating how one can organize a mass of Basic researchers are interested in formu-
observations into some coherent typology lating and testing theoretical constructs and
a major analytical tool of qualitative in- propositions that ideally generalize across
quiry. The sections that follow examine each time and space. The most powerful kinds of
type of research: basic research, applied re- findings in basic science are those findings
search, summative evaluation research, for- that are universal, such as Boyle's law in
mative evaluation, and action research. physics that the volume of a gas at constant
temperature varies inversely with the pres-
Basic Research sure exerted on it. Basic researchers, then,
are searching for fundamental patterns of
The purpose of basic research is knowl- the uni verse, the earth, nature, society, and
edge for the sake of knowledge. Research- human beings. For example, biologists have
ers engaged in basic research want to under- discovered that "changes in DNA are inher-
stand how the world operates. They are in- ited, but changes in proteins (specifically, in
terested in investigating a phenomenon to their amino acid sequence) are n o t . . . , per-
get at the nature of reality with regard to that haps the only universal truth biologists
phenomenon. The basic researcher 's pur- have" (Smith 2000:43). Social science, to
pose is to understand and explain. date, is markedly short of "universal truths."
Basic researchers typically work within Nevertheless, generalizations across time and
specific disciplines, such as physics, biology, space remain the Holy Grail of basic re-
psychology, economics, geography, and so- search and theory.
ciology. The questions and probleins they The findings of basic research are pub-
study emerge from traditions within those lished in scholarly books, journals, and dis-
disciplines. Each discipline is organized sertations. Each discipline has its own tradi-
around attention to basic questions, and the tions, norms, and rules for deciding what
research within that discipline derives from constitutes valid research in that discipline.
concem about those basic questions. Exhibit To be published in the major journals of any
5.1 presents examples of fundamental ques- particular discipline, scientists must engage
tions in several disciplines. in the kind of research that is valued by the
The fundamental questions undergirding researchers in that disciplinary tradition.
each discipline flow from the basic concerns Chapter 3 reviewed theoretical traditions
and traditions of that discipline. Researchers closely associated with qualitative inquiry,
working within any specific disciplinary tra- for example, ethnography and phenomenol-
dition strive to make a contribution to ogy. Qualitative inquiry also contributes to
knowledge in that discipline and thereby basic research through inductive theory de-
contribute to answering the fundamental velopment, a prominent example being the
questions of the discipline. The most presti- "grounded theory" approach of Glaser and
gious contribution to knowledge takes the Strauss (1967), essentially an inductive strat-
form of a theory that explains the phenome- egy for generating and confirming theory
non under investigation. Basic researchers that emerges from close involvement and di-
lUIBH Fundamental Disciplinary Questions
Anthropology What is the nature of culture?

How does culture emerge?
How is it transmitted?
What are the functions of culture?
Psychology Why do individuais behave as they do?

How do human beings behave, think, feel, and know?
What is normal and abnormal in human development and behavior?
Sociology What holds groups and societies together?

How do various forms of social organization emerge and what are their
functions?
What are the structures and processes of human social organizations?
Political science What is the nature of power?

How is power organized, created, distributed, and used?
Economics How do societies and groups generate and distribute scarce resources?
How are goods and services produced and distributed?
What is the nature of wealth?
Geography What is the nature of and variations in the earth's surface and atmosphere?
How do various forms of life emerge in and relate to variations in the earth?
What is the relationship between the physical characteristics of an area
and the activities that take place in that area?
Biology What is the nature of life?

What are variations in the forms of life?
How have life forms emerged and how do they change?
rect contact with the empirical world. Basic such as those in rural Mxico, typically inno-
qualitative research typically requires a rela- vate and develop specialties in an attempt to
tively lengthy and intensive period of field- establish a niche for themselves in a complex
work. The rigor of field techniques will be economic environment. Chibnik's (2000) ba-
subject to peer review. Particular attention sic research on commercial woodcarving in
must be given to the accuracy, validity, and Oaxaca has led to the theory that such mar-
integrity of the results. ket segmentation resembles the later stages
An example of interdisciplinary theory of product life cycles described in the busi-
development comes from work in basic eco- ness literature and is somewhat analogous
nomic anthropology studying craft com- to the proliferation of equilibrium species in
mercialization and the product differentia- mature or climax stages of ecological succes-
tion that ordinarily accompanies increased sions. Chibnik examined both market de-
craft sales. Artisans in emerging markets, mands and the initiative of artisans and
found that local artisans do not have total on environmental studies often involves re-
freedom in their attempts to create market searchers from a number of disciplines. In
niches since they are restricted by their abili- agricultural research, the field of integrated
ties and the labor and capital they can mobi- pest management (IPM) includes research-
lize. This is a classic example of interdisci- ers from entomology, agronomy, agricul-
plinary theory generation and testing tural economics, and horticulture. Fields of
bridging economics, ethnology, and ecology. interdisciplinary research in the social sci-
ences include gerontology, criminal justice
Applied Research studies, women's studies, and family re-
search. Exhibit 5.2 offers examples of ap-
Applied researchers work on human and plied interdisciplinary research questions for
societal problems. In the example just cited, economic anthropology, social psychology,
had Chibnik examined the problem of creat- political geography, and educational and or-
ing new markets for rural artisans and of- ganizational development. Notice the dif-
fered possible solutions for increased mar- ference between these questions and the
keting, the work would have constituted kinds of questions asked by basic research-
applied rather than basic research. The pur- ers in Exhibit 5.1. Applied researchers are
pose of applied research is to contribute trying to understand how to deal with a sig-
knowledge that will help people understand nificant societal problem, while basic re-
the nature of a problem in order to intervene, searchers are trying to understand and ex-
thereby allowing human beings to more ef- plain the basic nature of some phenomenon.
fectively control their environment. While in Applied qualitative researchers are able
basic research the source of questions is the to bring their personal insights and experi-
traditions within a scholarly discipline, in ences into any recommendations that may
applied research the source of questions is in emerge because they get especially close to
the problems and concerns experienced by the problems under study during fieldwork.
people and articulated by policymakers. Audiences for applied research are typically
Applied researchers are often guided by policymakers, directors and managers of in-
the findings, understandings, and explana- tervention-oriented organiza tions, and pro-
tions of basic research. They conduct studies fessionals working on problems. Timelines
that test applications of basic theory and dis- for applied research depend a great deal on
ciplinary knowledge to real-world problems the timeliness and urgency of the problem
and experiences. The results of applied re- being researched. A good example of ap-
search are published in journals that special- plied research is Emerging Drug Problems, a
ize in applied research within the traditions work sponsored by the U.S. General Ac-
of a problem area or a discipline. counting Office (1998) that examined new
Societal concerns have given rise to a vari- street drugs, recent research on addiction,
ety of new fields that are interdisciplinary in and alternatives for public policy.
nature. These emerging fields reflect the In contrast to basic researchers, who ulti-
long-standing crticism by policymakers mately seek to generalize across time and
that universities have departments but soci- space, applied research findings typically
ety has problems. Applied interdisciplinary are limited to a specific time, place, and con-
fields are especially problem oriented rather dition. For example, a researcher studying
than discipline oriented. For example, work the nature of family problems in the 1980s
E X H I B I T 5.2 Sample Interdisciplinary Applied Research Questions
Applied economic anthropology How can the prosperous economy of an isolated, small minor-
ity group be preserved when that group encounters new com-
petition from the encroaching global economy?
Applied social psychology How can a group within a large organization develop cohe-
sion and identity within the mission and vaues of its parent
structure and culture?
Applied political geography How can people of previously isolated towns, each with its
own history of local governance, come together to share
power and engage in joint decision making at a regional
levei?
Applied educational and How can students from different neighborhoods with varied
organizational development ethnic and racial backgrounds be integrated in a new magnet
school?
would not expect those problems to be the tial solutions, evaluations examine and
same as those experienced by families in the judge the processes and outcomes aimed at
1880s. While the research might include attempted solutions. Evaluators study pro-
making such comparisons, applied re- grams, policies, personnel, organiza tions,
searchers understand that problems emerge and products. Evaluation research can be
within particular time and space bound- conducted on virtually any explicit attempt
aries. to solve problems or bring about planned
change. As illustrated in the Daniel story of
Evaluation Research: history's "first evaluation" that opened this
Summative and Formative chapter, evaluators distinguish two quite
different purposes for evaluation: (1) sum-
Once solutions to problems are identi- mative evaluations that judge overall effec-
fied, policies and programs are designed to tiveness to inform major decisions about
intervene in society and bring about change. whether a program should continue and (2)
It is hoped that the mtervention and changes formative evaluations that aim to improve
will be effective in helpmg to solve prob- programs.
lems. However, the effectiveness of any Summative evaluations serve the pur-
givenhuman intervention is a matter subject pose of rendering an overall judgment about
to study. Thus, the next point on the theory- the effectiveness of a program, policy, or
to-action research continuum is the conduct product for the purpose of saymg that the
of evaluation and policy research to test the evaluand (thing being evaluated) is or is not
effectiveness of specific solutions and hu- effective and, therefore, should or should
man interventions. not be continued, and has or does not have
While applied research seeks to under- the potential of being generalizable to other
stand societal problems and identify poten- situations. A summative decision implies a
summing-up judgment or a summit (from tion) funding to more permanent funding,

the mountaintop) decision, for example, to or it may lead to program or policy termina-
expand a pilot program to new sites or move tion. Summative evaluations seldom rely
it from temporary (pilot test or demonstra- entirely, or even primarily, on qualitative
data and naturalistic inquiry because of de- evaluation takes place. Formative evalua-
cision makers' interest in measuring stan- tions rely heavily on process studies,
dardized outcomes, having controlled com- implementation evaluations, case studies,
parisons, and making judgments about and evaluability assessments (see Chapter
effectiveness from relatively larger samples 4). Formative evaluations often rely heavily,
with statistical pre-post and follow-up re- even primarily, on qualitative methods.
sults. Qualitative data in summative evalua- Findings are context specific.
tions typically add depth, detail, andnuance Although formative and summative re-
to quantitative findings, rendering insights main the most basic and classic distinctions
through illuminative case studies and examin evaluation, other evaluation purposes
ining individualized outcomes and issues of have emerged in recent years in "a world
quality or excellenceapplications dis- larger than formative and summative"
cussed in Chapter 4. Harkreader and Henry (Patton 1996b). New purposes include ongo-
(2000) have provided an excellent discus- ing "developmental evaluation" for pro-
sion of the challenges of rendering sum- gram and organizational development and
mative judgments about merit and worth; learning (Patton 1994; Preskill and Torres
they use as their example comparative 1999); empowering local groups through
quantitative performance data from Gergia evaluation partcipation (Fetterman 2000a;
schools to assess a democratic reform initia- Patton 1997b); and using the processes of
tive. Fetterman (2000b) shows how qualita- evaluation (process use) to build staff capac-
tive data can be the primary basis for a ity for data-based decision making and con-
summative evaluation. His evaluation of tinuous improvement (Patton 1997a: 87-113,
STEP, a 12-month teacher education pro- 1998). For our analysis here, however, these
gram in the Stanford University School of and related approaches to evaluation share
Education, included fieldwork immersion the general purpose of improvement and
in the program, open-ended interviews with can be included within the broad category
ali students, focus groups, observatons of of formative research along our theory-to-
classrooms, interviews with faculty, digital action continuum. In addition, some evalua-
photography of classroom activities, and tion studies are now designed to generate
qualitative analysis of curricular materiais, generalizable knowledge about effective
as well as a variety of surveys and outcome practices across different projects or pro-
measures. The summative evaluations of a grams based on cluster evaluations, lessons
democratic reform initiative m Gergia and learned, "better" practices, and meta-analy-
of Stanford's STEP program both followed ses (Patton 1997a:70-75). This knowledge-
and built on extensive formative evalua- generating approach to evaluation research,
tion work, the purpose of which we now to the extent that it aims to discover general
examine. principies of effective practice rather than
Formative evaluations, in contrast to render judgment about the merit or worth
summative ones, serve the purpose of im- of a specific intervention, falls roughly
proving a specific program, policy, group of within the category "applied research" in
staff (in a personnel evaluation), or product. this theory-to-action continuum. However,
Formative evaluations aim at forming (shap- the emergence and increased importance of
ing) the thing being studied. No attempt is knowledge-generating evaluations illus-
made in a formative evaluation to generalize trate why these five categories (basic, ap-
findings beyond the setting in which the plied, summative, formative, and action re-
search) cannot be thought of as fixed or informal, the people in the situation are of-
exhaustive; rather, this typology provides ten directly involved in gathering the
general guidance to major formations in the information and then studying themselves,
research landscape without charting every and the results are used internally to attack
hill and valley in that varied and complex specific problems within a program, organi-
territory that research has become. zation, or community. While action research
may be used as part of an overall organiza-
tional or community development process,
Action-Oriented, it most typically focuses on specific prob-
Problem-Solving Research lems and issues within the organization or
community rather than on the overall ef-
The final category along the theory-to- fectiveness of an entire program or organi-
action continuum is action research. Action zation. Thus, along this theory-to-action-
research aims at solving specific problems research continuum, action research has the
within a program, orgariization, or commu- narrowest focus.
nity. Action research explicitly and purpose- The findings of formative evaluation and
fully becomes part of the change process by action research are seldom disseminated be-
engaging the people in the program or orga- yond the immediate program or organiza-
nization in studying their own problems in tion within which the study takes place. In
order to solve those problems (Whyte 1989). many instances, there may not even be a full
As a result, the distinction between research written research report. Publication and dis-
and action becomes quite blurred and the re- semination of findings are more likely to be
search methods tend to be less systematic, through briefings, staff discussions, and oral
more informal, and quite specific to the communications. Summaries of findings
problem, people, and organization for and recommendations will be distributed
which the research is undertaken. for discussion, but the formality of the re-
Both formative evaluation and action reporting and the nature of the research publi-
search focus on specific programs at specific cations are quite different from those in ba-
points in time. There is no intention, typi- sic, applied, or even summative evaluation
cally, to generalize beyond those specific set- research.
tings. The difference between formative An example of action research comes
evaluation and action research centers on from a small rural community in the Mid-
the extent to which the research is system- west in which the town board needed to de-
atic, the different kinds of problems studied, cide what to do with a dilapidated building
and the extent to which there is a special role on a public park. They got a high school class
for the researcher as distinct from the people to put together a simple telephone survey to
being researched. solicit ideas about what to do with the build-
In formative evaluation, there is a formal ing. They also conducted a few focus groups
design and the data are collected and/or an- in local churches. The results showed that
alyzed, at least in part, by an evaluator. For- the townspeople preferred to fix up the
mative evaluation focuses on ways of im- building and restore it as a community cen-
proving the effectiveness of a program, a ter rather than tear it down. The action re-
policy, an organization, a product, or a staff search process took about a month. Based on
unit. In action research, by way of contrast, the findings, a local committee was formed
design and data collection tend to be more to seek volunteers and funds for the restora-
INACT-A P :?f!AnC u
fd-i. ! ;:i! ' \ .'!' .!":i' !.'i! i' < !'ki i ' kl i!i!!!!! / !<< ! \ l. >;:,! i n!'!! v .Vi r n.n i' ij=.-';:V.'. ! Y n' T I .vryi r* ; "
s:=;r ri//L.':- /5! .V! iiitfSM.ri.1 uitt ti .=. ! J R' =:. i!'N V lli-r I - P V T * V ! I- : . = , V R D - ITI i"R . 1 ' R r v n
F D F F L L L B T ^ N I.!!'di/iVVi ' I . I I / I . I J = ' : I M U P I . ; ! = = I.V: I I V I R hr M R I : Y - : \ ' : T Y S-RS..>.| : I I T !.:! RI '
1
it ! i,'. . .,' !' !1AV i.i !;=i 1! IVfE-g i.:, IVi h t -! > - :i ii r.! .1 i =i i iV: i" rvi vi ! i.' Vi S "s i i ' i V.TI vVi i\'i r.;i - -
!
tVr:fa iJi iVrilif iVi it Ari liJ k; :i::-:.i : ! '.=: i i'i i. \ V v. IVi : i! " mv! f':'i vh i; ;Vi i' i;.Vi , ?[ vij V, i"y;j |rl
:= ! :
!
: ! ILIS!^!!,. : hES ifl! iS. N ' ! ^ II!.' i.==': ,'i - = V"! r.i: i" . iVi ::V ::ir ::vi/: n!,:: ii'|".'IVI." .'.'i r ; cvri !. ri! , M ' -
C R U . ! K P nmh. IVSI' IIIRI :! !:' !J
, ! :::.: !. ;"= . ? TF* > i VI. i\ =: \h i- r W I I.-
: ; ': I! vi , > UV .! - n T I : - I ; T I :RR-iVi;i
fdfiTr P!:^. J iiili/i:! =;!J1:: : i ! iritiiV.'. I.' !>=: i i i i. i 'i I ; : i i Vi iVi i".' s ' i I.;I ytiy i; |i1 \ \ 11 '.-vi T I ' I. i' ri 'i
llllllllllllllllJIJIJjjJJJIl
I'.'I ITV I-:; i.vi i,:. i r :.=.'' W! I : i I !" r '.:-V I i TI
. i:iirtt .IMVi^:/!:! tij: ii:!v . -1 ri ]!;'j::i rllTIitiii:!:! i:!,i' -ii:!!: |! ii i!, i':'-, iV i'
: Y .. s.!;:: i:=i'!.' i '"i =:'i I'" îVVVI V-
i:' i ,: :' :! :>!' Ir, ' :!=i : i.vi!'.': 1-i.i.'5 "i i.'.' br!! iVi. .'i i-1,1-
1
... ztt* ":::':!=:T: ! -INLIISIHT!;!: :=::! !: :,-=:.'!:'.';:; I===

.:!!: ! !=:; !=:T!J;!;:! =::; A 1 = :
: !
:
! V :: tu.- i : v i ! ^ u '. I !
ifi: iTiiilVvl^.iVi!^'n":in!e iTi-âii iliî^JiÎkiiiîiTi.i-îvri^,iir r>y}>.i": ii' IVir vi|:V|V:!' IT'.-i;i;it!'.""
!: i'-.!iTf 'ti-! ii!.': jj! = , n r, !; iiMil /j i!.;: in \ ri; =- I>:=_ " L J - P S ' : " ; i ! < m 11.; i I ( ! I |.i.;! i' i'
i;i i y.^, >. j, ' ' i j' j.' j- j j,,, | ii ' I.VftM ..
lllllllllllllllJIJJIIIJ^
I (ws '! .! >:| HÎ !'
! : '::.;!; :::J; !;!
! ! != I: !:: :;=;', : ii- :',- s:a-.s:: i i.'! I <:V hi *iw> i.-; J,* .i U IVI'YI
! !
:!i**i!'i:i i! 1!;=:!; ii!i.i !!:;.;:!: ItIiJ :- amtfrfl .!i i^îM lIJi rp k& IVi lIi-iiS itn ':> j'i: i vi I 'i! i i'i'!' i !V ii 'Vi i Vi
!! ^Tiiii' iri1!" i-i -i" =i' fVi iririVin P I !=:; i:i mh C A T A M l;^:; =!:" i.. ' i " ii! S: C ;' !'I'
;.''!'!'!: "i' "VVIV.".: :!;; .;=:;! J| I :;':.=. V . ! - : .1'M.
tion, thereby solving the town board's prob- Researchers engaged in inquiry at vari-
lem of what to do with the budingan ex- ous points along the continuum can have
ample of action-oriented, problem-solving very strong opinions and feelings about re-
research. searchers at other points along the contin-
uum, sometimes generating opposing opin-
The Purpose of ions and strong emotions. Basic and applied
Purpose Distinctions researchers, for example, would often dis-
pute even calling formative and action re-
It is important to understand variations search by the name research. The standards
in purpose along this theory-to-ac tion con- that basic researchers apply to what they
tinuum because different purposes typically would consider good research excludes
lead to different ways of conceptualizing even some applied research because it may
problems, different designs, different types not manifest the conceptual clarity and theo-
of data gathering, and different ways of pub- retical rigor in real-world situations that ba-
licizing and disseminating findings. These sic researchers value. Formative and action
are only partly issues of scholarship. Poli- researchers, on the other hand, may attack
tics, paradigms, and values are also part of basic research for being esoteric and irrele-
the landscape. vant.
Debates about the meaningfulness, rigor, those choices for both the kind of research
significance, and relevance of various ap- undertaken and the researcher's status as a
proaches to research are regular features of professional within various social groups.
university life. On the whole, within univer- Exhibit 5.3 summarizes some of the major
sities and among scholars, the status hierar- differences among the different kinds of re-
chy in science attributes the highest status to search.
basic research, secondary status to applied
research, little status to summative evalua- Examples of Types
tion research, and virtually no status to for-
of Research Questions:
mative and action research. The status hier-
A Family Research Example
archy is reversed in real-world settings,
where people with problems attribute the
greatest significance to action and formative To further clarify these distinctions, it
research that can help them solve their prob- may be helpful to take a particular issue and
lems in a timely way and attach the least im- look at how it would be approached for each
por tance to basic research, which they con- type of research. For illustrative purposes,
sider remote and largely irrelevant to what let's examine the different kinds of questions
they are doing on a day-to-day basis. that can be asked about families for different
research purposes. Ali of the research ques-
The distinctions along the continuum are
tions in Exhibit 5.4 focus on families, but the
not only distinctions about purpose and
purpose and focus of each type of research
how one conducts research, but they also in-
are quite different. With clarity about pur-
volve the issue of what one calls what one
pose, it is possible to turn to consideration of
does. In other words, a person conducting
specific design alternatives and strategies.
basic research for the purpose of contribut-
Clarity about purpose helps in making deci-
ing to theory within a discipline may find it
sions about criticai trade-offs in research and
helpful to call that work applied research to
designs, our next topic.
get certain kinds of funding. Summative
evaluation researchers may describe what
they are doing as formative evaluation to
Criticai Trade-Offs
make their work more acceptable to pro-
in Design
gram staff resistant to being studied. On the
other hand, applied researchers may call Purposes, strategies, and trade-offs these
what they are doing basic research to increase themes go together. A discussion of design
its acceptability among scholars. strategies and trade-offs is necessitated by
In short, there are no clear lines dividmg the fact that there are no perfect research
the points along the continuum. Part of what designs. There are always trade-offs.
determines where a particular kind of re- Limited resources, limited time, and limits
search falls along the continuum is how the on the human ability to grasp the complex
researcher describes what is being done and nature of social reality necessitate trade-
its purpose. Different reviewers of the same offs.
piece of research might well use a differ- The very first trade-offs come in framing
ent label to describe it. What is important for the research or evaluation questions to be
our purposes is that researchers understand studied. The problem here is to determine
the implications of these distinctions, the the extent to which it is desirable to study
choices involved, and the implications of one or a few questions in great depth or to
AGI3 S 9 A Typology of Research Purposes
Types of Desired Levei

Research Purpose Focus oFReseorch Desired Results of Generalizador Key Assumptions Publication Mode Standord forJudging
Basic Knowledge as Questions deemed Contribution to Across time and The world is Major refereed Rigor of research,
research an end in itself; important by one's theory space (ideal) patterned; those scholarly journals universality and
discover truth discipline or personal patterns are knowable in one's discipline, verifiability of theory
intellectual interest and explainable. scholarly books
Applied Understand the Questions deemed Contributions to Within as general Human and societal Specialized academic Rigor and theoretical
research nature and sources important by society theories that can be a time and space as probiems can be journals, applied insight into the
of human and used to formulate possible, but clearly understood and research journals problem
societal problems problem-solving limited application soived with within disciplines,
programs and context knowledge. interdisciplinary
interventions problem-focused
journals
Summative Determine Goals of the Judgments and Ali interventions What works one Evaluation reports Generalizability to
evaluation effective ness of intervention generalzation5 with similar goals place under specified for program funders future efforts and to
human interventions about effective types conditions should and policymakers, other programs and
and actions (programs, of interventions and work elsewhere. specialized journals policy issues
policies, personnel, the conditions under
products) which those efforts
are effective
Formative Improve an Strengths and Recommendations Limited to specific People can and will Oral briefings; Usefulness to and
evaluation intervention: weaknesses of the for improvements setting studied use information to conferences; internai actual use by
A program, policy, specific program, improve what they're report; limited intended users in
organization, or policy, product, or doing. circulaton to similar the setting studied
product personnel being programs, other
studied evaluators
Action Solve problems Organization and immediate action; Here and now People in a setting Interpersonal Feelings about the
resea rch in a program, community problems solving problems as can solve problems by interactions among process among
organization, or quickly as possible studying themseives. research participants; research participants,
community informal unpublished feasibility of the
solution generated
Family Research Example: Research Questions Matched to

E X H I B I T 5.4 Research Category
Basic research What are variations in types of famifes and what functions do those
variations serve?
Applied research What is the divorce rate among different kinds of families in the United States
and what explains different rates of divorce among different groups?
Summative What is the overall effectiveness of a publicly funded educational program
evaluation that teaches family members communication skills where the desired
program outcomes are enhanced communications among family members,
a greater sense of satisfaction with family life, effective parenting
practices, and reduced risk of divorce?
Formative How can a program teaching family communications skills be improved? What
evaluation are the program's strengths and weaknesses? What do participants like and
dislike?
Action research A self-study in a particular organization (e.g., church, neighborhood center) to
figure out what activities would be attractive to families with children of dif-
ferent ages to soive the probiem of low participation in family activities.
study many questions but in less depth And always there are fundamental con-
the "boundary probiem" in naturalistic in- straints of time and resources.
quiry (Guba 1978). Once a potential set of in- Converging on focused priorities typi-
quiry questions has been generated, it is nec- cally proves more difficult than the chal-
essary to begin the process of prioritizing lenge of generating potential questions at
those questions to decide which of them the beginning of a study or evaluation. Doc-
ought to be pursued. For example, for an toral students can be especially adept at
evaluation, should ali parts of the program avoiding focus, conceiving instead of
be studied or only certain parts? Should ali sweeping, comprehensive studies that make
clients be interviewed or only some subset the whole world their fieldwork oyster. In
of clients? Should the evaluator aim at de- evaluations, once involved users begin to
scribing ali program processes or only cer- take seriously the notion that they can learn
tain selected processes in depth? Should ali from finding how whether what they think
outcomes be examined or only certain out- is being accomplished by a program is what
comes of particular interest to inform a is really being accomplished, they soon gen-
pending decision? These are questions that erate a long list of things they'd like to find
are discussed and negotiated with intended out. The evaluation facilitator's role is to
users of the evaluation. In basic research, help them move from a rather extensive list
these kinds of questions are resolved by the of potential questions to a much shorter list
nature of the theoretical contribution to be of realistically possible questions and finally
made. In dissertation research, the doctoral to a focused list of essential and necessary
committee provides guidance on focusing. questions.
226 LEI. QUALITATIVE DESIGNS AND DATA COLLECTION
Review of relevant literature can also attempt to study. In a narrowly focused

bring focus to a study. What is already study, we might select one particular set of
known? Unknown? What are the cutting- interactions and limit our study to those
edge theoretical issues? Yet, reviewing the for example, the social interactions between
literature can present a quandary in qualita- teachers and children. Broadening the scope
tive inquiry because it may bias the re- somewhat, we might decide to look only at
searcher^ thinking and reduce openness to those interactions that occur in the class-
whatever emerges in the field. Thus, some- room, thereby increasing the scope of the
times a literature review may not take place study to include interactions not only be-
until after data collection. Alternatively, the tween teacher and child but also among
literature review may go on simultaneously peers in the classroom and between any vol-
with fieldwork, permitting a creative inter- unteers and visitors to the classroom and the
play among the processes of data collection, children. Broadening the scope of the study
literature review, and researcher introspec- still more, we might decide to look at ali of
tion (Marshall and Rossman 1989:38-40). As the social relationships that children experi-
with other qualitative design issues, ence in schools; in this case we would move
trade-offs appear at every turn, for there are beyond the classroom to look at interactions
decided advantages and disadvantages to with other personnel in the schoolfor ex-
reviewing the literature before, during, or ample, the librarian, school counselors, spe-
after fieldworkor on a continuai basis cial subject teachers, the custodian, and/or
throughout the study. school administrative staff. Broadening the
A specific example of possible variations scope of the study still further, the educators
in focus will illustrate the kinds of trade-offs might decide that it is important to look at
involved in designing a study. Suppose the social relationships children experience
some educators are interested in studying at home as well as at school so as to under-
how a school program affects the social de- stand better how children experience and
velopment of school-age children. They are affected by both settings, so we would in-
want to know how the interactions of chil- clude in our design interactions with par-
dren with others in the school setting con- ents, siblings, and others in the home.
tribute to the development of social skills. Finally, one might look at the social relation-
They believe that those social skills will be ships experienced throughout the full range
different for different children, and they are of societal contacts that children have, in-
not stire of the range of social interactions cluding church, clubs, and even mass media
that may occur, so they are interested in a contacts.
qualitative inquiry that will capture varia- A case could be made for the importance
tions in program experience and relate those and value of any of these approaches, from
experiences to individualized outcomes. the narrowest focus, looking only at student-
What, then, are trade-offs in determining the teacher interactions, to the broadest focus,
final focus? looking at students' full, complex social
We begin with the fact that any given world. Now lefs add the real-world con-
child has social interactions with a great straint of limited resourcessay, $50,000
many people. The first problem in focusing, and three monthsto conduct the study. At
then, is to determine how much of the social some levei, any of these research endeavors
reality experienced by children we should could be undertaken for $50,000. But it be-
comes dear, immediately, that there are cally produce a wealth of detailed data
trade-offs between breadth and depth. A about a much smaller number of people and
highly focused study of the student-teacher cases.
relationship could consume our entire bud- However, the breadth versus depth
get but allow us to investigate the issue in trade-off also applies within qualitative de-
great depth. On the other hand, we might at- sign options. Human relations specialists
tempt to look at ali social relationships that tellus that we can never fully understand the
children experience but to look at each of experience of another person. The design
them in a relatively cursory way in order, issue is how much time and effort we are
perhaps, to explore which of those relation- willing to rnvest in trying to increase our un-
ships is primary. (If school relationships derstanding about any single person's expe-
have very little impact on social develop- riences. So, for example, we could look at a
ment in comparison with relationships out- narrow range of experiences for a larger
side the school, policymakers could use that number of people or a broader range of ex-
Information to decide whether the school periences for a smaller number of people.
program ought to be redesigned to have Take the case of interviews. Interviewing
greater impact on social development or, al- with an instrument that provides respon-
ternatively, if the school should forget about dents with largely open-ended stimuli typi-
trying to directly affect social development cally takes a great deal of time. In an educa-
at ali.) The trade-offs involved are the clas- tion study, I developed an open-ended
sic trade-offs between breadth and depth, interview for elementary students consist-
which we now turn to in more depth. ing of 20 questions that included items such
as "What do you like most about school?"
and "What don't you like about school?"
Breadth Versus Depth These interviews took between half an hour
and two hours depending on students' ages
In some ways, the differences between and how articulate they were. It would cer-
quantitative and qualitative methods in- tainly have been possible to have longer in-
volve trade-offs between breadth and depth. terviews. Indeed, Ihave conducted in-depth
Qualitative methods permit inquiry into se- interviews with people that ran 6 to 16 hours
lected issues in great depth with careful at- over a period of a couple of days. On the
tention to detail, context, and nuance; that other hand, it would have been possible to
data collection need not be constrained by ask fewer questions, make the interviews
predetermined analytical categories con- shorter, and probe in less depth.
tributes to the potential breadth of qualita- Or consider another example with a fuller
tive inquiry. Quantitative instruments, on range of possibilities. It is possible to study a
the other hand, ask standardized questions single individual over an extended period of
that limit responses to predetermined cate- timefor example, the study, in depth, of
gories (less breadth and depth). This has the one week in the life of one child. This in-
advantage of making it possible to measure volves gathering detailed information about
the reactions of many respondents to a lim- every occurrence in that child's life and ev-
ited set of questions, thus facilitating com- ery interaction involving that child during
parison and statistical aggregation of the the week of the study. With more focus, we
data. By contrast, qualitative methods typi- might study several children during that
week but capture fewer events. With a still selected as the unit of analysis when there is
more limited approach, say, a daily half- some important characteristic that separates
hour interview, we could mterview yet a people into groups and when that character-
larger number of children on a smaller num- istic has important implications for the pro-
ber of issues. The extreme case would be to gram.
spend ali of our resources and time asking a A different unit of analysis involves fo-
single question of as many children as we cusing on different parts of a program. Dif-
could interview given time and resource ferent classrooms within a school might be
constraints. studied, making the classrooin the unit of
No rule of thumb exxsts to tell a re- analysis. Outpatient and inpatient programs
searcher precisely how to focus a study. in a medicai facility could be studied. The in-
The extent to which a research or evalua- take part of a program might be studied sep-
tion study is broad or narrow depends on arately from the service delivery part of a
purpose, the resources available, the time program as separate units of analysis. Entire
available, and the interests of those in- programs can become the unit of analysis. In
volved. In brief, these are not choices be- state and na tional programs where there are
tween good and bad but choices among al- a number of local sites, the appropriate unit
ternatives, ali of which have merit. of analysis may be local projects. The analyt-
ical focus in such multisite studies is on vari-
ations among project sites more than on
Units of Analysis variations among individuais within pro-
jects.
A design specifies the unit or units of Different units of analysis are not mutu-
analysis to be studied. Decisions about sam- ally exclusive. However, each unit of analy-
ples, both sample size and sampling strate- sis implies a different kind of data collection,
gies, depend on prior decisions about the a different focus for the analysis of data, and
appropriate unit of analysis to study. Often a different levei at which statements about
individual people, clients, or students are findings and conclusions would be made.
the unit of analysis. This means that the pri- Neighborhoods can be units of analysis or
mary focus of data collection will be on what communities, cities, states, cultures, and
is happening to individuais in a setting and even nations in the case of International pro-
how individuais are affected by the setting. grams.
Individual case studies and variation across One of the strengths of qualitative analy-
individuais would focus the analysis. sis is looking at program units holistically.
Comparing groups of people in a pro- This means doing more than aggregating
gram or across programs involves a differ- data from individuais to get overall program
ent unit of analysis. One may be interested m results. When a program, group, organiza-
comparing demographic groups (males tion, or community is the unit of analysis,
compared with females, Whites compared qualitative methods involve observations
with African Americans) or programmatic and description focused directly on that
groups (dropouts vs. people who complete unit: The program, organization, or commu-
the program, people who do well vs. people nity, not just the individual people, becomes
who do poorly, people who experience the case study focus in those settings.
group therapy vs. people who experience in- Particular events, occurrences, or inci-
dividual therapy). One or more groups are dents may also be the focus of study (unit of
analysis). For example, a quality assurance the kids want to be outside, so that's not an
effort in a health or mental health program effective time to gather data. In African vil-
might focus only on those criticai incidents lages, I was given similar scenarios about
in which a patient fails to receive expected or the difficulties of data collection for every
desirable treatment. A criminal justice eval- month in the annual cycle of the agricultural
uation could focus on violent events or in- season. A particular period of time, then, is
stances in which juveniles run away from both an important context for a study and a
treatment. A cultural study may focus on sampling issue.
celebrations. There are limits to how much one can ap-
Sampling can also involve time period ply logic in trying to calculate ali of the pos-
strategies, for example, continuous and on- sible consequences of sampling options,
going observation versus fixed-interval whether the decision is about which time pe-
sampling in which one treats units of time riods to sample or which activities to ob-
(e.g., 15-minute segments) as the unit of observe. The trick is to keep coming back to the
servation. "The advantage of fixed-interval criterion of usefulness. What data collected
sampling over continuous monitoring are during what time period describing what
that fieldworkers experience less fatigue activities will most likely illuminate the in-
and can collect more information at each quiry? For evaluation, what focus of inquiry
sampling interval than they could on a con- will be most useful? There are no perfect
tinuous observation routine" (Johnson and evaluation designs, only more and less use-
Sackett 1998:315). Time sampling (sampling ful ones.
periods or units of time) can be an especially The key issue in selecting and making
important approach because programs, or- decisions about the appropriate unit of
ganiza tions, and communities may function analysis is to decide what it is you want to
in different ways at different times during be able to say something about at the end of
the year. Of course, in some programs there the study. Do you want to have findings
never seems to be a good time to collect data. about individuais, families, groups, or some
In doing school evaluations in the United other unit of analysis? For scholarly inqui-
States, I've been told by educators to avoid ries, disciplinary traditions provide guid-
collecting data before Halloween because ance about relevant units of analysis. For
the school year is just getting started and evaluations, one has to determine what deci-
the kids and teachers need time to get set- sion makers and primary intended users re-
tled in. But the period between Halloween ally need information about. Do they want
and Thanksgiving is really too short to do findings about the different experiences of
very much, and, then, of course, after individuais in programs, or do they want to
Thanksgiving everybody's getting ready for know about variations in program processes
the holidays, so that's not a typical or conve- at different sites? Or both? Such differences
nient period. It then takes students a few in focus will be criticai to the design but may
weeks after the winter break to get their at- not be easy to determine. A decision maker
tention focused back on school and then the is unlikely to say to the evaluator, "The unit
winter malaise sets in and both teachers and of analysis we want to study is ." The
students become deeply depressed with the evaluator mustbe able to hear the real issues
endlessness of winter (at least in northern involved in the decision maker's questions
climes). Then, of course, once spring hits, at- and translate those issues into the appropri-
tention is focused on the close of school and ate unit of analysis, then check out that
translation with the intended evaluation us- pling as one of the core distinguishing
ers. strategic themes of qualitative inquiry. The
Exhibit 5.5 presents some alternative next section presents variations in, ratio-
units of analysis. Clarity about the unit of nales for, and the details of how to design a
analysis is needed to select a study sample. study based on a purposeful sample.
In Chapter 2, I identified purposeful sam-
!3. Purposeful S a m p l i n g
the motive exalts the action.
Margaret Preston, 1875
learn a great deal about issues of central

Perhaps nothing better captures the dif-
impor tance to the purpose of the inquiry,
ference between quantitative and qualita-
thus the term purposeful sampling.
tive methods than the different logics that
Studying information-rich cases yields in-
undergird sampling approaches.
sights and in-depth understanding rather
Qualitative inquiry typically focuses in
than empirical generalizations. For exam-
depth on relatively small samples, even sin-
ple, if the purpose of an evaluation is to in-
gle cases (N = 1), selected purposefully. Quan-
crease the effectiveness of a program in
titative methods typically depend on larger
reaching lower-socioeconomic groups,
samples selected randomly. Not only are the
one may learn a great deal more by study-
techniques for sampling different, but the
ing in depth a small number of carefully
very logic of each approach is unique be-
selected poor families than by gathering
cause the purpose of each strategy is differ-
standardized information from a large,
ent.
statistically representative sample of the
The logic and power of random sampling
whole program. Purposeful sampling fo-
derive from statistical probability theory A
cuses on selecting information-rich cases
random and statistically representative
whose study will illuminate the questions
sample permits confident generalization
under study. Purposeful sampling is
from a sample to a larger population. Ran-
sometimes called purposive or judgment
dom sampling also controls for selection
sampling: "In judgment sampling, you de-
bias. The purpose of probability-based ran-
cide the purpose you want inforinants (or
dom sampling is generalization from the
communities) to serve, and you go out to
sample to a population and control of selec-
find some" (Bernard 2000:176). There are
tivity errors.
several different strategies for purpose-
What would be "bias" in statistical sam-
fully selecting information-rich cases. The
pling, and therefore a weakness,becomes in-
logic of each strategy serves a particular
tended focus in qualitative sampling, and
purpose.
therefore a strength. The logic and power of
purposeful sampling lie in selecting informa-
tion-rich cases for study in depth. Informa- 1. Extreme or deviant case sampling. This
tion-rich cases are those from which one can strategy involves selecting cases that are
1
|l(Mg|gM Examples of Units of Analysis for Case Studies and
SBilIslIEHsB Comparisons
People Focused Structure Focused
individuais Projects
Small, informal groups (friends, gangs) Programs
Families Organizations
Units in organizations
Perspective/Worldview Based
People who share a culture

People who share a common experience or perspective, for example, dropouts,
graduates, leaders, parents, Internet listserv participants, survivors
Geoaraphv Focused
Neighborhoods Villages
Cities Farms
States Regions
Countries Markets
Ac ti vi tv Focused
Criticai incidents Time periods

Celebrations Crises
Quality assurance violations Events
Time Based
Particular days, weeks, or months Vacations

Winter break Rainy season
Ramadan Dry season
Full moons School term
A political term of office An election period
NOTE: These are not mutually exclusive categories.
information rich because they are unusual fectly representative of U.S. industry as a
or special in some way, such as outstanding whole . . . [but] a list of companies consid-
successes or notable failures. The influential ered to be innovative and excellent by an in-
study of America's best-run companies, formed group of observers of the business
published as In Search ofExcellence, exempli- scene" (Peters and Waterman 1982:19).
fies the logic of purposeful, extreme group Lisbeth Schorr (1988) used a similar strategy
sampling. This study was based on a sample in studying especially effective programs
of 62 companies "never intended to be perfor families in poverty, published as the
influential book Within Our Reach. Stephen An excellent applied research example
Covey's (1990) best-selling book The 7 Hab- is Angela Browne's (1987) study When Bat-
its of Highly Effective People is based on a pur- tered Women Kill. She conducted in-depth
poseful, extreme group sampling strategy. studies of the most extreme cases of do-
Studies of leadership have long focused on mestic violence to elucidate the phenome-
identifying the characteristics of highly suc- non of battering and abuse. The extreme
cessful leaders, as in Collins's (2001) case nature of the cases is what renders them so
studies of 11 corporate executives in whom powerful. Browne'sbookis an exemplar of
"extreme personal humility blends para- qualitative inquiry using purposeful sam-
doxically with intense professional will" pling for applied research.
(p. 67), what he calls "Levei 5 leaders," the In evaluation, the logic of extreme case
highest levei in his model. In the early days sampling is that lessons may be learned
of AIDS research when HTV infections al- about unusual conditions or extreme out-
most always resulted in death, a small num- comes that are relevant to improving more
ber of cases of people infected with HIV typical programs. Let's suppose that we
who did not develop AIDS became crucial are interested in studying a national pro-
outlier cases that provided important in- gram with hundreds of local sites. We
sights into directions researchers should know that many programs are operating
take in combating AIDS. reasonably well, even quite well, and that
Sometimes cases of dramatic failure offer other programs verge on being disasters.
powerful lessons. The legendary UCLAbas- We also know that most programs are do-
ketball coach John Wooden won 10 national ing "OK." This information comes from
championships from 1964 through 1975, an knowledgeable sources who have made
unparalleled sports achievement. But the site visits to enough programs to have a
game he remembered the most and said he basic idea about what the variation is. The
learned the most from was UCLA's 1974 question is this: How should programs be
overtime loss to North Carolina State in the sampled for the study? If one wanted to
semifinais (quoted in the Los Angeles Times, precisely document the natural variation
December 21,2000). Wooden's focus on that among programs, a random sample
gamethat extreme caseillustrates the would be appropriate, preferably a ran-
learning psychology of extreme group pur- dom sample of sufficient size to be truly
poseful sampling. representative and permit generalizations
This is also the sampling psychology be- to the total population of programs. How-
hind Jim Paul's (1994) book What I Learned ever, some information is already avail-
Losing a Million Dollars. A former governor able on what program variation is like. The
of the Chicago Mercantile Exchange, he question of more immediate interest may
made thousands of trades in many com- concern illuminative cases. With limited re-
modities over a long and distinguished ca- sources and limited time, an evaluator
reer, but what he reports learning the most might learn more by intensively studying
from was a highly unusual combination of one or more examples of really poor pro-
mistakes in which he lost more than $1 mil- grams and one or more examples of really
lion in a few weeks of trading soy beans an excellent programs. The evaluation focus,
extreme but illuminative case. He reports then, becomes a question of understand-
that he ultimately learned to be a winner by ing under what conditions programs get
carefully studying and learning from losing. into trouble and under what conditions
SM/ HMO THE K5-T

=:JH!- :i > L M M I I ! .Ifr.ii I J L'ii3! K I I ! C i:!!!1!:!.::.:'.:!' ir-wi-wpiv!i- :::;:! ^'iiv:'
uvri P R A TF-R I jki VS:;' ; : J : I : ! . V ' T ! : \ k IV IÎMRIIIN! ::=! i'.!!:::;.!i."im' ':i>! == :=.:
: ;
1 ! i-1"O
F I A I S F R V I ! iVi'L"i ': .'I' J:
! , , niI:vi3. :ii J,i '::':;' PAl*
, ! , s , : iVi - sii .=31 -j=
!:: iV r .!:!.'I|<-.!tii:; ..=ij! ii! -d !." iVC l>.Y ii:!:5 V!.; .igi" -jis-::s !.=- ' = j - îi? "O
:"!; 1 rri!:t'i:!.' rh tiq"\ .!:,'ifcliifi-î!iiihiM Cii:ii-' esrtihjpoaw :!:;:;::;( n.- rILTCI ".:
': N ::. I - =7 V- . IVi r =.! . I I I.I : I.= !
P 1 :. I : ! i- :; -i:; -:ij S :: !!=:'.:!; a i : ' : <<=> hi
LMSVNJJTTTVFT SI!''!.!!.!?! ! T ! !\..:i :T.1H P I L M W -
! l.Vv.iVry .ivi
i: !:i i5 .
':! n>= i
: i! ij-: n o ====!
jea-f h lAVa;!. IV h:!::'*:j>iihi V : : I j'fi-5!fa!!! edv.'!'.': h: rui ; tflE^:
! :: p:;.!:. - ::!: :r >;!!
VKS ! ;:' RI I RN.'1 ;:; Y I I J . Y R U M !,!.:!! !"!ii;:'i !iJi.s 0 .v^scrM iV-: ir! !>.I:-::;;Y =:=.':!f::.: = :=s i-n;
:;i'ij::".!i i!;i: ::::tri i WM, ,'f t '.. >S j: i:'il! :Tr:j:l.-:i:"i:i! ' M.ii-:i"ii:, ==!:':;Ori!iri!'i I ! ! i ! i ' " ' !:':-'"ii':i
! '!:;
! J:S.' I:;I I..T-IV=.I'I ;R . S I I D .Ir!'!:!.!'! H I * ! JiVlj:1 :F'. SFT: ! J
!
J^- i: : :':. ': w: .V: :!.;; . =! i-:: === i- K f
!|?iViiliV:::':!"-.^",'; nYl iV-i' i'.!.'i i:!:T:|j-. iJ!!:!:!i.i: !:: i;yiri!ri'i:i;i :!'^! !::! i.!! : !:'!' inixuirs
: :
IVii-
:;r::i;4 if if pvi \ rn h:;ri M:=i!. ,li h:|,ri:':! hihii5 n:i :=!:i: !:-: ". " ii'
! !i.iVi.i'i:- -:'!- : ":; : ) ' : , N :r
:
"JGVT:! I I T 11-5 == . . " I I v'J.'}.R ir>, \!;I'!," II 1';.=!^

1
*:':; IYVÎ-i-iVli'::- Y I I V I I J / I . ! ! : X =!'!I-Iil
! I I ; I I " ' ! -LI!!
j i ;: h -:! :'!:.!";;! :;: ;M:|!|T!;;:W!

! iYird i-lMrij:!!:!; ClsilsillM.shl^.'! i:'i;! SiViV ' !='.'' i! !',! ! i!'..i i
:;:!:!' YXV" In livy !Y"P;:Iri R N FRI''"Irn. .'I!::- :T!ii) i i hi i:i: i!'i !' iV :i f i'i iVi l! H iX! , :v,.i" !V!i!
!!"i!::Cis"!! !":' isi""'" x* "15fr".s:.?"!!:':ii i.:
! ! !
i.r'iI'." T!:!":li' Lra i.i\'iW i!'i' I i!' :;!:
:\'>kr!!i:!sv ;:s!'!i:s !wi\iTI ;iTh"! ,1..Ei! ,!:..- 1 j!'i i j ' . . i,Mi!!i:;
!".;::;: J N R;.;I !| !;{ : ;; N N J I V I V sVi.R. I " ; ; " R W - K - M|:|I I !
; D V R , ' ! ! I'!:" / I T I P I N I V . ! O ? ! ! ' : ; !'.': ! ' I i/I .i ,'-!'. ; i'.;
!:';:;:: O" 1'.U'! w-rVC!. .iV:!i: FÎIWIIHII tlDIE hivii'. '; I!^ AWll" i'i.'J i . i
1
f-5-:5j:==:-=: i : : \ > ^ - v i : : i r ^ V n i;i.;.i! !

:: : HJ' ! I:i'I:::!ii-I:.'I-I : I^ri ' r
: !: ! ! : : TS J''I j ,-i'i :' : .
iVii! OV. ! :A1\\\!i/f i=i=.;!i !V ;:!!:V!;= j"jj^nnifii '!j!!;i!;!-ri i.^!S'!'!!ri!,'i!.:!!:!'.!,! iVis.= i!' !'>i":ir" jll.'Hi
:;=: -5=-B:. = :: ' : I S; S: I; R . ! '
5
:
! : . : , : = : :' !=;:;! MN LV! ;:=I.! !:! h"-i I'- r ri T " ! I'- N.L ! .!!;= ! .'! . U
iiiii s: h :i;!.n : ,!!!:.: r^J ..hivvi.:;^:^! .i;^^!^::':! !::il ;: i" i'.': r m-ii vi y . ' =:! i ,'n!:. i i r I '"'
:!! i i1;;' : \ oo?x< !" :' n!; lyix j;; :;v! v cg !-!; :H; :.T N B N - S :! ;R' i V. i.=N -J!:T! -I
! ! !
; I I:' i iVr
k- i.iiv,! vii^îrK!'!.';!;! :ip-.t:.hi. ''.i j-fe! -

! ! ;
| i i: !i:!.!l
: :!' R " : " ' ! ! ! !ri'i"." |i ViSITS' r! !' i i,".! " . -""'r
. v. .. .. ,...1 , ! ..: .. ... . .. j j . I . . ,
; ) , 'i , 1-1 | ! Vi"
;;Ji.-.I;!:'.'::'. J-w xR-!.: T!: -:!S:! - Y!
!! -
iViivi iVrjs' = r i! :':.'T; \'I i'tvi ::S-I!- . : IV!' prt !:,: .-.Ml-, J i;.
!
,1 i s i. vj i !.". ! 11 ,.,,,j
i.!.:. i ViV i V ! i i:!1 ii i! 1 =:-' C!:'V:!'!:i' ! ? i h \Xi w::L.VIII.:i -II !-: HRI
! \ .j:I;YY!.*:T! P brS |: : I I.!." I-S^VS'.
programs exemplify excellence. It is not standing selected cases of special interest,

even necessary to randomly sample poor for example, unexpected dropouts or out-
programs or excellent programs. The re- standing successes. In an evaluation of the
searchers and intended users involved in the Caribbean Agricultural Extension Project,
study think through what cases they could we did case studies of the "outstanding ex-
learn the most from and those are the cases tension agent" selected by peers in each of
that are selected for study. eight Caribbean countries to help the pro-
In a single program the same strategy gram develop curriculum and standards
may apply. Instead of studying some repre- for improving extension practice. The
senta tive sample of people in the setting, the sample was purposefully "biased," not to
evaluator may focus on studying and under- make the program look good, but rather to
learn from those who were exemplars of nomenon of interest intensely (but not ex-
good practice. In many instances, more can tremely). Extreme or deviant cases may be
be learned from intensively studying exem- so unusual as to distort the manifesta tion of
plary (information-rich) cases than can be the phenomenon of interest. Using the logic
learned from statistical depictions of what of intensity sampling, one seeks excellent or
the average case is like. In other evaluations, rich examples of the phenomenon of inter-
detailed information about special cases can est, but not highly unusual cases.
be used to supplement statistical data about Heuristic research (Chapter 3) uses inten-
the normal distribution of participants. In sity sampling. Heuristic research draws
statistical terms, extreme case sampling fo- explicitly on the intense personal expe-
cuses on outliers (the endpoints of the bell- riences of the researcher, for example, ex-
shaped curve normal distribution) that are periences with loneliness or jealousy. Co-
often ignored in aggregate data reporting. researchers who have experienced these
Ethnomethodologists use a form of ex- phenomena intensely also participate in the
treme case sampling when they do their study. The heuristic researcher is not typi-
field experiments. Ethnomethodologists are cally seeking pathological or extreme mani-
interested in everyday experiences of rou- festations of loneliness, jealousy, or what-
tine living that depend on deeply under- ever phenomenon is of interest. Such
stood, shared understandings among peo- extreme cases might not lend themselves to
ple in a setting (see Chapter 3). One way of the reflective process of heuristic inquiry.
exposing these implicit assumptions and On the other hand, if the experience of
norms on which everyday life is based is to the heuristic researcher and his or her
create disturbances that deviate greatly from coresearchers is quite mild, there won't be
the norm. Observing the reactions to some- much to study. Thus, the researcher seeks a
one eating like a pig in a restaurant and then sample of sufficient intensity to elucidate the
interviewing people about what they saw phenomenon of interest.
and how they felt would be an example of The same strategy can be applied in a pro-
studying a deviant sample to illuminate the gram evaluation. Extreme successes or un-
ordinary. usual failures may be discredited as being
In essence, the logic of extreme group too extreme or unusual to yield useful infor-
sampling is that extreme cases may be informa tion. Therefore, the evaluator may select
mation-rich cases precisely because, by be- cases that manifest sufficient intensity to il-
ing unusual, they can illuminate both the luminate the nature of success or failure, but
unusual and the typical. In proposing an ex- not at the extreme.
treme group sample, as in ali purposeful Intensity sampling involves some prior
sampling designs, the researcher has an obli- information and considerable judgment.
gation to present the rationale and expected The researcher or evaluator must do some
benefits of this strategy as well as to note its exploratory work to determine the nature of
weakness (lack of generalizability). the variation in the situation under study,
then sample intense examples of the phe-
2. Intensity sampling. Intensity sampling nomenon of interest.
involves the same logic as extreme case
sampling but with less emphasis on the ex- 3. Maximum variation (heterogeneity) sam-
tremes. An intensity sample consists of in- pling. This strategy for purposeful sam-
formation-rich cases that manifest the phe- pling aims at capturing and describing the
central themes that cut across a great deal of case studies of individual fellowship recipi-
variation. For small samples, a great deal of ents. Over 20 years of awards, more than 600
heterogeneity can be a problem because in- people had received fellowships. We had
dividual cases are so different from each sufficient resources to do only 40 case stud-
other. The maximum variation sampling ies. We naximized sample variation by cre-
strategy turns that apparent weakness into ating a matrix in which each person in the
a strength by applying the following logic: sample was as different as possible from oth-
Any common patterns that emerge from ers using dimensions such as nature of
great variation are of particular interest and work, stage in career, public visibility, insti-
value in capturing the core experiences and tutional affiliation, age, gender, ethnicity,
central, shared dimensions of a setting or geographic location, mobility, health status,
phenomenon. nationality, and field of endeavor. The the-
How does one maximize variation in a matic patterns of achievement that emerged
small sample? One begins by identifying di- from this diversity allowed us to construct a
verse characteristics or criteria for construct- model to illuminate the primary dimensions
ing the sample. Suppose a statewide proof and factors in the award's impact. A
gram has project sites spread around the theme song emerged from ali the scattered
state, some in rural areas, some in urban ar- noise. That's the power of maximum varia-
eas, and some in suburban areas. The evalu- tion (heterogeneity) sampling.
ation lacks sufficient resources to randomly Thus, when selecting a small sample of
select enough project sites to generalize great diversity, the data collection and anal-
across the state. The evaluator can study a ysis will yield two kinds of findings: (1)
few sites from each area and at least be sure high-quality, detailed descriptions of each
that the geographical variation among sites case, which are useful for documenting
is represented in the study. While the evalua- uniquenesses, and (2) important shared pat-
tion would describe the uniqueness of each terns that cut across cases and derive their
site, it would also look for common themes significance from having emerged out of
across sites. Any such themes take on added heterogeneity. Both are important findings
importance precisely because they emerge in qualitative inquiry.
out of great variation. For example, in study-
ing community-based energy conservation
efforts statewide using a maximum hetero- 4. Homogeneous samples. In direct contrast
geneity sampling strategy, I constructed a to maximum variation sampling is the strat-
matrix sample of 10 commuriities m which egy of picking a small, homogeneous sam-
each community was as different as possible ple, the purpose of which is to describe
from every other community on such char- some particular subgroup in depth. A pro-
acteristics as size, form of local government gram that has many different kinds of par-
(e.g., strong mayor/weak mayor), ethnic di- ticipants may need in-depth information
versity, strength of the economy, demo- about a particular subgroup. For example, a
graphics, and region. In the analysis, what parent education program that involves
stood out across these diverse cases was the many different kinds of parents may focus a
importance of a local, committed cadre of qualitative evaluation on the experiences of
people who made things happen. single-parent female heads of household
In a study of the MacArthur Foundation because that is a particularly difficult group
Fellowship Program, the design focused on to reach and hold in the program.
Focus group interviews are based typi- illuminates key issues that must be consid-
cally on homogeneous groups. Focus ered in any development project aimed at
groups involve open-ended interviews with that kind of village.
groups of five to eight people on specially In evaluation and policy research, the in-
targeted or focused issues. The use of focus terests of decision makers will shape the
groups in evaluation will be discussed at sampling strategy. I remember an evalua-
greater length in the chapter on interview- tion in which the key decision makers had
ing. The point here is that sampling for focus made their peace with the fact that there will
groups typically involves bringing together always be some poor programs and some
people of similar backgrounds and experi- excellent programs, but the programs they
ences to participate in a group interview really wanted more information about were
about major issues that affect them. what they called "those run-of-the-mill pro-
grams that are so hard to get a handle on pre-
5. Typicai case sampling. In describing a cisely because they are so ordinary and don't
culture or program to people not familiar stand out in any definitive way." Given that
with the setting studied, it can be helpful to framing, we employed typical case sam-
provide a qualitative profile of one or more pling. It is important, when using this strat-
typical cases. These cases are selected with egy, to attempt to getbroad consensus about
the cooperation of key informants, such as which cases are typicaland what criteria
program staff or knowledgeable partici- are being used to define typicality.
pants, who can help identify who and what
are typical. Typical cases can also be selected 6. Criticai case sampling. Criticai cases are
using survey data, a demographic analysis those that can make a point quite dramati-
of averages, or other statistical data that cally or are, for some reason, particularly
provide a normal distribution of character- important in the scheme of things. A clue to
istics from which to identify "average-like" the existence of a criticai case is a statement
cases. Keep in mind that the purpose of a to the effect that "if it happens there, it will
qualitative profile of one or more typical happen anywhere," or, vice versa, "if it
cases is to describe and illustrate what is doesn't happen there, it won't happen any-
typical to those unfamiliar with the set- where." Another clue to the existence of a
tingnot to make generalized statements criticai case is a key informant observation
about the experiences of ali participants. to the effect that "if that group is having
The sample is illustrative not definitive. problems, then we can be sure ali the groups
When entire programs or communities are having problems."
are the unit of analysis, the processes and ef- Looking for the criticai case is particularly
fects described for the typical program may important where resources may limit the
be used to provide a frame of reference for evaluation to the study of only a single site.
case studies of "poor" or "excellent" sites. Under such conditions, it makes strategic
When the typical site sampling strategy is sense to pick the site that would yield the
used, the site is specifically selected because most information and have the greatest im-
it is not in any major way atypical, extreme, pact on the development of knowledge.
deviant, or intensely unusual. This strategy While studying one or a few criticai cases
is often appropriate in sampling villages for does not technically permit broad general-
community development studies in Third izations to ali possible cases, logical general-
World countries. A study of a typical village izations can oftenbe made from the weight of
evidence produced in studying a single, crit- it could work anywhere. That makes the crit-
icai case. icai case an especially information-rich
Physics provides a good example of such exemplar, therefore worthy of study as the
a criticai case. In Galileo's study of gravity, centerpiece in a small or "N of 1" sample.
he wanted to find out if the weight of an ob- World-renowned medicai hypnotist Mil-
ject affected the rate of speed at which it ton H. Erickson became a criticai case in the
would fali. Rather than randomly sampling field of hypnosis. Erickson was so skillful
objects of different weights in order to gener- that he became widely known for "his abil-
alize to ali objects in the world, he selected a ity to succeed with 'impossibles'people
criticai casethe feather. If in a vacuum, as who have exhausted the traditional medicai,
he demonstrated, a feather fell at the same dental, psychotherapeutic, hypnotic and re-
rate as some heavier object (a coin), then he ligious avenues for assisting them in their
could logically generalize from this one criti- need, and have not been able to make the
cai comparison to ali objects. His finding changes they desire" (Grinder, DeLozier,
was both useful and credible because the and Bandler 1977:109). If Milton Erickson
feather was a convincing criticai case. couldn't hypnotize a person, no one could.
Criticai cases can be found in social sci- He was able to demonstrate that, under his
ence and evaluation research if one is cre- defnition of hypnosis, anyone could be
ative in looking for them. For example, sup- hypnotized.
pose national policymakers want to get local
communities involved in making decisions 7. Snowballor chain sampling. This isan ap-
about how their local program will be run, proach for locating information-rich key in-
but they aren't sure that the communities formants or criticai cases. The process be-
will understand the complex regulations gins by asking well-situated people: "Who
governing their involvement. The first criti- knows a lot about ? Whom should I
cai case is to evaluate the regulations in a talk to?" By asking a number of people who
community of well-educated citizens; if they else to talk with, the snowball gets bigger
can't understand the regulations, then less and bigger as you accumulate new informa-
educated folks are sure to find the regula- tion-rich cases. In most programs or sys-
tions incomprehensible. Or conversely, one tems, a few key names or incidents are men-
might consider the criticai case to be a com- tioned repeatedly. Those people or events,
munity consisting of people with quite low recommended as valuable by a number of
leveis of education: "lf they can understand different informants, take on special impor-
the regulations, anyone can." tance. The chain of recommended infor-
Identification of criticai cases depends on mants would typically diverge initially as
recognition of the key dimensions that make many possible sources are recommended,
for a criticai case. For example, a criticai case then converge as a few key names get men-
might come from a particularly difficult pro- tioned over and over.
gram location. If the funders of a new pro- The Peters and Waterman (1982) study In
gram are worried about recruiting clients or Search of Excellence began with snowball
participants into a program, it may make sampling, asking a broad group of knowl-
sense to study the site where resistance to edgeable people to identify well-run compa-
the program is expected to be greatest to pro- nies. Rosabeth Moss Kanter's (1983) study of
vide the most rigorous test of program re- innovation reported in The Change Masters
cruitment. If the program works in that site, focused on 10 core case studies of the "most
innovative" companies. She began by ask- tem that exhibit certain predetermined crite-
ing corporate experts for candidate compa- rion characteristics are routinely identified
nies to study. Nominations snowballed as for in-depth, qualitative analysis. Criterion
she broadened her inquiry and then con- sampling also can be used to identify cases
verged into a small number of core cases from standardized questionnaires for in-
nominated by a number of different expert depth follow-up, for example, ali respon-
informants. dents who report having experienced ongo-
ing workplace discrimination. (This strategy
can only be used where respondents have
8. Criterion sampling. The logic of criterion willingly supplied contact information.)
sampling is to review and study ali cases
that meet some predetermined criterion of
importance, a strategy common in quality 9. Theory-based sampling, operational con-
assurance efforts. For example, the expected struct sampling, and theoretical sampling. A
range of participation in a mental health more conceptually oriented version of crite-
outpatient program might be 4 to 26 weeks. rion sampling is theory-based sampling.
Ali cases that exceed 28 weeks are reviewed The researcher samples incidents, slices of
to find out why the expected range was ex- life, time periods, or people on the basis of
ceeded and to make sure the case was being their potential manifestation or representa-
appropriately handled. Or a quality assur- tion of important theoretical constructs.
ance standard may be that ali patients enter- Buckholt (2001) studied people who met
ing a hospital emergency room, who are not theory-derived criteria for being "resilient"
in a life-threatening situation, receive care in a study of resilience among adult abuse
within 2 hours. Cases that exceed this stan- survivors. The sample becomes, by defini-
dard are reviewed. tion and selection, representative of the phe-
Criticai incidents can be a source of crite- nomenon of interest.
rion sampling. For example, ali incidents of When one is studying people, programs,
client abuse in a program may be objects of organizations, or communities, the popula-
in-depth evaluation in a quality assurance tion of interest can be fairly readily deter-
effort. Ali former mental health clients who mined. Constructs, however, do not have as
commit suicide within three months of re- clear a frame of reference:
lease may constitute a sample for in-depth,
qualitative study. In a school setting, ali stu- For sampling operational instances of con-
dents who are absent 25% or more of the structs, there is no concrete target popula-
time may merit the in-depth attention of a tion. . . . Mostly, therefore, we are forced to
case study. The point of criterion sampling is select on a purposive basis those particular in-
to be sure to understand cases that are likely stances of a construct that past validity stud-
to be information rich because they may re- ies, conventional practice, individual intuition,
veal major system weaknesses that become or consultation with critically minded persons
targets of opportunity for program or sys- suggest offer the closest correspondence to the
tem improvement. construct of interest. Altematively, we can use
Criterion sampling can add an important the same procedures to select multiple opera-
qualitative component to a management in- tional representa tions of each construct, cho-
formation system or an ongoing program sen because they overlap in representing the
monitoring system. Ali cases in the data sys- criticai theoretical components of the con-
struct and because they differ from each other data gathered during fieldwork. The con-
on irrelevant dimensions. This second form of stant comparative method involves sys-
sampling is called multiple operationalism, tematically examining and refining varia-
and it depends more heavily on individual tions in emergent and grounded concepts.
judgment than does the random sampling of Variations in the concept must be sampled
persons from a well-designated, target popu- to rigorously compare and contrast those
lation. Yet, such judgments, while inevitable, variations. (See Chapters 3 and 8 for more
are less well understood than formal sampling detailed discussions of grounded theory.)
methods and are largely ignored by sampling
experts. (Cook, Leviton, and Shadish 1985: 10. Confirming and disconfirming cases. In
163-64) the early part of qualitative fieldwork, the
evaluator is exploringgathering data and
Operational construct sampling simply watching for patterns to emerge. Over time,
means that one samples for study real- the exploratory process gives way to confir-
world examples (i.e., operational examples) matory fieldwork. This involves testing
of the constructs in which one is interested. ideas, confirming the importance and
Studying a number of such examples is meaning of possible patterns, and checking
called "multiple operationalism" (Webb out the viability of emergent findings with
et al. 1966). For example, classic diffusion of new data and additional cases. This stage of
innovations theory (Rogers 1962) predicts fieldwork requires considerable rigor and
that early adopters of some innovation will integrity on the part of the evaluator in look-
be different in significant ways from later ing for and sampling confirming as ivell as
adopters. Doing cases studies on early and disconfirming cases.
late adopters, then, would be an example of Confirmatory cases are additional exam-
theory-based sampling. Such samples are ples that fit already emergent patterns; these
often necessarily purposefully selected be- cases confirm and elabora te the findings,
cause the population of ali early and late adding richness, depth, and credibility.
adopters may not be known, so random Disconfirming cases are no less important at
sampling is not an option. this point. These are the examples that don't
Theoretical sampling is what grounded fit. They are a source of rival interpretations
theorists define as "sampling on the basis of as well as a way of placing boundaries
the emerging concepts, with the aim being to around confirmed findings. They may be
explore the dimensional range or varied "exceptions that prove the rule" or excep-
conditions along which the properties of tions that disconfirm and alter what ap-
concepts vary" (Strauss and Corbin 1998: peared to be primary patterns.
73). In grounded theory, theoretical sam- The source of questions or ideas to be con-
pling supports the constant comparative firmed or disconfirmed may be from stake-
method of analysis. That is, one does theo- holders or previous scholarly literature
retical sampling in grounded theory in order rather than the evaluator's fieldwork. An
to use the constant comparative method of evaluation may in part serve the purpose of
analysis. The two go hand in glove, connect- confirming or disconfirming stakeholders'
ing design and analysis. Theoretical sam- or scholars' hypotheses, these having been
pling permits elucidation and refinement identified during early, conceptual evalua-
of the variations in, manifestations of, and tor-stakeholder design discussions or litera-
meanings of a concept as it is found in the ture reviews.
Thinking about the challenge of finding ever the data lead is a primary strength of
confirming and disconfirming cases empha- qualitative fieldwork strategies. This per-
sizes the relationship between sampling and mits the sample to emerge during field-
research conclusions. The sample deter- work.
mines what the evaluator will have some- During fieldwork, it is impossible to ob-
thing to say aboutthus the importance of serve everything. Decisions must be made
sampling carefully and thoughtfully. about what activities to observe, which peo-
ple to observe and interview, and when to
11. Stratified purposeful sampling. Strat- collect data. These decisions cannot ali be
ified samples are samples within samples. A made in advance. The purposeful sampling
stratified random sample, for example, strategies discussed above provide direction
might stratify by socioeconomic status for sampling but often depend on some
within a larger population so as to make knowledge of the setting being studied. Op-
generalizations and statistically valid com- portunistic, emergent sampling takes ad-
parisons by social class as well as to gener- vantage of whatever unfolds as it unfolds.
alize to the total population. In Chapter 2,1 identified emergent flexi-
Purposeful samples can also be stratified ble designs as one of the core strategic
and nested by combining types of purpose- themes of qualitative inquiry and cited as an
ful sampling. So, for example, one might exemplar the anthropologist Brackette F.
combine typical case sampling with maxi- Williams and her fieldwork on how Ameri-
mum heterogeneity sampling by taking a cans view violence in America:
stratified purposeful sample of above aver-
age, average, and below average cases. This I do impromptu interviews. I don't have some
represents less than a full maximum varia- target number of interviews in mind or prede-
tion sample, but more than simple typical termined questions. It depends on the person
case sampling. The purpose of a stratified and the situation. Airports, for example, are a
purposeful sample is to capture major varia- good place for impromptu interviews with
tions rather than to identify a common core, people. So sometimes, instead of using airport
although the latter may also emerge in the time to write, I interview people about the
analysis. Each of the strata would constitute death penalty or about killing or about death
a fairly homogeneous sample. This strategy in their life. It's called opportunity sampling
differs from stratified random sampling in I'm following where the data take me, where
that the sample sizes are likely to be to o my questions take me. (personal interview)
small for generalization or statistical repre-
sentativeness. Few qualitative studies are as fully emergent
and open-ended as the fieldwork of Wil-
12. Opportunistic or emergent sampling. liams. Her approach exemplifies emergent
Fieldwork often involves on-the-spot deci- opportunity sampling.
sions about sampling to take advantage of
new opportunities during actual data col- 13. Purposeful random sampling. A purpose-
lection. Unlike experimental designs, emer- ful sampling strategy does not automati-
gent qualitative designs can include the op- cally eliminate any possibility for random
tion of adding to a sample to take advantage selection of cases. For many audiences, ran-
of unforeseen opportunities after fieldwork dom sampling, even of small samples, will
has begun. Being open to following wher- substantially increase the credibility of the
results. I recently worked with a program about why certain cases were selected for
that annually appears before the state legis- study, but such a sample still does not permit
lature and tells "war stories" about client statistical generalizations.
successes and struggles, sometimes even in-
cluding a few stories about failures to pro- 14. Sampling politically important cases. Eval-
vide balance. To enhance the credibility of uation is inherently and inevitably political
their reports, the director and staff decided (see Turpin 1989; Falumbo 1987; Patton
to begin collecting evaluation information 1987b). A variation on the criticai case strat-
more systematically. Because they were egy involves selecting (or sometimes avoid-
striving for individualized outcomes, they ing) a politically sensitive site or unit of
rejected the notion of basing the evaluation analysis. For example, a statewide program
entirely on a standardized pre-post instru- may have a local site in the district of a state
ment. They wanted to collect case histories legislator who is particularly influential. By
and do in-depth case studies of clients, but studying carefully the program in that dis-
they had very limited resources and time to trict, evaluation data may be more likely to
devote to such data collection. In effect, staff attract attention and get used. This does not
at each program site, many of whom serve mean that the evaluator then undertakes to
200 to 300 families a year, felt that they could make that site look either good or bad, de-
only do 10 or 15 detailed, in-depth clinicai pending on the politics of the moment. That
case histories each year. We systematized would clearly be unethical. Rather, sam-
the kind of information that would be going pling politically important cases is simply a
into the case histories at each program site strategy for trying to increase the usefulness
and then set up a random procedure for se- and relevance of information where re-
lecting those clients whose case histories sources permit the study of only a limited
would be recorded in depth, thereby sys- number of cases.
tematizing and randomizing their collec- The same political perspective (broadly
tion of war stories. While they cannot gener- speaking) may inform case sampling in ap-
alize to the entire client population on the plied or even basic research studies. A politi-
basis of 10 cases from each program site, cal scientist or historian might select the
they will be able to tell legislators that the election year 2000 Florida vote-counting
stories they are reporting were randomly se- case, the Clinton impeachment effort,
lected in advance of knowledge of how the Nixon's Watergate crisis, or Reagan's Iran-
outcomes would appear and that the infor- Contra scandal for study not only because of
mation collected was comprehensive. The the insights they provide about the Ameri-
credibility of systematic and randomly se- can system of government but because of the
lected case examples is considerably greater likely attention such a study would attract.
than the personal, ad hoc selection of cases A sociologisfs study of a riot or a psycholo-
selected and reported after the factthat is, gisfs study of a famous suicide would likely
after outcomes are known. involve some attention during sampling to
It is criticai to understand, however, that the public and political importance of the
this is a purposeful random sample, not a repre- case.
sentative random sample. The purpose of a
small random sample is credibility, not 15. Convenience sampling. Finally, there is
representa ti veness. A small, purposeful the strategy of sampling by convenience:
random sample aims to reduce suspicion doing what's fast and convenient. This is
probably the most common sampling strat- ments that will lend credibility to the study
egyand the least desirable. Too often eval- as well as the kinds of arguments that
uators using qualitative methods think that might be used to attack the findings. Rea-
because the sample size they can study will sons for site selections or individual case
be too small to permit generalizations, it sampling need to be carefully articulated
doesn't matter how cases are picked, so they and made explicit. Moreover, it is important
might as well pick ones that are easy to ac- to be open and clear about a study's lrmita-
cess and inexpensive to study While conve- tions, that is, to anticipate and address criti-
nience and cost are real considerations, cisms that may be made of a particular sam-
they should be the last factors to be taken pling strategy, especially from people who
into account after strategically deliberating think that the only high-quality samples are
on how to get the most information of great- random ones.
est utility from the limited number of cases Having weighed the evidence and con-
to be sampled. Purposeful, strategic sam- sidered the alternatives, evaluators and pri-
pling can yield crucial information about mary stakeholders make their sampling de-
criticai cases. Convenience sampling is cisions, sometimes painfully, but always
neither purposeful nor strategic. with the recognition that there are no perfect
designs. The sampling strategy must be se-
Information-Rich Cases lected to fit the purpose of the study, the re-
sources available, the questions being asked,
Exhibit 5.6 summarizes the 15 purposeful and the constraints being faced. This holds
sampling strategies discussed above, plus a true for sampling strategy as well as sample
16th approachcombination or mixed pur- size.
poseful sampling. For example, an extreme
group or maximum heterogeneity approach
S a m p l e Size
may yield an initial potential sample size
that is still larger than the study can handle.
The final selection, then, may be made ran- Qualitative inquiry is rife with ambiguities.
domlya combination approach. Thus, There are purposeful strategies instead of
these approaches are not mutually exclu- methodological rules. There are inquiry ap-
sive. Each approach serves a somewhat dif- proaches instead of statistical formulas.
ferent purpose. Because research and evalu- Qualitative inquiry seems to work best for
ations often serve multiple purposes, more people with a high tolerance for ambiguity.
than one qualitative sampling strategy may (And we're still only discussing design. It
be necessary. In long-term fieldwork, ali of gets worse when we get to analysis.)
these strategies may be used at some point. Nowhere is this ambiguity clearer than in
The underlying principie that is common the matter of sample size.
to ali these strategies is selecting informa- I get letters. I get calls. I get e-mails.
tion-rich casescases from which one can
learn a great deal about matters of impor- Is 10 a large enough sample to achieve maxi-
tance and therefore worthy of in-depth mum variation?
study
In the process of developing the research I started out to interview 20 people for two
design, the evaluator or researcher is trying hours each, but I've lost 2 people. Is 18 large
to consider and anticipate the kinds of argu- enough, or do I have to find 2 more?
Designing Qualitative Studies 243
n&P&l
HBw B fflsiJSiBB
swasin D/wflras^SBfiBSWBSsasaasa
Sampling Strategies
i ^
Type Purpose
Random probabiiity sampling Representativeness: Sample size a function of population size

and desired confidence levei.
1. Simple random sample Permit generalization from sample to the population it
re presents.
2. Stratified random and Increase confidence in making generalizations to particular
cluster samples subgroups.
Purposeful sampling Select information-rich cases strategically and purposefully;

specific type and number of cases selected depends on study
purpose and resources.
1. Extreme or deviant case Learning from unusual manifestations of the phenomenon of
(outlier) sampling interest, for example, outstanding successes/notable failures;
top of the class/dropouts; exotic events; crises.
2. Intensity sampling Information-rich cases that manifest the phenomenon
intensely, but not extremely, for example, good students/
poor students; above average/below average.
3. Maximum variation Document unique or diverse variations that have emerged in
samplingpurposefully adaptng to different conditions. Identify important common
picking a wide range of patterns that cut across variations (cut through the noise of
cases to get variation on variation).
dimensions of interest
4. Homogeneous sampling Focus; reduce variation; simplify analysis; facilitate group
interviewing.
5. Typical case sampling Illustrate or highlight what is typical, normal, average.
6. Criticai case sampling Permits logical generalization and maximum application of

information to other cases because if it's true of this one case,
i f s likely to be true of ali other cases.
7. Snowball or chain Identify cases of interest from sampling people who know
sampling people who know people who know what cases are information
rich, that is, good examples for study, good interview
participants.
8. Criterion sampling Picking ali cases that meet some criterion, for example, ali
children abused in a treatment facility. Quality assurance.
9. Theory-based sampling, Finding manifestations of a theoretical construct of interest so
operational construct as to elaborate and examine the construct and its variations.
sampling, or theoretical
sampling
(continued)
I
E X H I B I T 5.6
Type Purpose
10. Confirming and Elaborating and deepening initial analysis; seeking exceptions;
disconfirming cases testing variation.
11. Stratified purposefui Illustrate characteristics of particular subgroups of interest;
sampling faciiitate comparisons.
12. Opportunistic or Following new leads during fieldwork; taking advantage of
emergent sampling the unexpected; flexibility.
13. Purposefui random Add credibility when potential purposefui sample is larger than
sampling {still small one can handle. Reduces bias within a purposefui category.
sample size) (Not for generalizations or representativeness.)
14. Sampling polticaNy Attract attention to the study (or avoid attracting undesired
important cases attention by purposefully eliminating from the sample
politically sensitive cases).
15. Convenience sampling Do what's easy to save time, money, and effort Poorest
rationale; lowest credibility. Yieds information-poor cases.
16. Combination or mixed Trianguation; flexibility; meet multiple interests and needs.
purposefui sampling
I want to study just one organization, but in- a small number of people can be very valu-
terview 20 people in the organization. Is my able, especially if the cases are information
sample size 1 or 20 or both? rich. Less depth from a larger number of
people can be especially help fui in exploring
My universal, certain, and confident reply to a phenomenon and trying to document di-
these questions is this: "It depends." versity or understand variation. I repeat, the
There are no rules for sample size in size of the sample depends on what you
qualitative inquiry. Sample size depends on want to find out, why you want to find it out,
what you want to know, the purpose of the how the findings will be used, and what re-
inquiry, what's at stake, what will be useful, sources (including time) you have for the
what will have credibility, and what can be study.
done with available time and resources. To understand the problem of small sam-
Earlier in this chapter, I discussed the ples in qualitative inquiry, it's necessary to
trade-offs between breadth and depth. With place these small samples in the context of
the same fixed resources and limited time, a probability sampling. A qualitative inquiry
researcher could study a specific set of expe- sample only seems small in comparison with
riences for a larger number of people (seek- the sample size needed for representa tive-
ing breadth) or a more open range of experi- ness when the purpose is generalizing from
ences for a smaller number of people a sample to the population of which it is a
(seeking depth). In-depth information from part. Suppose there are 100 people in a pro-
gram to be evaluated. It would be necessary mulated their widely followed eight princi-
to randomly sample 80 of those people pies for organizational excellence by study-
(80%) to make a generalization at the 95% ing 62 companies, a very small sample of the
confidence levei. If there are 500 people in thousands of companies one might study.
the program, 217 people must be sampled Sands (2000) did a fine dissertation studying
(43%) for the same levei of confidence. If a single school principal, describing the
there are 1,000 people, 278 people must be leadership of a female leader who entered a
sampled (28%), and if there are 5,000 people challenging school situation and brought
in the population of interest, 357 must be about constructive change.
sampled (7%) to achieve a 95% confidence Clair Claiborne Park's (2001) single case
levei in the generalization of findings. At the study of her daughter's autism reports 40
other extreme, if there are only 50 people in years of data on every stage of her develop-
the program, 44 must be randomly sampled ment, language use, emotions, capacities,
(88%) to achieve a 95% levei of confidence. barriers, obsessions, communication pat-
(See Fitz-Gibbon and Morris [1987:163] for a terns, emergent artistry, and challenges
table on determining sample size from a overcome and challenges not overcome.
given population.) Park and her husband made systematic ob-
The logic of purposeful sampling is quite servations throughout the years. Eminent
different. The problem is, however, that the medicai anthropologist Oliver Saks re-
utility and credibility of small purposeful viewed the data and determined in his pref-
samples are often judged on the basis of the ace to the book that more data are available
logic, purpose, and recommended sample on the woman in this extraordinary case
sizes of probability sampling. Instead, pur- study than on any other autistic human be-
poseful samples should be judged according ing who has ever lived. Here, then, is the
to the purpose and rationale of the study: epitome of N = 1, in-depth inquiry.
Does the sampling strategy support the The validity, meaningfulness, and in-
study's purpose? The sample, like ali other sights generated from qualitative inquiry
aspects of qualitative inquiry, must be have more to do with the information rich-
judged in contextthe same principie that ness of the cases selected and the obser-
undergirds analysis and presentation of vational/analytical capabilities of the re-
qualitative data. Random probability sam- searcher than with sample size.
ples cannot accomplish what in-depth, pur- This issue of sample size is a lot like the
poseful samples accomplish, and vice versa. problem students have when they are as-
Piaget contributed a major breakthrough signed an essay to write.
to our understanding of how children think
by observing his own two children at length Student: "How long does the paper have to
and in great depth. Freud established the be?"
field of psychoanalysis based originally on
fewer than 10 client cases. Bandler and Instructor: "Long enough to cover the as-
Grinder (1975a, 1975b) founded neurolin- signment."
guistic programming (NLP) by studying
Student: "But how many pages?"
three renowned and highly effective thera-
pists: Milton Erickson, Fritz Perls, and Vir- Instructor: "Enough pages to do justice to
gnia Satir. Peters and Waterman (1982) for- the subjectno more, no less."
Lincoln and Guba (1985) recommend evaluator is obligated to discuss how the
sample selection "to the point of redun- sample affected the findings, the strengths
dancy.... In purposeful sampling the size of and weaknesses of the sampling proce-
the sample is determined by informational dures, and any other design decisions that
considerations. If the purpose is to maxi- are relevant for interpreting and under-
mize information, the sampling is termi- standing the reported results. Exercising
nated when no new information is forth- care not to overgeneralize from purposeful
commg from new sampled units; thus samples, while maximizing to the full the
rednndancy is the primary criterion" (p. 202). advantages of in-depth, purposeful sam-
This strategy leaves the question of sampling, will do much to alleviate concerns
ple size open, another example of the emer- about small sample size.
gent nature of qualitative inquiry. There re-
mains, however, the practical problem of
how to negotiate an evaluation budget or get s. Emergent Designs
a dissertation committee to approve a de- and Protection of
sign if you don't have some idea of sample Human Subjects
size. Sampling to the point of redundancy is
an ideal, one that works best for basic re- Emergent designs pose special problems for
search, unlimited timelines, and uncon- institutional review boards (IRBs) charged
strained resources. with approving research designs to ensure
The solution is judgment and negotiation. protection of human subjects. Such boards
I recommend that qualitative sampling de- typically want to know, in advance of field-
signs specify minimum samples based on ex- work, who will be interviewed and the pre-
pected reasonable coverage of the phenome- cise questions that will be asked. If the topic
non given the purpose of the study and is fairly innocuous and the general line of
stakeholder mterests. One may add to the questioning relatively unobtrusive, an IRB
sample as fieldwork unfolds. One may may be willing to approve the framework of
change the sample if information emerges an emergent design with sample questions
that indicates the value of a change. The de- included, but without full sample specifica-
sign should be understood to be flexible and tion and a formal interview instrument.
emergent. Ye t, at the beginning, for planning Another approach is to ask for approval
and budgetary purposes, one specifies a in stages. This means initially asking for
minimum expected sample size and builds a approval for the general framework of the
rationale for that minimum, as well as crite- inquiry and specifically for the first ex-
ria that would alert the researcher to inade- ploratory stage of fieldwork, including pro-
quacies in the original sampling approach cedures for assuring confidentiality and in-
and/or size. formed consent, then returning periodically
In the end, sample size adequacy, like ali (e.g., quarterly or annually) to update the
aspects of research, is subject to peer review, design and its approval. This is cumber-
consensual validation, and judgment. What some for both the researcher and the IRB, but
is crucial is that the sampling procedures it is a way of meeting IRB mandates and still
and decisions be fully described, explained, implementing an emergent design. This
and justified so that information users and staged-approval approach can also be used
peer reviewers have the appropriate context when the evaluator is developing the design
for judging the sample. The researcher or jointly with program staff and/or partici-
pants and therefore cannot specify the full types of data. The chapters on interviewing,
design at the beginning of the participatory observation, and analysis will include infor-
process. mation that will help in making design deci-
sions. Before turning to those chapters,
Methodological Mixes however, I want to briefly discuss the value
A study inay employ more than one sam- of using multiple methods in research and
pling strategy. It may also include multiple evaluation.
Triangulation
T he method must follow the question. Campbell, many decades ago,

promoted the concept of triangulationthat every method has its
limitations, and multiple methods are usually needed.
Gene V. Glass eulogizing pioneering methodologist
Donald T. Campbell, quoted in Tashakkori and Teddlie (1998:22)
Triangulation strengthens a study by sic domes la Buckminster Fuller). The logic

combining methods. This can mean using of triangulation is based on the premise that
several kinds of methods or data, including
using both quantitative and qualitative ap- no single method ever adequately solves the
proaches. Denzin (1978b) has identified four problem of rival causai factors. Because each
basic types of triangulation: (1) data triangu- method reveals different aspects of empirical
lation, the use of a variety of data sources in a reality, multiple methods of observations
study; (2) investigator triangulation, the use of must be employed. This is termed triangula-
several different researchers or evaluators; tion. I now offer as a final methodological rule
(3) theory triangulation, the use of multiple the principie that multiple methods should be
perspectives to interpret a single set of data, used in every investigation. (Denzin 1978b:28)
and (4) methodological triangulation, the use of
multiple methods to study a single problem Triangulation is ideal. It can also be ex-
or program. pensive. A study's limited budget and time
The term triangulation is taken from land frame will affect the amount of triangulation
surveying. Knowing a single landmark only that is practical, as will political constraints
locates you somewhere along a line in a di- (stakeholder values) in an evaluation. Cer-
rection from the landmark, whereas with tainly, one important strategy for inquiry is
two landmarks (and your own position be- to employ multiple methods, measures, re-
ing the third point of the triangle) you can searchers, and perspectives but to do so
take bearings in two directions and locate reasonably and practically.
yourself at their intersection (Fielding and
Fielding 1986:23). The term triangulation also Most good researchers prefer addressing their
works metaphorically to call to mind the research questions with any methodological
world's strongest geometric shapethe tri- tool available, using the pragmatist credo of
angle (e.g., the formused to construct geode- "what works." For most researchers commit-
ted to the thorough study of a research prob- or examining how competing theoretical
lem, method is secondary to the research perspectives inform a particular analysis
question itself, and the underlying worldview (e.g., the transcendental phenomenology of
hardy enters the picture, except in the most ab- Husserl vs. the hermeneutic phenomenol-
stract sense. (Tashakkori and Teddlie 1998:22) ogy of Heidegger). A study can also be de-
signed to cut across inquiry approaches and
A rich variety of methodological combi- achieve triangulation by combining qualita-
nations can be employed to illuminate an tive and quantitative methods, a strategy
inquiry question. Some studies intermix in- discussed and illustrated in the next section.
terviewing, observation, and document
analysis. Others rely more on interviews
Mixing Data, Design,
than observatons, and vice versa. Studies
and Analysis Approaches
that use only one method are more vulnera-
ble to errors linked to that particular method
(e.g., loaded interview questions, biased or Borrowing and combining distinct ele-
untrue responses) than studies that use mul- ments from pure or coherent methodologi-
tiple methods in which different types of cal strategies can generate creative mixed in-
data provide cross-data validity checks. quiry strategies that illustrate variations on
Using multiple methods alio ws inquiry into the theme of triangulation. We begin by
a research question with "an arsenal of distinguishing measurement, design, and
methods that have nonoverlapping weak- analysis components of the hypothetico-
nesses in addition to their complementary deductive (quantitative/experimental) and
strengths" (Brewer and Hunter 1989:17). holistic-inductive (qualitative/naturalistic)
However, a common misunderstanding paradigms. The ideal-typical qualitative
about triangulation is that the point is to methods strategy is made up of three parts:
demonstrate that different data sources or (1) qualitative data, (2) a holistic-inductive
inquiry approaches yield essentially the design of naturalistic inquiry, and (3) con-
same result. But the point is really to test for tent or case analysis. In the traditional
such consistency. Different kinds of data hypothetico-deductive approach to re-
may yield somewhat different results be- search, the ideal study would include (a)
cause different types of inquiry are sensitive quantitative data from (b) experimental (or
to different real-world nuances. Thus, un- quasi-experimental) designs and (c) statisti-
derstanding inconsistencies in findings cal analysis.
across different kinds of data can be Measurement, design, and analysis alter-
illuminative. Finding such inconsistencies natives can be mixed to create eclectic de-
ought not be viewed as weakening the credi- signs, like customizing an architectural plan
bility of results, but rather as offering oppor- to tastefully integra te modern, postmodern,
tunities for deeper insight into the relation- and traditional elements, or preparing an el-
ship between inquiry approach and the egant dinner with a French appetizer, a Chi-
phenomenon under study. nese entre, and an American dessertnot
Triangulation within a qualitative in- to everyone's taste, to be sure, but the possi-
quiry strategy can be attained by combining bilities are endless. At least that's the con-
both interviewing and observations, mixing cept. To make the idea of mixed elements
different types of purposeful samples (e.g., more concrete and to illustrate the creative
both intensity and opportunity sampling), possibilities that can emerge out of a flexible
Triangulation
approach to research, it will help to examine style, high drug use), and who are likely can-
alterna tive design possibilities for a single didates for delinquency (ahenated from
program evaluation. The examples that fol- dominant societal values, running with a
low have been constructed under the artifi- "bad" crowd, and angry). The program con-
cial constraint that only one kind of mea- sists of experiential education internships
surement, design, and analysis could be through which these high-risk students get
used in each case. In practice, of course, the individual tutoring in basic skills, part-time
possible mixes are much more varied, be- job placements that permit them to earn in-
cause any given study could include several come while gaining work exposure, and
measurement approaches, varying design participation in peer group discussions
approaches, and varying different analytical aimed at changing health values, establish-
approaches to achieve triangulation. ing a positive peer culture, and increasing
social integra tion. Several evaluation ap-
proaches are possible.
The Case of Opera tion
Reach-Out: Variations in
Program Evaluation Design PURE HYPOTHET1CA L-DEDUCTIVE
APPROACH TO EVALUATION:
EXPERIMENTAL DESIGN,
Let's consider design alternatives for a QUANTITATIVE DATA, AND
comprehensive program aimed at high STATISTICAL ANALYSIS
school students at high risk educa tionally
(poor grades, poor attendance, poor atti- The program does not have sufficient re-
tudes toward school), with highly vulnera- sources to serve ali targeted youth in the
ble health (poor nutrition, sedentary life- population. A pool of eligible youth is estab-
lished with admission into the program on a ing. Near the end of the program, in-depth
random basis and the remaining group re- interviews are conducted with the partici-
ceives no immediate treatment intervention. pants to learn whatbehaviors have changed,
Before the program begins and one year how they view things, and what their expec-
later, ali youth, both those in the program tations are for the future. Interviews are also
and those in the control group, are adminis- conducted with program staff and some par-
tered standardized instruments measuring ents. These data are content analyzed to
school achievement, self-esteem, anomie, identify the patterns of experiences partici-
alienation, and locus of control. Rates of pants bring to the program, what patterns
school attendance, illness, drug use, and de- characterize their participation in the pro-
linquency are obtained for each group. gram, and what patterns of change are re-
When ali data have been collected at the end por ted by and observed in the participants.
of the year, comparisons between the control
and experimental groups are made using in- MIXED FORM: EXPERIMENTAL
ferential statistics. DESIGN, QUALITATIVE DATA, AND
CONTENT ANALYSIS
PURE QUALITATIVE STRATEGY: As in the pur experimental form, poten-
NATURALISTIC INQUIRY, tial participants are randomly assigned to
QUALITATIVE DATA, AND treatment and control groups. In-depth in-
CONTENT ANALYSIS terviews are conducted with ali youth, both
those in the treatment group and those in the
Procedures for recruiting and selecting control group, and both before the program
participants for the program are determined begins and at the end of the program. Con-
entirely by the staff. The evaluator finds a tent and thematic analyses are performed so
convenient time to conduct an in-depth in- that the control and experimental group pat-
terview with new participants as soon as terns canbe compared and contrasted. (For a
they are admitted into the program, asking detailed example combining experimental
students to describe what school is like for controls and ethnography, see Maxwell,
them, what they do m school, how they typi- Bashook, and Sandlow 1987.)
cally spend their time, what their family life
is like, how they approach academic tasks,
MIXED FORM: EXPERIMENTAL
their views about health, and their behav-
DESIGN, QUALITATIVE DATA,
iors/attitudes with regard to delinquent
AND STATISTICAL ANALYSIS
and criminal activity. In brief, participants
are asked to describe themselves and their Participants are randomly assigned to
social world. The evaluator observes pro- treatment and control groups, and in-depth
gram activities, collecting detailed descrip- interviews are conducted both before the
tive data about staff-participant interactions program and at its end. These interview data,
and conversa tions, staff intervention efforts, in raw form, are then given to a panei of
and youth reactions. The evaluator finds op- judges, who rate each interview along sev-
portunities for additional in-depth inter- eral outcome dimensions operationalized as
views with participants to find out how they a 10-point scale. For both the "pre" inter-
view the program, what kinds of experi- view and the "post" interview, the judges as-
ences they are having, and what they're do- sign ratings on such dimensions as likeli-
hood of success in school (low - 1, high = MIXED FORM: NATURALISTIC

10), likelihood of committing criminal of- INQUIRY, QUANTITATIVE DATA,
fenses (low = 1, high = 10), commitment to AND STATISTICAL ANALYSIS
education, commitment to engaging in pro-
Students are selected for the program ac-
ductive work, self-esteem, and manifesta-
cording to staff criteria. The evaluator enters
tion of desired nutritional and health habits.
the program setting without any predeter-
Inferential statistics are then used to com-
mined categories of analysis or presupposi-
pare these two groups. Judges make the rat-
tions about important variables or variable
ings without knowledge of which partici-
relationships. The evaluator observes im-
pants were in which group. Outcomes on the
portant activities and events in the program,
rated scales are also statistically related to
looking for the types of behaviors and inter-
background characteristics of participants.
actions that will emerge. For each significant
type of behavior or interaction observed, the
MIXED FORM: NATURALISTIC evaluator creates a category and then uses a
INQUIRY, QUALITATIVE DATA, time and space sampling design to count the
AND STATISTICAL ANALYSIS frequency with which those categories of
behavior and interaction are exhibited. The
As in the pure qualitative form, students frequency of the manifestation of observed
are selected for the program on the basis of behaviors and interactions are then statisti-
whatever criteria staff members choose to cally related to such characteristics as group
apply. In-depth interviews are conducted size, duration of the activity, staff-student
with ali students before and at the end of the ratios, and social/physical density.
program. These data are then submitted to a
panei of judges, who rate them on a series of
dimensions similar to those listed in the pre- Alternative Pure
vious example. Change scores are computed and Mixed Strategies
for each individual, and changes are statisti-
cally related to background characteristics Exhibit 5.7 summarizes the six alternative
of the students to determine in a regression design scenarios we created and just re-
format which characteristics of students are viewed for evaluation of "Operation Reach-
likely to predict success in the program. In Out." As these alternative designs illustrate,
addition, observations of program activities purity of approach is only one option. In-
are rated on a set of scales developed to quiry strategies, measurement approaches,
quantify the climate attributes of activities: and analysis procedures can be mixed and
for example, the extent to which the activity matched in the search for relevant and use-
involved active or passive participation, the ful information. That said, it is worth consid-
extent to which student-teacher interaction ering the case for maintaining the integrity
was high or low, the extent to which interac- and purity of qualitative and quantitative
tions were formal or informal, and the extent paradigms. The 12 themes of qualitative
to which participants had input into pro- inquiry described in the second chapter (Ex-
gram activities. Quantitative ratings of ac- hibit 2.1) do fit together as a coherent strat-
tivities based on qualitative descriptions are egy. The openness and personal involve-
then aggregated to provide an overview of ment of naturalistic inquiry mesh well with
the treatment environment of the program. the openness and depth of qualitative data.
Measurement, Design, and Analysis:

E X H I B I T 5.7 Pure and Mixed Combinations
Naturalistic Experimental <0

cg Inquiry Design
t
\ /
\
N
S ?? collect collect collect collect

S
J5 ^^ qualitative quantitative qualitative quantitative
Cv data data data data
L,
& i
perform
V
11
L I
perform perform perform
s content
analysis
statistical
analysis
content
analysis
statistical
analysis
Mixed Strategies
(middle paths)
Genuine openness flows naturally from an gies. Their cautions are not to be dismissed
inductive approach to analysis, particularly lightly. Mixing parts of different approaches
an analysis grounded in the immediacy of is a matter of philosophical and method-
direct fieldwork and sensitized to the desir- ological controversy. Yet, the practical man-
ability of holistic understanding of unique date in evaluation (Patton 1981) to gather
human settings. the most relevant possible information for
Likewise, there is an internai consistency evaluation users outweighs concerns about
and logic to experimental designs that test methodological purity based on episte-
deductive hypotheses derived from theoret- mological and philosophical arguments.
ical premises. These premises identify the The intellectual mandate to be open to what
key variables to consider in testing theory or the world has to offer surely includes meth-
measurmg, controlling, and analyzing hy- odological openness. In practice, it is alto-
pothesized relationships between program gether possible, as we have seen, to combine
treatments and outcomes. The rules and approaches, and to do so creatively (Patton
procedures of the quantitative-experimental 1987a). Just as machines that were originally
paradigm are aimed at producing internally created for separate functions such as print-
valid, reliable, replicable, and generalizable ing, faxing, scanning, and copying have
findings. now been combined into a single integrated
Guba and Lincoln (1988) have argued technological unit, so too methods that were
that the internai consistency and logic of originally created as distinct, stand-alone
each approach, or paradigm, mitigates approaches can now be combined into
against methodological mixing of different more sophisticated and multifunctional
inquiry modes and data collection strate- designs.
Advocates of methodological purity ar- the ideal; quasi-experimental designs often

gue that a single evaluator cannot be both represent what is possible and practical.
deductive and inductive at the same time, or Likewise, full participant observation over
cannot be testing predetermined hypotheses an extended period of time is the qualitative
and still remain open to whatever emerges ideal. In practice, many acceptable and
from open-ended, phenomenological obser- meaningful variations to qualitative inquiry
vation. Yet, in practice, human reasoning is can be designed.
sufficiently complex and flexible that it is This spirit of adaptability and creativity
possible to research predetermined ques- in designing studies is aimed at being prag-
tions and test hypotheses about certain as- matic, responsive to real-world conditions
pects of a program while being quite open and, when doing evaluations, to meeting
and naturalistic in pursuing other aspects of stakeholder information needs. Mixed
a program. In principie, this is not greatly methods and strategies allow creative re-
different from a questionnaire that includes search adaptations to particular settings and
both fixed-choice and open-ended ques- questions, though certain designs pose con-
tions. The extent to which a qualitative ap- straints that exclude other possibilities. It is
proach is inductive or deductive varies not possible, for example, for a program to
along a continuum. As evaluation fieldwork operate as an experiment by assigning par-
begins, the evaluator may be open to what- ticipants to treatment and control groups
ever emerges from the data, a discovery or while at the same time operating the pro-
inductive approach. Then, as the inquiry re- gram under naturalistic inquiry conditions
veals pattems and major dimensions of in- in which ali eligible participants enter the
terest, the evaluator will begin to focus on program (and thus there is no control group
verifying and elucidating what appears to and no random assigmnent). Another in-
be emerginga more deductively oriented compatibility: Qualitative descriptions can
approach to data collection and analysis. be converte d into quantitative scales for
The extent to which a study is naturalistic purposes of statistical analysis, but it is not
in design is also a matter of degree. This ap- possible to work the other way around and
plies particularly with regard to the extent to convert purely quantitative measures into
which the investiga tor places conceptual detailed, qualitative descriptions.
constraints on or makes presuppositions
about the program or phenomenon under
study. In practice, the naturalistic approach Design and Methods Decisions
may often involve moving back and forth
between inductive, open-ended encounters Which research design is best? Which
and more hypothetical-deductive attempts strategy will provide the most useful infor-
to verify hypotheses or solidify ideas that mation to decision makers? No simple and
emerged from those more open-ended expe- universal answer to that question is possi-
riences, sometimes even manipulating ble. The answer in each case will depend on
something to see what happens. the purpose of the study, the scholarly or
These examples of variations in qualita- evaluation audience for the study (what in-
tive approaches are somewhat like the dif- tended users want to know), the funds avail-
ferences between experimental and quasi- able, the political context, and the interests/
experimental designs. Pure experiments are abilities/biases of the researchers. Exhibit 5.8
E X H I B I T 5.8 Design Issues and Options
issues Design Options and Concems
1. What is the primary Basic research, applied research, summative evaluation,

purpose of the study? formative evaluation, action research. (See Exhibits 5.1-5.4)
2. What is the focus of study? Breadth versus depth trade-offs.
3. What are the units of Individuais, groups, program components, whole program,
analysis? organizations, communities, criticai incidents, time periods, etc.
(See Exhibit 5.5)
4. What will be the sampling Purposeful sampling, probability sampling. Variations in
strategy or strategies? sample size from a single case study to a generalizable
sample. (See Exhibit 5.6)
5. What types of data will Qualitative, quantitative, or both. (See Exhibit 5.7)
be collected?
6. What type and degree of Naturalistic inquiry (no control), experimental design, quasi-
control will be exercised? experimental.
7. What analyttcal approach Inductive, deductive. Content or thematic analysis, statistical
or approaches will be used? analysis, combinations. (See Exhibit 5.7)
8. How will the validity of and Triangulation options, multiple data sources, multiple
confidence in the findings methods, multiple perspectives, multiple investigators.
be addressed?
9. Time issues: When will the Long-term fieldwork, rapid reconnaissance, exploratory
study occur? How will the phase to confirmatory phase, fixe d times versus open timelines.
study be sequenced or
phased?
10. How will logistics and Gaining entry to the setting, access to people and records,
practicalities be handled? contracts, training, endurance, etc.
11. How will ethical issues and Informed consent, protection of human subjects, reactivity,
matters of confidentiality presentation of self, etc.
be handled?
12. What resources will be Personnel, supplies, data collection, materiais, analysis time
available? What will the and costs, reporting/publishing costs.
study cost?
summarizes the issues discussed in this gests a very specific blueprint, but "design
chapter that must be addressed in designing in the naturalistic sense . . . means planning
a study. for certain broad contingencies without,
In qualitative inquiry, the problem of de- however, indicating exactly what will be
sign poses a paradox. The term design sug- done in relation to each" (Lincoln and Guba
TVJO niÎRlM PlilRSi-llcriV']:.::!'

/iLi "l j "!'!! iVi.V! ! ; .VYti': :.= ! i", .'-I W
M " iTiVi1!!; J: ni." I ..! yrt i I 3'rJ I , - . , : ! i,;; | .!= r=; i ; V ! !n j.-yp! r!i.l.Vvi i.i!- i ii I Wfr-^ * t.fi.1i"i:!'!" n.
V IV "] ! H" I J: i!!' }: i- i M \ vi ! V BFL !:!. =V=:;I "j/i ;: s 5 r i " i !=;; N; Y n '.'i !;j!!!':.!!'' V.S i, 1 M h I! ' I.:: ro ' i! I:
'V!!!'=!''=!:! ' ' T ! U ' i'i! iVii- :i!ir. JfEipr !;: I;:! ji'
.}/::!!= !!=>:= ,"::=;,!i-i<:".M ST! nViriiii 1 1 hj"j'. hU !-.!Hrr t " u - "'i ''"::<i.i ':>.i! i'iVii'riiri.lA:'I MYi!,':V!i.!!
1 1
:'!;: iXijj!!|!I I I ! . . T I Y ! ' ! ! ' i;|- :!|Y,HJJ! I ' . ' I F R ! ' ! ! H * H - R Y LI? i'I!":'.;I:"
R
' ' : M T ! !T:.|; > ; ! ; ' R.3;" ^ri.VH.' ''fiu,' kiiV ':.'' i'v. A!">r .'i 1r
- !
"fite I.ii.isii.:'.V!:' i-ri !:;!'

A I T ! : . H T F I M ' ! \ ! lI,Y>;!I!:>! .H!..IIS:S:'!Rri ! ! ' . ; ' . ! I : "P!!::r". I"'.'IVII 1 Ivi
H
. ''.i.:n"' ii'ri i'i:! ::'! ': i.1;:'!:! "i::;.;. \ \ >i !' :?Cl 1" ! T =:V 'r. r !! ^.'i I; i ! 'T- !i ,!<:Y :'. f T H ! r i i. i i 'i . i'
!h> 5 .' i v !i-:i! '.'!:'!.': %"!' ii !!i:'-ii! yiwffln-J-mV ch^^l t A i 1 .ii .\.':Vi!':; V!.': i.V.'! WVI rr rn! r *: 1l:r ','
: !
! ir!:' u:- .jrd. ;::; :!!!:: l.- "

p 5 1
rm I !:'.' i:=:i l i \. ^ViVi :i =: ;Y V! ;'. > : : H.in ;I; !=;: !,; i r M v-,fti' ==i., i,^-..! 1 > \ L.M*: "! i n!! il! i'0 |i ? " V' . n
i-ihiv.Vin V.:,'jW.::!! : ! : : j V i h :!.!i'V'!:;i m''^^hn-n =:IVsi: II: ^V n! n i:l-yi 1 i;v.:;s =j" n-i>
lVi.Vii! !:!: !3' i ;V ! H: t:;y :;i jj-! n^.Ti'.: ?}, ; " T; ' f,Wi I'- A i . S! i I!1: "'
1985:226). A qualitative design needs to re- writer, have their favorite research methods
main sufficiently open and flexible to permit with which they are familiar and have some
exploration of whatever the phenomenon skill in using. And I suspect we mostly choose
under study offers for inquiry. Qualitative to investigate problems that seem vulnerable
designs continue to be emergent even after to attack through these methods. But we
data collection begins. The degree of flexibil- should at least try to be less parochial than
ity and openness is, however, also a matter cobblers. Let us be done with the arguments of
of great variation among designs. participant observation versus interviewing
What is certain is that different methods as we have largely dispensed with the argu-
can produce quite different findings. The ments for psychology versus sociologyand
challenge is to figure out which design and get on with the business of attacking our
methods are most appropriate, productive, problems with the widestarray of conceptual
and useful in a given situation. Martin Trow and methodological tools that we possess and
(1970) points out (quite nicely, I think) the they demand. This does not preclude discus-
difference between arguments about which sion and debate regarding the relatve useful-
methods are most appropriate for studying ness of different methods for the study of
a particular problem and arguments about specific problems or types of problems. But
the intrinsic and universal superiority of one that is very different from the assertion of the
method over another: general and inherent superiority of one
method over another on the basis of some
Every cobbler thinks leather is the only thing. intrinsic qualities it presumably possesses.
Most social scientists, including the present (P-149)
Professor, for the last

Sounds like it's
four years I've been time to take it to
struggling to define my the next levei.
research problem, but it Fuzzy methods.
still seems too fuzzy.
Sophisticated emergent design strategy

Choices
very path we take leads to fantasies about the path not taken.
Halcolm
This chapter has suggested that research will be able to make the same rational deci-
and evaluation should be built on the foun- sion that I have made.
dation of a "paradigm of choices" rather "Honey comes conveniently packaged in
than become the handmaiden of any single beautifully shaped prisms of the most deli-
and inevitably narrow disciplinary or meth- cate texture. It's ready to eat, slides down the
odological paradigm. But be careful, the throat ever so easily, is a highly nutritious
Sufis would warn us, for the exercise of real source of energy, digests smoothly, and
choice can be elusive. Trow admonishes us leaves a lingering taste of sweetness on the
to "at least try to be less parochial than cob- palate that provides pleasure for hours.
blers." The evaluation sage, Halcolm, might Honey is readily available and requires no
suggest that ali too often the methods deci- special labor to produce since bees do ali the
sions made become like the bear's "deci- work. Its pleasing aroma, light weight, resis-
sion" to like honey. tance to spoilage, and uniformly high qual-
One day, in a sudden impulse of generos- ity make it a food beyond compare. It comes
ity, a bear decided to enlighten the other ani- ready to consumeno peeling, no killing, no
mais in the forest about the marvelous prop- tearing openand there's no waste. Whafs
erties of honey. The bear assembled ali the more, it has so many uses; it can be eaten
other animais together for his momentous alone or added to enhance any other food.
announcement. "I could go on and on, but suffice it to say
"I have studied the matter at great that I have studied the situation quite objec-
length," began the bear, "and I have decided tively and at great length. A fair and rational
that honey is the best of ali foods. Therefore, analysis leads to only one conclusion.
I have chosen to like honey I am going to de- Honey is the supreme food and any reason-
scribe to you the perfect qualities of honey, able animal will undoubtedly make the
which, due to your past prejudices and lack same conscious decision I have made. I have
of experience, you have ignored. Then you chosen to like honey."
Fieldwork Strategies
and Observation Methods
X o LAnde^s+and +ke W o ^ l d
And the children said unto Halcolm, "We want to understand the world. Tell
us, O Sage, what must we do to know the world?"
"Have you read the works of our great thinkers?"
"Yes, Master, every one of them as we were instructed."
"And have you practiced diligently your meditations so as to become One
with the infinity of the universe?"
"We have, Master, with devotion and discipline."
"Have you studied the experiments, the surveys, and the mathematical mod-
els of the Sciences?"
"Beyond even the examinations, Master, we have studied in the innermost
chambers where the experiments and surveys are analyzed, and where the
mathematical models are developed and tested."
"Still you are not satisfied? You would know more?"
"Yes, Master. We want to understand the world."
"Then, my children, you must go out into the world. Live among the peoples
of the world as they live. Learn their language. Participate in their rituais and
routines. Taste of the world. Smell it. Watch and listen. Touch and be touched.
Write down what you see and hear, how they think and how you feel.
"Enter into the world. Observe and wonder. Experience and reflect. To under-
stand a world you must become part of that world while at the same time re-
maining separa te, a part of and apart from.
"Go then, and return to tell me what you see and hear, what you learn, and
what you come to understand."
From Halcolm's Methodological Chronicle
l 259
Folk Wisdom About Human Observation

W
mna
I n the fields of observation, chance favors the prepared mind.
Louis Pasteur (1822-1895)
P eople only see what they are prepared to see.
Ralph Waldo Emerson (1803-1882)
Every student who takes an introductory range, and lasted only twenty seconds. But the
psychology or sociology course learns that observers could not observe ali that hap-
human perception is highly selective. When pened. Some readers chuckled because the ob-
looking at the same scene or object, different servers were researchers, but similar experi-
people will see different things. What people ments have been reported numerous times.
"see" is highly dependent on their inter- They are alike for ali kinds of people. (Katzer
ests, biases, and backgrounds. Our culture et al. 1978:21-22)
shapes what we see, our early childhood so-
cialization forms how we look at the world, Using this story to cast doubt on ali vari-
and our value systems tell us how to inter- eties of observational research manifests
pret what passes before our eyes. How, then, two fundamental fallacies: (1) These re-
can one trust observational data? searchers were not trained as social science
In their classic guide for users of social observers, and (2) they were not prepared to
science research, Katzer, Cook, and Crouch make observations at that particular mo-
(1978) titled their chapter on observation ment. Scientific inquiry using observa-
"Seeing Is Not Believing." They open with tional methods requires disciplined train-
an oft-repeated story meant to demonstrate ing and rigorous preparation.
the problem with observational data. The fact that a person is equipped with
functioning senses does not make that per-
Once at a scientific meeting, a man suddenly son a skilled observer. The fact that ordinary
rushed into the midst of one of the sessions. persons experiencing any particular inci-
Another man with a revolver was chasing dent will highlight and report different
him. They scuffled in plain view of the assem- things does not mean that trained and pre-
bled researchers, a shot was fired, and they pared observers cannot report with accuracy,
rushed out. About twenty seconds had authenticity, and reliability that same inci-
elapsed. The chairperson of the session imrne- dent.
diately asked ali present to write down an ac- Training to become a skilled observer in-
count of what they had seen. The observers cludes
did not know that the ruckus had been
planned, rehearsed, and photographed. Of the B learning to pay attention, see what there
forty reports turned in, only one was less than is to see, and hear what there is hear;
20 percent mistaken about the principal facts,
and most were more than 40 percent mistaken. H practice in writing descriptively;
The event surely drew the undivided atten- H acquiring discipline in recording field
tion of the observers, was in full view at close notes;
Fieldwork Strategies and Observation Methods LET. 261
0 knowing how to separate detail from and ears, my observational senses. A scien-
trivia to achieve the former without be- tific observer cannot be expected to engage
ing overwhelmed by the latter; in systematic observation on the spur of the
using rigorous methods to validate and moment any more than a world-class boxer
triangulate observations; and can be expected to defend his title spontane-
ously on a street corner or an Olympic run-
reporting the strengths and limitations ner can be asked to dash off at record speed
of one's own perspective, which requires because someone suddenly thinks it would
both self-knowledge and self-disclosure. be nice to test the runner's time. Athletes,
artists, musicians, dancers, engineers, and
Training observers can be particularly scientists require training and mental prepa-
challenging because so many people think ration to do their best. Experiments and sim-
that they are "natural" observers and there- ulations that document the inaccuracy of
fore have little to learn. Training to become a spontaneous observations made by un-
skilled observer is a no less rigorous process trained and unprepared observers are no
than the training necessary to become a more indicative of the potential quality of
skilled survey researcher or statistician. Peo- observational methods than an amateur
ple don't "naturally" know how to write community talent show is indicative of what
good survey items or analyze statistics professional performers can do.
and people don't "naturally" know how to Two points are criticai, then, in this intro-
do systematic research observations. Ali ductory section. First, the folk wisdom about
forms of scientific inquiry require training observation being nothing more than selec-
and practice. tive perception is true in the ordinary course
Careful preparation for entering into of participating in day-to-day events. Sec-
fieldwork is as important as disciplined ond, the skilled observer is able to improve
training. Though I have considerable experi- the accuracy, authenticity, and reliability of
ence doing observa tional fieldwork, had I observations through intensive training and
been present at the scientific meeting where rigorous preparation. The remainder of this
the shooting scene occurred my recorded chapter is devoted to helping evaluators and
observations might not have been signifi- researchers move their observations from
cantly more accurate than those of my less the levei of ordinary looking to the rigor of
trained colleagues because I would not have systematic seeing.
been prepared to observe what occurred and,
lacking that preparation, would have been
The Value of
seeing things through my ordinary eyes
Direct Observations
rather than my scientific observer's eyes.
Preparation has mental, physical, intel- I'm often asked by students: "Isn't inter-
lectual, and psychological dimensions. Pas- viewing just as good as observation? Do you
teur said, "In the fields of observation, really have to go see a program directly to
chance favors the prepared mind." Part of evaluate it? Can't you find out ali you need
preparing the mind is learning how to con- to know by talking to people in the program
centrate during the observation. Observa- without going there and seeing it first-
tion, for me, involves enormous energy and hand?"
concentration. I have to "turn on" that con- I reply by relating my experience evaluat-
centration"turn on" my scientific eyes ing a leadership development program with
two colleagues. As part of a formative evalu- the reader to enter into and understand the
ation aimed at helping staff and funders situation described. In this way, evaluation
clarify and improve the program's design users, for example, can come to understand
before undertaking a comprehensive fol- program activities and impacts through de-
low-up study for a summative evaluation, tailed descriptive information about what
we went through the program as participant has occurred in a program and how the peo-
observers. After completing the six-day ple in the program have reacted to what has
leadership retreat, we met to compare expe- occurred.
riences. Our very first conclusion was that Naturalistic observations take place in the
we would never have understood the pro- field. For ethnographers, the field is a cul-
gram without personally experiencing it. It tural setting. For qualitative organizational
bore little resemblance to our expectations, development researchers, the field will be an
what people had told us, or the official pro- organization. For evaluators, the field is the
gram description. Had we designed the fol- program being studied. Many terms are
low-up study without having participated used for talking field-based observations in-
in the program, we would have completely cluding participant observation, fieldwork,
missed the mark and asked inappropriate qualitative observation, direct observation, and
questions. To absorb the program/s lan- field research. "Ali these terms refer to the cir-
guage, understand nuances of meaning, ap- cumstance of being in or around an on-going
preciate variations m participants7 experi- social setting for the purpose of making a
ences, capture the importance of what qualitative analysis of that setting" (Lofland
happened outside formal activities (during 1971:93),
breaks, over meals, in late-night gatherings Direct, personal contact with and obser-
and parties), and feel the intensity of the re- vations of a setting have several advantages.
treat environmentnothing could have First, through direct observations the in-
substituted for direct experience with the quirer is better able to understand and cap-
program. Indeed, what we observed and ture the context within which people inter-
experienced was that participants were act. Understanding context is essential to a
changed as much or more by what hap- holistic perspective.
pened outside the formal program structure Second, firsthand experience with a set-
and activities as by anything that happened ting and the people in the setting allows an
through the planned curriculum and exer- inquirer to be open, discovery oriented, and
cises. inductive because, by being on-site, the ob-
The first-order purposes of observational server has less need to rely on prior concep-
data are to describe the setting that was ob- tualizations of the setting, whether those
served, the activities that took place in that prior conceptualizations are from written
setting, the people who participated in those documents or verbal reports.
activities, and the meanings of what was ob- A third strength of observational field-
served from the perspectives of those ob- work is that the inquirer has the opportunity
served. The descriptions should be factual, to see things that may routinely escape
accurate, and thorough without being clut- awareness among the people in the setting.
tered by irrelevant minutiae and trivia. The For someone to provide information in an
quality of observational reports is judged by interview, he or she must be aware enough
the extent to which that observation permits to report the desired information. Because
ali social systems involve routines, partici- ural result of the bonding among partici-
pants in those routines may take them so pants. We learned that neither explanation
much for granted that they cease to be aware was true. What actually occurred was that,
of important nuances that are apparent only unbeknownst to program staff, the dining
to an observer who has not become fully im- hostess for the hotel where participants
mersed in those routines. stayed initiated the roast. After the second
The participant observer can also dis- evening's meai, when staff routinely de-
cover things no one else has ever really paid parted for a meeting, the hostess would tell
attention to. One of the highlights of the participants what was expected. She even
leadership training program we experi- brought out a photo lbum of past ban-
enced was the final evening banquet at quets and offered to supply joke books, cos-
which staff was roasted. For three nights, tumes, music, or whatever. This 60-year-old
after training ended, participants worked to woman had begun playing what amounted
put together a program of jokes, songs, and to a major staff role for one of the most im-
skits for the banquet. Staff were never portant processes in the programand the
around for these preparations, which lasted staff didn't know about it. We learned about
late into the night, but they had come to it by being there.
count on this culminating event. Month after A fourth value of direct observation is the
month for two years each completely new chance to learn things that people would be
training group had organized a final ban- unwilling to talk about in an interview. In-
quet event to both honor and make fun of terviewees may be unwilling to provide in-
staff. Staff assumed that either prior partici- formation on sensitive topics, especially to
pants passed on this tradition or it was a nat- strangers. A fifth advantage of fieldwork is
the opportunity to move beyond the selec- succeeding observations. (Becker and Geer
tive perceptions of others. Interviews pres- 1970:32)
ent the understandings of the people being
interviewed. Those understandings consti- Observation-Based
tute important, indeed criticai, information.
Evaluation and Applied
However, it is necessary for the inquirer to
Research in a Political World
keep in mind that interviewees are always
reporting perceptionsselective percep- The preceding review of the advantages
tions. Field observers will also have selective of fieldwork strikes me as fairly straightfor-
perceptions. By making their own percep- ward but a bit abstract. In a moment, well
tions part of the dataa matter of training, consider the details of how to do fieldwork,
discipline, and self-awarenessobservers but to inform that transition and reinforce
can arrive at a more comprehensive view of the importance of direct observation in the
the setting being studied than if forced to real world, let me offer a perspective from
rely entirely on secondhand reports through the world of children's stories. Some of the
interviews. most delightful, entertaining, and suspense-
Finally, getting close to the people in a set- ful fairy tales and fables concern tales of
ting through firsthand experience permits kings who discard their royal robes to take
the inquirer to draw on personal knowledge on the apparel of peasants so that they can
during the formal interpretation stage of move freely among their people to really un-
analysis. Reflection and introspection are derstand what is happening in their king-
important parts of field research. The im- doms. Our modern-day kings and political
pressions and feelings of the observer be- figures are more likely to take television
come part of the data to be used in attempt- crews with them when they make excur-
ing to understand a setting and the people sions among the people. They are unlikely to
who inhabit it. The observer takes in infor- go out secretly disguised, moving through
mation and forms impressions that go be- the streets anonymously, unless they're up
yond what can be fully recorded in even the to mischief. It is left, then, to applied re-
most detailed field notes. searchers and evaluators to play out the fa-
ble, to take on the appropriate appearance
Because [the observer] sees and hears the peo- and mannerisms that will permit easy
ple he studies in many situations of the kind movement among the people, sometimes se-
that normally occur for them, rather than just cretly, sometimes openly, but always with
in an isolated and formal interview, he builds the purpose of better understanding what
an ever-growing fund of impressions, many the world is really like. They are then able to
of them at the subliminal levei, which give report those understandings to our mod-
him an extensive base for the interpretation ern-day version of kings so that policy wis-
and analytic use of any particular datum. This dom canbe enhanced and programmatic de-
wealth of information and impression sen- cisions enlightened. At least thafs the
sitizes him to subtleties which might pass fantasy. Turning that fantasy into reality in-
unnoticed in an interview and forces him to volves a number of important decisions
raise continually new and different questions, about what kind of fieldwork to do. We turn
which he brings to and tries to answer in now to those decisions.
Fieldwork Strategies and Observation Methods LET. 2 6 5
L Variations in Observational Methods
W e shall not cease from exploration

And the end of ali our exploring
Will be to arrive where we started
And know the place for the first time.1
T. S. Eliot (1888-1965)
Observational research explores the along the continuum between these two end
world in many ways. Deciding which obser- points.
vational approaches are appropriate for Nor is it simply a matter of deciding at the
evaluation or action research involves dif- beginning how much the observer will par-
ferent criteria than those same decisions ticipate. The extent of participation can
made to undertake basic social scientific re- change over time. In some cases, the re-
search. These differences emerge from the searcher may begin the study as an onlooker
nature of applied research, the politics of and gradually become a participant as field-
evaluation, the nature of contract funding in work progresses. The opposite can also oc-
most evaluations, and the accountability of cur. An evaluator might begin as a complete
evaluators to information users. Thus, while participant to experience what it is like to be
field methods have their origins in basic an- initially immersed in the program and then
thropological and sociological field meth- gradually withdraw participation over the
ods, using these methods for evaluation of- period of the study until finally taking the
ten requires adaptation. The sections that role of occasional observer from an onlooker
follow will discuss both the similarities and stance.
differences between evaluation field meth- Full participant observation constitutes
ods and basic research field methods. an omnibus field strategy in that it "simulta-
neously combines document analysis, inter-
viewing of respondents and inforinants, di-
Variations in Observer rect participation and observation, and
Involvement: Participant introspection" (Denzin 1978b: 183). If, on the
or Onlooker or Both? other hand, an evaluator observes a pro-
gram as an onlooker, the processes of obser-
The first and most fundamental distinc- vation can be separated from interviewing.
tion that differentiates observational strate- In participant observation, however, no
gies concerns the extent to which the ob- such separation exists. Typically, anthropo-
server will be a participant in the setting logical fieldworkers combine in their field
being studied. This involves more than a notes data from personal, eyewitness obser-
simple choice between participation and vation with information gained from infor-
nonparticipation. The extent of participa- mal, natural interviews and informants' de-
tion is a continuum that varies from com- scriptions (Peito and Peito 1978:5). Thus, the
plete immersion in the setting as full partici- participant observer employs multiple and
pant to complete separation from the setting overlapping data collection strategies: being
as spectator, with a great deal of variation fully engaged in experiencing the setting
(participation) while at the same time ob- can become a full participant. For example, a
servmg and talking with other participants researcher who is not chemically dependent
about whatever is happening. will not be able to become a full participant,
In the leadership program I evaluated physically and psychologically, in a chemi-
through participant observation, I was a full cal dependency program, even though it
participant in ali exercises and program ac- may be possible to participate in the pro-
tivities using the field of evaluation as my gram as a client. Such participation in a treat-
leadership arena (since ali participants had ment program can lead to important in-
to have an arena of leadership as their focus). sights and understanding about what it is
As did other participants, I developed close like to be in the program; however, the eval-
relationships with some people as the week uator must avoid the delusion that partici-
progressed, sharing meals and conversing pation has been complete. This point is illus-
late into the night. I sometimes took detailed trated by an exchange between an inmate
notes during activities if the activity permit- and a student who was doing participant ob-
ted (e.g., group discussion), while at other servation in a prison.
times I waited until later to record notes
(e.g., after meals). If a situation suddenly be- Inmate: "What are you in here for, man?"
came emotional, for example during a small
Student: "I'm here for a while to find out
group encounter, I would cease to take notes
what it's like to be in prison."
so as to be fully present as well as to keep my
note taking from becoming a distraction. Inmate: "What do you mean 'find out
Unlike other participants, I sat in on staff what it's like' ?"
meetings and knew how staff viewed what
Evaluator: 'Tm here so that I can experi-
was going on. Much of the time I was fully
ence prison from the inside instead of just
immersed in the program experience as a
studying what it's like from out there."
participant, but I was also always aware of
my additional role as evaluation observer. Inmate: "You got to be jerkin' me off, man.
The extent to which it is possible for an 'Experience from the inside . . . ' ? Shit,
evaluator to become a participant in a pro- man, you can go home when you decide
gram will depend partly on the nature of the you've had enough can't you?"
program. In human service and education
Evaluator: "Yeah."
programs that serve children, the evaluator
cannot participate as a child but may be able Inmate: "Then you ain't never gonna know
to participate as a volunteer, parent, or staff what it's like from the inside."
member in such a way as to develop the per-
spective of an insider in one of those adult Social, cultural, political, and interper-
roles. Gender can create barriers to partici- sonal factors can limit the nature and degree
pant observation. Males can't be partici- of participation in participant observation.
pants in female-only programs (e.g., bat- For example, if the participants in a program
tered women's shelters). Females doing ali know each other intimately they may ob-
fieldwork m nonliterate cultures may not be ject to an outsider trying to become part of
permitted access to male-only councils and their close circle. Where marked social class
ceremonies. Programs that serve special differences exist between a sociologist and
populations may also involve natural limita- people in a neighborhood, access will be
tions on the extent to which the evaluator more difficult; likewise, when, as is often the
case, an evaluator is well educated and mid- search, the purpose, scope, length, and set-
dle class while welfare program clients are ting for the study will dictate the range and
economically disadvantaged and poorly ed- types of participant observation that are
ucated, the participants in the program may possible.
object to any ruse of "full" participant obser- One final caution: The researcher 's plans
vation. Program staff will sometimes object and intentions regarding the degree of pro-
to the additional burden of including an gram involvement to be experienced may
evaluator in a program where resources are not be the way things ac tually turn out. Lang
limited and an additional participant would and Lang (1960) report that two scientific
unbalance staff-client ratios. Thus, in evalu- participant observers who were studying
ation, the extent to which full participation is audience behavior at a Billy Graham evan-
possible and desirable will depend on the gelical crusade made their "decision for
precise nature of the program, the political Christ" and left their observer posts to walk
context, and the nature of the evaluation down the aisle and join Reverend Graham's
questions being asked. Adult training pro- campaign. Such are the occupational haz-
grams, for example, may permit fairly easy ards (or benefits, depending on your per-
access for full participation by evaluators. spective) of real-world fieldwork.
Offender treatment programs are much less
likely to be open to participant observation
Insider and Outsider Perspectives:
as an evaluation method. Evaluators must
therefore be flexible, sensitive, and adaptive Emic Versus Etic Approaches
in negotiating the precise degree of partici-
pation that is appropriate in any particular People who are insiders to a setting being
observational study, especially where re- studied often have a view of the setting and
porting timelines are constrained so entry any findings about it quite different from that
into the setting must be accomplished rela- of the outside researchers who are conducting
tively quickly. Social scientists who can take the study. (Bartunek and Louis 1996)
a long time to become integrated into the set-
ting under study have more options for Ethnosemanticist Kenneth Pike (1954)
fuller participant observation. coined the terms emic and etic to distinguish
As these examples illustrate, full and classification systems reported by anthro-
complete participation in a setting, what is pologists based on (1) the language and cate-
sometimes called "going native," is fairly gories used by the people in the culture stud-
rare, especially for a program evaluation. ied, an emic approach, in contrast to (2)
Degree of participation and nature of obser- categories created by anthropologists based
vation vary along a wide continuum of on their analysis of important cultural dis-
possibilities. The ideal in evaluation is to tinctions, an etic approach. Leading anthro-
design and negotiate that degree of partici- pologists such as Franz Boas and Edward
pation that will yield the most meaningful Sapir argued that the only meaningful dis-
data about the program given the charac- tinctions were those made by people within
teristics of the participants, the nature of a culture, that is, from the emic perspective.
staff-participant interactions, the socio- However, as anthropologists turned to more
political context of the program, and the in- comparative studies, engaging in cross-cul-
formation needs of intended evaluation tural analyses, distinctions that cut across
users. Likewise, in applied and basic re- cultures had to be made based on the anthro-
pologisfs analy tical perspective, that is, an emic perspective. This means that the partic-
etic perspective. The etic approach involved ipant observer not only sees what is
"standing far enough away from or outside happeningbut feels what it is like to be a part
of a particular culture to see its separate of the setting or program. Anthropologist
events, primarily in relation to their similari- Hortense Powdermaker (1966) has de-
ties and their differences, as compared to scribed the basic assumption undergirding
events in other cultures" (Pike 1954:10). For participant observation as follows: "To un-
some years a debate raged in anthropology derstand a society, the anthropologist has
about the relative merits of emic versus etic traditionally immersed himself in it, learn-
perspectives (Peito and Peito 1978:55-60; ing, as far as possible, to think, see, feel and
Headland, Pike, and Harris 1990), but, as of- sometimes act as a member of its culture and
ten happens over time, both approaches at the same time as a trained anthropologist
came to be understood as valuable, though from another culture" (p. 9).
each contributes something different. Nev- Experiencing the setting or program as an
ertheless, tension between these perspec- insider accentuates the participant part of
tives remains: participant observation. At the same time,
the inquirer remains aware of being an out-
Today, despite or perhaps because of the new sider. The challenge is to combine participa-
recognition of cultural diversity, the tension tion and observation so as to become capa-
between universalistic and relativistic values ble of understanding the setting as an
remains an uriresolved conundrum for the insider while describing it to and for outsid-
Western ethnographer. In practice, it becomes ers.
this question: By which values are observa-
tions to be guided? The choices seem to be ei- Obtaining something of the understanding of
ther the values of the ethnographer or the an insider is, for most researchers, only a first
values of the observedthat is, in modem step. They expect, in time, to become capable
parlance, either the etic or the emic.... Herein of thinking and acting within the perspective
lies a deeper and more fundamental problem: of two quite different groups, the one m which
How is it possible to understand the other they were reared andto some degreethe
when the other's values are not one's own? one they are studying. They will also, at times,
This problem arises to plague ethnography at be able to assume a mental position peripheral
a time when Western Christian values are no to both, a position from which they willbe able
longer a surety of truth and, hence, no longer to perceive and, hopefully, describe those rela-
the benchmark from which self-confidently tionships, systems and pattems of which an
valid observations can be made. (Vidich and inextricably involved insider is not likely to be
Lyman 2000:41) consciously aware. For what the social scien-
tist realizes is that while the outsider simply
Methodologically, the challenge is to do jus- does not know the meanings or the pattems,
tice to both perspectives during and after the insider is so immersed that he may be
fieldwork and to be clear with one's self and oblivious to the fact that pattems exist. . . .
one's audience how this tension is managed. What fieldworkers eventually produce out of
A participant observer shares as inti- the tension developed by this ability to shift
mately as possible in the life and activities of their point of view depends upon their sophis-
the setting under study in order to develop tication, ability, and training. Their task, in any
an insider's view of what is happening, the case, is to realize what they have experienced
and learned and to communicate this in terms useful, a supplementary agenda is often to
that will illurrne. (Wax 1971:3) increase participants' sense of being in con-
trol of, deliberative about, and reflective on
their own lives and situations. Chapter 4 dis-
Who Conducts the Inquiry? cussed these approaches as examples of how
Solo and Team Versus Participatory qualitative inquiry can be applied in sup-
and Collaborative Approaches port of organizational or program develop-
ment and community change.
The ultimate in insider perspective Degrees of collaboration vary along a
comes from involving the insiders as continuum. At one end is the solo field-
coresearchers through collaborative or par- worker or a team of professionals; what
ticipatory research. Collaborative forms of characterizes this end of the continuum is
fieldwork, participatory action research, that researchers completely control the in-
and empowerment approaches to evalua- quiry. At the other end are collaborations
tion have become sufficiently important and with people in the setting being studied,
widespread to make degree of collaboration a sometimes called "coresearchers"; they help
dimension of design choice in qualitative in- design the inquiry, collect data, and are in-
quiry. Participatory action research has a volved in analysis. Along the middle of the
long and distinguished history (Kemmis continuum are various degrees of partial
and McTaggart 2000; Whyte 1989). Collabo- and periodic (as opposed to continuous)
rative principies of feminist inquiry include collaboration.
connectedness and equality between re-
searchers and researched, participatory pro- Overt Versus Covert Observations
cesses that support consciousness-raising
and researcher reflexivity, and knowledge A traditional concern about the validity
generation that contributes to women's lib- and reliability of observational data has
eration and emancipation (Olesen 2000; been the effects of the observer on what is
Guerrero 1999a:15-22; Thompson 1992). In observed. People may behave quite differ-
evaluation, Cousins and Earl (1995) have ad- ently when they know they are being ob-
vocated participatory and collaborative ap- served versus how they behave naturally
proaches to evaluation primarily to increase when they don't think they're being ob-
use of findings. Empowerment evaluation, served. Thus, the argument goes, covert ob-
often using qualitative methods (Fetterman servations are more likely to capture what is
2000a; Fetterman, Kaftarian, and Wanders- really happening than are overt observa-
man 1996), involves the use of evaluation tions where the people in the setting are
concepts and techniques to foster self-deter- aware they are being studied.
mination and help people help themselves Researchers have expressed a range of
by learning to study and report on their own opinions concerning the ethics and morality
issues and concerns. of conducting covert research, what Mitchell
What these approaches have in common (1993:23-35) calls "the debate over secrecy."
is a style of inquiry in which the researcher One end of the continuum is represented by
or evaluator becomes a facilitator, collabora- Edward Shils (1959), who absolutely op-
tor, and teacher in support of those engaging posed ali forms of covert research including
in their own inquiry. While the findings "any observations of private behavior, how-
from such a participatory process may be ever technically feasible, without the explicit
and fully informed permission of the person problems are (1) misinformation, (2) evasions,
to be observed." He argued that there (3) lies, and (4) fronts. (Douglas 1976:55, 57)
should be full disclosure of the purpose of
any research project and that even partici- Just as degree of participation in field-
pant observation is "morally obnoxious . . . work turned out to be a continuum of varia-
manipulation" unless the observer makes tions rather than an all-or-none proposition,
explicit his or her research questions at the so too is the question of how explicit to be
very beginning of the observation (Shils about the purpose of fieldwork. The extent
1959, quoted in Webb et ai. 1966:vi). to which participants in a program under
study are informed that they are being ob-
At the other end of the continuum is the
served and are told the purpose of the re-
"investigative social research" of Jack
search has varied historically from full dis-
Douglas (1976). Douglas argued that con-
closure to no disclosure, with a great deal of
ventional anthropological field methods
variation along the middle of this contin-
have been based on a consensus view of so-
uum (Junker 1960). Discipline-based ethics
ciety that views people as basically coopera-
statements (e.g., American Psychological
tive, helpful, and willing to have their points
Association, American Sociological Associ-
of view understood and shared with the rest
ation) now generally condemn deceitful and
of the world. In contrast, Douglas adopted a
covert research. Likewise, institutional re-
conflict paradigm of society that led him to
view board (IRB) procedures for the protec-
believe that any and ali covert methods of re-
tion of human subjects have severely con-
search should be considered acceptable op-
strained such methods. They now refuse to
tions in a search for truth.
approve protocols in which research partici-
pants are deceived about the purpose of a
The investigative paradigm is based on the as- study, as was commonly done in early psy-
sumption that profound conflicts of interest, chological research. One of the more infa-
values, feelings and actions pervade social mous examples was Stanley Milgrain's New
life. It is taken for granted that many of the Haven experiments aimed at studying
people one deals with, perhaps ali people to whether ordinary people would follow the
some extent, have good reason to hide from orders of someone in authority by having
others what they are doing and even to lie to these ordinary citizens administer what
them. Instead of trusting people and expect- they were told were behavior modification
ing trust in return, one suspects others and ex- electric shocks to help students learn, shocks
pects others to suspect him. Conflict is the that appeared to the unsuspecting citizens to
reality of life; suspicion is the guiding princi- go as high as 450 volts despite the screams
p i e It's a war of ali and no one gives anyone and protests heard from supposed students
anything for nothing, especially truth on the other side of a wall. The real purpose
Ali competent adults are assumed to know of the study, participants later learned, was
that there are at least four major problems ly- to replicate Nazi prison guard behavior
ing in the way of getting at social reality by among ordinary American citizens (Mil-
asking people what is going on and that these gram 1974).
problems must be dealt with if one is to avoid IRBs also refuse to approve research in
being taken in, duped, deceived, used, put on, which people are observed and studied
fooled, suckered, made the patsy, left holding without their knowledge or consent, as in
the bag, fronted out and so on. These four the infamous Tuskegee Experiment. For
40 years, physicians and medicai research- missioned and used by colonial administra-
ers, under the auspices of the U.S. Public tors to maintain control over indigenous
Health Service, studied untreated syphilis peoples. Protection of human subjects pro-
among Black men in and around the county cedures are now an affirmation of our com-
seat of Tuskegee, Alabama, without the in- mitment to treat ali people with respect. And
formed consent of the men studied, men that is as it should be. But the necessity for
whose syphilis went untreated so that the such procedures comes out of a past littered
progress of the disease could be docu- with scientific horrors for which those of us
mented (Jones 1993). Other stories of abuse engaging in research today may still owe
and neglect by researchers doing covert penance. At any rate, we need to lean over
studies abound. In the late 1940s and early backward to be sure that such history is truly
1950s, schoolboys at the Walter E. Fernald behind usand that means being ever vigi-
State School in Massachusetts were rou- lant in fully informing and protecting the
tinely served breakfast cereal doused with people who honor us by agreeing to partici-
radioactive isotopes, without permission of pate in our research, whether they be home-
the boys or their guardians, for the disserta- less mothers (Connolly 2000) or corporate
tion of a doctoral student in nutritional bio- executives (Collins 2001).
chemistry. In the 1960s, the U.S. Army se- However, not ali research and evaluation
cretly sprayed a potentially hazardous falls under IRB review, so the issue of what
chemical from downtown Minneapolis roof- type and how much disclosure to make re-
tops onto unsuspecting citizens to find out mains a matter of debate, especially where
how toxic materiais might disperse during the inquiry seeks to expose the inner work-
biological warfare. Native American chil- ings of cults and extremist groups, or those
dren on the Standing Rock Sioux Reserva- whose power affects the public welfare, for
tion in the Dakotas were used to test an un- example, corporations, labor union boards,
approved and experimental hepatitis A political parties, and other groups with
vaccine without the knowledge or approval wealth and/or power. For example, Maurice
of their parents. In the 1960s and 1970s, sci- Punch (1985, 1989, 1997), formerly of the
entists teste d skin treatments and drugs on Nijenrode Business School in the Nether-
prisoners in a Philadelphia county jail with- lands, has written about the challenges of
out informing them of potential dangers. doing ethnographic studies of corruption in
Doctoral students frustrated by having both private and public sector organiza-
their fieldwork delayed while they await tions, notably the police.
IRB approval need to remember that they One classic form of deception in field-
are paying for the sins of their research fore- work involves pretending to share values
bears for whom deception and covert obser- and beliefs in order to become part of the
vations were standard ways of doing their group being studied. Sociologist Richard
work. Those most subject to abuse were of- Leo carefully disguised his liberal political
ten the most vulnerable in societychil- and social views, instead feigning conserva-
dren, the poor, people of color, the sick, peo- tive beliefs, to build trust with police and
ple with little education, women and men thereby gain admission to interrogation
incarcerated in prisons and asylums, and rooms (Allen 1997:32). Sociologist Leon
children in orphanages or state correctional Festinger (1956) infiltrated a doomsday cult
schools. Anthropological research was com- by lying about his profession and pretend-
ing to believe in the cult's prophecies. Sociol- know, and, of course, there are the classic sit-
ogist Laud Humphreys (1970) pretended to uations where everyone involved knows
be gay to gather data for his dissertation on that a study is being done and who the eval-
homosexual encounters in public parks. An- uator isbut the evaluator doesn't know
thropologist Carolyn Ellis (1986) pretended that everyone else knows.
to be just visiting friends when she studied a In undertaking participant observation of
Chesapeake Bay fishing culture. Her nega- the community leadership program men-
tive portrayals made their way back to the tioned earlier, my two evaluation colleagues
local people, many of whom were infuri- and I agreed with the staff to downplay our
ated. She later expressed remorse about her evaluation roles and describe ourselves as
deceptions (Allen 1997). "educational researchers" interested in
In traditional scholarly fieldwork, the de- studying the program. We didn't want par-
cision about the extent to which observa- ticipants to think that they were being evalu-
tions would be covert was made by re- ated and therefore worry about our judg-
searchers balancing the search for truth ments. Our focus was on evaluating the
against their sense of professional ethics. In program, not participants, but to avoid in-
evaluation research, the information users creasing participant stress we simply at-
for whom the evaluation is done have a tempted to finesse our evaluation role by
stake in what kind of methods are used, so calling ourselves educational researchers.
the evaluator alone cannot decide the extent Our careful agreement on and rehearsal
to which observations and evaluation pur- of this point with the staff fell apart during
poses will be fully disclosed. Rather, the introductions (at the start of the six-day re-
complexities of program evaluation mean treat) when the program director proceeded
that there are several leveis at which deci- to tell participantsfor 10 minutesthatwe
sions about the covert-overt nature of evalu- were just participants and they didn't have
ation observations must be made. Some- to worry about our evaluating them. The
times only the funders of the program or of longer he went on reassuring the group that
the evaluation know the full extent and pur- they didn't have to worry about us, the more
pose of observations. On occasion, program worried they got. Sensing that they were
staff may be informed that evaluators will be worried, he increased the intensity of his re-
participating in the program, but clients will assurances. While we continued to refer to
not be so informed. In other cases, a re- ourselves as educational researchers, the
searcher may reveal the purpose and nature participants thereafter referred to us as eval-
of program participation to fellow program uators. It took a day and a half to recover our
participants and ask for their cooperation in full participating roles as the participants
keeping the evaluation secret from program got to know us on a personal levei as indi-
staff. On still other occasions, a variety of viduais.
people intimately associated with the pro- Trying to protect the participants (and the
gram may be informed of the evaluation, but evaluation) had backfired and made our en-
public officials who are less closely associ- try into the group even more difficult than it
ated with the program may be kept "in the otherwise would have been. However, this
dark" about the fact that observations are experience sensitized us to what we subse-
under way. Sometimes the situation be- quently observed to be a pattern in many
comes so complex that the evaluator may program situations and activities through-
lose track of who knows and who doesn't out the week, and became a major finding of
the evaluation: staff overprotection of and reports conceal names, locations, and other
condescending attitudes toward partici- identifying information so that the people
pants. who have been observed will be protected
Based on this and other evaluation expe- from harm or punitive action. Because the
riences, I recommend full and complete dis- basic researcher is interested in truth rather
closure. People are seldom really deceived than action, it is easier to protect the identity
or reassured by false or partial explana- of inforinants or study settings when doing
tionsat least not for long. Trying to run a scholarly research. In evaluation research,
ruse or scam is simply too risky and adds to however, while the identity of who said
evaluator stress while holding the possibil- what may be possible to keep secret, it is sel-
ity of undermining the evaluation if (and dom possible to conceal the identity of a pro-
usually when) the ruse becomes known. gram, and doing so may undermine the util-
Program participants, over time, will tend to ity of the findings.
judge evaluators first and foremost as peo- Evaluators and decision makers will have
ple not as evaluators. to resolve these issues in each case in accor-
The nature of the questions being studied dance with their own consciences, evalua-
in any particular evaluation will have a pri- tion purposes, political realities, and ethical
mary effect on the decision about who will sensitivities.
be told that an evaluation is under way. In
formative evaluations where staff members
Variations in Duration
and/or program participants are anxious to
of Observations
have information that will help them im-
prove their program, the quality of the data Another important dimension along
gathered may be enhanced by overtly solic- which observational studies vary is the
iting the cooperation of everyone associated length of time devoted to data gathering. In
with the program. Indeed, the ultimate ac- the anthropological tradition of field re-
ceptance and usefulness of formative infor- search, a participant observer would expect
mation may depend on such prior disclo- to spend six months at a minimum, and of-
sure and agreement that a formative ten years, living m the culture being ob-
evaluation is appropriate. On the one hand, served. The fieldwork of Napoleon Chag-
where program funders have reason to be- non (1992) among the Yanomami Indians in
lieve that a program is corrupt, abusive, in- the rain forest at the borders of Venezuela
competently administered, and/or highly and Brazil spanned a quarter century. To de-
negative in impact on clients, it may be develop a holistic view of an entire culture or
cided that an externai, covert evaluation is subculture takes a great deal of time, espe-
necessary to find out what is really happen- cially when, as in the case of Chagnon, he
ing in the program. Under such conditions, was documenting changes in tribal life and
my preference for full disclosure may be nei- threats to the continued existence of these
ther prudent nor practical. On the other once-isolated people. The effects of his
hand, Whyte (1984) has argued that "in a long-term involveinent on the people he
community setting, maintaining a covert studied became controversial (Geertz 2001;
role is generally out of the ques tion" (p. 31). Tiemey 2000a, 2000b), a matter we shall take
Finally, there is the related issue of confi- up later. The point here is that fieldwork in
dentiality. Those who advocate covert re- basic and applied social science aims to un-
search usually do so with the condition that veil the interwoven complexities and funda-
mental pattems of social lifeactual, per- and organizational development. Such has
ceived, constructed, and analyzed. Such been the case with the extraordinary work of
studies take a long time. Patricia Carini (1975, 1979) at the Prospect
Educational researcher Alan Peshkin School in North Bennington, Vermont.
offers a stellar example of a committed Working with the staff of the school to collect
fieldworker who lived for periods of time in detailed case records on students of the
varied settings in order to study the intersec- school, she established an archive with as
tions between schools and communities. He much as 12 years of detailed documentation
did fieldwork in a Native American commu- about the learning histories of individual
nity; in a high school m a stable, multiethnic students and the nature of the school pro-
midsized city in Califrnia; in rural, east- grams they experienced. Her data included
central Illinois; in a fundamentalist Chris- copies of the students' work (completed as-
tian school; and in a private, residential signments, drawings, papers, projects),
school for elites (Peshkin 1986,1997,2000b). classroom observations, teacher and parent
To collect data, he and his wife Maryann observations, and photographs. Any organi-
lived for at least a year m and with the com- zation with an internai evaluation informa-
munity that he was studying. They shopped tion system can look beyond quarterly and
locally, attended religious services, and de- annual reporting to building a knowledge
veloped close relationships with civic lead- archive of data to document development
ers as well as teachers and students. and change over years instead of just
In contrast, evaluation and action re- months. Participant observations by those
search typically in volve much shorter dura- who manage such systems can and should
tions in keeping with their more modest be an integral part of this kind of knowl-
aims: generating useful information for ac- edge-building organizational data system
tion. To be useful, evaluation findings must that spans years, even decades.
be timely. Decision makers cannot wait for On the other end of the time continuum
years while fieldworkers sift through moun- are short-term studies that involve observa-
tains of field notes. Many evaluations are tions of a single segment of a program,
conducted under enormous pressures of sometimes for only an hour or two. Evalua-
time and limited resources. Thus, the dura- tions that include brief site visits to a number
tion of observations will depend to a consid- of program locations may serve the purpose
erable extent on the time and resources of simply establishing the existence of cer-
available in relation to the information tain leveis of program operations at different
needs and decision deadlines of primary sites. Chapter 1 presented just such an obser-
evaluation users. Later in this chapter we'11 vation of a single two-hour session of an
include reflections from an evaluator about early childhood parent education program
what it was like being a part-time, in which mothers discussed their child-rear-
in-and-out observer of a program for eight ing practices and fears. The site visit obser-
months, but only present 6 hours a week out vations of some 20 such program sessions
of the program's 40-hour week. throughout Minnesota were part of an im-
On the other hand, sustained and ongo- plementation evaluation that reported to the
ing evaluation research may provide annual state legislature how these innovative (at the
findings while, over years of study, accumu- time) programs were operating in practice.
lating an archive of data that serves as a Each site visit lasted no more than a day, of-
source of more basic research into human ten only a half day.
Sometimes an entire segment of a pro- which the observer participates in the set-
gram may be of sufficiently short duration ting being studied, the tension between
that the evaluator can participate in the com- insider versus outsider perspectives, the ex-
plete program. The leadership retreat we ob- tent to which the purpose of the study is
served lasted 6 days, plus three 1-day fol- made explicit, and the duration of the obser-
low-up sessions during the subsequent year. vations. A major factor affecting each of
The criticai point is that the length of time these other dimensions is the scope or focus
during which observations take place de- of the study or evaluation. The scope can be
pends on the purpose of the study and the broad, encompassing virtually ali aspects of
questions being asked, not some ideal about the setting, or it can be narrow, involving a
what a typical participant observation must look at only some small part of what is hap-
necessarily involve. Field studies may be pening.
massive efforts with a team of people partic- Parameswaran (2001) wanted to inter-
ipating in multiple settings in order to do view young women in ndia who read West-
comparisons over several years. At times, ern romance novis. Thus, her fieldwork
then, and for certain studies, long-term had a very narrow focus. But to contextu-
fieldwork is essential. At other times and for alize what she learned from interviews, she
other purposes, as in the case of short-term sought "active involvement in my infor-
formative evaluations, it can be helpful for mants' livesbeyond their romance reading."
program staff to have an evaluator provide How did she do this?
feedback based on just one hour of onlooker
observation at a staff meeting, as I have also I ate snacks and lunch at cafs with groups of
done. women, went to the movies, dined with them
My response to students who ask me how at their homes, and accompanied them on
long they have to observe a program to do a shopping trips. I joined women's routine con-
good evaluation follows the line of thought versations during break times and inter-
developed by Abraham Lincoln during one viewed informants at a range of everyday
of the Douglas-Lincoln debates. In an obvi- sites, such as college grounds, homes, and res-
ous reference to the difference in stature be- taurants. I visited used-book vendors, book-
tween Douglas and Lincoln, a heckler asked, stores, and lending libraries with several
"Tell us, Mr. Lincoln, how long do you think readers and observed social interactions be-
a man's legs ought to be?" tween library owners and young women. To
Lincoln replied, "Long enough to reach gain insight into the multidimensional rela-
the ground." tionship between women's romance reading
Fieldwork should last long enough to get and their experiences with everyday social
the job doneto answer the research ques- discourse about romance readers, I inter-
tions being asked and fulfill the purpose of viewed young women's parents, siblings,
the study. teachers, bookstore managers, and owners of
the lending libraries they frequented. (p. 75)
Variations in
The tradition of ethnographic fieldwork
Observational Focus has emphasized the importance of under-
standing whole cultural systems. The vari-
The preceding sections have discussed ous subsystems of a society are seen as inter-
how observations vary in the extent to dependent parts so that the economic
system, the cultural system, the political sys- program and how many are studied will
tem, the kinship system, and other special- clearly affect such issues as the extent to
ized subsystems could only be understood which the observer is a participant, who will
in relation to each other. In reality, fieldwork know about the evaluation's purpose, and
and observations have tended to focus on a the duration of observations.
particular part of the society or culture be- Chapter 5 discussed how decisions about
cause of specific investigator interests and the focus and scope of a study involve
the need to allocate the most time to those trade-offs between breadth and depth. The
things that the researcher considered most very first trade-off comes in framing the re-
important. Thus, a particular study might search questions to be studied. The problem
present an overview of a particular culture is to determine the extent to which it is desir-
but then go on to report in greatest detail able and useful to study one or a few ques-
about the religious system of that culture. tions in great depth or to study more ques-
In evaluating programs, a broad range of tions but each in less depth. Moreover, in
possible foci makes choosing a specific focus emergent designs, the focus can change over
challenging. One way of thinking about fo- time.
cus options involves distinguishing various
program processes sequentially: (1) pro- Dimensions Along Which
cesses by which participants enter a pro- Fieldwork Varies: An Overview
gram (the outreach, recruitment, and intake
components); (2) processes of orientation to WeVe examine d five dimensions that can
and socialization into the program (the initi- be used to describe some of the primary
ation period); (3) the basic activities that variations in fieldwork. Those dimensions,
comprise program implementation over the discussed in the previous sections, are
course of the program (the service delivery graphically summarized in Exhibit 6.1.
system); and (4) the activities that go on These dimensions canbe used to help design
around program termination, including fol- observational studies and make decisions
low-up activities and client impacts over about the parameters of fieldwork. They can
time. It would be possible to observe only also be used to organize the methods section
one of these program components, some of a report or dissertation in order to docu-
combination of components, or ali of the ment how research or evaluation fieldwork
components together. Which parts of the actually unfolded.
13. What to Observe: A Sensitizing Framework
I keep six honest serving men.

K ^ s They taught me ali I knew:
Their names are What and Why and When
And How and Where and Who.2
Rudyard Kipling
fffWfflfffffflfflM
iaUEn Dimensions Showing Fieldwork Variations
1. Role of the observer
Full participant I I Onooker observer

in the setting Part participant/part observer (spectator)
2. Insider versus outsiderperspective
Insider (emic) I I Outsider (etic)

perspective dominant Balance perspective dominant
3. Who conducts the inquiry
Solo researchers, I I People in the

teams of professionals Variations in colaboration setting being studied
and participatory research
4, Disciosure ofthe observer's role to others
Overt: I I Covert:
Fuil disciosure Selective disciosure No disciosure
5. Duration of observations and fieldwork
Short, I I Long-term,
single observation Ongoing over time multiple observations
(e.g., 1 site, 1 hour) (e.g., months, years)
6. Focus of observations
Narrow focus: I I Broad focus:

Single eement Evolving, emergent Holistic view
NASA space scientist David Morrison fields never require the authors to state a
(1999) has noted that in astronomy, geology, 'hypothesis' in order to publish their re-
and planetary science, observation precedes sults" (p. 8).
theory generation "and the journals in these
A Tecent example is the famous Hubble Space cept as a guide to fieldwork with special at-
Telescope Deep Field in which the telescope tention to the words and meanings that are
obtained a single exposure of many days du- prevalent among the people being studied.
ration of one small field in an unremarkable More generally, however, "a sensitizing con-
part of the sky. The objective was to see fainter cept is a starting point in thinking about the
and farther than ever before, and thus to find class of data of which the social researcher
out what the universe was like early in its his- has no definite idea and provides an initial
tory. No hypothesis was requiredjust the guide to her research" (van den Hoonaard
unique opportunity to look where no one had 1997:2). Sensitizing concepts in the social sci-
ever looked before and see what nature herself ences include loosely operationalized no-
had to tell us. tions such as victim, stress, stigina, and
In many other sciences the culture de- learning organization that can provide some
mands that funding proposals and published initial direction to a study as a fieldworker
papers be written in terms of formulating and inquires into how the concept is given mean-
testing a hypothesis. But I wonder if this is re- ing in a particular place or set of circum-
ally the way the scientific process works, or is stances being studied (Schwandt 2001).
this just an artificial structure imposed for the Rudyard Kipling's poem about his "six
sake of tradition. (Morrison 1999:8) honest serving men," quoted above, consti-
tutes a fundamental and msightful sensitiz-
Part of the value of open-ended naturalis- ing framework identifying the central ele-
tic observations is the opportunity to see ments of good description. In social science,
what there is to see without the blinders of "group process" is a general sensitizing con-
hypotheses and other preconceptions. Pure cept as is the focus on outcomes in evalua-
observation. As Morrison put it so elegantly, tion. Kinship, leadership, socialization,
just the unique opportunity to look where power, and similar notions are sensitizing in
no one has ever looked before and see what that they alert us to ways of organizing ob-
the world has to show us. servations and making decisions about
That's the ideal. However, it's not possi- what to record. Qualitative methodologist
ble to observe everything. The human ob- Norman Denzin (1978a) has captured the es-
server is not a movie camera, and even a sence of how sensitizing concepts guide
movie camera has to be pointed in the right fieldwork:
direction to capture what is happening. For
both the human observer and the camera The observer moves from sensitizing concepts
there must be focus. In fieldwork, this focus to the immediate world of social experience
is provided by the study design and the na- and permits that world to shape and modify
ture of the questions being asked. Once in his conceptual framework. In this way he
the field, however, the observer must some- moves contmually between the realm of more
how organize the complex stimuli experi- general social theory and the worlds of native
enced so that observing thatbecomes and re- people. Such an approach recognizes that so-
mains manageable. cial phenomena, while displaying regulari-
Experienced observers often use "sensi- ties, vary by time, space, and circumstance.
tizing concepts" to orient fieldwork. Quali- The observer, then, looks for repeatable regu-
tative sociologist and symbolic interaction- larities. He uses ritual patterns of dress and
ist Herbert Blumer (1954) is credited with body-spacing as indicators of self-image. He
originating the idea of the sensitizing con- takes special languages, codes, and dialects as
indicators of group boundaries. He studies his A note of caution about sensitizing con-
subjecfs prized social objects as indicators of cepts: When they become part of popular
prestige, dignity, and esteem hierarchies. He culture, they can lose much of their original
studies moments of interrogation and deroga- meaning. Philip Tuwaletstiwa, a Hopi geog-
tion as indicators of socialization strategies. rapher, relates the story of a tourist cruising
He attempts to enter his subjecfs closed world through Native American areas of the
of interaction so as to examine the character of Southwest. He overheard the tourist, "ali
private versus public acts and attitudes. (p. 9) agog at half-heard tales about Hopi land,"
ask his wife, "Where are the-powerplaces?"
The notion of "sensitizing concepts" re- "Tell her that's where we plug-in TV," he
minds us that observers do not enter the said (quoted m Milius 1998:92).
field with a completely blank slate. While Overused sensitizing concepts can be-
the inductive nature of qualitative inquiry come desensitizing.
emphasizes the importance of being open to
whatever one can learn, some way of orga-
nizing the complexity of experience is virtu-
Sources of Data
ally a prerequisite for perception itself. Ex-
hibit 6.2 presents examples of common Poet David Wagoner (1999) tells those ob-
sensitizing concepts for program evaluation serving the modern world and afraid of be-
and organizational studies. These common ing lost to follow the advice Native Ameri-
program concepts and organizational di- can elders gave the young when they were
mensions constitute ways of breaking the afraid of being lost in the forest:
complexities of planned human interven-
tions into distinguishable, manageable, and Lost
observable elements. The examples in Ex-
hibit 6.2 are by no means exhaustive of eval- Stand still. The trees ahead and bushes
uation and organizational sensitizing con- beside you
cepts, but they illustrate oft-used ways of Are not lost. Where you are is called
organizing an agenda for inquiry. These con- Here,
cepts serve to guide initial observations as And you must trust it as a powerful
the evaluator or organizational analyst stranger,
watches for incidents, interactions, and con- Must ask permission to know it and
versations that illuminate these sensitizing be known.
concepts in a particular program setting or The forest breathes. Listen. It answers,
organization. Highly experienced evalua- I have made this place around you.
tors and organizational consultants have in- If you leave it, you may come back
ternalized some kind of sensitizing frame- again, saying Here.
work like this to the point where they would No two trees are the same to Raven.
not need to list these concepts in a formal No two branches are the same to Wren.
written design. Less experienced research- If what a tree or a bush does is lost on
ers and dissertation students will usually you,
benefit from preparing a formal list of major You are surely lost. Stand still. The
sensitizing concepts in the formal design forest knows
and then using those concepts to help orga- Where you are. You must let it find
nize and guide fieldwork, at least initially. you.3
isâaaMl | Examples of Sensitizing Concepts

p^nMaffiqipfMi ||||i|yi[i j n |im
il MMgBB
Program Evaluation Orgonizational Dimensions
Context Mission/vision language

Goals Centralization/decentralization of participation and decision making
Inputs Externai boundary relationships: Open/closed
Recruitment Routinness/nonroutinness of work, products, decisions
Intake Leadership
Implementation Communication patterns
Processes Organizational culture
Output Hierarchy: Layered/flat
Outcomes Authority patterns
Products Formal/informal networks
Impacts Rewards/punishments, incentives/disincentives
Program theory Success and failure messages/stories
Logic model Degree of integration
Perceived Competition/cooperation
Assets
Barriers
Strengths
Weakn esses
Incentives
Disincentives
The what and how of qualitative inquiry coherent and in-depth illustrations, the ex-
are closely linked. Sources of data are de- amples that follow focus on program
rived from inquiry questions. Knowing evaluation.
what we want to illuminate helps us deter-
mine sources of data for that illumination.
The examples and illustrations that follow The Setting
derive from and build on the sensitizing
framework for program evaluation. Inter- Describing a setting, like a program set-
spersed with this presentation of sources of ting, begins with the physical environment
evaluation data are examples of how to col- within which the program takes place. The
lect observational data. These strategies ap- description of the program setting should be
ply to other inquiry settings, but to provide sufficiently detailed to permit the reader to
Fieldwork Strategies and Observation Methods LET. 2 8 1
visualize that setting. In writing a program interpretive phrases. But such writing can
description, the observer, unlike the novel- also be dull. Metaphors and analogies can
ist, should avoid interpretive adjectives ex- enliven and enrich descriptions, helping
cept as they appear in quotes from partici- readers connect through shared under-
pants about their reactions to and standings and giving them a better feel for
perceptions of that environment. Such ad- the environment being described. I once
jectives as comfortable, beautiful, drab, and evaluated a wilderness education program
stimulating interpret rather than describe that included time at the Grand Canyon. Ex-
and interpret vaguely at that. More purely hibit 6.3 presents my feeble attempt to cap-
descriptive adjectives include ture in words our first view of the Grand
Canyon. Notice the metaphors that run
colors ("a room painted blue with a through the description. Of course, this is
one of those instances where a picture
blackboard at one end"),
would be worth a mountain of words, which
space ("a 40-foot-by-20-foot classroom is why qualitative fieldwork increasingly in-
with windows on one side"), and cludes photography and videography. This
purpose ("a library, the walls lined with excerpt aims at offering a sense ofthe physical
books and tables in the center"). environment more than it offers a literal de-
scription because unless one has been there
Beginners can practice learning to write or seen pictures, the landscape is outside or-
descriptively by sharing a description of a dinary experience.
setting observed with a couple of people and The physical environment of a setting can
asking them if they can visualize the setting be important to what happens in that envi-
described. Another helpful exercise in- ronment. The way the walls look in rooms,
volves two people observing the same envi- the amount of space available, how the
ronment and exchanging their descriptions, space is used, the nature of the lighting, how
watching in particular for the use of inter- people are organized in the space, and the
pretive adjectives instead of descriptive interpretive reactions of program partici-
ones. Vivid description provides sufficient pants to the physical setting can be impor-
information that the reader does not have to tant information about both program imple-
speculate at what is meant. For example, menta tion and the effects of the program on
simply reporting "a crowded room" re- participants.
quires interpretation. Contrast with this:
A common mistake among observers is to
take the physical environment for granted.
The meeting room had a three-person couch Thus, an evaluator may report that the pro-
across one side, six chairs along the adjoining gram took place in "a school." The evaluator
walls next to the couch, and three chairs along may have a mental image of "school" that
the wall facing the couch, which included the matches what was observed, but schools
door. With 20 people in the room, ali standing, vary considerably in size, appearance, and
there was very little space between people. neighborhood setting. Even more so, the in-
Several participants were overheard to say, teriors of schools vary considerably. The
"This room is really crowded." same can be said for criminal justice settings,
health settings, community mental health
Such descriptive writing requires atten- programs, and any other human service ac-
tion to detail and discipline to avoid vague, tivity.
Example of Combining Description and Metaphor to

E X H I B I T 6.3 Provide a Sense of Place
Context for a Wilderness Program: First View From Briqht Anqel Point at the Grand Canvon
We foilowed an asphalt path from the tion supporting a series of sloping sedimentary
lodge a quarter mileto Bright Angel Point, per- rock terraces, the Supai. These sweeping ter-
haps the most popular tourist site at the Grand races, spotted green with sparse desert vege-
Canyon because of its relatively easy accessi- tation, point upward like arrow feathers to a
bility. With cameras amed in ali directions at white sandstone pedestal, the Coconino. A
the spectacular panorama, in a sea of domes- dark red pinnacle of Hermit shae uniquely
tic accents and foreign tongues, we waited our crowns each temple. Eons of erosion have
turn at the edge to behold the magnificent sculpted dramatic variations in every aspect
rock temples of Ottoman Amphitheater: Deva, save one: their common geologic history. I
Brahma, Zoroaster and, in the distance, Thor. studied each separately, wanting to fix in my
Each rises a half mile above the undulating mind the differences between them, but the
grayness of the stark Tonto Platform defining shared symmetry of strata melded them into a
the eight-mile descent of Bright Angel Can- single, massive formation, a half mile high and
yon, a narrow slit hiding the inner gorge that many miles around. Behind me I heard a par-
looks like it had been drawn in black ink to ticipant say softly to no one in particular, al-
outline the base of the temples. Each beginsas most under her breath, "lt's too awesome. I
sheer Redwall that forms a massive founda- feel overwhelmed."
SOURCE: Adapted from Patton (1999a).
During site visits to early childhood edu- cal environments clearly affect people and
cation programs, we found a close associa- programs.
tion between the attractiveness of the facility Variations in the settings for a wilderness
(child-made decorations and colorful posttraining program for which I served as par-
ers on the walls, well-organized learning ticipant observer provide an interesting ex-
materiais, orderly teacher area) and other ample of how physical environments affect a
program attributes (parent involvement, program. The explicit purpose of holding
staff morale, clarity of the program's goals the "field conferences" in the wilderness
and theory of action). An attractive, well-or- was to remove people from their everyday
dered environment corresponded to an en- settings in largely urban environments sur-
gaging, well-ordered program. In observing rounded by human-made buildings and the
as well as conducting workshops, I have paraphernalia of modern industrial society.
noted how the arrangement of chairs affects Yet, wilderness environments are no more
participa tion. It is typically much easier to uniform than the environments of human
generate discussion when chairs are in a cir- service programs. During the yearlong pro-
cle rather than in lecture style. The dim light- gram, participants were exposed to four dif-
ing of many hotel conference rooms seems to ferent wilderness environments: the au-
literally drain energy from people sitting in tumn forest in the Gila wilderness of New
those rooms for long periods of time. Physi- Mxico; the rough terrain of Arizona's Kofa
Mountains in winter; the muddy, flooding pants behave toward each other in those en-
San Juan River in the canyon lands of Utah vironments. Rudolf Moos (1975) described
during the spring; and among the magnifi- the social-ecological view of programs as
cent rock formations of the Grand Canyon in follows:
suinmer, a desertenvironment. One focus of
the evaluation, then, was to observe how The social climate perspective assumes that
participants responded to the opportunities environments have unique "personalities,"
and constraints presented by these different just like people. Personality tests assess per-
environments: forest, mountains, canyon- sonality traits or needs and provide informa-
lined river, and Grand Canyon desert. tion about the characteristic ways in which
In addition, weather and seasonal differ- people behave. Social environments can be
ences accentuated variations among these similarly portrayed with a great deal of accu-
environments. Program activities were racy and detail. Some people are more sup-
clearly affected by the extent to which there portive than others. Likewise, some social
was rain, cold, wmd, and shelter. In the pro- environments are more supportive than oth-
gram^ theory, weather uncertainties were ers. Some people feel a strong need to control
expected to be a natural part of the program, others. Similarly, some social environments
offering natural challenges for the group to are extremely rigid, autocratic, and control-
deal with. But the program theory also ling. Order, clarity, and structure are impor-
called for participants to engage deeply with tant to many people. Correspondingly, many
each other during evening group discus- social environments strongly emphasize or-
sions. During one 10-day winter field con- der, clarity, and control. (p. 4)
ference that was unusually cold and wet,
participants were miserable, and it became In describing the social environment, the
increasingly difficult to carry on group dis- observer looks for the ways in which people
cussions, thus reducing considerably the organize theinselves into groups and sub-
amount of group process time available and groups. Pattems and frequency of interac-
rushing the interactions that did occur be- tions, the direction of communication pat-
cause of participants' discomfort. Program tems (from staff to participants and
staff learned that they needed to anticipate participants to staff), and changes in these
more clearly the possible variations m phys- pattems tell us things about the social envi-
ical environments, plan for those variations, ronment. How people group together can be
and include the participants in thatplanning illuminative and important. All-male versus
so as to increase their commitment to contin- all-female groupings, male-female interac-
uing the process under difficult physical tions, and interactions among people with
conditions. different background characteristics, racial
identities, and/or ages alert the observer to
pattems in the social ecology of the pro-
The Human/ gram.
Social Environment Decision-making patterns can be a partic-
ularly important part of a program's social
Just as physical environments vary, so too environment. Who makes decisions about
do social environments. The ways in which the activities that take place? To what extent
human beings interact create social-ecologi- are decisions made openly, so that partici-
cal constellations that affect how participants are aware of the decision-making pro-
cess? How are decisions by staff communi-

cated to participants? Answers to these aBSEFttlNG NATURij-
questions are an important part of the de- 0'i.p!i':'is.Vi ij! i.i V I ! :-II!: i.Vi! -r.' Lrr ;; s R; i; ; SJ-S |! r SO
scription of a prograin's decision environ- !: ' : ^ : ; ; ' ! . : .*!! M-IH :.i.!.'ri !.'
! , !hh ! rn.vrn "y !',;!
ment. lMiTifo.Mi:' rî/iri h.Vivr.-* ::V" !!:"rn
An observer's descriptions of a social en- S V I w 'I-V!\'X IÎ1'!'T:''HI'I\A!.T!:;f .
vironment will not necessarily be the same =!>=
: !! ii; iiVi- :'i > i !'!"i'r! ri Vi ;< ' ! " I " O ; : - N
'.' ! :'.;
as the perceptions of that environment ex- i.>!
! Vi r.';;: "Vri n !v'v.' vir sk-i" ' v .!!=: !!
pressed by participants. Nor is it likely that ;:.ri.;-i:'IV:-i3 .J:-I:V dYiYVi \C >. r.h- s -iivi ;;-i;vj
ali participants will perceve the setting's irii,!'V-:;i:!:i:i!i.i.'!'.i+iS! ;Vi \.'r, ! rr! hy !. ; n iirVi >y
human climate in the same way. At ali times
it is criticai that the observer record partici-
pants' comments in quotation narks, indi-
cating the sourcewho said what?so as to
keep perceptions of participants separate ley, we traced the beginnings of the differenti-
from the observer's or evaluator's own de- ation of Huayopampa from Pa caros back
scriptions and interpreta tions. more than a century. (Whyte 1984:153)
Historical Perspectives Documenting and understanding the

context of a program will require delving
Historical information can shed impor- into its history. How was the program cre-
tant light on the social environment. The his- ated and initially funded? Who were the
tory of a program, community, or organiza- original people targeted for program ser-
tion is an important part of the context for vices, and how have target populations
research. Distinguished qualitative sociolo- change d over time? To what extent and in
gist William Foote Whyte, sometimes called what ways have goals and intended out-
the father of sociological field research, has comes changed over time? What have staff-
reflected on how he came to value historical ing patterns been over time? How has the
research as a criticai part of his fieldwork. program's governance (board) been in-
volved at various stages in the program's
When we began our Peruvian research pro- history? What crises has the program en-
gram, I viewed history as having little value dured? If the program is embedded within a
for understanding the current scene. I thought larger organizational context, what is the
I was only being sympathetic to the interests of history of that organization in relation to the
our Peruvian researchers in suggesting that program? How has the larger political and
they gather historical data on each village for economic environment changed over time,
the last 50 years. and how have those changes affected pro-
Fortunately, the Peruvians refused to ac- gram development? What are the stories
cept the 50-year limit and in some cases people tell about the program's history?
probed up to 500 years in the history of vil- These kinds of questions frame inquiry into
lages or areas. Much of these data on rural the program's history to illuminate context.
commuriities would be of interest only to his- In the 1990s, I evaluated a "free high
torians, However, understanding the paradox school" that had been created during the
of the Mantaro Valley required us to go back to struggles and turmoil of the 1960s. Little
the conquest of Peru, and, in the Chancay Val- about the program's current programming
Fieldzuork Strategies and Observation Methods lj. 285
could be understood outside the context of own feelings as part of the observation.)
its historical emergence. The schooTs image How do behaviors and feelings change over
of itself, its curriculum, and its policies had the course of the activity?
been handed down and adapted from that Finally, the observer looks for closure
intense period of early development. Doing points. What are the signals that a particular
fieldwork in the 1990s could only be done by activity is being ended? Who is present at
traversing the memories and legends of the that time? What is said? How do partici-
school's historical emergence in the 1960s. pants react to the ending of the activity?
How is the completion of this unit of activity
Planned Program related to other program activities and fu-
Implementation Activities ture plans?
and Formal Interactions Each unit of activity is observed and
treated as a self-contained event for the pur-
Most evaluations focus at least some ob- pose of managing field notes. The observa-
servations on planned program activities. tion of a single session of the early childhood
What goes on in the program? What do par- parent education program presented in
ticipants and staff do? What is it like to be a Chapter 1 is an example. Each observed
participant? These are the kinds of questions event or activity can be thought of as a
evaluators bring to the program setting to mini-case write-up of a discrete incident, ac-
document program implementation. tivity, interaction, or event. During analysis,
Build observations around activities that one looks across these discrete units-of-
have a kind of unity about them: a begin- activity cases for patterns and themes, but
ning, some middle point, and a closure during the initial stages of fieldwork the ob-
pointsuch things as a class session, a coun- server will be kept busy just trying to cap-
seling session, meai time in the residential ture self-contained units of activity without
facility, a meeting of some kind, a home visit worrying yet about looking for patterns
in an outreach program, a consultation, or a across activities.
registration procedure. Attending to se- Observing and documenting formal pro-
quence illustrates how the inquiry program activities will constitute a central ele-
gresses over the course of an observation. ment in evaluating planned program imple-
Initially, the observer will focus on how the mentation, but to fully understand a
activity is introduced or begun. Who is pres- program and its effects on participants, ob-
ent at the beginning? What exactly was said? servations should notbe restricted to formal,
How did participants respond or react to planned activities. The next section dis-
what was said? cusses observation of the things that go on
These kinds of basic descriptive ques- between and around formal, planned pro-
tions guide the evaluator throughout the full gram activities.
sequence of observation. Who is involved?
WJwt is being done and saidby staff and par-
Informal Interactions
ticipants? How do they go about what they
do? Where do activities occur? When do and Unplanned Activities
things happen? What are the variations in
how participants engage in planned activi- If observers put a way their seeing and ob-
ties? How does it feel to be engaged in this serving selves as soon as a planned, formal
activity? (The observer records his or her activity ends, they will miss a great deal of
data. Some programs build in "free" or un- Was it clear to you what they were trying to
structured time between activities, with the get at?
clear recognition that such periods provide
What did you think of the session today?
opportunities for participants to assimilate
what has occurred during formal program- How do you think what went on today fits
matic activities as well as to provide partici- into this whole thing that we're involved in?
pants with necessary breathing space.
Rarely, if ever, can a program or institution Such questioning should be done in an easy,
plan every moment of participants' time. conversational manner so as not to be intru-
During periods of informal interaction sive or so predictable that every time some-
and unplanned activity, it can be particu- one sees you coming they know what ques-
larly difficult to organize observations be- tions you're going to ask. "Get ready, here
cause people are likely to be milling around, comes the evaluator with another endless
coming and going, moving in and out of set of questions." Also, when doing infor-
small groups, with some sitting alone, some mal, conversational interviewing, be sure
writing, some seeking refreshments, and that you are acting in accordance with ethi-
otherwise engaging in a full range of what cal guidelines regarding informed consent
may appear to be random behaviors. How, and confidentiality. (See the earlier discus-
then, can the evaluator observer collect data sion in this chapter about overt versus co-
during such a time? vert fieldwork.)
This scenario illustrates beautifully the How something is said should be re-
importance of staying open to the data and corded along with what is said. At a mom-
doing opportunity sampling. One can't an- ing break in the second day of a two-day
ticipate ali the things that might emerge dur- workshop, I joined the other men in the
ing unplanned program time, so the ob- restroom. As the men lined up to use the fa-
server watches, listens, and looks for cilities, the first man to urinate said loudly,
opportunities to deepen observations, re- "Here's what I think of this program." As
cording what people do, the nature of infor- each man finished he turned to the man be-
mal interactions (e.g., what subgroups are in hind him and said, "Your turn to piss on the
evidence), and, in particular, what people program." This spontaneous group reaction
are saying to each other. This last point is spoke volumes more than answers to formal
particularly important. During periods of interview questions and provided much
unplanned activity, participants have the greater depth of expression than checking
greatest opportunity to exchange views and "very dissatisfied" on an evaluation ques-
to talk with each other about what they are tionnaire.
experiencing in the program. In some cases, Everything that goes on in or around the
the evaluator will simply listen in on conver- program is data. The fact that none of the
sations or there may be opportunities to con- participants talk about a session when it is
duct informal interviews, either with a sin- over is data. The fact that people immedi-
gle participant in natural conversation or ately split in different directions when a ses-
with some small group of people, asking sion is over is data. The fact that people talk
normal, conversational questions: about personal interests and share gossip
that has nothing to do with the program is
So what did you think of what went on this data. In many programs, the most sig-
moming? nificant participant learnings occur during
unstructured time as a result of interactions feeling among some staff members that they
with other participants. To capture a holistic had a responsibility to plan and account for
view of the program, the evaluator observer every moment during the program.
must stay alert to what happens during Participant observation necessarily com-
these informal periods. While others are on bines observing and informal interviewing.
break, the observer is still working. No Observers need to be disciplined about not
breaks for the dedicated field-worker! Well, assuming they know the meaning to partici-
not really. You've got to pace yourself and pants of what they observe without check-
take care of yourself or your observations ing with those participants. During one pe-
will deteriorate into mush. But you get the riod of unstructured time in the wilderness
idea. You may be better off taking a break program, following a fairly intensive group
during part of a formal session time so you activity in which a great deal of interper-
can work (collect data) while others are on sonal sharing had taken place, I decided to
break. pay particular attention to one of the older
As happens in many programs, the par- men in the group who had resisted involve-
ticipants in the wilderness education pro- ment. Throughout the week he had taken
gram I was observing/evaluating began every available opportunity to make it
asking for more free, unstructured time. known that he was unimpressed with the
When we weren't hiking or doing camp program and its potential for impact on him.
chores, a lot of time was spent in formal dis- When the session ended, he immediately
cussions and group activities. Participants walked over to his backpack, pulled out his
wanted more free time to journal. Some sim- writing materiais, and went off to a quiet
ply wanted more time to reflect. Most of ali, spot where he could write. He continued
they wanted more time for informal interac- writing, completely absorbed, until dinner-
tions with other participants. I respected the time an hour later. No one interrupted him.
privacy of one-to-one interactions when I With his legs folded, his notebook in his lap,
observed them and would never attempt to and his head and shoulders bent over the
eavesdrop. I would, however, watch for such notebook, he gave off clear signals that he
interactions and, judging body language was involved, concentrating and working
and facial expressions, I would speculate on something to which he was giving a
when serious interpersonal exchanges were great deal of effort.
taking place. I would then look for natural I suspected as I watched that he was vent-
opportunities to engage each of those partic- ing his rage and dissatisfaction with the pro-
ipants in conversational interviews, telling gram. I tried to figure out how I might read
them I had noticed the intensity of their in- what he had written. I was so intrigued that I
teraction and inquiring whether they were momentarily even considered covert means
willing to share what had happened and of getting my hands on his notebook, but
what significance they attached to the inter- quickly dismissed such unethical mvasion
action. Most appreciated my role in docu- of his privacy. Instead, I looked for a natural
menting the program's unfolding and its ef- opportunity to mitiate a conversation about
fects on participants and were open to his writing. During the evening meai
sharing. It was on the basis of those informal around the campfire, I moved over next to
interviews and observations that I provided him, made some small talk about the
formative feedback to staff about the impor- weather, and then began the following con-
tance of free time and helped alleviate the versation:
"You know, in documenting experiences letter may have been more for me than for
people are having, I'm trying to track some him. But the most important thing that's
of the different things folks are doing. The been happening for me during this week is
staff have encouraged people to keep jour- the time to think about my family and how
nals and do writing, and I noticed that you important it is to me and I haven't been hav-
were writing fairly intensely before dinner. ing a very good relationship with my son. In
If you're willing to share, it would be helpful fact, it's been pretty shitty and so I wrote him
for me to know how you see the writing fit- a letter. That's ali."
ting into your whole experience with the This short conversation revealed a very
program." different side of this man and an important
He hesitated, moved his food about in his impact of the program on his personal and
bowl a little bit, and then said, "I'm not sure family life. We had several more conversa-
about the program or how it fits in or any of tions along these lines, and he agreed to be a
that, but I will tell you what I was writing. I case example of the family impacts of the
was writing . . . , " and he hesitated because program. Until that time, impacts on family
his voice cracked, "a letter to my teenage son had not even been among the expected or in-
trying to tell him how I feel about him and tended outcomes of the program. It turned
make contact with him about some things. I out to be a major area of impact for a number
don't know if I'll give the letter to him. The of participants.
The Native Language of the Program
T he lunatic, the lover, and the poet

Are of imagination ali compact.
One sees more devils than vast hell can hold;
That is, the madman. The lover, ali as frantic,
Sees Helen's beauty in a brow of Egypt.
The poet's eye, in a fine frenzy rolling,
Doth glance from heaven to earth, from earth to heaven;
And as imagination bodies forth the forms of things unknown,
The poet's pen turns them to shapes
And gives to airy nothing
A local habita tion and a name.
William Shakespeare,
A Midsummer Nighfs Dreatn, Act V, scene 1
As noted in Chapter 2, the Whorf hypoth- that truck drivers were entering "empty"
esis (Schultz 1991) alerts us to the power of warehouses smoking cigarettes and cigars.
language to shape our perceptions and ex- The warehouses often contained invisible,
periences. As an insurance investigator, but highly flammable gases. He interviewed
Benjamin Whorf was assigned to look into truckers and found that they associated the
explosions in warehouses. He discovered word empty with harmless and acted accord-
ingly. Whorfs job, in Shakespeare's terms,i, plex system of language to distinguish dif-
was to turn the truckers' perception of "airyy ferent degrees and types of retarda tion, a
nothing" into the shape of possible danger. language that changes as cultural and politi-
An anthropological axiom insists that one2 cal sensitivities change. People in criminal
cannot understand another culture withoutt justice generate language for distinguishing
understanding the language of the people inn types of offenders or "perps" (perpetrators).
that culture. Language organizes our worldi Fieldwork involves learning the "native lan-
for us by shaping what we see, perceive, and1 guage" of the setting or program being stud-
pay attention to. The things for which peo- ied and attending to variations in connota-
ple have special words tell others what is im- tioris and situational use. The field notes and
portant to that culture. Thus, as studentss reports of the observer should include the
learn in introductory anthropology, Eski- exact language used by participants to com-
mos have many words for snow and Arabss municate the flavor and meaning of "na-
have many words for camel. Likewise, thee tive" program language.
artist has many words for red and differentt Language was especially important in the
kinds of brushes. wilderness education program I evaluated.
Roderick Nash (1986), in his classic study/ These were highly verbal people, well edu-
Wilderness and the American Mind, traces how/ cated, reflective and articulate, who spent a
changing European American perceptionss lot of program time in group discussioris.
of "wilderness" has affected at the deepestt Program staff understood how words can
leveis our cultural, economic, and political1 shape experiences. They wanted partici-
perspectives on deserts, forests, canyons, pants to view the time in the wilderness as a
and rivers. He traced the very idea of wilder- professional development learning experi-
ness to the eighth-century heroic epic char- ence not a vacation, so staff called each week
acter Beowulf, whose bravery was definedi in the wilderness a "field conference." They
by his courage in entering the wildeoraa hoped participants would see the program
place of wild and dangerous beasts, dark< as a "conference" held in the "field." Despite
and foreboding forests, and untamed, pri- the determined efforts of staff, however, the
mordial spirits. In the Judeo-Christian tradi- participants never adopted this language.
tion, wilderness came to connote a place off Almost universally they referred to the
uncontrolled evil that needed to be tamedi weeks in the wilderness as "trips." During
and civilized, while Eastern cultures and re- the second "field conference" the staff capit-
ligions fostered love of wilderness ratherr ulated. Interestingly enough, that capitula-
than fear. Nash credits the Enlightenmentt tion coincided with negative reactions by
with offering new ways of thinking aboutt participants to some logistical inadequacies,
wildernessand new language to shapee unsuccessful program activities, and bad
that changed thinking. weather, ali of which undercut the "confer-
Moving from the wilderness to the inte- ence" emphasis. Staff language reflected
rior territory of organizations, agencies, and1 that change.
programs, language still shapes experiencee Other language emerged that illuminated
and is therefore an important focus during5 participants' experiences. One of the partici-
fieldwork. Programs develop their own lan- pants expressed the hope of "detoxifying" in
guage to describe the problems they deal1 the wilderness. He viewed his return to his
with in their work. Educators who workk everyday world as "poisonous retoxifica-
with learning disabled students have a com- tion." The group immediately adopted this
language of detoxification and retoxification tendency to personify natural forms. Thus

to refer to "wilderness time" versus ordi- began a sustamed personal interest in how
nary "urban civilization time," ultimately names in the wilderness shaped our experi-
shortening the words to detox and retox. This ences there (Patton and Patton 2001).
language carne to permeate the program's When I took my son into the Grand Can-
culture. yon for a "coming of age" initiation experi-
The discussions in the wilderness often ence (Patton 1999a), he reacted to the prob-
reflected the physical environment in which lem of finding words for that awesome
program activities took place. Participants environment by making up words. For ex-
became skilled at creating analogies and ample, upon seeing the Canyon for the very
metaphors to contrast their urban work lives first time he whispered, "Bue dden," which
with their wilderness experiences. After became our way of describing things too
backpacking ali day, participants could be beautiful and awesome for ordinary words.
heard talking about learning how to "pace Capturing the precise language of partici-
myself in my work," or "shifting the bur- pants honors the emic tradition in anthro-
dens of responsibilities that I carry so that pology: recording participants' own under-
the load is more evenly balanced" (a refer- standings of their experiences. Observers
ence to the experience of adjusting the must learn the language of participants in
weight of the backpack). In the mountains, the setting or program they are observing
after rock climbrng, participants referred to in order to faithfully represent participants
"the danger in taking risks at work without in their own terms and be true to their
support" (a reference to the balay system of worldview.
climbing where someone supports the
climber with a safety rope below). One dis- Nonverbal Communication
cussion focused on how to "find toeholds
and handholds" to bring about change back Social and behavioral scientists have re-
home, "to get on top of the steep wall of reported at length the importance of both ver-
sistance in my institution." They even as- bal and nonverbal communication in hu-
signed numbers to degrees of back-home in- man groups. While recording the language
stitutional resistance corresponding to the of participants, the observer should also at-
numbers used to describe the degree of diffi- tend to nonverbal forms of communication.
culty of various rock climbs. On the ri ver, For example, in educational settings nonver-
participant language was filled with phrases bal communication would include how stu-
like "going with the flow," "learning to mon- dents get the attention of or otherwise ap-
itor professional development like you read proach instructors, such as waving their
and monitor the current," and "trying to hands in the air. In group settings a great
find my way out of the eddies of life." deal of fidgeting and noving about may re-
Because of the power of language to veal things about attention and involve-
shape our perceptions and experiences, ment. How participants dress, express affec-
most participants wanted to know the tion, and sit together or apart are examples
names of the rock forinations, winding can- of nonverbal cues about social norms and
yons, and river rapids we encountered, patterns.
while others, following Desert Solitaire au- Again, the wilderness program provides
thor Edward Abbey (1968), set for them- informative examples. Hugging emerged as
s eives the goal of suppressing the human a nonverbal way of providing support at
times of emotional distress or celebration, how potential interviewees reacted to her

for example, a way to recognize when some- subject matter, the study of young middle-
one had overcome some particularly diffi- class women in ndia who read Western ro-
cult challenge, like making it up across a mance novis. She had to depend on reading
ledge along a cliff face. But subgroups dif- body language to pick up hostility, disap-
fered in amount of and comfort with hug- proval, support, or openness because the
ging, and different field conferences mani- verbal formalities of some interactions of-
fested different amounts of hugging. When fered fewer cues than nonverbal reactions.
the group felt disparate, separated, with Among the young women, giggles, winks,
people on their own "trips," isolated from animated interactions, lowered eyes, and
each other, little hugging occurred either in direct gaze became cues about how the field-
pairs or around the group campfire. When work was progressing.
the depth of connection was deeper, shoul- A caution is in order here. Nonverbal be-
der-to-shoulder contact around the camp- haviors are easily misinterpreted, especially
fire was common and group singing was cross-culturally. Therefore, whenever possi-
more likely. Over time, it became possible to ble and appropriate, having observed what
read the tenor of the group by observing the appear to be significant nonverbal behav-
amount and nature of physical contact par- iors, some effort should be made to follow
ticipants were having with each otherand up with those involved to find out directly
participants in groups with a lot of hugging from them what the nonverbal behaviors re-
and connectedness reported noticeably ally meant. I confirmed with other partici-
greater personal change. pants in the wilderness program the impor-
In evaluating an international develop- tance of hugging as a mechanism that they
ment project, I observed that the three host themselves used to sense the tenor of the
country nationals ("locais") had developed group.
a subtle set of hand signals and gestures that
the American staff never noticed. In meet- Unobtrusive Observations
ings, the host country nationals regularly
communicated with each other and oper- Being observed can make people self-
ated as a team using these nonverbal signals. conscious and generate anxiety, especially
Later, having gained their confidence, I when the observations are part of an evalua-
asked the local staff members about the gestion. Regardless of how sensitively observa-
tures. They told me that the Americans had tions are made, the possibility always exists
insisted that each person participate as an that people will behave differently under
individual on an equal footing in staff meet- conditions where an observation or evalua-
ings and, to support an atmosphere of open- tion is taking place than they would if the ob-
ness, the Americans asked them not to use server were not present.
their own language during staff meetings.
But the locais wanted to operate as a unit to Even when well-intentioned and cooperative,
counter the power of the Americans, so they the research subject's knowledge that he is
developed subtle gestures to communicate participating in a scholarly search may con-
with each other since they were denied use found the invs tigator's data.... It is impor-
of their own language. tant to note early that the awareness of testing
Parameswaran (2001) has described hw need not, by itself, contaminate responses. It is
she relied on reading nonverbal cues to tell a question of probabilities, but the probability
of bias is high in any study in which a respon- ing. Ali participants were provided with
dentis aware ofhis subjectstatus. (Webb etal. learning logs at the beginning of the first
1966:13) field conference and were encouraged to use
them for private reflections and journaling.
Concern about reactions to being ob- These three-ring binders contained almost
served has led some social scientists to rec- no paper when first given to participants.
ommend covert observations as discussed Participants brought the learning logs back
earlier in this chapter. An alternative strat- each time they returned to the wilderness.
egy involves searching for opportunities to (The program involved four different trips
collect "unobtrusive measures" (Webb et al. over the course of a year.) The extent to
1966). Unobtrusive measures are those which paper had been added to the note-
made without the knowledge of the people books was one indicator of the extent to
being observed and without affecting what which the logs were being used.
is observed. The personnel of the National Forest Ser-
Robert L. Wolf and Barbara L. Tymitz vice and the Bureau of Land Management
(1978) included unobtrusive measures in have a kind of unobtrusive measure they use
their naturalistic inquiry evaluation of the in "evaluating" the wilderness habits of
National Museum of Natural History at the groups that go through an area such as the
Smithsonian Institution. They looked for San Juan River in Utah. The canyons along
"wear spots" as indicators of use of particu- the San Juan River are a very fragile environ-
lar exhibit areas. They decided that worn ment. The regulations for use of that land are
rugs would indicate thepopularity of partic- essentially "take only photographs, leave
ular areas in the museum. The creative eval- only footprints." This means that ali gar-
uator can learn a number of things about a bage, including human waste and feces, are
program by looking for physical clues. to be carried out. It takes several days to go
Dusty equipment or files may indicate down the river. By observing the amount
things that are not used. Areas that are used and types of garbage groups carry out, one
a great deal by children in a school will look can learn a great deal about the wilderness
differentthat is, more wornthan areas habits of various groups and their compli-
that are little used. ance with river regulations.
In a week-long staff training program for The creative observer, aware of the vari-
300 people, I asked the kitchen to systemati- ety of things to be learned from studying
cally record how much coffee was con- physical and social settings, will look for op-
sumed in the morning, afternoon, and eve- portunities to incorporate unobtrusive mea-
ning each day. Those sessions that I judged sures into fieldwork, thereby manifesting a
to be particularly boring had a correspond- "sympathy toward multi-method mquiry,
ingly higher levei of coffee consumption. triangulation, playfulness in data collection,
Active and involving sessions showed less outcroppings as measures, and alternatives
coffee consumption, regardless of time of to self report" (Webb and Weick 1983:210).
day. (Participants could get up and getcoffee A particularly powerful example of un-
whenever they wanted.) obtrusive fieldwork is Laura Palmer's (1988)
In the wilderness program, the thickness study of letters and remembrances left at the
of notebooks called "learning logs" became Vietnam Veterans Memorial in Washington,
an unobtrusive indicator of how engaged D.C., a work she called Shrapnel in the Heart.
participants were in self-reflective journal- For the unobtrusive part of her fieldwork,
Palmer sampled items left at the memorial, intriguing form of analysis involves com-
ali of which are saved and warehoused by paring official statements found in public
the U.S. government. She categorized and documents (brochures, board minutes, an-
analyzed types of items and the content of nual reports) with private memos and what
messages. In some cases, because of identi- the evaluation observer actually hears or
fying information contained in letters or in- sees occurring the program. Client files are
cluded with objects (photographs, baby another rich source of case data to supple-
shoes, artwork), she was able, through in- ment field observations and interviews. For
tensive investigative work, to locate the peo- example, Vesneski and Kemp (2000) coded
ple who left the materiais and interview and analyzed intake sheets and copies of
them. Their stories, the intrusive part of her family plans produced during more than
study, combined with vivid descriptions of 100 "family conferences" involving the ex~
the objects that led her to them, offer dra- tended families of abused or neglected chil-
matic and powerful insights into the effects dren in child welfare decision making in the
of the Vietnam War on the lives of survivors. state of Washington.
In one sense, her analysis of letters, journals, At the very beginning of an evaluation or
photos, and messages can be thought of as a organizational fieldwork, access to poten-
nontraditional and creative form of docu- tially important documents and records
ment analysis, another important fieldwork should be negotiated. The ideal situation
strategy. would include access to ali routine records
on clients, ali correspondence from and to
Documents program staff, financial and budget records,
organizational rules, regulations, memo-
Records, documents, artifacts, and ar- randa, charts, and any other official or unof-
chiveswhat has traditionally been called ficial documents generated by or for the pro-
"material culture" in anthropologyconsti- gram. These kinds of documents provide the
tute a particularly rich source of information evaluator with information about many
about many organizations and programs. things that cannot be observed. They may
Thus, archival strategies and techniques reveal things that have taken place before
constitute part of the repertoire of field re- the evaluation began. They may include pri-
search and evaluation (Hill 1993). In con- vate interchanges to which the evaluator
temporary society, ali kinds of entities leave would not otherwise be privy. They can re-
a trail of paper and artifacts, a kind of spoor veal goals or decisions that might be other-
that can be mined as part of fieldwork. Fam- wise unknown to the evaluator.
ilies keep photographs, children's school- In evaluating the mission fulfillment of a
work, letters, old Bibles with detailed gene- major philanthropic foundation, I examined
alogies, bronze d baby shoes, and other 10 years of annual reports. Each report was
sentimental objects that can inform and en- professionally designed, elegantly printed,
rich family case studies. People who commit and widely disseminatedand each report
suicide leave behind suicide notes that can state d a slightly different mission for the
reveal pattems of despair in a society foundation. It turned out that the president
(Wilkinson 1999). Gangs and others inscribe of the foundation wrote an annual introduc-
public places with graffiti. Organizations of tion and simply stated the mission from
ali kinds produce mountains of records, memory. The publication designer routinely
both public and private. Indeed, an oft- lifted this "mission statement" from the
presidenfs letter and highlighted it in bold utive staff in the project. Disagreements
font at the beginning of the report, often on about program finances constituted but one
the cover page. From year to year the focus arena of communication difficulties during
changed until, over the course of 10 years, the program, including time in the wilder-
the stated mission had changed dramati- ness. Interviews with those involved re-
cally without official board action, approval, vealed quite different perceptions of the na-
or even awareness. Further invstigation ture of the conflicts, their intensity, and their
through years of board minutes revealed potential for resolution. While participants
that, in fact, the board had never adopted a became aware of some arguments among
mission statement at ali, a matter of consid- staff, for the most part they were unaware of
erable surprise to ali involved. the origins of those conflicts and the extent
As this example shows, documents prove to which program implementation was
valuable not only because of what can be hampered by them.
learned directly from them but also as stimu- My review of files also revealed the enor-
lus for paths of inquiry that can be pursued mous complexity of the logistics for the wil-
only through direct observation and inter- derness education program. Participants
viewing. As with ali information to which an (college deans, program directors, adminis-
evaluator has access during observations, trators) were picked up at the airport in vans
the confidentiality of program records, par- and driven to the wilderness location where
ticularly client records, must be respected. the field conference would take place. Par-
The extent to which actual references to and ticipants were supplied with ali the gear nec-
quotations from program records and docu- essary for surviving in the wilderness. Prior
ments are included in a final report depends to each field trip, staff had many telephone
on whether the documents are considered and written exchanges with individual par-
part of the public record and therefore able ticipants about particular needs and fears.
to be publicized without breach of confiden- Letters from participants, especially those
tiality. In some cases, with permission and new to the wilderness, showed how little
proper safeguards to protect confidentiality, they understood about what they were get-
some information from priva te documents ting into. One seasoned administrator and
can be quoted directly and cited. hard-core smoker inquired, with reference
Program records can provide a behind- to the first 10-day hike in the heart of the Gila
the-scenes look at program processes and wilderness, "Will there be a place to buy cig-
how they came into being. In the wilderness arettes along the way?" Talk about being
program evaluation, program staff made clueless! But by the end of the year of field
their files available to me. I discovered a trips, he had given up smoking. His letter of
great deal of information not available to inquiry alerted me to the impor tance of this
other program participants: letters detailing pre-post observation.
both conceptual and financial debates be- Without having looked over this corre-
tween the technical staff (who led the wil- spondence, I would have missed the extent
derness trips) and the project directors (who to which preparation for the one-week expe-
had responsibility for the overall manage- riences in the wilderness consumed the time
ment of the program). Without knowledge and energy of program staff. The intensity of
of those arguments it would have been im- work involved before the field conferences
possible to fully understand the nature of helped explain the behavior of staff once the
the interactions between field staff and exec- field trips got under way. So much had gone
into the preparations, virtually none of tion that I include among the tasks of the ob-
which was appreciated by or known to pro- server that of noting what does not occur.
gram participants, that program staff would If social science theory, program goals,
sometimes experience a psychological let- implementa tion designs, and/or proposals
down effect and have difficulty energizing suggest that certain things ought to happen
themselves for the actual wilderness experi- or are expected to happen, then it is appro-
ence. priate for the observer or evaluator to note
Learning to use, study, and understand that those things did not happen. If a com-
documents and files is part of the repertoire munity where water is scarce shows no evi-
of skills needed for qualitative inquiry. For dence of conflict over water rights, an an-
an extended discussion of the interpreta tion thropologist could be expected to report and
of documents and material culture, see explain this absence of community conflict.
Hodder (2000). If a school program is supposed to, accord-
ing to its funding mandate and goals, pro-
vide children with opportunities to explore
Observing What the community and no such explorations
Does Not Happen occur, it is altogether appropriate for the
evaluator to note said implementation fail-
The preceding sections have described ure. If the evaluator repor ted only what oc-
the things one can observe in a setting or curred, a question might be left in the mind
program. Observing activities, interactions, of the reader about whether the other activi-
what people say, what they do, and the na- ties had occurred but had simply not been
ture of the physical setting is important in a observed. Likewise, if a criminal justice pro-
comprehensive approach to fieldwork. But gram is supposed to provide one-to-one
what about observing what does not hap- counseling to juveniles and no such counsel-
pen? ing takes place, it is entirely appropriate for
The potential absurdity of speculating the evaluator to note the absence of coun-
about what does not occur is illustrated by a seling.
Sufi story. During a plague of locusts, the In observing early childhood programs,
wise-fool Mulla Nasrudin, always looking the absence of children's art on the walls in
on the bright side, went from village to vil- one center stood out. Indeed, the absence of
lage encouraging people by observing how any colorful posters or art of any kind stood
fortunate they were that elephants had no out because ali other centers' walls were
wings. "You people don't realize how lucky covered with colorful displays. When I
you are. Imagine what life would be like pointed this out, embarrassed staff members
with elephants flying overhead. These lo- explained that they had set in motion a plan-
custs are nothing." ning process for decorating the walls that
To observe that elephants have no wings had become bogged down and they had just
is indeed data. Moreover, elephants have no neglected to get back to the issue because,
fins, claws, feathers, or branches. Clearly, they realized, they got gotten used to the
once one ventures into the area of observ- way things were.
ing what does not happen, there are a near- Thus, it can be appropriate to note that
infinite number of things one could point something did not occur when the ob-
out. The "absence of occurrence" list could serveis basic knowledge of and experience
become huge. It is therefore with some cau- with the phenomenon suggests that the ab-
sence of some particular activity or factor is goals would include statements about the
noteworthy. This clearly calls for judgment, necessity of staff being sensitive to the par-
common sense, and experience. As eminent ticular needs, interests, and cultural pat-
qualitative methodologist Bob Stake (1995) terns of minorities, but there may not be
has asserted: specific mention of the desired racial com-
position of program staff. If, then, the eval-
One of the principal qualifications of qualita- uator observes that the staff of the pro-
tive researchers is experience. Added to the gram consists entirely of Caucasians, it is
experience of ordnary looking and thinking, appropriate to report that the staff is ali
the experience of the qualitative researcher is White, that is, no people of color are
one of knowing what leads to significant un- among the program staff, the importance
derstanding, recognizmg good sources of of which derives from the location and na-
data, and consciously and unconsciously test- ture of the program.
ing out the veracity of their eyes and robust- Observations of staff interaction and
ness of their interpretations. It requires decision-making processes also provide
sensitivity and skepticism. Much of this meth- opportunities for evaluators to note things
odological knowledge and personality come that do not happen. If, over time, the ob-
from hard work under the criticai examina tion server notes that program planning pro-
of colleagues and mentors. (pp. 49-50) cesses never include participants' rnput in
any systematic or direct way, it may well
Making informed judgments about the be appropriate for the evaluator to pornt
significance of nonoccurrences can be out the absence of such rnput based on ex-
among the most important contributions an periences indicating the significance of
evaluator can make because such feedback participant input in the planning pro-
can provide program staff members or other cesses of other programs.
evaluation users with information that they My evaluation of the wilderness educa-
may not have thought to request. Moreover, tion program included observations about
they may lack the requisite experience or a number of things that did not occur. No
awareness to have noticed the absence of serious injuries occurred at any of the six
that which the evaluator observes. For ex- field conferences in the wildernessim-
ample, the absence of staff conflict is typi- portant information for someone thinking
cally noteworthy because staff conflict is about the possible risks involved in such a
common. Similarly, absence of conflict be- program. No participant refused to shoul-
tween administrative leveis (local, state, and der his or her share of the work that had to
federal) would be noteworthy because such be done in order for the group to live and
conflict is, in my experience, virtually uni- work together in the wilderness. This ob-
versal. servation emerged from discussions with
In many such cases, the observation technical field staff who often worked
about what did not occur is simply a restate- with juveniles in wilderness settings
ment, in the opposite, of what did occur. where uneven sharing of cooking, clean-
That restatement, however, will attract ating, and related responsibilities often led
tention in a way that the initial observation to major group conflicts. The fact that the
might not. For example, if one were observ- groups I observed never had to deal with
ing a program being conducted in a multira- one ortwo people not helping out was
cial community, it is possible that program worth noting.
Perhaps the most important observation tion, may also be presented within the
about what did not happen came from ob- larger case. The qualitative analysis pro-
serving staff meetings. Over time, I noticed a cess typically centers on presentation of
pattern in which s taff held meetings to make specific cases and thematic analysis across
decisions about important issues, but no cases. Knowing this, fieldwork can be or-
such decisions were made. Staff sometimes ganized around nested and layered case
thought that a decision had been made, but studies, which means that some form of
closure was not brought to the decision- nested case sampling must occur.
making process and no responsibility for Let me briefly review the centrality of
follow-up was assigned. Many subsequent case studies as a qualitative inquiry strat-
implementation failures and staff conflicts egy. Chapter 1 opened by citing a number
could be traced to ambiguities and differ- of well-known and influential books
ences of opinion that were left uriresolved at based on case studies, for example, In
staff meetings. By hearing me describe both Search of Excellence: Lessons From America's
what was and was not occurring, staff be- Best-Run Companies by Peters and Water-
came more explicit and effective in making man (1982), Angela Browne's important
decisions. Reporting what did happen in book When Battered Women Kill (1987), and
staff meetings was important, but it was also Sara Lawrence-Lightfoofs six detailed
extremely important to observe what did case studies in Respect (2000:13). Chapter 2
not happen. presented the construction of unique case
studies as a major strategic theme of quali-
Nested and Layered Case tative inquiry. Chapter 3 reviewed theoret-
ical perspectives that are inductively case
Studies During Fieldwork
based. Chapter 4 reviewed at some length
the importance in qualitative evaluation of
A case study is expected to catch the complex- capturing and reporting individualized out-
ity of a single case. The single leaf, even a sin- comes based on case studies of how partici-
gle toothpick, has unique complexitiesbut pants in programs change during a pro-
rarely will we care enough to submit it to case gram and whether they maintain those
study. We study a case when it itself is of very changes afterward. To illustrate this point,
special interest. We look for the detail of inter- in the wilderness education program our
action with its context. Case study is the study evaluation team constructed case studies
of the particularity and complexity of a single of participants using multiple sources of
case, coming to understand its activity within data from fieldwork: (1) background data
important circuinstances. (Stake 1995:xi) gathered through interviews about partic-
ipants' situations and perspectives upon
Months of fieldwork may result in a sin- entering the year of field conferences; (2)
gle case study that describes a village, com- observations of their experiences during
munity, neighborhood, organization, or pro- field conferences; (3) informal and conver-
gram. However, that single case study is sational interviews with them during the
likely to be made up of many smaller wilderness trips; (4) quotations from for-
casesthe stories of specific individuais, mal group interviews (focus groups) held
families, organizational units, and other at various times during the trips; (5) ex-
groups. Criticai incidents and case studies of cerpts from their journals and other per-
specific bounded activities, like a celebra- sonal writings when they were willing to
share those with us, as they often were; and tiative. As Exhibit 6.4 (p. 300) shows, how-
(6) follow-up telephone interviews with par- ever, within that overall evaluation case
ticipants after each field trip and after the en- study were nested individual case studies
tire program was completed to track the im- documenting individual experiences and
pact of the program on individuais over outcomes; case studies of each yearlong
time. group cohort; and case studies of each sepa-
Let me pause at this point and note some ra te field conference, for example, the 10
confusion in the qualitative literature about days in the Gila wilderness or the 10 days in
terminology. For example, sociologists the Kofa Mountains. Slicing through the
Hamel, Dufour, and Fortin (1993) ask: fieldwork and analysis in other ways were
case studies of particular incidents, for ex-
But is the case study a method? Or is it an ap- ample, the emotional catharsis experienced
proach? . . . Case studies employ various meth- by one participant when she finally man-
ods. These can include interviews, participant aged to overcome her terror and rappel
observation, and field studies. Their goals are down a cliff face, the whole group watching
to reconstruct and analyze a case from a socio- and urging her on, a process that took some
logical perspective. It would thus be more ap- 45 tense minutes. Other mini-cases consisted
propriate to define the case study as an of different units of analysis. A full day's
approach, although the term case method sug- hike could be a case, as could running a spe-
gests that it is indeed a method. (p. 1) cific dangerous rapid on the San Juan River.
Each evening discussion constituted a case
Whatever term or phrase is used, case stud- such that that over the three years, we had
ies depend on clearly defining the object of notes on over 80 discussions of various kinds.
study, that is, the case. But this too is com- Staff meetings made for a different unit of
plex. analysis and therefore a different series of
When more than one object of study or case studies. Thus, extended fieldwork can
unit of analysis is included in fieldwork, and typically does involve many mini- or
case studies may be layered and nested micro-case studies of various units of analy-
within the overall, primary case approach. sis (individuais, groups, specific activities,
William Foote Whyte's (1943) classic study specific periods of time, criticai incidents),
Street Comer Society has long been recog- ali of which together make up the overall
nized as an exemplar of the single-commu- case study, in this example, the final evalua-
nity (N = 1) case study (e.g., Yin 1989) even tion of the wilderness education program.
though his study of "Cornerville" includes Chapter 5 discusses at length various units
the stories (case studies) of several individ- of analysis and sampling strategies for case
ual lower-income youth, some of whom studies (see especially Exhibit 5.5 [p. 231] on
were striving to escape the neighborhood. units of analysis and Exhibit 5.6 [pp. 243-
The wilderness program illustrates how 244] on purposeful sampling strategies).
case studies often are layered and nested. Fieldwork, then, can be thought of as en-
The three-year wilderness program consti- gaging in a series of multilayered and nested
tuted the overall, one might say macro, case case studies, often with intersecting and
study. The final evaluation report presented overlapping units of analysis. One final case
conclusions about the processes and out- study deserves considerationthe observ-
comes of the overall program, a case exam- e i s experiences and reactions. We turn to
ple of a three-year wilderness education ini- that now.
Observing Oneself
P hysician, heal thyself.

Observer, observe thyself.
Halcolm
In the second chapter, I identified voice class, I did not have the language to engage in
and perspective, or reflexivity, as one of the a systematic feminist critique of patriarchy or
central strategic themes of contemporary, nationalism. Feminism for me had been unfor-
postmodern qualitative inquiry. The term tunately constructed as an illness that struck
reflexivity has entered the qualitative lexicon highly Westernized mtellectual lndian wom-
as a way of emphasizing the importance of en who were out of touch with reality.... [I]t
self-awareness, political/ cultural conscious- was my dislocation from ndia to the relatively
ness, and ownership of one's perspective. radicalized context of the United States that
Reflexivity reminds the qualitative inquirer prompted my political development as a femi-
to observe herself or himself so as to be at- nist and a woman of color. (p. 76)
tentive to and conscious of the cultural, po-
litical, social, linguistic, and ideological ori- Given this background and the contro-
gins of her or his own perspective and voice versial focus of her fieldwork (reading of
as well asand often in contrast tothe Western romance novis by young lndian
perspectives and voices of those she or he women), she identified reflective questions
observes and talks to during fieldwork. Re- to guide her reflexive inquiry during and af-
flexivity calls for self-reflection, indeed, crit- ter fieldwork:
icai self-reflection and self-knowledge, and
a willingness to consider how who one is af-
fects what one is able to observe, hear, and How do kinship roles assigned to native schol-
understand in the field and as an observer ars shape social interactions in the field? How
and analyst. The observer, therefore, during can commitments to sisterhood make it diffi-
fieldwork, must observe self as well as oth- cult for femmist ethnographers to achieve crit-
ers, and interactions of self with others. icai distance and discuss female informants'
prejudiced views? (p. 76)
Once again, for continuity, I cite Para-
meswaran (2001), who has written a won-
derfully self-reflective account of her ex- Her personal inquiry into these questions,
perience returning to her native ndia to do reflecting on her own fieldwork experiences
fieldwork as a feminist scholar after being (Parameswaran 2001), is a model of reflex-
educated in United States. ivity.
Many year ago, lndian philosopher J.
Because my parents were fairly liberal com- Krishnamurti (1964) commented on the
pareci to many of my friends' parents, I grew challenges of self-knowledge. Although his
up with a little more awareness than many reflections were directed to the importance
middle- and upper-class Indians of the differ- of lifelong learning rather than to being re-
ences between my life and that of the vast ma- flexive in fieldwork, his ruminations offer a
jority of Indians. Although I questioned some larger context for thinking about how to ob-
restrictions that were specific to women of my serve oneself, a context beyond concern
Nested, Layered, and Overlapping Mini-Case

Studies During Fieldwork: Example From the
E X H I B I T 6.4 Wilderness Education Program Evaluation
The wilderness education program evaluation illustrates how case studies often are layered and
nested. Evaluation of the three-year wilderness program constituted the overall macro case
study. Nested and layered within that overall evaluation were various mini-cases of overlap-
ping and intersecting units of analysis that helped organize and frame fieldwork.
Macro Case Study: Final Evaluation Study of the Three-Year Program
Possible nested, layered, and overlapping mini-case studies
about methodological authenticity, though mately deal with issues of authenticity, reac-
his advice applies to that as well. tivity, and how the observational process
may have affected what was observed as
Self-knowledge comes when you observe well as how the background and predispo-
yourself in your relationship with your fel- sitions of the observer may have con-
low-students and your teachers, with ali the strained what was observed and under-
people around you; it comes when you ob- stood. Each of these areas of methodological
serve the manner of another, his gestures, the inquiry depends on some degree of criticai
way he wears his clothes, the way he talks, his reflexivity.
contempt or flattery and your response; it comes
when you watch everything in you and about
you and see yourself as you see your face in Sources of Data Reviewed
the mirror. . . . Now, if you can look into the
mirror of relationship exactly as you look into This lengthy review of options in what to
the ordinary mirror, then there is no end to observe and sources of data for evaluation
self-knowledge. It is like entering a fathomless fieldwork began with the suggestion that a
ocean which has no s h o r e . . . . ; if you can just sensitizingframework canbe useful as a tool to
observe what you are and move with it, then guide fieldwork. The list of data sources
you will find that it is possible to go infinitely we've reviewed can be used to stimulate
far. Then there is no end to the journey, and thinking about evaluation fieldwork pos-
that is the mystery, the beauty of it. (Krishna- sibilities. Other phenomena and other ob-
murti 1964:50-51, emphasis added) servational arenas would have different
sensitizing frameworks or concepts. The fol-
I realize that Krishnamurti's phrase lowing summarizes the observation and in-
"There is no end to the journey" may strike quiry topics we've reviewed for evaluation:
terror in the hearts of graduate students
reading this in preparation for dissertation n Description of the program setting/
fieldwork or evaluators facing a report physical environment
deadline. But, remember, he's taking about
H Description of the social environment
lifelong learning, of which the dissertation
or a specific evaluation report is but one n Capturing historical perspectives
phase. Just as most dissertations and evalua- o Describing planned program implemen-
tions are reasonably expected to contribute tation activities and structured interac-
incrementai knowledge rather than make tions
major breakthroughs, so too the self-knowl-
Observing informal interactions and un-
edge of reflexive fieldwork is but one phase
planned activities
in a lifelong journey toward self-knowl-
edgebut it's an important phase and a a Recording participants' special program
commitment of growing significance as re- language
flexivity has emerged as a central theme in Observing nonverbal communication
qualitative inquiry.
Watching for unobtrusive indicators
The point here, which we shall take up in
greater depth in the chapters on analysis and H Analyzing documents, files, records,
credibility, is that the observer must ulti- and artifacts
Commenting on notable nonoccurrences L=J. Doing Fieldwork: The

(what does not happen) Data-Gathering Process
Constructing nested and layered case
studies during fieldwork for intersecting The purpose of the research has been clari-
and overlapping units of analysis fied. The primary research questions have
been focused. Qualitative methods using
Observing oneself: Reflexivity
observations have been selected as one of
o Document mg individualized and com- the appropriate methods of data gathering.
mon outcomes (Chapter 4) It is time to enter the field. Now begins the
arduous task of taking field notes.
Creativity in Fieldwork
Field Notes
No checklist can be relied on to guide ali Many options exist for taking field notes.
aspects of fieldwork. A participant observer Variations include the writing materiais
must constantly make judgments about used, the time and place for recording field
what is worth noting. Because it is impossi- notes, the symbols developed by observers
ble to observe everything, some process of as their own method of shorthand, and how
selection is necessary. Plans made during field notes are stored. No universal prescrip-
design should be revised as appropriate tions about the mechanics of and procedures
when important new opportunities and for taking field notes are possible because
sources of data become available. That's different settings lend themselves to differ-
where flexibility and creativity help. Cre- ent ways of proceeding and the precise orga-
ativity can be learned and practiced (Patton nization of fieldwork is very much a matter
1987a). Creative fieldwork means using ev- of personal style and individual work hab-
ery part of oneself to experience and under- its. What is not optional is the taking of
stand what is happening. Creative insights field notes.
come from being directly involved in the set-
Aside from getting along in the setting,
ting being studied.
the fundamental work of the observer is the
I shall return to the issue of creativity in
taking of field notes. Field notes are "the
considering the interpretation of field notes
most important determinant of later bring-
later in this chapter and again in the analysis
ing off a qualitative analysis. Field notes pro-
chapter. For the moment, it is sufficient to ac-
vide the observer's raison d'tre. I f . . . not
knowledge the centrality of creativity in nat-
doing them, [the observer] might as well not
uralistic inquiry and to concur with Virginia
be in the setting" (Lofland 1971:102).
Woolf:
Field notes contain the description of
what has been observed. They should con-
Odd how the creative power at oncebrings the tain everything that the observer believes to
whole universe to order. . . . I mark Henry be worth noting. Don't trust anything to fu-
James' sentence: observe perpetually. Observe ture recall. At the moment one is writing it is
the oncome of age. Observe greed. Observe very tempting, because the situation is still
my own despondency. By that means it be- fresh, to believe that the details or particular
comes serviceable. (quoted in Partnow 1978: elements of the situation can be recalled
185) later. If it's important as part of your con-
tions and conditions. Words such as poor, an-

[v- ger, and uneasy are insufficiently descriptive.
P / IT!!:!):!! nVWl^ M ! => ! ->": i; ri"i:!! ; ,:=; :! -- iit:'!:!.';-- Such interpretive words conceal what actu-
,'.i;i:=i1 | |; :'| h::j"j; i:=.hif i;!:" jl?i !:;=!;. " :!i Si!1- ally went on rather than reveal the details of
I|:i:5:!R;j;;=ji ] !;i,!ii'j::!;i :!;lij:!:!N !r;| i:i"ii' | ,:'I:!:.' i-!i I R - the situation. Such terms have little meaning
; :!i!:i!!i! M^TM '."Jm iSY-fe ' for a person present for the observation.
.ITKRIIRLJ;!: I-M;!^!;!;:: ! !';' " - P E ^ :' I ^ I J M ! ' I !;' U J
:
Moreover, the use of such terms in field
lri l!-' i!r! IliiiiiyK-jjtiiMITyill:':!; l!.iX ;'ii!:"j- !'! hi-r. notes, without the accompanying detailed
s-itra i-lil! fi;i'!!/l!5h;V'Of:i!n
: ! : ;Vi i'- description, means that the fieldworker has
:= !;=! IV! vi;;;':, lVii:n: 'm* i-i-fiV'i:irhnj.Vi:h.
! : ; fallen into the bad habit of primarily record-
'IrT! jlhbii-T- it^HiîTi-ili^f ing interpretations rather than description.
=!.=:I-I:=i: U !:!^J';I;I1!IXS!.'! I^^P:1! |;V;.!M!!:!; HI^TEAV:: Particularly revealing are terms that can
1'!:!' ifiiULiljr! j;!i:!i;!::iiii!:!i:i iî j;=.
! ; ; ; j 1j
; make sense only in comparison to some-
i'!."'!! ip. i!'!'!T l i ) rflt J!iJ!l 'Ji>!!
1 ! ... thing else. The phrase "poorly dressed" re-
:L.!;-iV ith!!:!; hiUiTlvi:!!:' !!:';i!:j;j!':'!: jrllii-^lr-^n quires some frame of reference about what
S'LI::;=V!=::' & I - ^ I ' ; . ; ; . ! J ; J : ; =';;
constitutes "good dress." No skill is more
criticai in fieldwork than learning to be de-
ihTS"! j;!!i!j-"::ii \\ \ . :;::;;!:;! !::L::!V: -: !
scriptive, concrete, and detailed.
Field notes also contain what people say.
Direct quotations, or as near as possible re-
sciousness as an observer, if it's information call of direct quotations, should be captured
that has helped you understand the context, during fieldwork, recording what was said
the setting, and what went on, then as soon during observed activities as well as re-
as possible that information should be cap- sponses garnered during interviews, both
tured in the field notes. formal and conversational. Quotations pro-
First and foremost, field notes are de- vide the "emic perspective" discussed ear-
scriptive. They should be dated and should lierthe insider 's perspectivewhich "is at
record such basic information as where the the heart of most ethnographic research"
observation took place, who was present, (Fetterman 1989:30).
what the physical setting was like, what so- Field notes also contam the observer's
cial interactions occurred, and what activi- own feelings, reactions to the experience,
ties took place. Field notes contain the de- and reflections about the personal meaning
scriptive information that will permit you to and significance of what has been observed.
return to an observation later during analy- Don't deceive yourself into thinking that
sis and, eventually, permit the reader of the such feelings can be conjured up again sim-
study's findings to experience the activity ply by reading the descriptions of what took
observed through your report. place. Feelings and reactions should be re-
The passages in Exhibit 6.5 on the next corded at the time they are experienced,
page illustrate different kinds of descriptive while you are in the field. Both the nature
field notes. On the left side are vague and and intensity of feelings should be recorded.
overgeneralized field notes. On the right In qualitative inquiry, the observer's own
side are more detailed and concrete field experiences are part of the data. Part of the
notes from the same observation. purpose of being in a setting and getting
These examples illustrate the probiem of close to the people in the setting is to permit
using general terms to describe specific ac- you to experience what it is like to be in that
E X H B I T 6.5 Fieldnotes Comparisons

Vague and
Overgenerolized Notes Detailed and Concrete Nates
1. The new client was 1. At first the new client sat very stiffiy on the chair next to the
uneasy waiting for receptionisfs desk. She picked up a magazine and let the pages
her intake interview. flutter through her fingers very quickly without really looking at
any of the pages. She set the magazine down, looked at her watch,
pulled her skirt down, picked up the magazine again, set it back
down, took out a cigarette and lit it. She watched the receptionist
out of the corner of her eye and glanced at the two or three other
people waiting in the room. Her eyes moved from people to the
magazine to the cigarette to the people to the magazine in rapid
succession, but avoided eye contact. When her name was finaIly
called, she jumped like she was startled.
2. The client was quite 2. When Judy, the snior staff member, told the client that she could
hostile toward the not just do whatever she wanted to do, the client began to yell,
staff person. screaming that Judy couldn't couldn't control her life, accused
Judy of being on a "power trip," and said that she'd "like to beat
the shit out of her," then told her to "go to hell." The client shook
her fist in Judy's face and stomped out of the room, leaving Judy
standing there with her mouth open, looking amazed.
3. The next student 3. The next student who carne into the room wore clothes quite
who carne in to take different from the three previous students. The other students had
the test was very hair carefully combed, clothes clean, pressed, and in good condition
poorly dressed. with colors coordinated. This new student wore soiled pants with
a tear in one knee and a threadbare seat. His flannel shirt was
wrinkled with one tail tucked into the pants and the other tail
hanging out. His hair was disheveled and his hands looked liked
he'd been playing in the engine of a car.
setting. If what it is like for you, the observer come a mechanical recording inachine on
or participant observer, is not recorded in entering the field. Insights, ideas, inspira-
your field notes, then much of the purpose tionsand yes, judgments, toowill occur
for being there is lost. while making observations and recording
Finally, field notes include your insights, field notes. It's not that you sit down early on
interpretations, beginning analyses, and and begin the analysis and, if you're an eval-
working hypotheses about what is happen- uator, make judgments. Rather, it's in the na-
ing in the setting and what it means. While ture of our intellects that ideas about the
you should approach fieldwork with a disci- meaning, causes, and significance of what
plined intention not to impose preconcep- we experience find their way into our
tions and early judgments on the phenome- minds. These insights and inspirations be-
non being experienced and observed, come part of the data of fieldwork and
nevertheless, as an observer you don't be- should be recorded in context in field notes.
I like to set off field interpretations with their journals, but some of the expansion
brackets. Others use parentheses, asterisks, had to be completed after the weeklong field
or some other symbol to distinguish inter- conference. In evaluating a leadership train-
pretations from description. The point is ing program as a participant observer, the
that interpretations should be understood to staff facilitator privately asked me not to
be just that, interpretations, and labeled as take notes during group discussions be-
such. Field-based insights are suffciently cause it made him nervous, even though
precious that you need not ignore them in most other participants were taking notes.
the hopes that, if really important, they will The extent to which notes are openly re-
return la ter. corded during the activities being observed
Field notes, then, contain the ongoing is a function of the observer's role and pur-
data that are being collected. They consist of pose, as well as the stage of participant ob-
descriptions of what is being experienced servation. If the observer or evaluator is
and observed, quotations from the people openly identified as a short-term, externai,
observed, the observer's feelings and reac- nonparticipant observer, participants may
tions to what is observed, and field-gener- expect him or her to write down what is go-
ated insights and interpretations. Field notes ing on. If, on the other hand, one is engaged
are the fundamental database for construct- in longer-term participant observation, the
ing case studies and carrying out thematic early part of the process may be devoted to
cross-case analysis in qualitative research. establishing the participant observer role
with emphasis on participation so that open
Procedurally Speaking taking of notes is deferred until the field-
worker's role has been firmly established
When field notes are written will depend within the group. At that point, it is often
on the kind of observations being done and possible to openly take field notes since, it is
the nature of your participa tion in the set- hoped, the observer is better known to the
ting being studied. In an evaluation of a par- group and has established some degree of
ent education program, I was introduced to trust and rapport.
the parents by the staff facilitator and ex- The wilderness program evaluation in-
plained the purpose of the evaluation and volved three 10-day trips ("field confer-
assured the parents that no one would be ences") with participants at different times
identified. I then openly took extensive notes during the year. During the first field confer-
without participating in the discussions. Im- ence, I never took notes openly. The only
mediately following those sessions, I would time I wrote was when others were also writ-
go back over my notes to fill in details and be ing. During the second field conference, Ibe-
sure what I had recorded made sense. By gan to openly record observations when dis-
way of contrast, in the wilderness education cussions were going on if taking notes did
program I was a full participant engaged not interfere with my participation. By the
in full days of hiking, rock climbing, and third week, I felt I could take notes whenever
rafting/kayaking. I was suffciently ex- I wanted to and I had no indication from
hausted by the end of each day that I seldom anyone that they even paid attention to the
stayed awake making field notes by flash- fact that I was taking notes. By that time I
light while others slept. Rather, each night I had established myself as a participant, and
jotted down basic notes that I could expand my participant role was more primary than
during the time that others were writing in my evaluator role.
The point here is that evaluator observers the program. By using a combination of ob-
must be strategic about taking field notes, servations, mterviewing, and document
timing their writing and recording in such a analysis, the fieldworker is able to use dif-
way that they are able to get their work done ferent data sources to validate and cross-
without unduly affecting either their partici- check findings. Each type and source of data
pation or their observations. Given those has strengths and weaknesses. Using a com-
constraints, the basic rule of thumb is to bination of data typestriangulation, a re-
write promptly, to complete field notes as currmg theme in this book increases va-
soon and as often as physically and pro- lidity as the strengths of one approach can
grammatically possible. compensate for the weaknesses of another
Writing field notes is rigorous and de- approach (Marshall and Rossman 1989:
manding work. Lofland (1971) has de- 79-111).
scribed this rigor quite forcefully: Limitations of observations include the
possibility that the observer may affect the
Let me not deceive the reader. The writing of situation being observed m unknown ways,
field notes takes personal discipline and time. program staff and participants may behave
It is ali too easy to put off actually writing in some atypical fashion when they know
notes for a given day and to skip one or more they are being observed, and the selective
days. For the actual writing of the notes may perception of the observer may distort the
take as long or longer than did the observa- data. Observations are also limited in focus-
tion! Indeed, a reasonable rule of thumb here ing only on externai behaviorsthe ob-
is to expect and plan to spend as much time server cannot see what is happening inside
writing notes as one spent in observing. This people. Moreover, observational data are of-
is, of course, not i n v a r i a n t . . . but one point is ten constrained by the limited sample of ac-
inescapable. Ali the fun of actually being out tivities actually observed. Researchers and
and about monkeying around in some setting evaluators need other data sources to find
must also be met by cloistered rigor in com- out the extent to which observed activities
mitting to paperand therefore to future use- are typical or atypical.
fulnesswhat has taken place. (p. 104) Interview data limitations include possi-
bly distorted responses due to personal bias,
anger, anxiety, politics, and simple lack of
Observations, Interviews, awareness since interviews can be greatly af-
and Documentation: fected by the emotional state of the inter-
Bringing Together viewee at the time of the interview. Inter-
view data are also subject to recall error,
Multiple Perspectives
reactivity of the interviewee to the inter-
Fieldwork is more than a single method or viewer, and self-serving responses.
technique. For example, evaluation field- Observations provide a check on what is
work means that the evaluator is on-site reported in interviews; interviews, on the
(where the program is happening) observ- other hand, permit the observer to go be-
ing, talking with people, and going through yond externai behavior to explore feelings
program records. Multiple sources of infor- and thoughts.
mation are sought and used because no sin- Documents and records also have limita-
gle source of information can be trusted to tions. They may be incomplete or maccurate.
provide a comprehensive perspective on Client files maintained by programs are no-
toriously variable in quality and complete- wilderness as an environment for experien-

ness, with great detail in some cases and vir- tial education. Each of the staff members
tually nothing in others. Document analysis, described in interviews his or her percep-
however, provides a behind-the-scenes look tions of how these former relationships af-
at the program that may not be directly ob- fected the field operations of the program,
servable and about which the interviewer including difficulties in communication that
might not ask appropriate questions with- had emerged during planning sessions prior
out the leads provided through documents. to the actual field conferences. Some of those
By using a variety of sources and re- conflicts were documented in letters and
sources, the evaluator observer can build on memos. Reading their files and correspon-
the strengths of each type of data collection dence gave me a deeper understanding of
while minimizing the weaknesses of any the different assumptions and values of var-
single approach. This mixed methods, trian- io us staff members. But the documentation
gulated approach to fieldwork is based on would not have made sense without the in-
pragmatism (Tashakkori and Teddlie 1998) terviews, and the focus of the interviews
and is illustrated in my attempt to under- carne from the field observations. Taken to-
stand some of the problems in volve d in staff gether, these diverse sources of informa-
communication during the wilderness edu- tion and data gave me a complete picture of
cation evaluation. I mentioned this example staff relationships. Working back and forth
earlier, but I'd like to expand it here. among individual staff members and group
As noted, two kinds of staff worked in the staff meetings, I was able to use this informa-
program: (1) those who had overall manage- tion to assist staff members in their efforts to
ment and administrative responsibility and improve their communication during the fi-
(2) the technical staff, who had responsibil- nal field conference. Ali three sources of in-
ity for wilderness skills training, field logis- formation proved criticai to my understand-
tics, and safety. The technical staff had exten- ing of the situation, and that understanding
sive experience leading wilderness trips, but enhanced my effectiveness in providing
they also were skilled at facilitating group feedback as a forma tive evaluator.
processes. During the trips, the lines of re-
sponsibility between technical staff and ad-
ministrative staff were often blurred and, on The Technology of
occasion, these ambiguities gave rise to con- Fieldwork and Observation
flicts. I observed the emergence of conflict
early on the first trip but lacked context for The classic image of the anthropological
knowing what wasbehind these differences. fieldworker is of someone huddled in an Af-
Through interviews and casual conversa- rican hut writing voluminously by lantern.
tions during fieldwork, I learned that ali of Contemporary researchers, however, have
the staff, both administrative and technical, available to them a number of technological
had known each other prior to the program. innovations that, when used judiciously, can
Indeed, the program administrative direc- make fieldwork more efficient and compre-
tors had been the college professors of the hensive. First and foremost is the battery-
technical staff while the latter were still un- operated tape recorder or dictaphone. For
dergraduate students. However, the techni- some people, myself included, dictating
cal staff had introduced the directors to the field notes saves a great deal of time while
increasing the comprehensiveness of the re- sual feedback to staff. Videotaping class-
port. Learning to dictate takes practice, ef- rooms, training sessions, therapeutic inter-
fort, and criticai review of early attempts. actions, and a host of other observational
Tape recorders must be used judiciously so targets can sometimes be less intrusive than
as not to become obtrusive and inhibit pro- a note-taking evaluator. We had great suc-
gram processes or participant responses. A cess taking videos of mothers and children
tape recorder is much more useful for re- playing together in early childhood educa-
cording field notes in priva te than it is as an tion centers. Of course, use of such equip-
instrument to be carried about at ali times, ment must be nego tiated with program staff
available to put a quick end to any conversa- and participants, but the creative and judi-
tion into which the observer enters. cious use of technology can greatly increase
Portable computers have emerged as a the quality of field observations and the util-
fieldwork tool that can facilitate writing ity of the observational record to others.
field notes. Cameras have become standard Moreover, comfort with tape recorders and
accessories in fieldwork. Photographs can video cameras has made it increasingly pos-
help m recalling things that have happened sible to use such technology without undue
as well as vividly capturing the setting for intrusion when observing programs where
others. Digital photography and advances professionals are the participants. In addi-
in printing and photo copying now make it tion, sometimes videotapes originally done
possible to economically reproduce photo- for research or evaluation can subsequently
graphs in research and evaluation reports. be used for future training, program devel-
In the wilderness education evaluation, I opment, and public relatioris, making the
officially became the group photographer, costs more manageable because of added
making photographs available to ali of the uses and benefits. Evaluators learn to bal-
participants. This helped legitimize taking ance costs against benefits and look for mul-
photographs and reduced the extent to tiple uses of more expensive techniques
which other people felt it necessary to carry where there is a need to make judicious deci-
their own cameras at ali times, particularly sions about reducing expenses.
at times when it was possible that the equip- Visual technology can add an important
ment might be damaged. Looking at photo- dimension to fieldwork if the observer
graphs during analysis helped me recall the knows how to use such technology and uses
details of certain activities that I had not it wellfor there is much to learn beyond
fully recorde d in my written notes. I relied how to click the camera or turn on the video
heavily on photographs to add details to de- recorder, especially about integrating and
scriptions of places where criticai events oc- analyzing visual data within a larger field-
curred in the Grand Canyon initiation story I work context (Bali and Smith 1992). More-
wrote about coming of age in modern soci- over, a downside to visual technology has
ety (Patton 1999a). emerged, since it is now possible to not only
Video photography is another technolog- capture images on film and video but also
ical innovation that has become readily ac- change and edit those images in ways that
cessible and common enough that it can distort. In his extensive review of "visual
sometimes be used unobtrusively. For exam- methods" in qualitative inquiry, Douglas
ple, in a formative evaluation of a staff train- Harper (2000) concludes that "now that im-
ing program I used videotapes to provide vi- ages can be created and/or changed digi-
tally, the connection between image and haps somewhat like a rolling chair. (Scott and
'truth' has been forever severed" (p. 721). Eklund 1979:9-11).
This means that issues of credibility apply to
using and reporting visual data as they do to The imagery of a fieldworker following a
other kinds of data. subject around through a day wearing a
Perhaps the ultimate in observer technol- Stenomask offers a stark contrast to that of
ogy for fieldwork is the Stenomask, a the traditional anthropologist doing partici-
sound-shielded microphone attached to a pant observation and trying covertly to
portable tape recorder that is worn on a write notes during informal field inter-
shoulder strap. The handle of the Stenomask views. Taking field notes canbe nearly as in-
contains the microphone switch. The trusive as wearing a Stenomask, as illus-
Stenomask allows the observer to talk into trated in the fieldwork of anthropologist
the recorder while an activity is occurring Carlos Castaneda. In the passage below,
without people in the area being able to hear Castaneda (1973) reports on his negotiations
the dictation. Its use is limited to externai, with Don Juan to become his Native Indian
onlooker observations, as the following pas- key informant on sorcery and indigenous
sage makes clear. drugs. The young anthropologist records
that Don Juan "looked at me piercingly."
Two procedures precede any data taking. The "What are you doing in your pocket?" he
first is orientation of the subject and as many asked, frowning. "Are you playing with your
other persons in the environment as are likely whanger?"
to be present during observations.... During He was referring to my taking notes on a
this phase, the observer goes into the habitat minute pad inside the enormous pockets of
and behaves exactly as he or she will during my wirvdbreaker.
the actual recording. They wear the Steno- When I told him what I was doing he
mask, follow the subject about and run the laughed heartily.
machine, taking mock records. The purpose of I said that I did not want to disturb him by
these activities is exactly what is implied in the writing in front of him.
title, to adapt the subject and others in the en- "If you want to write, write," he said. "You
vironment to the presence of the observer and don't disturb me." (pp. 21-22)
to reduce the effects of that presence to as near
zero as possible. The cardinal rule of the ob- Whether one uses modern technology to
server during this time is to be completely support fieldwork or simply writes down
nonresponding. It has been demonstrated what is occurring, some method of keeping
over and over again that if the observer contin- track of what is observed must be estab-
ues to resist ali social stimuli from the subject lished. In addition, the nature of the record-
and others (and some will occur despite the ing system must be worked out in accor-
most careful orientation) by simply keepmg dance with the participant observer's role,
the mask in place, looking busily at work and the purpose of the study, and consideration
remaining nonresponding, both subjects and of how the data-gathering process will affect
others soon cease emitting stimuli to the ob- the activities and persons being observed.
server and come to truly accept him or her as a Many of these issues and procedures must
present and sometimes mobile but completely be worked out during the initial phase (en-
nonresponding part of the environment, per- try period) of fieldwork.
13. Stages of Fieldwork cepted. If only he could figure out what to

do, if only he could understand the rules,
Thus far, fieldwork has been described as if then he would happily do what he was sup-
it were a single, integrated experience. Cer- posed to do. Such are the trials of entry into
tainly, when fieldwork goes well it flows the field.
with a certain continuity, but it is useful to Entry into the field for evaluation re-
look at the evolution of fieldwork through search involves two separate parts: (1) nego-'
identifiable stages. Three stages are most of- tia tion with gatekeepers, whoever they may
ten discussed in the participant observation be, about the nature of the fieldwork to be
literature: the entry stage, the routinization done and (2) actual physical entry into the
of data-gathering period, and the closing field setting to begin collecting data. These
stage. The following sections explore each two parts are closely related, for the negotia-
of these stages, again using evaluative re- tions with gatekeepers will establish the
search as the primary example. rules and conditions for how one goes about
playing the role of observer and how that
role is defined for the people being ob-
Entry Into the Field served. In traditional scholarly fieldwork for
the purpose of basic or applied research, the
The writings of anthropologists some- invs tigator unilaterally decides how best to
times present a picture of the early period of conduct the fieldwork. In evaluation stud-
fieldwork that reminds me of the character ies, the evaluator will need to take into ac-
in Franz Kafka's haunting novel The Castle. count the perspectives and interests of the
Kafka's character is a wandering stranger, primary intended users of the evaluation. In
K., with no more identity than that initial. He either case, interactions with those who con-
doesn't belong anywhere, but when he ar- trol entry into the field are primarily strate-
rives at the castle he wants to become part of gic, figuring out how to gain entry while pre-
that world. His efforts to make contact with serving the integrity of the study and the
the faceless authorities who run the castle invs tigator's interests. The degree of diffi-
lead to frustration and anxiety. He can't culty involved varies depending on the pur-
quite figure out what is going on, can't break pose of the fieldwork and the expected or
through their vagueness and impersonal na- real degree of resistance to the study. Where
ture. He doubts himself; then he gets angry the field researcher expects coopera tion,
at the way he is treated; then he feels guilty, gaining entry may be largely a matter of es-
blaming himself for his inability to break tablishing trust and rapport. At the other
through the ambiguous procedures for en- end of the continuum are those research set-
try. Yet, he remains determined to make tings where considerable resistance, even
sense out of the incomprehensible regula- hostility, is expected, in which case gaining
tions of the castle. He is convinced that, after entry becomes a matter of "infiltrating the
ali, where there are rulesand he does find setting" (Douglas 1976:167). And sometimes
that there are rulesthey must fit together entry is simply denied. A doctoral student
somehow, have some meaning, and mani- had negotiations for entry end abruptly in a
fest some underlying logic. There must be school district where she had developed
some way to make contact, to satisfy the good relationships with school personnel
needs of the authorities, to find some pattern and negotiations appeared to be going well.
of behavior that will permit him to be ac- She later learned that she was denied entry
far into the negotiation process because of propriate to consider some other term to de-
community opposition. The local commu- scribe the fieldwork. In our onlooker,
nity had had a very bad experience with a nonparticipatory observations for an imple-
university researcher more than 20 years mentation study of early childhood pro-
earlier and still viewed ali research with grams in Minnesota, we described our role
great suspicion. to local program participants and staff as fol-
A major difference between the entry pro- lows:
cess in anthropological or sociological re-
search and the entry process for evaluation
We're here to be the eyes and ears for state leg-
research is the extent to which fieldworkers
islators. They can't get around and visit ali the
are free to make up whatever story they
programs throughout the state, so they've
want to about the purpose of the study. In
asked us to come out and describe for them
scholarly research, the investigators repre-
what you're doing. That way they can better
sent only themselves and so they are rela-
understand the programs they have funded.
tively free to say whatever they want to say
We're not here to make any judgments about
about why they are doing the research
whether your particular programs is good or
guided by the ethics of their discipline with
bad, We are just here tobe the eyes and ears for
regard to informed consent. The usual
the legislature so that they can see how the leg-
cross-cultural explanation is some variation
islation they've passed has tumed into real
of 'Tm here because I would like to under-
programs. This is your chance to inform them
stand you better and learn about your way
and give them your point of view.
of life because the people from my culture
would like to know more about you." While
anthropologists admit that such an explana- Other settings lend themselves to other
tion almostnever makes sense to indigenous terms that are less threatening than evaluator.
peoples in other cultures, it remains a main- Sometimes a fieldwork project can be de-
stay initial explanation until mutual reci- scribed as documentation. Another term I've
procities can be established with enough lo- heard used by community-based evaluators
cal people for the observation process to is process historian. In the wilderness educa-
become established and accepted in its own tion program I was a full participant ob-
right. server, and staff described my role to partici-
Evaluators and action researchers, how- pants as "keeper of the community record/'
ever, are not just doing fieldwork out of per- making it clear that I was not there to evalu-
sonal or professional interest. They are do- ate individual participants. The staff of the
ing the fieldwork for some decision makers project explained that they had asked me to
and information users who may be either join the project because they wanted some-
known or unknown to the people being one who did not have direct ego involve-
studied. It becomes criticai, then, that evalu- ment in the success or outcomes of the pro-
ators, their funders, and evaluation users gram to observe and describe what went on,
give careful thought to how the fieldwork is both because they were too busy running the
going to be presented. program to keep detailed notes about what
Because the word evaluation has suchneg- occurred and because they were too in-
ative connotations for many people, having volved with what happened to be able to
had negative experiences being evaluated, look at things dispassionately. We had
for example, at school or work, it may be ap- agreed from the beginning that the commu-
nity record I produced would be accessible While the observer must learn how to be-
to participants as well as staff. have in the new setting, the people in that
In none of these cases did changing the setting are deciding how to behave toward
language automatically make the entry pro- the observer. Mutual trust, respect, and co-
cess smooth and easy. Earlier in this chapter, operation are dependent on the emergence
1 described our atteinpt to be viewed as "ed- of an exchange relationship, or reciprocity
ucational researchers" in evaluating a com- (Jorgensen 1989:71; Gallucci and Perugini
munity leadership program. Everyone fig- 2000), in which the observer obtains data
ured out almost immediately that we were and the people being observed find some-
really evaluators and thafs what partici- thing that makes their cooperation worth-
pants called us. Regardless of the story told while, whether that something is a feeling of
or the terms used, the entry period of field- importance from being observed, useful
work is likely to remain "the first and most feedback, pleasure from interactions with
uncomfortable stage of field work" (Wax the observer, or assistance in some task. This
1971:15). It is a time when the observer is get- reciprocity model of gaining entry assumes
ting used to the new setting, and the people that some reason can be found for partici-
in that setting are getting used to the ob- pants to cooperate in the research and that
server. Johnson (1975) suggests that there some kind of mutual exchange can occur.
are two reasons why the entry stage is both Infiltration lies at the opposite end of the
so important and so difficult: continuum from a negotiated, reciprocity
model of entry. Many field settings are not
First, the achievement of successful entree is a open to observation based on cooperation.
precondition for doing the research. Put sim- Douglas (1976:167-71) has described a num-
ply, no entree, no research [P]ublished re- ber of infiltration strategies, including
ports of researchers' entree experiences "worming one's way in," "using the crow-
describe seemmgly unlimited contingencies bar to pry them open for our observations,"
which may be encountered, ranging from be- showing enough "saintly submissiveness"
ing gleefully accepted to being thrown out on to make members guilty enough to provide
one's ear. But there is a more subtle reason help, or playing the role of a "spineless
why the matter of one's entrance to a research boob" who could never possibly hurt the
setting is seen as so important. This concems people being observed. He has also sug-
the relationship between the initial entree to gested using various ploys of misdirection
the setting and the validity of the data that is where the researcher diverts people's atten-
subsequently collected. The conditions under tion away from the real purpose of the study.
which an initial entree is negotiated may have There is also the "phased-entre tactic" by
important consequences for how the research which the researcher who is refused entree
is socially defined by the members of the set- to one group begins by studying another
ting. These social definitions will have a bear- group until it becomes possible to get into
ing on the extent to which the members trust a the group that is the real focus of the re-
social researcher, and the existence of relations searcher^ attention, for example, begin by
of trust between an observer and the members observing children in a school when what
of a setting is essential to the production of an you really want to observe are teachers or
objective report, one which retains the integ- administrators.
rity of the actor's perspective and its social Often the best approach for gaining
context. (pp 50-51) entre is the "known sponsor approach/'
When employing this tactic, observers use perience the same socialization process that
the legitimacy and credibility of another regular participants experience by becom-
person to establish their own legitimacy and ing part of the initiation process and timing
credibility, for example, the director of an or- their observations to coincide with the be-
ganization for an organizational study, a lo- ginning of a program. Such timing makes
cal leader, elected official, or village chief tain the evaluator one among a number of nov-
for a community study. Of course, it's im- ices and substantially reduces the disparity
portant to make sure that the known spon- between the evaluator's knowledge and the
sor is indeed a source of legitimacy and cred- knowledge of other participants.
ibility. Some prior assessment must be made Beginning the program with other partic-
of the extent to which that person can pro- ipants, however, does not assure the evalua-
vide halo feelings that will be positive and tor of equal status. Some participants may be
helpful. For example, in an evaluation, using suspicious that real difficulties experienced
a program administrator or funders as a by the evaluator as a novice participant are
known sponsor may increase suspicion and phonythat the evaluator is play-acting,
distrust among program participants and only pretending to have difficulty. On the
staff. first day of my participation in the wilder-
The initial period of fieldwork can be ness education program, we had our first
frustrating and give rise to self-doubt. The backpacking experience. The staff leader be-
fieldworker may lie awake at night worry- gan by explaining that "your backpack is
ing about some mistake, some faux pas, your friend." I managed to both pack and
made during the day. There may be times of adjust my "friend" incorrectly. As a result, as
embarrassment, feeling foolish, of question- soon as we hit the trail, I found that the belt
ing the whole purpose of the project, and around my waist holding the backpack on
even feelings of parania. The fact that one is my hips was so tight that my friend was
trained in social science does not mean that making my legs fali asleep. I had to stop sev-
one is immune to ali the normal pains of eral times to adjust the pack. Because of
learning in new situations. On the other these delays and other difficulties I was hav-
hand, the initial period of fieldwork can also ing with the weight and carriage of the pack,
be an exhilarating time, a period of rapid I ended up as the last participant along the
new learning, when the senses are height- trail. The next morning when the group was
ened by exposure to new stimuli, and a time deciding who should carry the map and
of testing one's social, intellectual, emo- walk at the front of the group to learn map
tional, and physical capabilities. The entry reading, one of the participants immediately
stage of fieldwork magnifies both the joys volunteered my name. "Let Patton do it.
and the pains of doing fieldwork. That way he can't hang back at the end of the
Evaluators can reduce the "stick-out- group to observe the rest of us." No amount
like-a-sore-thumb syndrome" by beginning of protest from me seemed to convince the
their observations and participation in a participants that I had ended up behind
program at the same time that participants them ali because I was having trouble hiking
are beginning the program. In traditional (working out my "friendship" with my
fieldwork, anthropologists cannot become backpack). They were convinced I had taken
children again and experience the same so- that position as a strategic place from which
cialization into the culture that children ex- to evaluate what was happening. It is well to
perience. Evaluators, however, can often ex- remember, then, that regardless of the na-
ture of the fieldwork, during the entry stage about his research, but by the style in which he
more than at any other time, the observer is lives and acts, by the way in which he treats
also the observed. them. In a somewhat shorter run, they will ac-
cept or tolerate him because some relative,
friend, or person they respect has recom-
What You Say and What You Do
mended him to them. (Wax 1971:365)
Fieldworkers' actions speak louder than
their words. Researchers necessarily plan William Foote Whyte (1984:37-63) has ex-
strategies to present themselves and their tracted and summarized entry strategies
function, but participant reactions to state- used in a number of groundbreaking socio-
ments about the researcher's role are quickly logical studies, including the Lynds' study
superseded by judgments based on how the of Middletown, W. Lloyd Warner's study of
person actually behaves. Yankee City, Burleigh Gardner's fieldwork
The relative impor tance of words versus in the deep South, Elliot Liebow's hanging
deeds in establishing credibility is partly a around Tally's Corner, Elijah Anderson's
function of the length of time the observer fieldwork in a Black neighborhood, Ruth
expects to be in a setting. For some direct on- Horowitz's study of a Chicano neighbor-
looker observations, the fieldworker may be hood, Robert Cole's work in Japan, and
present in a particular program for only a Whyte's own experiences in Cornerville.
few hours or a day. The entry problem in They each had to adapt their entry strategy
such cases is quite different from the situa- to the local setting and they ali ended up
tion where the observer expects to be partici- changing what they had planned to do as
pating in the program over some longer pe- they learned from the initial responses to
riod of time, as anthropologist Rosalie Wax their efforts to gain acceptance. These exam-
has noted: ples from those who paved for way for mod-
em fieldworkers demonstrate the impor-
Ali field workers are concerned about explain- tance of careful attention to entry and the
ing their presence and their work to a host of variety of approaches that are possible. The
people. "How shall I introduce myself?" they next section presents a concrete example
wonder, or, "what shall I say I am doing?" If from an evaluation by Joyce Keller.
the field worker plans to do a very rapid and
efficient survey, questions like these are ex- AN ENTRY CASE EXAMPLE:
tremely important. The manner in which an THE PART-TIME OBSERVER
interviewer introduces himself, the precise
words he uses, may mean the difference be- Introduction. The prevous section contrasted the
tween a first-rate job and a failure But if the entry challenges for the one-shot onlooker observer
field worker expects to engage in some variety with those of the long-term participant observer;
of participant observation, to develop and but a great deal of middle ground exists between
maintain long-term rela tionships, to do a these extremes. In this section, loyce Keller, a snior
study that involves the enlargement of his staff member of the Minnesota Center for Social
own understanding, the best thing he can do is Research at the time, describes her entry intofield-
relax and remember that most sensible people ivork as a part-time observer.4 Because limitations
do not believe what a stranger tells them. In of time and resources are common in evaluation,
the long run, his host will judge and trust him, many situations call for a part-time observer.
not because of what he says about himself or Ioyce's reflections capture some ofthe special entry
problems associated zuith this "now yoii're here, be notified in the event of schedule changes.
now you're gone" role. I would have firmly in mind that a subgroup
was to meet on Tuesday at 10:00 a.m. in a cer-
One word can describe my role, at least tain place. I would arrive to find no one
initially, in a recent evaluation assignment: there. Later, I would discover that on Mon-
ambiguous. I was to be neither a participant day the meeting had been changed to
observer nor an outsider coming in for a Wednesday afternoon and no one had been
brief but intensive stint. I was to allocate ap- delegated to tell me. At no time did I seri-
proximately six hours a week for seven ously feel that the changes were planned to
months to observing the team development exclude me; on the contrary, the members'
of a group of 23 professionals in an educa- contrition about their oversight seemed
tional setting. At first, the ambiguity was quite genuine. They had simply forgotten
solely on my side: What, really, was I to do? me.
The team, too busy in the beginning with de- Another area of sudden change that
fining their own roles, had little time to con- caused me difficulty was in policy and pro-
sider mine. Later on, as I became accus- cedure. What had seemed to be firm com-
tomed to my task, the team's curiosity about mitments on ways to proceed or tasks to be
my function began to grow. tackled were being ignored. I came to realize
In their eyes, I served no useful purpose that while a certain amount of this instability
that they could see. I was in the way a great was inherent in the program itself, other
deal of the time inhibiting their private con- shifts in direction were outgrowths of plan-
versations. On the other hand, they ap- ning sessions I had not attended or had not
peared to be concerned about what I was heard the results from after they had oc-
thinking. Some of themmost of thembe- curred. Therefore, keepmg current became
gan to be friendly, to greet me as I came in, to for me a high-priority activity. Not to do so
comment when I missed a team meeting. would have added to my feeling of ambigu-
They came to see me as I saw myself: neither ity. Also, if I had not operated with a certain
really part of the group nor a separate, re- degree of self-confidence, I would have felt
moved force. somehow at fault for coming to a meeting at
Observing their interaction perhaps six the wrong time or place or assuming that a
hours a week out of their 40-hour work week certain decision, which the team had previ-
obviously meant that I missed a great deal. I ously made, was still valid.
needed to develop a sense of when to be I began my observation of this team in its
present, to choose among group meetings, forma tive stage. Had I begun after the team
subgroup meetings, and activities when ali was well established, my difficulties would
the members were to come together. At the have been greater. Nevertheless, many of
same time, I was working on other contracts the team members were already well ac-
which Iimited the amount of adjustable time quainted with each other; ali had been em-
available. "Flexible" was the way I came to ployees of the same school district over a pe-
define my weekly schedule; others, not as riod of time. They were much better versed
charitable, would probably have defined it in what they had come together to accom-
as "shifty." plish than I, whose only orientation was
A hazard that I encountered as I filled my reading the proposal which, upon accep-
ambiguous, flexible role was that I soon distance, had brought them together. I found
covered I was not high on the priority list to also that the proposal and the way they
plaimed to proceed were, in actuality, far they were interacting about. I noted (and ig-
from identical. nored) a few passing suggestions that
With my observer role to continue over since I was obviously taking notes maybe I
many months, I realized that I must main- could
tain the difficult position of being impartial. I took copious notes before I began to de-
I could not be thought of by the team mem- velop a sense of what was or was not impor-
bers as being closely aligned with their lead- tant to record. When I relaxed more and
ers, nor could I expect the leaders to talk can- aimed for the tone of the meeting my under-
didly and openly with me if they believed standing of the group increased. I had to re-
that I would repeat their confidences to the alize that, as a part-time observer, it was im-
group members. Reluctantly, for I discov- possible for me to understand ali of what
ered several team members with whom was said. My decision frequently was to let
friendships could easily have developed, I this portion of the meeting pass or to jot
declined invitations to social activities out- down a reminder to myself to ask clarifying
side of working hours. questions later.
When I inet with the group for the first Side-stepping sensitive questions from
time, I directed most of my energies to both leaders and team members had to be
matching names and faces. I would be tak- developed into a fine art. As I became more
ing notes at most of the sessions and it was finely tuned to the interactions, and most be-
essential that I could record not only what came aware that I was, I was frequently que-
was said but who said it. At the first session ried as to my perceptions of a particular indi-
everyone, including me, wore a name tag. vidual or situation. On one occasion, I found
But within a few days, they were ali well ac- a team member jumping into an elevator to
quainted and had discarded their name tags; ride two floors with me in a direction he
I was the only one still fumbling for names. didn't want to go so that he could ask me pri-
While being able to greet each member by vately what I thought of another team mem-
name was important, so was knowing some- ber. My response was, "I think she's a very
thing about each one's background. Coffee interesting person," or something equally
breaks allowed me to circulate among the innocuous, and received from him a highly
group and carry on short conversations with raised eyebrow, since the woman in ques-
as many as possible to try to fix in my mind tion had just behaved in a very peculiar
who they were and where they came from, manner at the meeting we had both just at-
which provided insights into why they be- tended.
haved in the group as they did. In-depth interviews with each team
Team members at first expressed a certain member began in the fourth month of my
amount of enthusiasm for minutes to be observation and was the mechanism which
taken of their meetings. This enthusiasm filled in many of the gaps in my understand-
was short-lived, for willing volunteers to ing. The timing was perfect: I had gained
serve as secretary did not emerge. I was dis- enough familiarity with both personnel and
appointed, for, had minutes been kept of the project by that time so that I was knowledge-
meetings and had I been able to rely on re- able, they had come to trust me, and they
ceiving copies, I would have concentrated still cared deeply about the project. (This
solely on observing the interactions and caring diminished for some as the project
would not have had to keep track of what year drew to a close without any real hopes
of refunding for a second year.) My inter- tions and conditions reinforced for me what
view design was intentionally simple and were sometimes at best only vague percep-
open-ended. What I wanted most was for tions. Team members who appeared to be
them to talk about their experiences in terms passive and quiet when I saw them at group
of strengths and weaknesses. meetings were often referred to by their
The amount of new information dimin- team members as hard-working and cre-
ished throughout the six weeks or so that ative when they were out in the field. The in-
was required to interview ali team members. terviews also helped me become aware of
My own performance unquestionably di- misconceptions on my part caused by seeing
minished too as the weeks went on. It was only part of the picture, due to time con-
difficult to be animated and interesting as I straints.
asked the same questions over and over, de- The experience was a new one for me, that
vised strategies with which to probe, and re- of part-time observer. Quite frankly, this
corded perceptions and incidents which I mode of evaluation probably will never be a
had heard many times before. favorite one. On the other hand, it provided
Nevertheless, the interviews appear in a picture that no "snap-shot" evaluation
retrospect to have been a necessary tool of method could have accomplished as interac-
the part-time observer. Bit by bit team mem- tions changed over time and in a situation
bers filled in holes in my information and where the full participant observer role was
their repeated references to particular situa- clearly not appropriate.
Routinization of Fieldwork:
The Dynamics of the Second Stage
hat did you learn in your readings today?" asked Master Halcolm.
"We learned that a journey of a thousand miles begins with the first
step," replied the learners.
"Ah, yes, the importance of beginnings," smiled Halcolm.
"Yet, I am puzzled," said a learner. "Yesterday I read that there are a
thousand beginnings for every ending."
"Ah, yes, the importance of seeing a thing through to the end," affirmed
Halcolm.
"But which is more important, to begin or end?"
"Two great self-deceptions are asserted by the world's self-congratulators:
that the hardest and most important step is the first and that the greatest and
most resplendent step is the last.
"While every journey must have a first and last step, my experience is that
what ultimately determines the nature and enduring value of the journey are
the steps in between. Each step has its own value and importance. Be present
for the whole journey, learners that you are. Be present for the whole journey."
Halcolm
During the second stage, after the awakening identification involves some re-
fieldworker has established a role and pur- alization of how much I have in common
pose, the focus moves to high-quality data with these people whose world I have been
gathering and opportunistic investigation permitted to enter. At times during field-
following emergent possibilities and build- work I feel a great separation from the peo-
ing on what is observed and learned each ple I'm observing, then at other times I feel a
step along the way. The observer, no longer strong sense of our common humanity. For a
caught up in adjustments to the newness of fieldworker to identify, however briefly,
the field setting, begins to really see what is with the people in a setting or for an evalua-
going on instead of just looking around. As tor to identify with the clients in a program
Florence Nightingale said, "Merely looking can be a startling experience because social
at the sick is not observing." science observers are often quite separated
Describing the second stage as "rou- from those they study by education, experi-
tinization of fieldwork" probably overstates ence, confidence, and income. Such differ-
the case. In emergent designs and ever- ences sometimes make the world of pro-
deepening inquiry, the human tendency to- grams as exotic to evaluators as nonliterate
ward routines yields to the ups and downs cultures are exotic to anthropologists.
of new discoveries, fresh insights, sudden There come times, then, when a field-
doubts, and ever-present questoning of worker must deal with his or her own feel-
othersand often of self. Discipline is ings about and perspectives on the people
needed to maintain high-quality, up-to-date being observed. Part of the sorting-out pro-
field notes. Openness and perseverance are cess of fieldwork is establishing an under-
needed to keep exploring, looking deeper, standing of the relationship between the
diverging broader, and focusing narrower, observed and the observer. When that hap-
always going where the inquiry and data pens, and as it happens, the person involved
take you. Fieldwork is intellectually chal- in fieldwork may be no less startled than Jo-
lenging at times, mind-numbingly dull at seph Conrad's infamous character Marlowe
times, and for many, an emotional roller in Heart o/Darkness. Marlowe had followed
coaster. Appendix 9.1 at the end of Chap- Kurtz, the European ivory trader, up the
ter 9, "A Documenter's Perspective," offers deep river into the Congo where Kurtz had
the reflections of a participant observer con- established himself as a mangod to the tribal
ducting a school evaluation and grappling people there. He used his position to acquire
with changes in fieldwork over time. ivory, but to maintain his position he had to
One of the things that can happen in the perform the indigenous rituais of human
course of fieldwork is the emergence of a sacrifice and cannibalism. Marlowe, deeply
strong feeling of connection with the people enmeshed in the racism of his culture and
being studied. As you come to understand time, was initially horrified by the darkness
the behaviors, ideais, anxieties, and feelings of the jungle and its peoples, but as he
of other people, you may find yourself iden- watched the rituais of those seeming sav-
tifying with their lives, their hopes, and their ages, he found an emergent identification
pain. This sense of identification and con- with them and even entertained the suspi-
nection can be a natural and logical conse- cion that they were not inhuman. He became
quence of having established relationships aware of a linkage between himself and
of rapport, trust, and mutuality. For me, that them:
They howled and leaped and spun, and madee self in the middle of deep generational divi-
horrid faces; but what thrilled you was thee sions between mothers and their daughters,
thought of their humanitylike oursthee teachers and students, bookstore owners
thought of your remote kinship with this wild1 and their clients. She could not risk deeply
and passionate uproar. Ugly. Yes, it was uglyy alienating or completely acquiescing to any
enough; but if you were man enough youx of these important and competing groups,
would admit to yourself that there was in youx for they ali affected her access and the ulti-
just the faintest trace of a response to the terri- mate success of her fieldwork.
ble frankness of that nose, a dim suspicion off In evaluations, the evaluator can be
there being a meaning in it which youyou so3 caught in the middle of tensions between
remote from the night of the first agescom- competing groups and conflicting perspec-
prehend. And why not? (Conrad 1960:70) tives. For example, where divisions exist
among the staff and/or the participants in a
In this passage, Conrad chronicles the2 program, and such divisions are common,
possibility of awakening to unexpected real- the evaluator will be invited, often subtly, to
izations and intense emotions in the course align with one subgroup or the other. In-
of encounters with the unknown and those2 deed, the evaluator may want to become
who are different from us. In many ways, itt part of a particular subgroup to gain further
is our common humanity, whether we are^ insight into and understanding of that sub-
fully aware of it at any given moment or not, group. How such an alliance occurs, and
that makes fieldwork possible. As humani how it is interpreted by others, can greatly
beings, we have the amazing capability to) affect the course of the evaluation.
become part of other people's experiences,,, My experience suggests that it is imprac-
and through watching and reflecting, weB tical to expect to have the same kind of re-
can come to understand soinething aboutt lationshipclose or distantwith every
those experiences. group or faction. Fieldworkers, human be-
As fieldwork progresses, the intrica te^ ings with their own personalities and inter-
web of human relationships can entangle5 ests, will be naturally attracted to some peo-
the participant observer in ways that will1 pie more than others. Indeed, to resist those
create tension between the desire to become2 attrac tions may hinder the observer from
more enmeshed in the setting so as to learni acting naturally and becoming more thor-
more and the need to preserve some dis- oughly integrated into the setting or pro-
tance and perspective. Participant observers3 gram. Recognizing this, the observer will be
carry no immunity to the political dynamicss faced with ongoing decisions about per-
of the settings being observed. Virtually anyj sonal relationships, group involvement, and
setting is likely to include subgroups of peo- how to manage differential associations
ple who may be in conflict with other sub- without losing perspective on what the ex-
groups. These factions or cliques may eitherr perience is like for those with whom the
woo or reject the participant observer, butt fieldworker is less directly involved.
they are seldom neutral. During her field- Perhaps the most basic division that will
work interviewing young women in ndia, always be experienced in program evalua-
Parameswaran (2001) reports efforts by par- tion is the separation of staff and partici-
ents and teachers to get her to inform on the pants. While the rhetoric of many programs
women she interviewed or to influence? attempts to reduce the distinction between
them in a desired direction. She found her- staff and participants, there is almost always
a distinction between those who are paid for servers among people on both sides of those
their responsibilities in the program (staff) fences. (pp. 96-97)
and those who are primarily recipients of
what the program has to offer (participants). In contrast to Lofland's advice, in evalu-
Sociologically, it makes sense that staff and ating the wilderness education program I
participants would be differentiated, creat- found myself moving back and forth be-
ing a distance that can evolve into conflict or tween a full participant role, where I was
distrust. Participants will often view the identified primarily as a participant, and a
evaluator as no different from the staff or ad- full staff role, where I was identified primar-
ministration, or even the funding sources ily with those who carried responsibility for
virtually any group except the partici- directing the program. During the first field
pants. If the evaluator observer is attempt- conference, I took on the role of full partici-
ing to experience the program as a participant and made as visible as possible my alle-
pant, special effort will be required make giance to fellow participants while main-
participation real and meaningful and to taining distance from the staff. Over time,
become accepted, even trusted, by other however, as my personal relationships with
participants. On the other hand, staff and the staff increased, I became more and more
administrators may be suspicious of the aligned with the staff. This coincided with a
evaluator's relationships with funders or change of emphasis in the evaluation itself,
board members. with the earlier part of the fieldwork being
The point is not to be naive about the tan- directed at describing the participant experi-
gled web of relationships the participant ob- ence and the latter part of the fieldwork be-
server will experience and to be thoughtful ing aimed at describing the workings of the
about how fieldwork, data quality, and the staff and providing formative feedback.
overall inquiry are affected by these connec- However, I was always aware of a ten-
tions and interrelationships, ali of which sion, both within myself and within the
have to be negotiated. group at large, about the extent to which I
Lofland (1971) has suggested that partici- was a participant or a staff member. I found
pant observers can reduce suspicion and that as my observational skills became in-
fear about a study by becoming openly creasingly valued by the program staff I had
aligned with a single broad grouping within to more consciously and actively resist their
a setting while remairiing aloof from that desire to have me take on a more active and
grouping's own internai disputes. explicit staff role. They also made occasional
attempts to use me as an informer, trying to
seduce me into conversations about particu-
Thus, known observers of medicai schools lar participants. The ambiguities of my role
have aligned themselves only with the medi- were never fully resolved. I suspect that
cai students, rather than attempting to parti ci- such ambiguities were inherent in the situa-
pate extensively with both faculty and tion and are to be expected in many evalua-
students. In mental hospitais, known observ- tion fieldwork experiences.
ers have confined themselves largely to men- Managing field relationships involves a
tal patients and restricted their participation different set of dynamics when the inquiry is
with staff. To attempt to participate with both, collaborative or participatory. Under such
extensively and simultaneously, would prob- designs, where the researcher involves oth-
ably have generated suspicion about the ob- ers in the setting in fieldwork, a great deal of
the work consists of facilitating the interac- tagonisms among those who may resent or
tions with co-inquirers, supporting their distrust the special relationships between
data collection efforts, ongoing training in the fieldworker and the key informant. In-
observation and interviewing, managing indeed, howand how muchto make visi-
tegration of field notes ainong different par- ble this relationship involves strategic think-
ticipant researchers, and inonitoring data ing about how others will react and how
quality and consistency. These coliaborative their reactions will affect the inquiry. There's
management responsibilities will reduce the no formal announcement that the "position"
primary researcher's own time for field- of key informant is open, or that it's been
work and will affect how others in the set- filled; the key informant is simply that per-
tings, those who aren't participatory or col- son or those persons with whom the re-
iaborative researchers, view the inquiry and searcher or evaluator is likely to spend con-
the fieldwork director, if that is the role taken siderable time.
on. In some cases, management of the coliab- Key informants mustbe trained or devel-
orative inquiry effort is done by one of the oped in their role, not in a formal sense, but
participants and the trained fieldworker because they will be more valuable if they
serves primarily as a skills and process understand the purpose and focus of the in-
trainer and consultant to the group. Clarity quiry, the issues and questions under inves-
about these roles and divisions of labor can tigation, and the kinds of information that
make or break coliaborative, participatory are needed and most valuable. Anthropolo-
forms of inquiry. Having shared values gists Peito and Peito (1978) made this point
about collaboration does not guarantee ac- in reflecting on their own fieldwork:
tually pulling it off. Coliaborative inquiry is
challenging work, often frustrating, but We noticed that humans differ in their will-
when it works, the findings will carry the ad- ingness as well as their capabilities for ver-
ditional credibility of coliaborative triangu- bally expressing cultural information. Conse-
lation, and the results tend to be rewarding quently, the anthropologist usually finds that
for ali involved, with enduring insights and only a small number of individuais in any
new inquiry skills for those involved. community are good key informants. Some of
the capabilities of key informants are system-
Key Informants atically developed by the field workers, as
they train the informants to conceptualize cul-
One of the mainstays of much fieldwork tural data in the frame of reference employed
is the use of key informants as sources of in- by anthropologists. . . . The key informant
formation about what the observer has not gradually learns the rules of behavior in a
or cannot experience, as well as sources of role vis--vis the interviewer-anthropologist.
explanation for events the observer has actu- (p- 72)
ally witnessed. Key informants are people
who are particularly knowledgeable about The danger in cultivating and using key
the inquiry setting and articulate about their informants is that the researcher comes to
knowledgepeople whose insights can rely on them too much and loses sight of the
prove particularly useful in helping an ob- fact that their perspectives are necessarily
server understand what is happening and limited, selective, and biased. Data from in-
why. Selecting key informants mustbe done formants represent perceptions, not truths.
carefully to avoid arousing hostility or an- Information obtained from key informants
should be clearly specified as such in the the turtles, who willingly kept me informed
field notes so that the researcher's observa- about the details of what went on in that
tions and those of the informants do not be- group. Without that key informant relation-
come confounded. This may seem like an ship, I would have missed some very impor-
obvious point, and it is, but over weeks and tant information about the kinds of experi-
months of fieldwork it can become difficult ences the turtle participants were having
to decipher what information came from and the significance of the project to them.
what sources unless the fieldworker has a While being part of any setting necessar-
routine system for documenting sources ily involves personal choices about social re-
and uses that system with great discipline, lationships and political choices about group
thoroughness, and care. alliances, the emphasis on making strategic
Key informants can be particularly help- decisions in the field should not be inter-
ful in learning about subgroups to which the preted as suggesting that the conduct of
observer does not or cannot have direct ac- qualitative research in naturalistic settings is
cess. During the second year of the wilder- an ever-exciting game of chess in which
ness education program, one informal players and pieces are manipula ted to ac-
group, mostly women, dubbed themselves complish some ultimate goal. Fieldwork cer-
the "turtles" to set themselves apart from tainly involves times of both exhilaration
participants, mostly men, who had more ex- and frustration, but the dominant motifs in
perience in the wilderness and wanted to fieldwork are hard work, long hours to both
hike at a fast pace, climb the highest peaks, do observations and keep up-to-date with
or otherwise demons trate their prowessa field notes, enormous discipline, attention
group they called somewhat disparagingly to details, and concentration on the mun-
the "truckers" (trucks being unwelcome in dane and day-to-day. The routinization of
the wilderness). Having had a full year of fieldwork is a time of concentrated effort
wilderness experiences the first year of the and immersion in gathering data. Alas, let
program, I didn't qualify to become an inti- the truth be told: The gathering of field
mate part of the turtles. I therefore estab- data involves very little glory and an abun-
lished an informant relationship with one of dance of nose-to-the-grindstone drudgery.
Bringing Fieldwork to a Close
ell, I've gotten to the end of the subjectof the pageof your
patience and my time.
Alice B. Toklas in a letter to Elizabeth Hansen, 1949
In traditional scholarly fieldwork within cific repor ting deadlines, stated in a con-
anthropology and sociology, it can be diffi- tract, that affect the length of and resources
cult to predict how long fieldwork will last. available for fieldwork, and the intended
The major determinant of the length of the uses of evaluative findings.
fieldwork is the investigator's own re- In the previous section, we looked at the
sources, interests, and needs. Evaluation many complex relationships that get formed
and action research typically have quite spe- during fieldwork, relationships with key in-
formarits, hosts, and sponsors in the setting was observed show up more in the field
who helped with entre and may have sup- notes. Some of these explanations have been
ported ongoing fieldwork, helping solve offered by others; some occur directly to the
problems and smooth over diffculties. In observer. In short, analysis and interpreta-
collaborative research, relationships with tion will have begun even before the ob-
coresearchers will have deepened. In any ex- server has left the field.
tended involvement within a setting, friend- Chapter 9 discusses analysis strategies at
ships and alliances are formed. As fieldwork length. At this point, I simply want to recog-
comes to an end, an exit or disengagement nize the fact that data gathering and analysis
strategy is needed. While a great deal of at- flow together in fieldwork, for there is usu-
tention has traditionally been paid to enter- ally no definite, fully anticipated point at
ing the field, much less attention has been which data collection stops and analysis be-
given to the disengagement process, what gins. One process flows into the other. As the
Snow (1980) has called the "neglected prob- observer gains confidence in the quality and
lem in participant observation research." meaningfulness of the data, sophisticated
One side of the coin is disengagement. about the setting under study, and aware
The other side is reentry back to one's life af- that the end draws near, additional data col-
ter extended fieldwork or an all-consuming lection becomes increasingly selective and
project. When I went to do graduate research strategic.
in Tarizania, our team received a lot of sup- As fieldwork draws to a close, the re-
port and preparation for entry, much of it searcher is increasingly concerned with veri-
aimed at avoiding culture shock. But when fication of already-collected data and less
we returned home, we were given no prepa- concerned with generating new inquiry
ration for what it would be like to return leads. While in naturalistic inquiry one
to America's highly commercial, materialis- avoids imposing preconceived analytical
tic, and fast-moving culture after months in categories on the data, as fieldwork comes to
an agrarian, community-oriented, slower- an end, experience with the setting will usu-
moving environment. The culture shock hit ally have led to thinking about prominent
coming home, not going to frica. themes and dimensions that organize what
Interpersonal, cross-cultural, disengage- has been experienced and observed. These
ment, and reentry issues ali deserve atten- emergent ideas, themes, concepts, and di-
tion as fieldwork comes to a close. Relation- mensionsgenerated inductively through
ships with people change and evolve from fieldworkcan also now be deepened, fur-
entry, through the middle days, and into the ther examined, and verified during the clo-
end of fieldwork. So does the fieldworker's sure period in the field.
relationship with the data and engagement Guba (1978) has described fieldwork as
in the inquiry process. That changed en- moving back and forth between the discov-
gagement in the inquiry process is what I ery mode and the verification mode like a
want to focus on here. wave. The ebb and flow of research involves
As you near completion of data gather- moving in and out of periods when the in-
ing, having become fairly knowledgeable vs tigator is open to new inputs, generative
about the setting being observed, more and data, and opportunistic sampling to periods
more attention can be shifted to fine-tuning when the investigator is testing out hunches,
and confirming observed patterns. Possible fine-tuning conceptualization, sifting ideas,
interpretations of and explanations for what and verifying explanations.
When fieldwork has gone well the ob- ator may have to bring the fieldwork to a
server grows increasingly confident that close before that state of real confidence has
things make sense and begins to believe in fully emerged. Nevertheless, I find that
the data. Glaser and Strauss (1967), com- there is a kind of Parkinson's law in field-
menting on grounded theory as an outcome work: As time runs out, the investigator
of fieldwork, have described the feelings feels more and more the pressure of making
that the traditional field observer has as sense out of things, and some form of order
fieldwork moves to a close, data-based pat- does indeed begin to emerge from the obser-
tems have emerged, and the whole takes vations. This is a time to celebrate emergent
shape: understandings even while retaining the
criticai eye of the skeptic, especially useful
The continuai intermeshing of data collection in questioning one's own confident conclu-
and analysis has directbearing on how the re- sions.
search is brought to a close. When the re-
searcher is convrnced that his conceptual
Evaluation Feedback
framework forms a systematic theory, that it is
a reasonably accurate statement of the matter In doing fieldwork for program evalua-
studied, that it is couched in a form possible tion, in contrast to theory-oriented scholarly
for others to use in studying a similar area, and field research, the evaluator observer must
that he can publish his results with confi- be concerned about providing feedback,
dence, then he has neared the end of his re- making judgments, and generating recom-
search mendations. Thus, as the fieldwork draws to
Why does the researcher trust what he a close, the evaluator observer must begin to
knows? . . . They are his perceptions, his per- consider what feedback is to be given to
sonal experiences, and his own hard-won whom and how.
analyses. A field worker knows that he knows, Giving feedback can be part of the verifi-
not orily because he has been in the field and cation process in fieldwork. My own prefer-
because he has carefully discovered and gen- ence is to provide the participants and staff
erated hypotheses, but also because "in his with descriptions and analysis, verbally and
bons" he feels the worth of his final analysis. informally, and to include their reactions as
He has been living with partial analyses for part of the data. Part of the reciprocity of
many months, testing them each step of the fieldwork can be an agreement to provide
way, until he has built this theory. What is participants with descriptive information
more, if he has participated in the social life of about what has been observed. I find that
is subject, then he has been living by his anal - participants and staff are hungry for such in-
yses, testing them not only by observation and formation and fascinated by it. I also find
interview but also by daily living, (pp. 224-25) that I learn a great deal from their reactions
to my descriptions and analyses. Of course,
This representation of bringing a ifs neither possible nor wise to report every-
grounded theory inquiry to a close repre- thing one has observed. Moreover, the infor-
sents the scholarly inquiry ideal. In the mal feedback that occurs at or near the end
"contracted deliverables" world of program of fieldwork will be different from the find-
evaluation, with limited time and resources, ings that are reported formally based on the
and reporting schedules that may not permit more systematic and rigorous analysis that
as much fieldwork as is desirable, the evalu- must go on once the evaluator leaves the
field. But that formal, systematic analysis lier during each field conference. During the
will take more time, so while one is still in second field conference in the second year,
the field it is possible to share at least some when a number of factors had combined to
findings and to learn from the reactions of make the program quite different from what
those who hear those findings. the staff had hoped for, the end-of-the-con-
Timing feedback in formative evalua- ference evaluation feedback session gener-
tions can be challenging. When the purpose ated an unusual amount of frustration from
is to offer recommendations to improve the the staff because my analyses of what had
program, the program staff will usually be happened had not been shared earlier.
anxious to get that information "ASAP" (as Again, I found some distrust of my insis-
soon as possible). The evaluator observer tence that those interpretations had
may even feel pressured to report findings emerged later rather than sooner as the pat-
prematurely, before having confidence in terns became clear to me.
the patterns that seem to be einerging. I ex- Evaluators who provide formative feed-
perienced this problem throughout the eval- back on an ongoing basis need to be consci-
uation of the wilderness education program. entious in resisting pressures to share find-
During the first year, we met with the staff at ings and interpretations before they have
the end of each field conference program confidence about what they have observed
(the three 10-day field conferences were and sorted out important patternsnot cer-
spread out over a year) to discuss what we tainty, but at least some degree of confi-
had observed and to share interpretations dence. The evaluator is caught in a dilemma:
about those observations. At the very first Reporting patterns before they are clearly
feedback session, the staff reaction was, "I established may lead program staff to inter-
wish you'd told us that in the middle of the vene inappropriately; withholding feedback
week, when we could have done something too long may mean that dysfunctional pat-
about it. Why'd you hold back? We could terns become so entrenched that they are dif-
have used what you've learned to change ficult, if not impossible, to change.
the program right then and there." No ideal balance has ever emerged for me
I tried to expiam that the implications of between continuing observations and pro-
what I observed had only become clear to viding feedback. Timing feedback is a mat-
me an hour or two before our meeting when ter of judgment and strategy, and it depends
my coevaluator and I had sat down with our on the nature of the evaluator's relationship
field notes, looked them over, and discussed with program staff and the nature of the
their significance together. Despite this ex- feedback, especially the balance between
planation, which struck me as altogether what staff will perceive as negative and pos-
reasonable and persuasive and struck the itive feedback. When in doubt, and where
staff as altogether disingenuous, from that the relationship between the evaluator and
moment forth a lingering distrust hung over program staff has not stabilized into one of
the evaluation as staff periodically joked long-term trust, I counsel evaluator observ-
about when we'd get around to telling them ers to err on the side of less feedback rather
what we'd learned next time. Throughout than more. As often happens in social rela-
the three years of the project, the issue of tim- tionships, negative feedback that was wrong
ing feedback surfaced several times a year. is long remembered and often recounted.
As they came increasingly to value our feed- On the other hand, it may be a measure of
back, they wanted it to come earlier and ear- the success of the feedback that program
staff so fully adopt it that they make it their surement. The physical world canbe altered
own and cease to credit the insights of the by the intrusion of the observer. How much
evaluator. more, then, are social worlds changed by the
Once feedback is given, the role of the intrusion of fieldworkers?
evaluator changes. Those to whom the feed- The effects of observation vary depend-
back was presented are likely to become ing on the nature of the observation, the type
much more conscious of how their behavior of setting being studied, the personality and
and language are being observed. Thus, procedures of the observer, and a host of un-
added to the usual effect of the fieldworker anticipated conditions. Nor is it simply in
on the setting being observed, this feedback fieldwork involving naturalistic inquiry that
dimension of fieldwork increases the impact scientific observers affect what is observed.
of the evaluator observer on the setting in Experimentalists, survey researchers, cost-
which he or she is involved. benefit analysts, and psychologists who ad-
Though this problem of reactivity is ac- minister standardized tests ali affect the sit-
centuated in evaluation, it exists in any ob- uations into which they introduce data col-
servational inquiry. As the researcher pre- lection procedures. The issue is not whether
pares to leave the field, and people react to or not such effects occur; rather, the issue is
that imminent departure, the impact of the how to monitor those effects and take them
researcher's presence on the setting may be- into consideration when interpreting data.
come visible in new ways. Because those ef- A strength of naturalistic inquiry is that
fects have been of such major concern to the observer is suffciently a part of the situa-
people who engage in naturalistic inquiry, tion to be able to understand personally
the final section in this chapter considers this what is happening. Fieldworkers are called
question of how the observer affects what is on to inquire into and be reflective about
observed. how their inquiry intrudes and how those
mtrusions affect findings. But that's not al-
ways easy. Consider the case of anthropolo-
gist Napoleon Chagnon, who did fieldwork
!=f. The Observer and for a quarter century among the isolated and
What Is Observed: primitive Yanomami Indians who lived
Unity and Separation deep in the rain forest at the borders of
Venezuela and Brazil. He studied mortality
The question of how the observer affects rates by dispensing steel goods, including
what is observed has natural as well as so- axes, as a way of persuading people to give
cial science dimensions. The Heisenberg him the names of their dead relatives in vio-
uncertainty principie states that the instru- lation of tribal taboos. Brian Ferguson, an-
ments used to measure velocity and posi- other anthropologist knowledgeable about
tion of an electron alter the accuracy of mea- the Yanomami, believes that Chagnon's
surement, When the scientist measures the fieldwork destabilized relationships among
position of an electron, its velocity is villages, promoted warfare, and introduced
changed, and when velocity is measured, it disease. Chagon denies these charges but
becomes difficult to capture precisely the acknowledges extracting tribal secrets by
electron's position. The process of observ- giving informants gifts like beads and
ing affects what is observed. These are real fishhooks, capitalizing on animosities be-
effects, not just errors of perception or mea- tween individuais, and bribing children for
information when their elders were not server's degree of participation in the
around. He gave away machetes in ex- setting, the visibility and openness of field-
change for blood samples for his genealogi- work, and the duration of fieldwork (see Ex-
cal studies. The long-term effects of his field- hibit 6.1 earlier in this chapter) to anticipate
work have become a matter of spirited certain of the situations that may arise and to
debate and controversy within anthropol- establish strategies for how those situations
ogy (Geertz 2001; Tierney 2000a, 2000b). will be handled. For example, I have been in-
At the other end of the intrusion contin- volved as a participant observer- evaluator
uum we find those qualitative designs in a number of professional development
where "intrusions" are intentionally de- programs where participants were expected
signed because the qualitative inquiry is to exercise increasing control over the curric-
framed as an intended form of desired inter- ulum as the program evolved. Had I fully
vention. This is the case, for example, with participated in such participatory decision
collaborative and participatory forms of in- making, I could have influenced the direc-
quiry in which those people in the setting tion of the program. Anticipating that prob-
who become coresearchers are expected to lem and reviewing the implications with
be affected by participation in the inquiry. program staff, in each case I decided not to
The processes of participation and collabo- participate actively in participant-led deci-
ration can be designed and facilitated to sion making to the full extent I might have
have an impact on participants and collabo- had I not been involved in the role of evalua-
rators quite beyond whatever findings they tor observer. The participatory and empow-
may generate by working together. In the ering philosophy of these programs called
process of participating in an evaluation, for each participant to articulate interests
participants are exposed to and have the op- and help make happen those things that he
portunity to learn the logic of research and or she wanted to have happen. In my role as
the discipline of data-based reasoning, Skills evaluator observer, I had to reduce the ex-
are acquired in problem identification, crite- tent to which I acted out that philosophy so
ria specification, and data collection, analy- as to limit my impact on the direction of the
sis, and interpretation. Acquisition of re- group. 1 aimed my involvement at a levei
search skills and ways of thinking can have a where I would not appear withdrawn from
longer-term impact than the use of findings the process, yet at the same time attempted
from a particular evaluation study. This to minimize my influence, especially where
"learning from the process" as an outcome the group was divided on priorities.
of participatory and collaborative inquiry Another example comes from evaluation
experiences is called process use in contrast to of a community leadership program men-
findings use (Patton 1997a: Chapter 5,1998, tioned previously in this chapter. As a
1999c). three-person team of participant observers,
While it is not possible to know precisely we participated fully in small-group leader-
how collaboration will affect coresearchers ship exercises. When the groups in which we
or to fully anticipate how an observer will af- participated were using concepts inappro-
fect the setting observer, both cases illustrate priately or doing the exercise wrong, we
the need to be thoughtful about the intercon- went along with what participants said and
nections between observers and observed. It did without making corrections. Had we re-
is possible, however, when designing the ally been only participantsand not partici-
study and making decisions about the ob- pant evaluatorswe would have offered
corrections and solutions. Thus, our roles only a few. Ali imply that the way in which a
made us more passive than we tended natu- person construes his relationship to the phe-
rally to be in order not to domina te the small nomenal world is a function of his point of view
groups. We had anticipated this possibility about it. That is, relationship is not a given nor
in the design stage prior to fieldwork and an absolute, but depends upon a personal per-
had agreed on this strategy at that time. spective. It is also true that perspective can
The role and impact of the evaluator ob- shift, the only necessity of a person's human-
server can change over the course of field- ity being that he takes some stance in relation-
work. Early in the wilderness program, I ship to the events about him. (pp. 8-9)
kept a low profile during participant-led
planning discussions. Later in the program, Carini is here articulating the interde-
particularly during the final field conference pendence between the observer and what
of the second year, I became more engaged is observed. Prior to data collection, the
in discussions about the future direction of fieldworker plans and strategizes about the
the project. hoped-for and expected nature of that inter-
Reporting on the relationship between dependence. But things don't always unfold
the observer and the observed, then, and the as planned, so observers must make some
ways in which the observer may have af- effort to observe themselves observing
fected the phenomenon observed becomes and record the effects of their observations
part of the methodological discussion in on the people observed and, no less impor-
published fieldwork reports and evaluation tant, reflect on changes they've experienced
studies. In that methodological discussion from having been in the setting. This means
(or the methods chapter of a dissertation), being able to balance observation with re-
the observer presents data about the effects flection and manage the tension between en-
of fieldwork on the setting and people gagement and detachment.
therein and also the observer's perspective Bruyn (1966), in his classic work on par-
on what has occurred. As Patricia Carini ticipant observation, articulated a basic
(1975) has explained, such a discussion ac- premise of participant observation: the "role
knowledges that findings inevitably are in- of the participant observer requires both
fluenced by the observer's point of view detachment and personal involvement"
during naturalistic inquiry: (p. 14). To be sure, there is both tension and
ambiguity in this premise. How it plays out
The observer has a point of view that is central in any given situation will depend on both
to the datum and it is in the articula tionin the observer and the phenomenon being ob-
the revela tion of his point of viewthat the served.
datum of inquiry is assumed to emerge. In ef-
fect the observer is here construed as one mo- Thus, we may observe at the outset that while
ment of the datum and as such the fabric of his the traditional role of the scientist is that of a
thought is inextricably woven into the datum neutral observer who remains unmoved, un-
as he is assumed to be constituent of its mean- changed, and untouched in his examination of
ing. From this assumption it is possible to con- phenomena, the role of the participant ob-
sider the relationship of the observer to the server requires sharing the sentiments of peo-
phenomenon under inquiry. Relatedness can ple in social situations; as a consequence he
be stated in many ways: opposition, identity, himself is changed as well as changing to some
proximity, interpenetration, isolation, to name degree the situation in which he is a partici-
pant The effects are reciprocai for observer Fieldwork is not for everyone. Some, like
and observed. The participant observer seeks, Henry James, will find that "innocent and
on the one hand, to take advantage of the infinite are the pleasures of observation."
changes due to his presence in the group by re- Others will find observational research any-
cording these changes as part of his study, and thing but pleasurable. Some students have
on the other hand, to reduce the changes to a described their experiences to me as tedious,
minimum by the manner in which he enters frightening, boring, and "a waste of time,"
into the life of the group. (Bruyn 1966:14) while others have experienced challenge,
exhilaration, personal learning, and intellec-
Whether one is engaged in participant ob- tual insight. More than once the same stu-
servation or onlooker observation, what dent has experienced both the tedium and
happens in the setting being observed will, the exhilaration, the fright and the growth,
to some extent, be dependent on the role as- the boredom and the insight. Whatever the
sumed by the observer. Likewise, the nature adjectives used to describe any particular in-
of the data collected will, to some extent, be dividual^ fieldwork, of this much we are as-
dependent on the role and perspective of the sured: The experience of observing provides
observer. And just as the presence of the ob- the observer with both experience and ob-
server can affect people observed, so too the servations, the interconnection being ce-
observer can be affected. mented by reflection. No less an authority
than William Shakespeare gives us this as-
The Personal surance.
Experience of Fieldwork
Armado: "How hast thou purchased this
The intersection of social science proce- experience?"
dures with individual capabilities and situa- Moth: "By my penny of observation."
tional variation is what makes fieldwork a
highly personal experience. At the end of Love's Labour's Lost
her book Doing Fieldwork, Rosalie Wax (1971)
reflected on how fieldwork changed her:
A Part of and Apart
A colleague has suggested that I reflect on the From the World Observed
extent to which I was changed as a person by
doing field work. I reflected and the result as- The personal, perspective-dependent na-
tonished me. For what I realized was that I had ture of observations can be understood as
not been greatly changed by the things I suf- both a strength and a weakness, a strength
fered, enjoyed or endured; nor was I greatly in that personal involvement permits first-
changed by the things I did (though they hand experience and understanding, and a
strengthened my confidence in myself). What weakness in that personal involvement in-
changed me irrevocably and beyond repair troduces selective perception. In the deep
were the things learned. More specifically, engagement of naturalistic inquiry lies both
these irrevocable changes involved replacing its risks and its benefits. Reflection on that
mythical or ideological assumptions with the engagement, from inside and outside the
correct (though often painful) facts of the situ- phenomenon of interest, crowns fieldwork
ation. (p. 363) with reflexivity and makes the observer the
OilL^ cwim.iL, oisdm.n-JEP riyi^Ri^vrinN?-

i''i l:ii>ii.il;. i1i ::i!r-liil.i.Vi.Vi ri:,;i".V!:"V.îi\ .=="!V! / CTII, nD-Ifflrf-'fl!'(,Hl"CiTiFt !:V!i .<iiiVi!j.ii.l
uftirs =!VJ: .!.i'|ri!iiv i'i i.'i !" ': ;J L-i'i !::p, nhii' H i*PFTtatfiVfaj :'bf \TU ^h-ft !! P; ': W :^ h? "i.in
KVlH H=:'i .V..' i ViA
-.;: |;ai-|.-rrs .;>!;:>:>/Vi',.;;.';u.ijn Yj',1;kjiT-.;.in\s: tf!'MXi.:iV:ii!j-'i.!"i)",'!vn!!i^M-n-:'e*0'':
i'i.vh. I J t ;ii i! s: .M.^V Li.'t J A I H 1 I I ! =. ^ - . ! . ^ " V N I - I.V.V- ;;V- ^ rs :;:I : .:' ' : ; . T : ! . ' ' ! : ; ' ! ! ' . H K
!
toai* I.a!vr-:j ' !.':lYi' :.i.!.v ,!i n!! J1 iiVil{{lATifr n.:|;'.r PGTFL h 31 - tf E LV.VL ii" itfV i P/C, T !:: !! in'!i: '.i-riViWf
fhv n.'iT=i-h. !;iVi:r.'!: .';|:M |!i;i.;.i?:;f rijii {TI i\!j i-.YiVi iî !'*:. iVhin.!,i.! i'i y '.1piin!h:;"' ,'i!n;i!
Mil 5 fl ;!Mii^hY in>% ^Min iViS!p,:!.'-i:=!:r!:! !i,:ifi!\ ii!,'! .!:'' = hi".jvAi.VTM h:n;'!r pajip.;.:;. fl.'- W-H
!
! : ; ; ! ' J " M. :;::.-;!!- I:M.';;! rivn.s:-'J-JU

, , 1
': Ti 'ii"n H p i r ; ! " rVi TiT!.:: 1 v\;; ;! ;:;| W i,:; i \ ^ (I-i'.^1' j' n' ! , 1 , ! hi h ii h : v!;'"".= n ' ' VFTL -
!.;,!>:.:;;; r| nr-viV!r-.rnpni*1:.:;=.,h:;i i ! !:!.'I:'.H1 ! ' 'vJir^.Vy {yi^*Pi o' O!:=!'"!=::;iVv LI
i.si:li:i-iii/iVvi. .'.'mlii! J 4 H i
observedeven if only by oneself. So we re- the characteristics of the setting, and the
peat Halcolm's refrain that opened this skills, interests, needs, and point of view
chapter: that y o u , as observer, bring to y o u r engage-
ment. Yet, the conduct of observational re-
Go out into the world. Live among the peoples search is not without direction. Exhibit 6.6
of the world as they live. Learn their language. offers a modest list of 10 guidelines for field-
Participate in their rituais and routines. Taste work (not, please notice, commandments,
of the world. Smell it. Watch and listen. Touch just guidelines) by way of reviewing some
and be touched. Write down what you see and of the major issues discussed in this chapter.
hear, how they think and how you feel. Beyond these seemingly simple but decep-
Enter into the world. Observe and wonder. tively complex prescriptions, the point re-
Experience and reflect. To understand a world mains that what you do depends on a great
you must become part of that world while at number of situational variables, your own
the same time remaining separate, a part of capabilities, and careful judgment informed
and apart from. by the strategic themes for qualitative in-
Go then, and return to tell what you see quiry presented in the first chapter (Exhibit
and hear, what you learn, and what you come 2.1).
to understand.
Having considered the guidelines and
strategic themes for naturalistic field-based
Summary Guidelines research, and after the situational con-
for Fieldwork straints on and variations in the conduct of
fieldwork have been properly recognized
A reader who came to this chapter looking and taken into account in the design, there
for specific fieldwork rules and clear proce- remains only the core commitment of quali-
dures would surely be disappointed. tative inquiry to reaffirm. That core commit-
Looking back over this chapter, the major ment was articulated by Nobel laureate
theme seems to be, What you do depends Nicholas Tinbergen in his 1975 acceptance
on the situation, the nature of the i n q u i ^ speech for the Nobel Prize in physiology
E X H I B I T 6.6 Summary Guidelines for Fieldwork
1. Design the fieldwork to be clear about the role of the observer (degree of participation);
the tension between insider (emic) and outsider (etic) perspectives; degree and nature
of coiaboration with coresearchers; disclosure and explanation of the observer's role to
others; duration of observations (short vs. long); and focus of observation {narrow vs.
broad). (See Exhibit 6.1.)
2. Be descriptive in taking field notes. Strive for thick, deep, and rich description.
3. Stay open. Gather a variety of information from different perspectives. Be opportunistic
in following leads and sampling purposefully to deepen understanding. Allow the design
to emerge flexibly as new understandngs open up new paths of inquiry.
4. Cross-validate and trianguiate by gathering different kinds of data: observations,
interviews, documents, artifacts, recordings, and photographs. Use muitiple and mixed
methods.
5. Use quotations; represent people n their own terms. Capture participants' views of their
experiences in their own words.
6. Select key informants wisely and use them carefully. Draw on the wisdom of their
informed perspectives, but keep in mind that their perspectives are selective.
7. Be aware of and strategic about the different stages of fieldwork.
a. Build trust and rapport at the entry stage. Remember that the observer is also being
observed and evaluated.
b. Attend to relationships throughout fieldwork and the ways in which relationships
change over the course of fieldwork, including relationships with hosts, sponsors within
the setting, and coresearchers in collaborative and participatory research.
c. Stay alert and dscipned during the more routine, middle phase of fieldwork.
d. Focus on pulling together a useful synthesis as fieldwork draws to a close. Move
from generating possibilities to verifying emergent patterns and confirming themes.
e. Be disciplined and conscientious in taking detailed field notes at ali stages of
fieldwork.
f. In evaluations and action research, provide formative feedback as part of the
verifcation process of fieldwork. Time that feedback carefully. Observe its impact.
8. Be as involved as possible in experiencing the setting as fully as is appropriate and
manageable while maintaining an analytical perspective grounded in the purpose of
the fieldwork.
9. Separate description from interpretation and judgment.
10. Be reflective and reflexive. Include in your field notes and reports your own experiences,
thoughts, and feelings. Consider and report how your observations may have affected
the observed as well as how you may have been affected by what and how you've
participated and observed. Ponder and report the origins and implications of your own
perspective.
Fm a researcher.
This is great. A serendipitous
purposeful sample o f a criticai
incident involving conflict for
my study on posimodem en-
tanglements among strangers
in a globai viliage context.
Constructivist Rashonton heaven: Multiple and diverse perspectives
and medicine: "watching and wondering." behaviors, he was able to make a major med-
Tinbergen explained that it was by watching icai and scientific contribution. His research
and wondering that he had, despite being methodology: "watching and wondering."
neither a physiologist nor a medicai doctor,
discovered what tumed out to be a major S Notes
breakthrough in our understanding of au-
tism. His observations revealed that the ma- 1. Excerpt from "Little Gidding" in the Four
jor clinicai research on autism did not hold Quartis by T. S. Eliot. Copyright 1942 by T. S.
up outside clinicai settings. His "watching Eliot; renewed 1970 by Esme Valerie Eliot. Re-
printed by permission of Harcourt, Inc.
and wondering" allowed him to see that
2. Excerpt from "The Elephant's Child," from
normal individuais, those not clinically la- Just So Stories, by Rudyard Kipling. Used by per-
beled as autistic, exhibited under a variety of mission of A. P. Watt Ltd. on behalf of The Na-
circumstances ali of the behaviors described tional Trust for Places of Historical Interest or
as autistic in clinicai research. He also noted Natural Beauty. Original publication 1902.
that children diagnosed as autistic re- 3. From Traveling Light: Collected and Nezu
Poems. Copyright 1999 by David Wagoner.
sponded in nonautistic ways outside the
Used with permission of the University of Illinois
clinicai setting. By observing people in a va- Press.
riety of settings and watching a full range of 4. Used with permission of Joyce Keller.
Outside to Inside, Inside to Outside
Shifting Perspectives
Preface gram evaluator). She recounts those transi-

tions from outside to inside, and inside to
The preceding chapter on fieldwork in- outside, and how they have shaped her per-
cluded discussion of insider (emic) versus spective on research and evaluation. Her
outsider (etic) perspectives. Understand- reflections provide a poignant and insight-
ing different perspectives from inside and ful transition from our discussion of par-
outside a phenomenon goes to the core of ticipant observation in the last chapter to
qualitative inquiry. Experience affects per- interviewing strategies in the next chapter,
spective. Perspective shapes experience. In both methods aimed at bridging insider-
the next chapter, on interviewing, we shall outsider perspectives. Barbara Lee gener-
continue to explore ways of capturing the ously wrote these reflections especially for
experiences and getting deeply into the this book. I thank Barbara for her openness,
perspectives of those who have encoun- courage, commitment, and insights.
tered whatever phenomenon interests us.
The reflections that follow look at the ex-
perience of mental illness from the inside
S "Nothing About Us,
and outside, and how being on the inside
Without Us"
can dramatically change the view from the
outside. Barbara Lee moved from the out-
side (Ph.D. researcher and mental health I was a high school and jnior high school
professional) to the inside as an involun- science teacher, a school psychologist and
tary participant observer (a patient in a guidance counselor, and finally an educa-
locked mental health facility) and back tional researcher, before I completed my
again to the outside (as a professional pro- Ph.D. (on the second try!). I had specifically
a 335
sought that degree in my mid-40s to get rnaze. Probably the most serious is that what
evaluation and research skills, which I ac- I had assumed to be "treatment" from the
complished nicely at St. Louis University, in viewpoint of the researcher now looked like
one of the few programs in the 1980s de- mostly futile efforts that were most often ex-
signed to train graduate students in evalua- perienced as punishment and threat from
tion theory and methodology. the viewpoint of "patient." I was not asked if
Since I had been a scientist of some sort I would go into the hospital. I was told I had
for ali of my career life, I was a "natural" for to. If I got angry at something, someone gave
the field of evaluation. I thought like a scien- me powerful medications that made me
tist. I was familiar with the practice of re- feel like a zombie.
search inbiology, bacteriology, and field bot- While some of what happened in my hos-
any and had a special interest in medicine. pitaliza tion helped, much of what I experi-
But through some quirks of fate and person- enced made me feel much worse. I was in-
ality, I found myself working as a clircian, carcerated and my jailers looked at me
providing therapy and case management to kindly, certain that I was being locked up,
people with severe mental disorders. Then, strapped to a bed, and injected with medica-
after having been gainfully employed my tions for my own good. Even though I actu-
entire adult life, and successfully raising two ally entered the hospital "voluntarily," the
children who had now produced one grand- threat of involuntary treatment and perma-
child each, I was forced into the locked, psy- nent damage to my ability to earn a living
chiatric ward of a hospital for the third time was the driving force that got me there, and
in my life. kept me there, and taught me to "make nice"
Back at work, after nearly two months in for the staff, lest they refuse to certify me
the hospital, I found myself, for the first sane and let me be free again.
time, looking at my professional work and Asa human services program evaluator, I
reading professional literature with the eyes learned to do needs assessments, to use
of one from the other side of the locked proven treatment methods in a package deal
doors and medicai charts. called a program, to gather data of various
The irony of my situation was obvio us: I kindssometimes even from the people
treated people like me! Thus began a shift of who were getting the program. I learned
viewpoint that has radically altered my how to interpret the data gathered in the en-
practice of evaluation in the field of mental vironmental context in which the program
health. operated and how to get and report reliable
First, I had to throw out some grand as- and credible information to those who make
sumptions. As a scientist, I trusted scientific decisions about programs. Sometimes, I ad-
method and worshipped at the same shrine mit, I even offered my own "expert" judg-
of true experimental design and random as- ment about the value of the program. Now a
signment as everyone else. But now I was whole new set of questions confronted me as
much more conscious that the "lab rats," the a professional.
subjects of our research, literally have minds What can distort the perceptions of those
of their own. Some of the most treasured as- I ask about "needs" and how much is the dis-
sumptions of mental health research were tortion? The providers believe they have the
looking awfully different from inside the well-being of the clients at heart, but the cli-
Shifting Perspectives . 337
ents may experience the treatments as more threatening voices might just as likely be real
disabling than the symptoms of the disease. as hallucinated. The provider who is con-
Clients are taught to mistrust their own fined to the office and distance of a profes-
symptom-distorted thoughts and are flatly sional relationship will not know when
ignored when psychotic, yet virtually ali of there is abuse in the home the client never
them can make reasoned decisions if they speaks of, because the abuser is someone
have adequate information and are asked. It they love or who controls their money as
seems to be assumed that a psychiatric label payee. Mental health workers are put in the
defines the "needs" of people with severe role of defenders of the public purse, and
mental illness. But other needs may be a con- then we wonder why clients feel their safety
sequence of a stingy health care system that net of services threatened with every dollar
won't provide necessary and expensive they are given or earn and fail to trust their
medications unless you are completely dis- "providers."
abled, and social stignia makes it nearly im- In short, I may apply many of the same
possible to get a good job with benefits after ideas, theories, methods, and interpreta-
psychiatric hospitalizations. A clinician who tions as I always did as a program evaluator.
admits to having a psychiatric label of severe But now I always question, not just the va-
mental illness will never be hired, so it can be lidity, reliability, and generalizability of the
survival as well as denial to deny even to evaluation work itself but also the hidden
oneself that one is one of "those crazies." assumptions that surround it. I will always
Providers are taught, in ali sincerity and be seekng to empower those disenfran-
good intentions, to act in a kind of parental chised by custom, poverty, and stigma. Fur-
role toward clients, a benign dictatorship. thermore, I will always be conscious of the
But their "subjects" are people who have al- fact that my work is always limitedand
ready had their dignity as adults medically empoweredby the selection of data and
removed, their privacy invaded, and the job methods tobe used. But if I want to be part of
and relationship underpinnings of Ameri- the solution rather than part of the problem,
can self-esteem destroyed; been told they I better be sure I know what the experience
will be sick for life; and been medicated so of different stakeholders really is, not con-
they cannot perform sexually or sometimes strained by the limited questions I may think
even read a good book. Is it any wonder that to ask, or guided too narrowly by work done
many of them (us) will accept survival as an in the past.
adequate "quality of life"? As an evaluator, I now try to approach my
What are the limits of (a) theory about task with equal measures of chutzpah and
what the "problem" is; (b) the kinds, rele- humility so that I will not fail to challenge ali
vance, and quality of data collected in the the assumptions, especially my own, nor
past; and (c) the stakeholders' (mentally ill ever assume that I have ali of the questions,
people) freedom to express themselves? much less the answers, right. I have adopted
Evaluation designs that test the effects of the motto of the people who do not claim the
treatment programs on the individual don't title of "consumer," because they were not
address the problem of living in a neighbor- given true choice about treatment when they
hood where life is stressful and taxis won't found themselves pinned with psychiatric
take you home after dark, and where the labels: "Nothing about us, without us."
When was the last time you saw the cli- answer truthfully, "Just last week, or last
ents who are the intended target of the pro- month": Thank you!
gram, sitting at the table with the evaluators
and program providers, freely exchanging Barbara Lee, Ph.D., prosumer and
perspectives and ideas? For those who can mental health program evaluator
Qualitative Interviewing
S e y o n d .Silekrf O b s e ^ v a + i o ^
After much cloistered study, three youths carne before Halcolm to ask how
they might further increase their knowledge and wisdom. Halcolm sensed that
they lacked experience in the real world, but he wanted to have them make the
transition from seclusion in stages.
During the first stage he sent them forth under a six-month vow of silence.
They wore the identifying garments of the muted truth-seekers so that people
would know they were forbidden to speak. Each day, according to their instruc-
tions, they sat at the market in whatever village they entered, watching but
never speaking. After six months in this fashion they returned to Halcolm.
"So," Halcolmbegan, "you have returned. Your period of silence is over. Your
transition to the world beyond our walls of study has begun. What have you
learned so far?"
The first youth answered, "In every village the patterns are the same. People
come to the market. They buy the goods they need, talk with friends, and leave. I
have learned that ali markets are alike and the people in markets always the
same."
Then the second youth reported, "I too watched the people come and go in
the markets. I have learned that ali life is coming and going, people forever mov-
ing to and fro in search of food and basic material things. I understand now the
simplicity of human life."
fi. 339
|
Halcolm looked at the third youth: "And what have you learned?"
"I saw the same markets and the same people as my fellow travelers, yet I know
not what they know. My mind is filled with questions. Where did the people come
from? What were they thinking and feeling as they came and went? How did they
happen to be at this market on this day? Who did they leave behind? How was to-
day the same or different for them? I have failed, Master, for I am filled with ques-
tions rather than answers, questions for the people I saw. I do not know what I
have learned."
Halcolm smiled. "You have learned most of ali. You have learned the impor-
tance of finding out what people have to say about their experiences. You are
ready now to return to the world, this time without the vow of silence.
"Go forth now and question. Ask and listen. The world is just beginning to
open up to you. Each person you question can take you into a new part of the
world. The skilled questioner and attentive listener know how to enter into an-
other's experience. If you ask and listen, the world will always be new."
From Halcolm's Episte?nological Parables
Rigorous and social scientists have become suspect, as

Skillful Interviewing have our methods. The popular business
magazine Forbes (self-proclaimed "The Cap-
The very popularity of interviewing may italistTool") has opined, "People become so-
be its undoing as an inquiry method. In the ciologists because they hate society, and they
contemporary "interview society" (Fontana become psychologists because they hate
and Frey 2000:646), so much interviewing is themselves" (quoted in Geertz 2001:19).
being done so badly that its credibility may Such glib sarcasm, anti-intellectual at the
be undermined. Television, radio, maga- core, can serve to remind us that we bear the
zines, newsletters, and Web sites feature in- burden of demonstrating that our methods
terviews. In their ubiquity, interviews done involve rigor and skill. Interviewing, seem-
by social scientists become indistinguish- ingly straightforward, easy, and universal,
able in the popular mind from interviews can be done well or poorly. This chapter is
done by talk show hosts. The motivations of about doing it well.
Inner Perspectives
nterviewing is rather like a marriage: everybody knows what it is, an

awful lot of people do it, and yet behind each closed door there is a
world of secrets.
A. Oakley (1981:41)
We interview people to find out from serve. The issue is not whether observa-
them those things we cannot directly ob- tional data are more desirable, valid, or
Qualitative Intervieiving |J, 341
meaningful than self-report data. The fact is said in Es Lebe das Leben I, "I know how to lis-
that we cannot observe everything. We can- ten when clever men are talking. That is the
not observe feelings, thoughts, and inten- secret of what you call my influence." Evalu-
tions. We cannot observebehaviors that took ators must learn how to listen when knowl-
place at some previous point in time. We edgeable people are talking. That may be the
cannot observe situations that preclude the secret of their influence.
presence of an observer. We cannot observe An evaluator, or any interviewer, faces
how people have organized the world and the challenge of making it possible for the
the meanings they attach to what goes on in person being interviewed to bring the inter-
the world. We have to ask people questions viewer into his or her world. The quality of
about those things. the information obtained during an inter-
The purpose of interviewing, then, is to view is largely dependent on the inter-
allow us to enter into the other person's per- viewer. This chapter discusses ways of ob-
spective. Qualitative interviewing begins taining high-quality information by talking
with the assumption that the perspective of with people who have that information.
others is meaningful, knowable, and able to We'll be delving into "the art of hearing"
be made explicit. We interview to find out (Rubin and Rubin 1995).
what is in and on someone else's mind, to This chapter begins by discussing three
gather their stories. different types of interviews. Later sections
Program evaluation interviews, for ex- consider the content of interviews: what
ample, aim to capture the perspectives of questions to ask and how to phrase ques-
program participants, staff, and others asso- tions. The chapter ends with a discussion of
ciated with the program. What does the pro- how to record the responses obtained dur-
gram look and feel like to the people ining interviews. This chapter emphasizes
volved? What are their experiences in the skill and technique as ways of enhancing the
program? What thoughts do people knowl- quality of interview data, but no less impor-
edgeable about the program have concem- tant is a genuine interest in and caring about
ing program operations, processes, and out- the perspectives of other people. If what
comes? What are their expectations? What people have to say about their world is gen-
changes do participants perceive in them- erally boring to you, then you will never be a
selves as a result of their involvement in the great interviewer. Unless you are fascinated
program? It is the responsibility of the evalu- by the rich variation in human experience,
ator to provide a framework within which qualitative interviewing will become drudg-
people can respond comfortably, accurately, ery. On the other hand, a deep and genuine
and honestly to these kinds of questions. interest in learning about people is insuffi-
Evaluators can enhance the use of quali- cient without disciplined and rigorous in-
tative data by generating relevant and high- quiry based on skill and technique.
quality findings. As Hermann Sudermann
!?. Variations in Qualitative Interviewing
O n her deathbed Gertrude Stein asked her beloved companion, Alice B.

Toklas, "What is the answer?" When Alice, unable to speak, reinained
silent, Gertrude asked: "In that case, what is the question?"
fcdhf
The question in this section is how to for- pose and poses quite varying interviewer
mai questions. There are three basic ap- challenges.
proaches to collecting qualitative data
through open-ended interviews. They in-
volve different types of preparation, concep-
The Informal
tualization, and instrumentation. Each ap-
Conversational Interview
proach has strengths and weaknesses, and
each serves a somewhat different purpose.
The informal conversational interview is
The three alternatives are
the most open-ended approach to inter-
the informal conversational interview, viewing. It is also called "unstructured inter-
viewing" (Fontana and Frey 2000:652). The
the general interview guide approach, and
conversational interview offers maximum
the standardized open-ended interview. flexibility to pursue information in what-
ever direction appears to be appropriate, de-
These three approaches to the design of the pending on what emerges from observing a
interview differ in the extent to which inter- particular setting or from talking with one or
view questions are determined and stan- more individuais in that setting. Most of the
dardized before the interview occurs. questions will flow from the immediate con-
The informal conversational interview relies text. Thus, the conversational interview con-
entirely on the spontaneous generation of stitutes a major tool of fieldwork and is
questions in the natural flow of an interac- sometimes referred to as "ethnographic in-
tion, often as part of ongoing participant ob- terviewing." No predetermined set of ques-
servation fieldwork. The persons being tions would be appropriate under many
talked with may not even realize they are be- emergent field circumstances where the
mg interviewed. The general interview guide fieldworker doesn't know beforehand what
approach involves outlining a set of issues is going to happen, who will be present, or
that are to be explored with each respondent what will be important to ask during an
before interviewing begins. The guide event, incident, or experience.
serves as a basic checklist during the inter- Data gathered from informal conversa-
view to make sure that ali relevant topics tional interviews will be different for each
are covered. In contrast, the standardized person interviewed. The same person may
open-ended interview consists of a set of ques- be interviewed on different occasions with
tions carefully worded and arranged with questions specific to the interaction or event
the intention of taking each respondent at hand. Previous responses canbe revisited
through the same sequence and asking each and deepened. This approach works partic-
respondent the same questions with essen- ularly well where the researcher can stay in
tially the same words. Flexibility in probing the setting for some period of time so as not
is more or less limited, depending on the na- to be dependent on a single interview op-
ture of the interview and the skills of inter- portunity. Interview questions will change
viewers. The standardized open-ended in- over time, and each new interview builds on
terview is used when it is important to those already done, expanding information
minimize varia tion in the questions posed to that was picked up previously, moving in
interviewees. Lefs look at each approach in new directions, and seeking elucidations
greater depth for each serves a different pur- and elaborations frm various participants.
Being unstructured doesn't mean that rapid insights, formulate questions quickly
conversational interviews are unfocused. and smoothly, and guard against asking
Sensitizing concepts and the overall pur- questions that impose interpretations on the
pose of the inquiry inform the interviewing. situation by the structure of the questions.
But within that overall guiding purpose, the Data obtained from informal conversa-
interviewer is free to go where the data and tional interviews can be difficult to pull
respondents lead. together and analyze. Because different
The conversational interviewer must "go questions will generate different responses,
with the flow." Depending on how the inter- the researcher has to spend a great deal of
viewer^ role has been defined, the people time sifting through responses to find pat-
being interviewed may not know during terns that have emerged at different points
any particular conversation that data are be- in different interviews with different people.
ing collected. In many cases, participant ob- By contrast, interviews that are more sys-
servers do not take notes during such con- tematized and standardized facilitate analy-
versational interviews, instead writing sis but provide less flexibility and are less
down what they learned later. In other cases, sensitive to individual and situational dif-
it can be both appropriate and comfortable ferences.
to take notes or even use a tape recorder.
The strength of the informal conversa- The Interview Guide
tional method resides in the opportunities it
offers for flexibility, spontaneity, and re- An interview guide lists the questions or
sponsiveness to individual differences and issues that are to be explored in the course of
situational changes. Questions can be per- an interview. An interview guide is pre-
sonalized to deepen communication with pared to ensure that the same basic lmes of
the person being interviewed and to make inquiry are pursued with each person inter-
use of the immediate surroundings and situ- viewed. The interview guide provides top-
ation to increase the concreteness and imme- ics or subject areas within which the inter-
diacy of the interview questions. viewer is free to explore, probe, and ask
A weakness of the informal conversa- questions that will elucidate and illuminate
tional interview is that it may require a that particular subject. Thus, the interviewer
greater amount of time to collect systematic remains free to build a conversation within a
Information because it may take several con- particular subject area, to word questions
versations with different people before a spontaneously, and to establish a conversa-
similar set of questions has been posed to tional style but with the focus on a particular
each participant in the setting. Because this subject that has been predetermined.
approach depends on the conversational The advantage of an interview guide is
skills of the interviewer to a greater extent that it makes sure that the interviewer / eval-
than do more formal, standardized formats, uator has carefully decided how best to use
this go-with-the-How style of interviewing the limited time available in an interview sit-
may be susceptible to interviewer effects, uation. The guide helps make interviewing a
leading questions, and biases, especially number of different people more systematic
with novices. The conversational inter- and comprehensive by delimiting m ad-
viewer must be able to mteract easily with vance the issues to be explored. A guide is
people in a variety of settings, generate essential in conducting focus group inter-
views for it keeps the interactions focused ject on Reading, illustrates how it is possible
while allowing individual perspectives and to use a detailed outline to conduct a series
experiences to emerge. of interviews with the same respondents
Interview guides can be developed in over the course of a year. The flexibility per-
more or less detail, depending on the extent mitted by the interview guide approach will
to which the interviewer is able to specify become clearer after reviewing the third
important issues in advance and the extent strategy of qualitative interviewing: the stan-
to which it is important to ask questions in dardized open-ended interview.
the same order to ali respondents. Exhibit 7.1
provides an example of an interview guide The Standardized
used with participants in an employment Open-Ended Interview
training program. This guide provides a
framework within which the interviewer This approach requires carefully and
would develop questions, sequence those fully wording each question before the inter-
questions, and make decisions about which view. For example, the interview guide for
information to pursue in greater depth. the employment training program in Exhibit
Usually, the interviewer would not be ex- 7.1 simply lists "work experiences" as a
pected to go into totally new subjects that are topic for inquiry. In a fully structured inter-
not covered within the framework of the view instrument, the question would be
guide. The interviewer does not ask ques- completely specified:
tions, for example, about previous employ-
ment or education, how the person got into You've told me about the courses you've taken
the program, how this program compares in the program. Now I'd like to ask you about
with other programs the trainee has experi- any work experiences you've had. Let's go
enced, or the trainee's health. Other topics back to when you first entered the program
might still emerge during the interview, top- and go through each work experience up to
ics of importance to the respondent that are the present. Okay? So, what was your first
not listed explicitly on the guide and there- work experience?
fore would not normally be explored with
each person interviewed. For example,
Probes: Who did you work for?
trainees might comment on farnily support
What did you do?
(or lack thereof) or personal crises. Com-
ments on such concerns might emerge What do you feel you learned
when, in accordance with the interview doing that?
guide, the trainee is asked for reactions to What did you especially like about
program strengths, weaknesses, and so on, the experience, if anything?
but if farnily is not mentioned by the respon- What did you dislike, if anything?
dent, the interviewer would not raise the is- Transition: Okay, tell me about your next work
sue. experience.
An additional, more detailed example of
an interview guide is included as Appen- Why so much detail? To be sure that each
dix 7.1 at the end of this chapter. The exam- interviewee gets asked the same questions
ple in the chapter appendix, a "descriptive the same stimuliin the same way and
interview" developed by the Educational the same order, including standard probes.
Testing Service Collaborative Research Pro- A doctoral committee may want to see the
Evaluation Interview Guide for Participants in

an Employment Training Program
What has the trainee done in the program?

is* activities
is* courses
^ groups
work experiences
Achievements?
f skills attained
is* products produced
is* outcomes achieved
knowledge gained
is* things completed
what can the trainee do that is marketable?
How has the trainee been affected in areas other than job skills?
is* feelings about self
attitudes toward work
is* aspirations
is* interpersonal skills
What aspects of the program have had the greatest impacts?

is* formal courses
v 0 relationships with staff
is* peer relationships
W the way treated in the program
is* contacts
i > work experiences
What problems has the trainee experienced?

t ^ work related
v 0 program related
personal
v 0 family, friends, world outside program
What are the trainee's plans for the future?

work pans
income expectations
v 0 lifestyle expectations/plans
What does the trainee think of the program?

v 0 strengths, weaknesses
things liked, things disliked
^ best components, poorest components
^ things that should be changed
full interview protocol before approving a ipants before they enter the program, when
dissertation proposal. The institutional re- they leave the program, and again some pe-
view board for protection of human subjects riod of time (e.g., six months) after they have
may insist on approving a structured inter- left the program. For example, a chemical
view, especially if the topic is controversial dependency program would ask partici-
or intrusive. In evaluations, key stake- pants about sobriety issues before, during,
holders may want to be sure that they know at the end of, and after the program. To com-
what program participants will be asked. In pare answers across these time periods, the
team research, standardized interviews en- same questions need to be asked in the same
sure consistency across interviewers. In way each time. Such interview questions are
multisite studies, structured interviews pro- written out in advance exactly the way they
vide comparability across sites. are to be asked during the interview. Careful
In participatory or coliaborative studies, consideration is given to the wording of each
inexperienced and nonresearcher interview- question before the interview. Any clarifica-
ers may be involved in the process, so stan- tions or elabora tions that are to be used are
dardized questions can compensate for vari- written into the interview itself. Probes are
ability in skills. Some evaluations rely on placed in the interview at appropriate places
volunteers to do interviewing; at other times to minimize interviewer effects by asking
program staff may be involved in doing the same question of each respondent,
some interviewing; and in still other in- thereby reducing the need for interviewer
stances interviewers may be novices, stu- judgment during the interview. The stan-
dents, or others who are not social scientists dardized open-ended interview also makes
or professional evaluators. When a number data analysis easier because it is possible to
of different interviewers are used, variations locate each respondent's answer to the same
in data created by differences among inter- question rather quickly and to organize
viewers will become particularly apparent if questions and answers that are similar.
an informal conversational approach to data In summary, there are four major reasons
gathering is used or even if each interviewer for using standardized open-ended inter-
uses a basic guide. The best way to guard views:
against variations among interviewers is to
carefully word questions in advance and 1. The exac t instrument used in the evalua-
train the interviewers not to deviate from the tion is available for inspection by those
prescribed forms. The data collected are still who will use the findings of the study.
open-ended, in the sense that the respon- 2. Variation among interviewers can be
dent supplies his or her own words, minimized where a number of different
thoughts, and insights in answering the interviewers must be used.
questions, but the precise wording of the
questions is determined ahead of time. 3. The interview is highly focused so that
When doing action research or conduct- interviewee time is used efficiently.
ing a program evaluation, it may only be 4. Analysis is facilitated by making re-
possible to interview participants once for a sponses easy to find and compare.
short, fixed time, such as a half hour, so
highly focused questions serve to establish In program evaluations, potential prob-
priorities for the interview. At other times, it lems of legitimacy and credibility for quali-
is possible and desirable to interview partic- tative data can make it politically wise to
produce an exact interview form that the To illustrate the standardized open-
evaluator can show to primary decision ended interview, three interviews have been
makers and evaluation users. Moreover, reproduced in Appendix 7.2 at the end of
when generating a standardized form, eval- this chapter. These interviews were used to
uation users can participate more com- gather information from participants in an
pletely in writing the interview instrument. Outward Bound wilderness program for
They not only will know precisely what is disabled persons. The first interview was
going to be asked but, no less important, will conducted at the beginning of the program,
understand what is not going to be asked. the second interview was used at the end of
This reduces the likelihood of the data being the 10-day experience, and the third inter-
attacked later because certain questions view took place six months after the pro-
were missed or asked in the wrong way. By gram.
making it clear, in advance of data collection,
exactly what questions will be asked, the
limitations of the data canbe known and dis- Combining Approaches
cussed before evaluation data are gathered.
While the conversational and interview These contrasting interview strategies are
guide approaches permit greater flexibility by no means mutually exclusive.
and individualiza tion, these approaches A conversational strategy can be used
also open up the possibility, indeed, the like- within an interview guide approach, or you
lihood, that more information will be col- can combine a guide approach with a stan-
lected from some program participants than dardized format by specifying certain key
from others. Those using the findings may questions exactly as they must be asked
worry about how conclusions have been in- while leaving other items as topics to be ex-
fluenced by qualitative differences in the plored at the interviewer's discretion. This
depth and breadth of information received combined strategy offers the interviewer
from different people. flexibility in probing and in determining
In contrast, in fieldwork done for basic when it is appropriate to explore certain sub-
and applied research, the researcher will be jects in greater depth, or even to pose ques-
attempting to understand the holistic tions about new areas of inquiry that were
worldview of a group of people. Collecting not originally anticipated in the interview
the same information from each person instrumenfs development. A common com-
poses no credibility probiem when each per- bination strategy involves using a standard-
son is understood as a unique informant ized interview format in the early part of an
with a unique perspective. The political interview and then leaving the interviewer
credibility of consistent interview findings free to pursue any subjects of interest dur-
across respondents is less of an issue under ing the latter parts of the interview. An-
basic research conditions. other combination would include using the
The weakness of the standardized ap- informal conversational interview early in
proach is that it does not permit the inter- an evaluation project, foliowed midway
viewer to pursue topics or issues that were through by an interview guide, and then
not anticipated when the interview was closing the program evaluation with a stan-
written. Moreover, a structured interview dardized open-ended interview to get sys-
reduces the extent to which individual dif- tematic information from a sample of partic-
ferences and circumstances can be queried. ipants at the end of the program or when
conducting follow-up studies of partici- While the three strategies vary in the extent
pants. to which the wording and sequencing of
A sensitizing concept can provide the questions are predetermined, no variation
bridge across types of interviews. In doing exists in the principie that the response for-
follow-up interviews with recipients of Mac- mat should be open-ended. The interviewer
Arthur Foundation Fellowships, the sensi- never supplies and predetermines the
tizing concept "enabling," a concept cen- phrases or categories that must be used by
tral to the fellowship's purpose, allowed us respondents to express themselves as is the
to focus interviews on any ways in which case in fixed-response questionnaires. The
receiving the fellowship had enabled re- purpose of qualitative interviewing is to
cipients. "Enabling," or "being enabled," capture how those being interviewed view
broadly defined and open-ended, gave in- their world, to learn their terminology and
terviewees room to share a variety of experi- judgments, and to capture the complexities
ences and outcomes while also letting me of their individual perceptions and experi-
identify some carefully worded, standard- ences. This openness distinguishes qualita-
ized questions for ali interviewees, some in- tive interviewing from the closed question-
terview guide topics that might or might not naire or test used in quantitative studies.
be pursued, and a theme for staying cen- Such closed instruments force respondents
tered during completely open-ended con- to fit their knowledge, experiences, and feel-
versations at the end of the interviews. ings into the researcher's categories. The
fundamental principie of qualitative inter-
Summary of viewing is to provide a framework within
Interviewing Strategies which respondents can express their own un-
derstandings in their own terms.
Ali three qualitative approaches to inter- Exhibit 7.2 summarizes variations in in-
viewing share the commitment to ask genu- terview instrumentation. In reviewing this
inely open-ended questions that offer the summary table, keep in mind that these are
persons being interviewed the opportun- presented as pure types. In practice, any par-
ity to respond in their own words and to ticular study may employ ali or several of
express their own personal perspectives. these strategies together.
13. Question Options
f you ask me I'm gonna tell you.
Roseanne (2001:164)
Six kinds of questions can be asked of Experience and

people. On any given topic, it is possible to Behavior Questions
ask any of these questions. Distinguishing
types of questions forces the interviewer to Questions about what a person does or
be clear about what is being asked and helps has done aim to elicit behaviors, experi-
the interviewee respond appropriately. ences, actions, and activities that would
Variations in Interview Instrumentation
Type of Interview Characterstics Strengths Weaknesses
Informal conversational interview Questions emerge from the immediate context Increases the salience and relevance of Different information colected from different
and are asked in the natural course of things; questions; interviews are built on and emerge peopie with different questions. Less systematic
there is no predetermination of question topics from observations; the interview can be matched and comprehensive if certain questions do not
or wordng. to individuais and circumstances. arise naturally. Data organization and analysis
can be quite difficult.
Interview guide approach Topics and issues to be covered are specified The outline increases the comprehensiveness of Important and salient topics may be
in advance, in outline form; interviewer the data and makes data collection somewhat inadvertently omitted. Interviewer flexibility
decides sequence and wording of questions systematic for each respondent. Logical gaps in in sequencing and wording questions can
in the course of the interview. data can be anticipated and ciosed. Interviews result in substantially different responses
remain fairly conversational and situational. from different perspectives, thus reducing
the comparability of responses.
Standardized open-ended interview The exact wording and sequence of questions Respondents answer the same questions, thus Little flexibility in relating the interview to
are determined in advance. Ali interviewees increasing comparability of responses; data particular individuais and circumstances;
are asked the same basic questions in the are complete for each person on the topics standardized wording of questions may
same order. Questions are worded in a addressed in the interview. Reduces interviewer constrain and limit naturalness and relevance
completely open-ended format. effects and bias when several interviewers are of questions and answers.
used. Permits evaluation users to see and review
the instrumentation used in the evaluation.
Facilitates organization and analysis of the data.
Ciosed, fixed-response interview Questions and response categories are Data analysis is simple; responses can be directly Respondents must fit their experiences and
determined in advance. Responses are fixed; compared and easily aggregated; many questions feelings into the researcher's categories; may
respondent chooses from among these fixed can be asked in a short time. be perceived as impersonal, irrelevant, and
responses. mechanistic. Can distort what respondents
really mean or experienced by so completely
limiting their response choices.
have been observable had the observer been has not really been answered. Analytical, in-
present. "If I followed you through a typical terpretive, and opinion statements are not
day, what would I see you doing? What ex- answers to questions about feelings.
periences would I observe you having?" "If I This confusion sometimes occurs because
had been in the program with you, what interviewers give the wrong cues when ask-
would I have seen you doing?" ing questionsfor example, by asking opin-
ion questions using the format "How do you
feel about that?" instead of "What is your
Opinion and opinion about that?" or "What do you think
Values Questions about it?" When you want to understand the
respondents' emotional reactions, you have
Questions aimed at understanding the
to ask about and listen for feeling-level re-
cognitive and interpretive processes of peo-
sponses. When you want to understand
ple ask about opinions, judgments, and val-
what someone thinks about some thing, the
ues"head stuff" as opposed to actions and
question should explicitly tell the inter-
behaviors. Answers to these questions tell
viewee that you're searching for opinions,
us what people think about some experience
beliefs, and considered judgmentsnot
or issue. They tell us about people's goals,
feelings.
intentions, desires, and expectations. "What
do you believe?" "What do you think about Knowledge Questions
?" "What would you like to see hap-
pen?" "What is your opinion of ?" Knowledge questions inquire about the
respondent's factual informationwhat the
respondent knows. Certain things are facts,
Feeling Questions such as whether it is against the law to drive
while drunk and how the law defines
Emotional centers in the brain can be dis-
drunkenness. These things are not opinions
tinguished from cognitive areas. Feeling
or feelings. Knowledge about a program
questions aim at eliciting emotions feeling
may include knowing what services are
responses of people to their experiences and
available, who is eligible, what the rules and
thoughts. Feelings tap the affective dimen-
regula tions of the program are, how one en-
sion of human life. In asking feeling ques-
rolls in the program, and so on. Cooke
tions"How do you feel about that?"the
(1994), for example, reviews and assesses a
interviewer is looking for adjective re-
variety of knowledge elicitation methods
sponses: anxious, happy, afraid, intimi-
and techniques.
dated, confident, and so on.
Opinions and feelings are often confused. Sensory Questions
It is criticai that interviewers understand the
distinction between the two in order to Sensory questions ask about what is seen,
know when they have the kind of answer heard, touched, tasted, and smelled. Re-
they want to the question they are asking. sponses to these questions allow the inter-
Suppose an interviewer asks, "How do you viewer to enter into the sensory apparatus of
feel about that?" The response is, "I think it's the respondent. "When you walk through
probably the best that we can do under the the doors of the program, what do you see?
circumstances." The question about feelings Tell me what I would see if I walked through
the doors with you." Or again: "What does Colored, Black, African American, Person of
the counselor ask you when you meet with African descent), qualitative inquiry is a
him? What does he actually say?" Sensory particularly appropriate way of finding out
questions attempt to have interviewees de- how people perceive and talk about their
scribe the stimuli that they experience. Tech- backgrounds.
nically, sensory data are a type of behavioral
or experiential datathey capture the expe-
rience of the senses. However, the types of Distinguishing
questions asked to gather sensory data are Question Types
sufficiently distinct to merit a separa te cate-
gory. Behaviors, opinions, feelings, knowl-
edge, sensory data, and demographics are
common background questions possible to
Background/ ask in an interview. Any kind of question
Demographic Questions one might want to ask can be subsumed in
one of these categories. Keeping these dis-
Age, education, occupation, and the like
tinctions in mind can be particularly helpful
are standard background questions that
in planning an interview, designing the in-
identify characteristics of the person being
quiry strategy, focusing on priorities for in-
interviewed. Answers to these questions
quiry, and ordering the questions in some se-
help the interviewer locate the respondent
quence. Before considering the sequence of
in relation to other people. Asking these
questions, however, lef s look at how the di-
questions in an open-ended rather than
mension of time intersects with the different
closed manner elicits the respondenfs own
kinds of questions.
categorical worldview. Asked about age, a
person aged 55 might respond, "Fm 55" or The Time Frame of Questions
"Fm middle-aged" or "Fm at the cusp of old
age" or "I'm still young at heart" or "I'm in Questions can be asked in the present,
my mid-50s" or "Fm 10 years from retire- past, or future tense. For example, you can
ment" or "Fm between 40 and 60" (smiling ask someone what they're doing now, what
broadly) and so forth. Responses to they have done in the past, and what they
open-ended, qualitative background inqui- plan to do in the future. Likewise, you can
ries tell us about how people categorize inquire about present attitudes, past atti-
themselves in today's endlessly categoriz- tudes, or future attitudes. By combining the
ing world. Perhaps nowhere is such open- time frame of questions with the different
ness more important and illuminative than types of questions, we can construct a matrix
in asking about race and ethnicity. For exam- that generates 18 different types of ques-
ple, professional golfer Tiger Woods has Af- tions. Exhibit 7.3 shows that matrix.
rican, Thai, Chinese, American Indian, and Asking ali 18 questions about any partic-
European ancestryand resists being "as- ular situation, event, or programmatic activ-
signed" to any single ethnic category. He ity may become somewhat tedious, espe-
came up with the name "Cablinasian" to de- cially if the sequence is repeated over and
scribe his mixed heritage. In an increasingly over throughout the interview for different
diverse world with people of mixed ethnic- program elements. The matrix constitutes a
ity and ever-evolving labels (e.g., Negro, set of options to help you think about what
E X H I B I T 7.3 A Matrix of Question Options
Question Focus Past Present Future
Behaviors/experiences
Opinons/values
FeeMngs/emotions
Knowledge
Sensory
Background
information is most important to obtain. To are you currently working on in school?"

understand how these options are applied in Such questions ask for relatively straightfor-
an actual study, it may be helpful to review a ward descriptions; they require minimal re-
real interview. The Outward Bound stan- call and interpretation. Such questions are, it
dardized interview in Appendix 7.2 at the is hoped, fairly easy to answer. They encour-
end of this chapter can be used for this pur- age the respondent to talk descriptively.
pose. Try identifying which cell in the matrix Probes should focus on eliciting greater de-
(Exhibit 7.3) is represented by each question tailfilling out the descriptive picture.
in the Outward Bound interviews. Once some experience or activity has
been described, then opinions and feelings
can be solicited, building on and probing for
Sequencing Questions
interpretations of the experience. Opinions
No recipe for sequencing questions can or and feelings are likely to be more grounded
should exist, but the matrix of questions and meaningful once the respondent has
suggests some possibilities. The challenges verbally "relived" the experience. Knowl-
of sequencing vary, of course, for different edge and skill questions also need a context.
strategies of interviewing. Informal conver- Such questions can be quite threatening if
sational interviewing is flexible and respon- asked too abruptly. The interviewer doesn't
sive so that a predetermined sequence is sel- want to come across as a TV game show host
dom possible or desirable. In contrast, quizzing a contestant. So, for example, in
standardized open-ended interviews must evaluation interviewing, it can be helpful to
establish a fixed sequence of questions to fit ask knowledge questions ("What are the eli-
their structured format. I offer, then, some gibility requirements for this program?") as
suggestions about sequencing. follow-up questions about program activi-
I prefer to begin an interview with ques- ties and experiences that have a bearing on
tions about noncontroversial present behav- knowledge and skills ("How did you be-
iors, activities, and experiences like "What come part of the program?"). Finding out
from people what they know works best interviewed. How a question is worded and
once some rapport and trust have been es- asked affects how the interviewee responds.
tablished in the interview. As Payne (1951) observed in his classic book
Questions about the present tend to be on questioning, asking questions is an art.
easier for respondents than questions about In qualitative inquiry, "good" questions
the past. Future-oriented questions involve should, at a minimum, be open-ended, neu-
considerable speculation, and responses to tral, singular, and clear. Let's look at each of
questions about future actions or attitudes these criteria.
are typically less reliable than questions
about the present or past. I generally prefer
to begin by asking questions about the pres- Asking Truly
ent, then, using the present as a baseline, ask Open-Ended Questions
questions about the same activity or attitude
in the past. Only then will Ibroach questions Qualitative inquirystrategically, philo-
about the future. sophically, and therefore, methodologically
Background and demographic questions aims to minimize the imposition of prede-
are basically boring; they epitomize what termined responses when gathering data. It
people hate about interviews. They can also follows that questions should be asked in a
be somewhat uncomfortable for the respon- truly open-ended fashion so people can re-
dent, depending on how personal they are. I spond in their own words.
keep such questions to a minimum and pre- The standard fixed-response item in a
fer to space them strategically and unobtru- questionnaire provides a limited and prede-
sively throughout the interview. I advise termined list of possibilities: "How satisfied
never beginning an interview with a long list are you with the program? (a) very satisfied,
of routine demographic questions. In quali- (b) somewhat satisfied, (c) not too satisfied,
tative interviewing, the interviewee needs to (d) not at ali satisfied." The closed and limit-
become actively involved in providing de- ing nature of such a question is obvious to
scriptive information as soon as possible in- both questioner and respondent. Many re-
stead of becoming conditioned to providing searchers seem to think that the way to make
short-answer, routine responses to uninter- a question open-ended is simply to leave out
esting categorical questions. Some back- the structured response categories. But do-
ground information may be necessary at the ing so does not make a question truly
beginning to make sense out of the rest of the open-ended. It merely disguises what still
interview, but such questions should be tied amounts to a predetermined and implicit
to descriptive information about present life constraint on likely responses.
experience as much as possible. Otherwise, Consider the question "How satisfied are
save the sociodemographic inquiries (age, you with this program?" Asked without
socioeconomic status, birth order, and the fixed response choices, this can appear to be
like) for the end. an open-ended question. On closer inspec-
tion, however, we see that the dimension
along which the respondent can answer has
Is. Wording Questions already been identifieddegree of satisfac-
tion. The interviewee can use a variety of
An interview question is a stimulus aimed modifiers for the word satisfaction"pretty
at eliciting a response from the person being satisfied," "kind of satisfied," "mostly satis-
fied," and so on. But, in effect, the possible dures for enrolling in the program? Have
response set has been narrowly limited by you interacted much with the staff in the
the wording of the question. The typical program? By their grammatical form, ali of
range of answers will vary only slightly these questions invite a yes/no reply.
more than what would have been obtained In contrast, in-depth interviewing strives
had the categories been made explicit from to get the person being interviewed to talk
the start while making the analysis more to talk about experiences, feelings, opin-
complicated. ions, and knowledge. Far from encouraging
A truly open-ended question does not the respondent to talk, dichotomous re-
presuppose which dimension of feeling or sponse questions limit expression. They can
thought will be salient for the interviewee. even create a dilemma for respondents who
The truly open-ended question allows the may not be sure whether they are being
person being interviewed to select from asked a simple yes/no question or if, in-
among that person's full repertoire of possi- deed, the interviewer expects a more elabo-
ble responses those that are most salient. In- rate response. Often, in teaching interview-
deed, in qualitative inquiry one of the things ers and reviewing their fieldwork, I've
the inquiry is trying to determine is what di- found that those who report having diffi-
mensions, themes, and images/ words peo- culty getting respondents to talk are posing
ple use among themselves to describe their a string of dichotomous questions that pro-
feelings, thoughts, and experiences. Exam- gram the respondent to be largely reactive
ples, then, of truly open-ended questions and binary.
would take the following format: Consider this classic exchange between a
parent and teenager. Teenager returns home
How do you feel about ? from a date:
What is your opinion of ? Do you know that you're late?
What do you think of ? Yeah.
Did you have a good time?
The truly open-ended question permits
Yeah.
those being interviewed to take whatever di-
Did you go to a movie?
rection and use whatever words they want
to express what they have to say. Moreover, Yeah.
to be truly open-ended a question cannotbe Was it a good movie?
phrased as a dichotomy. Yeah, it was okay.
So, it was worth seeing?
Yeah, it was worth seeing.
The Horns of a Dichotomy
I've heard a lot about it. Do you think I
Dichotomous response questions pro- would like it?
vide the interviewee with a grammatical I don't know. Maybe.
structure suggesting a "yes" or "no" answer. Anything else happen?
Are you satisfied with the program? Have
No. That's about it.
you changed as a result of your participation
in this program? Was this an important ex- Teenager then goes off to bed. One parent
perience for you? Do you know the proce- turns to the other and says, "Sure is hard to
get him to talk to us. I guess he's at the age INTERVIEW

where kids just don't want to tell their par- DEMONSTRATION
ents anything."
Dichotomous questions can turn an inter- Instruction: Okay, now we're going to
view into an interrogation or quiz rather play an interviewing game. I want you to
than an in-depth conversation. In everyday ask me questions about an evaluation I just
conversation, our interactions with each completed. The program being evaluated
other are filled with dichotomous questions was a staff development demonstration pro-
that we unconsciously ignore and treat as if ject that involved taking professionals into a
they were open-ended questions. If a friend wilderness setting for a week. That's ali l'm
asks, "Did you have a good time?" you're going to tell you at this point. TH answer
likely to offer more than a yes/no answer. In your questions as precisely as I can, but IT1
a more formal interview setting, howe ver, only answer what you ask. I won't volunteer
the interviewee will be more conscious of any information that isn't directly asked for
the grammatical structure of questions and by your questions.
is less likely to elaborate beyond "yes" or The questions on the left (next page) illus-
"no" when hit with dichotomous queries. trate a fairly extreme example of posing di-
Indeed, the more intense the interview situa- chotomous questions in an interview. Notice
tion, the more likely the respondent reacts to that the open-ended questions on the right
the "deep structure" stimulus of questions side generate richer answers and quite dif-
which includes their grammatical fram- ferent information than was elicited from
ingand to take questions literally (Bandler the dichotomous questions. In addition, di-
and Grinder 1975a, 1975b). chotomous questions can easily become
In training interviewers, I like to play a leading questions. Once the interviewer be-
game where I will only respond literally to gins to cope with what appears to be a reluc-
the questions asked without volunteering tant or timid interviewee by asking ever
any information that is not clearly de- more detailed dichotomous questions,
manded in the question. I do this before ex- guessing at possible responses, the inter-
plaining the difficulties involved in asking viewer may actually impose those responses
dichotomous questions. I have played this on the interviewee. One sure sign that this is
game hundreds of times and the interaction happening is when the interviewer is doing
seldom varies. When getting dichotomous more talking than the person being inter-
responses to general questions, the interviewed. Consider the excerpt on page 357
viewer will begin to rely on more and more from an actual interview. The interviewee
specific dichotomous response questions, was a teenager who was participating in a
thereby digging a deeper and deeper hole, chemical dependency program. The inter-
which makes it difficult to pull the interview view took place during the time the teenager
out of the dichotomous response pattern. was resident in the program.
Transcribed on page 356 is an actual inter- The person conducting this interview
view from a training workshop. In the left said she wanted to find out two things in this
column, I have recorded the interview that portion of the interview: What experiences
took place; the right column records truly were most salient for John and how person-
open-ended alternatives to the dichotomous ally involved John was becoining in the ex-
questions that were asked. perience. She has learned that the hot seat
Actual Questions Asked Genuinely Open-Ended Alterna tives With Richer Responses
Question: Were you doing a Q: What were the purposes of the evaluation?
formative evaluation?
Answer: Mostly. A: First, to document what happened; then to provide feedback to staff and help
them identify their "model"; and finaliy to report to funders.
Q: Were you trying to find out if Q: What were you trying to find out through the evaluation?
the people changed from being
in the wilderness?
A: That was part of it. A: Several things. How participants experienced the wilderness, how they talked
about the experience, what meanings they attached to what they experienced,
what they did with the experience when they returned home, and any ways in
which it affected them.
Q: Did theychange? Q: What did you find out? How did participation in the program affect participants?
A: Some of them did. A: Many participants reported "transformative" experiencestheir termby which
they meant something iife-changing. Others became more engaged in experiential
education themselves. A few reported just having a good time. You'd need to read
the full case studies to see the depth of variation and impaets.
Q: Did you interview people both Q: What kinds of information did you collect for the evaluation?
before and after the program?
A: Yes. A: We interviewed participants before, during, and after the program; we did focus
groups; we engaged n participant observation with conversational interviews;
and we read their journals when they were willing. They also completed
open-ended evaluation forms that asked about aspects of the program.
Q: Did you find that being in Q: How do you think your participation in the program affected what happened?
the program affected what
happened?
A: Yes. A: We've reflected a lot on that and we talked with staff and participants about it.
Most agreed that the evaluation process made everyone more intentional and
reflective-and that increased the impact in many cases.
Q: Did you have a good time? Q: What was the wilderness experience ke for you?
A: Yes. A: First, learned a great deal about participant observation and evaluation. Second,
carne to love the wilderness and have become an avid hiker. Third, I began what 1
expect will be a deep and lifelong friendshp with one staff member.
was highly salient for John, but she really One person does it every day.
knows very little about the reasons for that
salience. With regard to the question of his Yeah, it depends.
personal involvement, the only data she has Okay, let's see, hmmm... there was this guy
come from his acquiescence to leading ques- yesterday who really got nailed. I mean
tions. In fact, if one lists the actual data from he really caught a lot of crap from the
the interviewhis verbatim responses group. It was really heavy.
there is very little there:
No, it was them others.
Okay.
Yeah, right, and it really got to him.
Yeah,... the hot seat.
He started crying and got mad and one guy
Right. really came down on him and afterwards
interview Comments
Question: Helo, John. it's nice to scc you again. !'m The opening is dominated by the interviewer. No informal
anxious to find out what's been happening with you. give-and-take. The interviewee is set up to take a passve/
Can I ask you some questions about your experience? reactive role.
Answer: Okay.
Q: l'd like you to think about some of the really important Introductory cue sentence is immediateiy foltowed by a
experiences youVe had here. Can you think of some- dichotomous response question.
thing thatstands out in your mind?
A: Yeah,... the hot seat. John goes beyond the dichotomous response.
Q: The hot seat is when one person is the focus of The interviewer has provided the definition rather than
attention for the whole group, right? getting John's own definition of the hot seat.
A: Right.
Q: So, what was it like . . . ? Was this the first time Began open-ended, then changed the question and posed
you've seen the hot seat used? a dichotomous question. The question is no longer singular
or open.
A: One person does it every day. Not really an answer to the question.
Q: Is it different with different people? Question foilows previous answer but still a dichotomous
format.
A: Yeah, it depends.
Q: Can you tell me about one that really stands out in "Can you?" Is this an inquiry about willingness or memory
your mind? or capacity or trust?
A: Okay, let's see, hmm . . . there was this guy yesterday Before responding to the open request, John reacts to the
who really got nailed. I mean, he really caught a lot dichotomous format.
of crap from the group. It was really heavy.
Q: Did you say anything? Dichotomous question.
A: No, it was them others.
Q: So what was it like for you? Did you get caught up in Multiple questions. Unclear connections. Ambiguous,
it? You said it was really heavy, Was it heavy for you multiple-choice format at the end.
or just the group?
A: Yeah, right, and it really got to him. John's positive answer ("Yeah, right") is actually
unsnterpretable, given the questions asked.
Q: Did you think it was good for him? Did it help him? Dichotomous questions.
A: He started crying and got mad and one guy really The question asks for a judgment. John wants to describe
carne down on him and afterwards they were talking what happened. The narrowness of the interview questions
and it seemed to be okay for him. are limiting his responses.
Q: So it was really intense? Leading question, setting up an easy acquiescence response.
A: Yeah, it really was. Acquiesces to leading question. Accepts interviewer's term,
"intense," so we don't learn what word he would have
chosen.
Q: And you got really involved. Another leading question.
A: It was pretty heavy. John doesn't actually respond to the question. Ambiguous
response.
Q: Okay, 1 want to ask you some about the lecture part of Transition. John is cued that the hot seat questions are over.
the program. Anything else you want to say about the No response really expected.
hot seat? (John doesn't answer verbaliy. Sits and waits
for the next questions.)
they were talking, and it seemed to be The lack of a coherent story line in these
okay for him. responses reveals how little we've actually
learned about John's perspective and expe-
Yeah, it really was.
riences. Study the transcript and you'll find
It was pretty heavy. that the interviewer was talking more than
the interviewee. The questions put the inter- the programwhat you think are the
viewee in a passive stance, able to confirm or strengths and weaknesses of the program.
deny the substance provided by the inter- What you like. What you don't like. What you
viewer but not really given the opportunity think could be improved or should stay the
to provide in-depth, descriptive detail in his same. Those kinds of thingsand any other
own words. comments you have.
Asking Singular Questions The evaluator who used this question regu-
larly in interviewing argued thatby asking a
series of questions, he could find out which
One of the basic rules of questionnaire
was most salient to the person being inter-
writing is that each item must be singular;
viewed because the interviewee was forced
that is, no more than one idea should be con-
to choose what he or she most cared about in
tained in any given question. Consider this
order to respond to the question. The evalu-
example:
ator would then probe more specifically in
those areas that were not answered in the
How well do you know and like the staff in
initial question.
this program?
Ifs necessary to distinguish, then, be-
(a) a l o t
tween giving an overview of a series of ques-
(b) pretty much
tions at the beginning of a sequence, then
(c) not too much
asking each one singularly, and laying out a
(d) not at ali
whole series of questions at once and seeing
which one strikes a respondenfs fancy. In
This item is impossible to interpret in analy- my experience, multiple questions create
sis because it asks two questions: tension and confusion because the person
being interviewed doesn't really know what
(1) How well do you know the staff? is being asked. An analysis of the strengths
(2) How much do you like the staff? and weaknesses of a program is not the same
as reporting what one likes and dislikes
When one turns to open-ended inter- about a program. Likewise, recommenda-
viewing, however, many people seem to tions for change may be unrelated to
think that singular questions are no longer strengths, weaknesses, likes, and dislikes.
called for. Precision gives way to vagueness The following is an excerpt from an inter-
and confused multiplicity, as in the illustra- view with a parent participating in a family
tion in this section. I've seen transcripts of in- education program aimed at helping par-
terviews conducted by experienced and ents become more effecve as parents.
well-known field researchers in which sev-
eral questions have been thrown together,
which they might think are related, but Question: Based on your experience, what
which are likely to confuse the person being would you say are the strengths of this
interviewed about what is really being program?
asked. Answer: The other parents. Different par-
ents can get together and talk about what
To help the staff improve the program, we'd being a parent is like for them. The pro-
like to ask you to talk about your opinion of gram is really parents with parents. Par-
Asking Ciear,
Singular Questions / ' N o w ! the first question l'd like to ask, just to get us started is
\ how you happened to find out about the program; what it
| was like when you first came--you know, how you fee about
. I it now, and what impact it's had on you...?
ents really need to talk to other parents

A: I'd put the staff right at the top of that. I
about what they do, and what works and
really like the program director. She's re-
doesn't work. It's the parents, it really is.
ally well educated and knows a lot, but
she never makes us feel dumb. We can
Q: What about weaknesses? say anything or ask anything. She treats
us like people, like equals even. I like the
A: I don't know . . . I guess I'm not always other parents. And I like being able to
sure that the program is really getting to bring my daughter along. They take her
the parents who need it the most. I don't into the child's part of the program, but
know how you do that, but I just think we also have some activities together.
there are probably a lot of parents out But it's also good for her to have her ac-
there who need the program and . . . es- tivities with other kids and I get some
pecially maybe single-parent families. time with other parents.
And fathers. It's really hard to get fa-
thers into something like this. It should Q: What about dislikes? What are some
just get to everybody and tht's real things you don't like so much about the
hard. program?
Q: Let me turn now to your personal likes A: I don't like the schedule much. We meet
and dislikes about the program. What in the afternoons after lunch and it kind
are some of the things that you have re- of breaks into the day at a bad time for
ally liked about the program? me, but there isn't any really good time
for ali the parents and I know they've Notice that each of these questions solic-
tried different times. Time is always go- ited a different response. Strengths, weak-
ing to be a hassle for people. Maybe they nesses, likes, dislikes, and recommendations
could just offer different things at differ- each question meant something differ-
ent times. The room we meet in isn't too ent and deserved to be asked separately.
great, but that's no big deal. Qualitative interviewing can be deepened
through thoughtful, focused, and distinct
Q: Okay, you've given us a lot of informa-
questions.
tion about your experiences in the pro-
A consistent theme runs through this dis-
gram, strengths and weaknesses you've
cussion of question formulation: The word-
observed, and some of the things you've
ing used in asking questions can make a
liked and haven't liked so much. Now I'd
significant difference in the quality of re-
like to ask you about your recommenda-
sponses elicited. The interviewer who
tions for the program. If you had the
throws out a bunch of questions ali at once to
power to change things about the pro-
see which one takes hold puts an unneces-
gram, what would you make different?
sary burden on the interviewee to decipher
A: Well, I guess the first thing is money. It's what is being asked. Moreover, multiple
always money. I just think they should questions asked at the same time suggest
put, you know, the legislature should that the interviewer hasn't figured out what
put more money into programs like this. question should be asked at that juncture in
I don't know how much the director gets the interview. Taking the easy way out by
paid, but I hear that she's not even get- asking several questions at once transfers
ting paid as much as schoolteachers. She the burden of clarity from the interviewer to
should get paid like a professional. I the interviewee.
think there should be more of these pro- Asking several questions at once can also
grams and more money in them. waste precious interview time. Given multi-
Oh, I know what I'd recommend. We ple stimuli and not being sure of the focus of
talked about it one time in our group. It the question, the interviewee is free to go off
would be neat to have some parents in any direction at ali, including tangents
who have already been through the pro- that are irrelevant to the issues under study.
gram come back and talk with new In evaluation interviews, for example, both
groups about what they've done with interviewers and respondents typically
their kids since they've been in the pro- have only so much time to give to an inter-
gram, you know, like problems that they view. To make the best use of that time, it is
didn't expect or things that didn't work helpful to think through priority questions
out, or just getting the benefit of the ex- that will elicit relevant responses. This
periences of parents who've already means that the interviewer must know what
been through the program to help new issues are important enough to ask ques-
parents. We talked about that one day tions about, and to ask those questions in a
and thought that would be a neat thing way that the person being interviewed can
to do. I don't know if it would work, but clearly identify what he or she is being
it would be a neat thing. I wouldn't askedthat is, to ask clear questions.
mind doing it, I guess.
Clarity of Questions
I f names are not correct, language will not be in accordance with the
C ^ / truth of things.
-Confucius
The interviewer bears the responsibility dated by the rulers in Ouagadougou was
to pose questions that make it clear to the in- "the people's government."
terviewee what is being asked. Asking un- Second, clarity can be sharpened by un-
derstandable questions facilitates establish- derstanding what language participants
ing rapport. Unclear questions can make the use among themselves in talking about a
person being interviewed feel uncomfort- setting, activities, or other aspects of life.
able, ignorant, confused, or hostile. Asking When we interviewed juveniles who had
singular questions helps a great deal to beenplaced in foster group homes by juve-
make things clear. Other factors also contrib- nile courts, we had to spend a good deal of
ute to clarity. preparatory time trying to find out how
First, in preparing for an interview, find the juveniles typically referred to the
out what special terms are commonly used group home parents, to their natural par-
by people in the setting. For example, state ents, to probation officers, and to each
and national programs often have different other in order to ask questions clearly
titles and language at the local levei. CETA about each of those sets of people. For ex-
(Comprehensive Employment and Train- ample, when asking about relationships
ing Act) was designed as a national program with peers, should we use the word juve-
in which local contractors were funded to es- niles, adolescents, youth, teenagers, or w h a t ?
tablish and implement services in their area. In preparation for the interviews, we
We found that participants only knew these checked with a number of juveniles, group
programs by the name of the local contrac- home parents, and court authorities about
tor, such as "Youth Employment Services/' the proper language to use. We were ad-
"Work for Youth," and "Working Opportu- vised to refer to "the other kids in the
nities for Women." Many participants in group home." However, we found no con-
these programs did not even know they sensus about how the kids in the group
were in CETA programs. Conducting an in- home referred to group home parents.
terview with these participants where they Thus, one of the questions we had to ask in
were asked about their "CETA experience" each interview was, "What do you usually
would have been confusing and disruptive call Mr. and Mrs. ?" We then used the
to the interview. language given to us by that youth
When I was doing fieldwork in Burkina throughout the rest of the interview to re-
Faso, the national government was run by fer to group home parents.
the military after a coup d'tat. Local offi- Third, providing clarity in interview
cials carried the title "commandant" (com- questions may mean avoiding using labels
mander). However, no one referred to the altogether. This means that when asking
government as a military government. To do about a particular phenomenon or pro-
so was not only politically incorrect but gram component, it may be better to first
risky. The appropriate official phrase man- find out what the interviewee believes that
phenomenon to be and then ask questions same children would be working on pro-
about the descriptions provided by the per- jects, such as the construction of a model of
son being interviewed. In studying officially their town using milk cartons that required
designated "open classrooms" in North Da- geometry, fractions, and reductions to scale,
kota, I interviewed parents who had chil- but they did not perceive of these activities
dren in those classrooms. However, many of as math because they associated math with
the teachers and local school officials did not worksheets and workbooks. Thus, to find
use the term "open" to refer to these class- out that kind of math activities children
rooms because they wanted to avoid politi- were doing, it was necessary to talk with
cal conflicts and stereotypes that were some- them in detail about specific projects and
times associated with the notion of "open work they were engaged in without asking
education." Thus, when interviewing par- them the simple question "What kind of
ents we could not ask their opinions about math do you do in the classroom?"
open education. Rather, we had to pursue a Another example of problems in clarity
sequence of questions like the following: comes from foliow-up interviews with
mothers whose children were victims of sex-
What kinds of differences, if any, have you no- ual abuse. A major part of the interview fo-
ticed between your child's classroom last year cused on experiences with and reactions to
and the classroom this year? (Parent re- the child protection agency, the police, the
sponds.) welfare workers, the court system, the
OK, you've mentioned several differences. school counselor, probation officers, and
Let me ask you your opinion about each of the other parts of the enormously complex sys-
things you've mentioned. What do you think tem constructed to deal with child sexual
about ? abuse. We learned quickly that mothers
could seldom differentiate the parts of the
This strategy avoids the probiem of col- system. They didn't know when they were
lecting data that later turn out to be uninter- dealing with the courts, the child protection
pretable because you can't be sure what re- people, the welfare system, or some treat-
spondents meant by their responses. Their ment program. It was ali "the system." They
opinions and judgments are grounded in de- had strong feelings and opinions about the
scriptions, in their own words, of what system, so our questions had to remam gen-
they've experienced and what they're as- eral, about the system, rather than specifi-
sessing. cally asking about the separate parts of the
A related probiem emerged in interview- system (Patton 1991).
ing children about their classrooms. We The theme running through these sugges-
wanted to find out how basic skills were tions for increasing the clarity of questions
taught in open classrooms. In preparing for centers on the importance of using language
the interviews, we learned that many teach- that is understand able and part of the frame
ers avoided such terms as "math time" or of reference of the person being interviewed.
"reading time" because they wanted to inte- It means taking special care to find out what
grate math and reading into other activities. language the interviewee uses. Questions
In some cases, we learned during parent in- that use the respondent's own language are
terviews, children reported to parents that most likely to be clear. This means being sen-
they didn't do any math in school. These sitive to "languaculture" by attending to
"meanings that lead the researcher beyond Why to Take Care Asking " W h y ? "
the words into the nature of the speaker's
world" (Agar 2000:93-94). This sensitivity to Three Zen Masters were discussing a flapping
local language, the emic perspective in an- flag on a pole. The first observed dryly: "The
thropology is usually discussed m relation flag moves."
to data analysis in which a major focus is il- " N o , " said the second, "wind is moving."
luminating a setting or culture through its
" N o , " said the third. "It is not flag. It is not
language. Here, however, we're discussing
wind. It is mind moving."
languaculture, not as an analytical frame-
work but as a way of enhancmg data collec-
"Why" questions presuppose that things
tion during interviewing by increasmg clar-
happen for a reason and that those reasons
ity, communicating respect, and facilitatmg
are knowable. "Why" questions presume
rapport.
cause-effect relationships, an ordered world,
Using words that make sense to the inter-
and rationality. "Why" questions move be-
viewee, words that reflect the respondent's
yond what has happened, what one has ex-
worldview, will improve the quality of data
perienced, how one feels, what one opines,
obtained during the interview. Without sen-
and what one knows to the making of ana-
sitivity to the impact of particular words on
lytical and deductive inferences.
the person being interviewed, an answer
may make no sense at alior there may be The problems in deducing causai infer-
no answer. A Sufi story makes this point ences have been thoroughly explored by
quite nicely. philosophers of science (Bunge 1959; Nagel
1961). On a more practical levei and more
illuminative of interviewing challenges, re-
A man had fallen between the rails in a sub-
ports from parents about "why" conversa-
way station. People were ali crowding around
tions with their children document the diffi-
trying to get him out before the train ran him
culty of providing causai explanations about
over. They were ali shouting. "Give me your
the world. The infinite regression quality of
hand!" but the man would not reach up.
"why" questions is part of the difficulty en-
Mulla Nasrudin elbowed his way through gendered by using them as part of an inter-
the crowd and leaned over the man. "Friend," view. Consider this parent-child exchange:
he asked, "what is your profession?"
"I am an income tax inspector," gasped the
Dad, why does it get dark at night?
man.
"In that case," said Nasrudin, "take my Because our side of the earth turns away
hand!" from the sun.
The man immediately grasped the Mulla's Dad, why does our side of the earth turn
hand and was hauled to safety. Nasrudin away from the sun?
turned to the amazed by-standers. "Never ask
a tax man to give you anything, you fools," he Because that's the way the world was made.
said. (Shah 1973:68) Dad, why was the world made that way?
So that there would be light and dark.
Before leaving the issue of clarity, let me
offer one other suggestion: Be especially Dad, why should there be dark? Why can't it
careful asking "why" questions. just be light ali the time?
Because then we would get too hot. "Because it takes place at a convenient
time." (programmatic reason)
Why would we get too hot?
a "Because Fm a joiner." (personality rea-
Because the sun would be shining on us ali
son)
the time.
a "Because a friend told me about the pro-
Why can't the sun be cooler sometimes?
gram." (information reason)
It is, thafs why we have night. a "Because my priest told me about the
But why can't we just have a cooler sun? program and said he thought it would be
good for me." (social influence reason)
Because thafs the way the world is.
a "Because it was inexpensive." (economic
Why is the world like that?
reason)
It just is. Because. H "Because I wanted to learn about the
Because why? things they're teaching in the program."
{outcomes reason)
Just because.
o "Because God directed me to join the
Oh.
program." (personal motivation reason)
Daddy?
o "Because it was there." (philosophical rea-
Yes. son)
Why don't you know why it gets dark?
Anyone being interviewed could re-
spond at any or ali of these leveis. The inter-
In a program evaluation interview, it
viewer must decide before conducting the
might seem that the context for asking a
interview which of these leveis carries suffi-
"why" question would be clearer. However,
cient importance to make it worth asking a
if a precise reason for a particular activity is
question. If the primary evaluation question
what is wanted, it is usually possible to ask
concerns characteristics of the program that
that question in a way that does not involve
attracted participants, then instead of ask-
using the word why. Let's look first at the dif-
ing, "Why did you join?" the interviewer
ficulty posed for the respondent by the
should ask something like the following:
"why" question, and then look at some alter-
"What was it about the program that at-
native phrases.
tracted you to it?" If the evaluator is inter-
"Why did you join this program?" The ac-
ested in learning about social influences that
tual reasons for joining the program proba-
led to participation in a program, either vol-
bly consist of some constellation of factors,
untary or involuntary participation, a ques-
including the influences of other people, the
tion like the following could be used:
nature of the program, the nature of the per-
son being interviewed, the interviewee's ex-
pectations, and practical considerations. It is Other people sometimes influence what we
unlikely that an interviewee can sort do. What other people, if any, played a role in
through ali of these leveis of possibility at your joining this program?
once, so the person to whom the question is
posed must pick out some levei at which to In some cases, the evaluator may be par-
respond. ticularly interested in the characteristics of
participants, so the question might be "Because we go outside and play on the

phrased in the foliowing fashion:
swings."
"Why do you go outside?" I asked.
I'm interested in learning more about you as a
"Because that's where the swings are!"
person and your personal involvement in this
program. What is it about youyour situa- She replied with a look of incredulity that
tion, your personality, your desires, what- adults could ask such stupid questions, then
everwhat is it about you that you think led explained helpfully: "If you want to swing
you to become part of this program? on the swings, you have to go outside where
the swings are."
Children take interview questions quite
When used as a probe, "why" questions
literally and so it becomes clear quickly
can imply that a person's response was
when a question is not well thought out. It
somehow inappropriate. "Why did you do
was during those days of interviewing chil-
that?" may sound like doubt that an action
dren in North Dakota that I learned about
(or feeling) was justified. A simple "Tell me
the problems with "why" questions.
more, if you will, about your thinking on
that" may be more inviting.
The point is that by thinking carefully Rapport and Neutrality
about what you want to know, there is a
greater likelihood that respondents will sup- Neutral Questions
ply answers that make senseand are rele-
vant, usable, and interpretable. My cautions As an interviewer, I want to establish rap-
about the difficulties raised with "why" port with the person I am questioning, but
questions come from trying to analyze such that rapport must be established in such a
questions when responses covered such a way that it does not undermine my neutral-
multitude of dimensions that it was clear ity concerning what the person tells me.
different people were responding to differ- Neutrality means that the person being in-
ent things. This makes analysis unwieldy. terviewed can tell me anything without en-
gendering either my favor or disfavor with
Perhaps my reservations about the use of
regard to the content of her or his response. I
"why" questions come from having ap-
cannot be shocked; I cannot be angered; I
peared the fool when asking such questions
cannot be embarrassed; I cannot be sad-
during interviews with children. In our
dened. Nothing the person tells me will
open classroom interviews, several teachers
make me think more or less of the person.
had mentioned that children often became
so involved in what they were doing that At the same time that I am neutral with re-
they chose not to go outside for recess. We gard to the content of what is being said to
decided to check this out with the children. me, I care very much that that person is will-
ing to share with me what she or he is saying.
Rapport is a stance vis--vis the person be-
"What's your favorite time in school?" ing interviewed. Neutrality is a stance
I asked a first grader. vis--vis the content of what that person
"Recess," she answered quickly. says. Rapport means that I respect the peo-
ple being interviewed, so what they say is
"Why do you like recess?" important because of who is saying it. I want
Why Questions
e>
ra
5
is
TI
<5
C
c
O
lg
3
a
"S
<c
to convey to them that their knowledge, examples format. When phrasing questions in
periences, attitudes, and feelings are impor- this way I want to let the person I'm inter-
tant. Yet, I will not judge them for the content viewing know that I've pretty much heard it
of what they say to me. alithe bad things and the goodand so
Rapport is built on the ability to convey I'm not interested in something that is par-
empathy and understanding without judg- ticularly sensational, particularly negative,
ment. Throughout this chapter, we have or especially positive. I'm really only inter-
been considering ways of phrasing ques- ested in what that person's genuine experi-
tions that facilita te the establishment of rap- ence has been like. I want to elicit open and
port through mutual understanding. In this honest judgments from people without
section, I want to focus on ways of wording making them worry about my judging what
questions that are particularly aimed at con- they say.
veying that important sense of neutrality. An example of the illustrative examples
format is provided by a question taken from
Using Illustrative interviews we conducted with juvenile de-
Examples in Questions linquents who had been placed in foster
group homes. One section of the interview
One kind of question wording that can was aimed at finding out how the juveniles
help establish neutrality is the illustrative ex- were treated by group home parents.
Okay, now I'd like to ask you to tell me how and

you were treated in the group home by the
parents. Some kids have told us they were We've already heard that this place has lots of
treated like one of the family; some kids have troubles, so feel free to tell us about the trou-
told us that they got knocked around and beat bles you've seen.
up by the group home parents; some kids have
and
told us about sexual things that were done to
them; some of the kids have told us about fun I imagine it must be horrible to have a child
things and trips they did with the parents; abused and have to deal with the system, so
some kids have felt they were treated really you can be honest with me. How bad was it?
well and some have said they were treated
pretty bad. What about youhow have you
Each of these questions builds in a re-
been treated in the group home?
sponse bias that communicates the inter-
viewer^ belief about the situation prior to
A closely related approach is the illustra- hearing the respondenfs assessment. The
tive extremes formatgiving examples only questions are leading in the sense that the in-
of extreme responses. This question is from a terviewee can be led into acquiescence with
follow-up study of award recipients who re- the interviewer's point of view.
ceived a substantial fellowship with no In contrast, the questions offered above to
strings attached. demonstrate the illustrative examples for-
mat included several dimensions to provide
How much of the award, if any, did you spend balance between what might be construed
on entirely personal things to treat yourself as positive and negative kinds of responses.
well? Some fellows have told us they spent a I prefer to use the illustrative examples for-
sizable portion of the award on things like a mat primarily as a clarifying strategy after
new car, a hot tub, fixing up their house, per- having begun with a simple, straightfor-
sonal trips, and family. Others spent almost ward, and truly open-ended question:
nothing on themselves and put it ali into their "What do you think about this program?" or
work. How about you? "What has been your experience with the
system?" Only if this initial question fails to
In both the illustrative examples format elicit a thoughtful response, or if the inter-
and the illustrative extremes format, it is viewee seems to be struggling, will I offer il-
criticai to avoid asking leading questions. lustrative examples to facilitate a deeper re-
Leading questions are the opposite of neu- sponse.
tral questions; they give the interviewee
hints about what would be a desirable or ap-
Role-Playing and
propriate kind of answer. Leading questions
Simulation Questions
"lead" the respondent in a certain direction.
Below are questions I found on transcripts
during review of an evaluation project car- Providing context for a series of questions
ried out by a reputable university center. can help the interviewee hone in on relevant
responses. A helpful context provides cues
We've been hearing a lot of really positive about the levei at which a response is ex-
comments about this program. So what's your pected. One way of providing such a context
assessment? is to role-play with persons being inter-
viewed, asking them to respond to the inter- were contemplating suicide. What would you
viewer as if he or she were someone else. tell them?
Suppose I was a new person who just came

The first question comes across as abrupt
into this program, and I asked you what I
and demanding, almost like an examination
should do to succeed here. What would you
to see if they know the right answer. The sec-
tell me?
ond question, with the interviewee allowed
or to create a personal context, is softened and,
it is hoped, more inviting. While this tech-
Suppose I was a new kid in this group home, nique canbe overused and can sound phony
and I didn't know anything about what goes if asked insensitively, with the right intona-
on around here. What would you tell me about tion communicating genuine interest and
the rules that I have to be sure to follow? used sparingly, with subtlety, the role-play-
ing format can ease the asking of difficult
These questions provide a context for questions to deepen answers and enhance
what would otherwise be quite difficult the quality of responses.
questions, for example, "How does one get Simulation questions provide context in a
the most out of this program?" or "What are different way, by asking the person being in-
the rules of this group home?" The terviewed to imagine himself or herself in
role-playing format emphasizes the inter- the situation about which the interviewer is
viewees' expertise; that is, itputs them in the interested.
role of expert because they know some thing
of value to someone else. The interviewee is Suppose I was present with you during a staff
the insider xuith inside information. The inter- meeting; what would I see going on? Take me
viewer, in contrast, as an outsider, takes on there.
the role of novice or apprentice. The "ex-
pert" is being asked to share his or her exper- or
tise with the novice. I've often observed in-
terviewees become more animated and Suppose I was in your classroom at the begin-
engaged when asked role-playing ques- ning of the day when the students first come
tions. They get into the role. in. What would I see happening as the stu-
A variation on the role-playing format in- dents came in? Take me to your classroom and
volves the interviewer dissociating some- let me see what happens during the first 10 to
what from the question to make it feel less 15 minutes as the students arrive, what you'd
personal and probing. Consider these two be doing, what they'd be doing, what those
difficult questions for a study of a tough sub- first 15 minutes are like.
ject: teenage suicide.
In effect, these questions ask the inter-
What advice would you give someone your
viewee to become an observer. In most cases,
age who was contemplating suicide?
a response to this question will require the
versus interviewee to visualize the situation to be
described. I frequently find that the richest
Think of someone you know and like who is and most detailed descriptions come from a
moody. Suppose that person told you they series of questions that ask a respondent to
reexperience and/or simulate some aspect Used in interviewing, presuppositions

of an experience. communicate that the respondent has some-
thing to say, thereby increasing the likeli-
hood that the person being interviewed will,
Presupposition Questions indeed, have something to say. Consider the
following question: "What is the most im-
Presupposition questions involve a twist portant experience you have had in the pro-
on the theme of empathic neutrality. Presup- gram?" This question presupposes that the
positions have been identified by linguists respondent has had an important experi-
as a grammatical structure that creates rap- ence. The person of whom the question is
port by assuming shared knowledge and as- asked, of course, has the option of respond-
sumptions (Kartunnen 1973; Bandler and mg, "I haven't had any important experi-
Grinder 1975a). Natural language is filled ences." However, it is more likely that the in-
with presuppositions. In the course of our terviewee will go directly to the issue of
day-to-day communications, we often em- which experience to report as important,
ploy presuppositions without knowing rather than dealing first with the question of
we're doing so. By becoming aware of the ef- whether or not an important experience has
fects of presupposition questions, we can occurred.
use them strategically in interviewing. The Contrast the presupposition format
skillful interviewer uses presuppositions to "What is the most important experience you
increase the richness and depth of re- have had in the program?"to the follow-
sponses. ing dichotomous question: "Have you had
What then, are presuppositions? Lin- any experiences in the program that you
guists Grinder and Bandler define presup- would call really important?" This dichoto-
positions as follows: mous framing of the question requires the
person to make a decision about what an im-
When each of us uses a natural language sys- portant experience is and whether one has
tem to communicate, we assume that the lis- occurred. The presupposition format by-
tener can decode complex sound structures passes this initial step by asking directly for
into meanings, i.e., the listener has the ability description rather than asking for an affir-
to derive the Deep-Structure meaning from mation of the existence of the phenomenon
the Surface-Structure we present to him in question. Listed onpage 70, on the left, are
auditorily. . . . [W]e also assume the complex typical dichotomous response questions
skill of listeners to derive extra meaning from that are used to introduce a longer series of
some Surface-Structures by the nature of their questions. On the right are presupposition
form. Even though neither the speaker nor the questions thatbypass the dichotomous lead-
listener may be aware of this process, it goes in query and, in some cases, show how add-
on ali the time. For example, if someone says: I ing "if any" creates a more neutral framing.
want to watch Kung Fu tonight on TV we must A naturalness of inquiry flows from pre-
understand that Kung Fu is on TV tonight in suppositions making more comfortable
order to process the sentence "I want to watch what might otherwise be embarrassing or
. . ." to make any sense. These processes are intrusive questions. The presupposition in-
called presuppositions of natural language. cludes the implication that what is presup-
(Bandler and Grinder 1975a:241) posed is the natural way things occur. It is
l Alternative Question Formats
Dichotomous Lead-ln Question Presupposition Lead-ln Question
Do you feel you know enough about the How effective do you think the program is?
program to assess its effectiveness? (presupposes that a judgment can be made)
Have you learned anything from this What, if anything, have you learned from
program? this program? (presupposes that some learning
is likely)
Do you do anything now in your work that What, if anything, do you do now that you
you didn't do before the program began? didn't do before the program began?
(presupposes change)
Are there any conflicts among the staff? What kinds of staff conflicts have occurred here?
(presupposes conflicts)
natural for there to be conflict in programs. were much more likely to generate open re-
The presupposition provides a stimulus that sponses with the presupposition format
asks the respondent to mentally access the than with the dichotomous response format.
answer to the question directly without The purpose of in-depth interviews is to
making a decision about whether or not find out what someone has to say. By pre-
something has actually occurred. supposing that the person being inter-
I first learned about interview presuppo- viewed does, indeed, have something to say,
sitions from a friend who worked with the the quality of the descriptions received may
agency in New York City that had responsi- be enhanced. However, a note of warning:
bility for interviewing carriers of venereal Presuppositions, like any single form of
disease. His job was to find out about the car- questioning, can be overused. Presupposi-
riers previous sexual contracts so that those tions are one option. There are many times
persons could be informed that they might when it is more coinfortable and appropri-
have venereal disease. He had learned to ate to check out the relevance of a question
avoid asking men, "Have you had any sex- with a dichotomous inquiry ("Did you go to
ual relationships with other men?" Instead, the lecture?") before asking further ques-
he asked, "How many sexual contacts with tions ("What did you think of the lecture?").
other men have you had?" The dichotomous
question carried the burden for the respon-
dent of making a decision about some ad- Prefatory Statements
mission of homosexuality and/or promis- and Announcements
cuity. The presupposition form of the
open-ended question implied that some sex- Another technique for facilitating re-
ual contacts with other men might be quite sponses involves alerting the interviewee to
natural and focused on the frequency of oc- what is about to be asked before it is actually
currence rather than whether or not such asked. Think of it as warming up the respon-
sexual contacts have occurred at ali. The ve- dent, or ringing the interviewee's mental
nereal disease interviewers found that they doorbell. This is done with prefatory state-
ments that introduce a question. These can An alternative format is the summarizing
serve two functions. First, a preface alerts in- transition. This involvesbringing closure to a
terviewees to the nature of the question that section of the interview by summarizing
is coming, directs their awareness, and fo- what has been said and asking if they have
cuses their attention. Second, an introduc- anything to add or clarify before moving on
tion to a question gives respondents a few to a new subject.
seconds to organize their thoughts before re-
sponding. Prefaces, transition announce- Before we move on to the next set of questions,
ments, and introductory statements help let me make sure I've got everything you said
smooth the flow of the interview. Any of sev- about the program goals and objectives. You
eral formats can be used. said the program had five goals. First,... Sec-
The transition format announces that one ond, . . .
section or topic of the interview has been Before I ask you some questions about pro-
completed and a new section or topic is gram activities related to these goals, are there
about to begin. any additional goals or objectives that you
want to add?
We've been talking about the goals and objec-
tives of the program. Now I'd like to ask you
The summarizing transition lets the person
some questions about actual program activi-
being interviewed know that the inter-
ties. What are the major activities offered to cli-
viewer is actively listening and recording
ents in this program?
what is being said. The summary invites the
or interviewee to make clarifications, correc-
tions, or additions before moving on to a
We've been talking about your personal expe- new topic.
riences with this program. Now I'd like to ask The direct announcement format simply
your opinions about the program more gener- states what will be asked next. A preface to a
ally, specifically, about the program's question that announces its content can
strengths and weaknesses. Lefs begin with soften the harshness or abruptness of the
strengths. What would you say are the basic question itself. Direct prefatory statements
strengths of this program, from your point of help make an interview more conversa-
view? tional and easy-flowing. The transcriptions
below show two interview sequences, one
The transition format essentially says to
without prefatory statements and the other
the interviewee: "This is where we've been
with prefatory statements.
. . . and this is where we're going " Ques-
tions prefaced by transition statements help
maintain the smooth flow of an interview.
Question Without Preface Interview With Direct Preface
How have you changed as a result of the Now, let me ask you to think about any changes
program? you see in yourself as a result of participating in
this program. (pause) How, if at ali, have you been
changed by your experiences in this program?
The attention-getting preface goes beyond unique quality that makes it particularly
just announcing the next question to make a worthy of being answered.
comment about the question. The comment Making statements about the questions
may concern the importance of the question, being asked is a way for the interviewer to
the difficulty of the question, the openness engage in some conversation during the in-
of the question, or any other characteristic of terview without commenting judgmen-
the question that would help set the stage. tally on the answers being provided by the
Consider these examples: interviewee. What is said concems the ques-
tions and not the respondenfs answers. In
This next question is particularly important to this fashion, the interview can be made more
the program staff. How do you think the pro- interesting and interactive. However, ali of
gram could be improved? these formats must be used selectively and
strategically. Constant repetition of the same
or format or mechanical use of a particular ap-
proach will make the interview more, rather
This next question is purposefully vague so
than less, awkward.
that you can respond in any way that makes
sense to you. What difference has this pro-
gram made to the larger community? Probes and
Follow-Up Questions
or
This next question may be particularly diffi- Probes are used to deepen the response to
cult to answer with certainty, but I'd like to get a question, increase the richness and depth
your thoughts on it. In thinking about how of responses, and give cues to the inter-
you've changed during the last year, how viewee about the levei of response that is de-
much has this program caused those changes sired. The word probe is usually best avoided
compared to other influences on your life at in interviewsa little too proctological. The
this time? expression "Let me probe that further" can
sound as if you're conducting an investiga-
or
tion of something illicit or illegal. Quite sim-
This next question is aimed directly at getting ply, a probe is a follow-up question used to
your perspective. W h a f s itlike to be a client in go deeper into the interviewee's responses.
this program? As such, probes should be conversational,
offered in a natural style and voice, and used
or to follow up initial responses.
One natural set of conversational probes
As you may know, this next issue has been
consists of detail-oriented questions. These are
both controversial and worrisome. What kind
the basic questions that fill in the blank
of staff is needed to run a program like this?
spaces of a response.
The common element in each of these ex-

When did that happen?
amples is that some prefatory comment is
made about the question to alert the inter- Who else was involved?
viewee to the nature of the question. The Where were you during that time?
attention-getting format communicates that What was your involvement in that
the question about to be asked has some situation?
u
o
U
<5
s
"O
t
<3
c
o
|
3
a
a
.c
u

O

Hozv did that come about? respondent has to say. Gentle and strategic
WJere did that happen? head nodding is aimed at communicating
that you are listening and want to go on lis-
These etail-oriente probes are the basic tening.
"who/' "where/' "what/' "when/' and The verbal corollary of head nodding is
"how" questions that are used to obtain a the quiet "uh-huh." A combination may be
complete and detailed picture of some activ- necessary; when the respondent seems
ity or experience. about to stop talking and the interviewer
At other times, an interviewer may want would like to encourage more comment, an
to keep a respondent talking about a subject "uh-huh" combined with a gentle rocking of
by using elaboration probes. The best cue to the whole upper body can communicate in-
encourage continued talking is nonver- terest in having the interviewee elaborate.
balgently nodding your head as positive
Elaboration probes also have direct ver-
reinforcement. However, overenthusiastic
bal forms:
head nodding may be perceived as endorse-
ment of the content of a response or as want-
Would you elaborate on that?
ing the person to stop talking because the in-
terviewer has already understood what the Could you say some more about that?
That's helpful. I'd appreciate a bit more A major characteristic that separates
detail. probes from general interview questions is
Fm beginning to get the picture. (The im- that probes are seldom written out in an in-
plication is that I don't have the full terview. Probing is a skill that comes from
picture yet, so please keep talking.) knowing what to look for in the interview,
listening carefully to what is said and what
is not said, and being sensitive to the feed-
If something has been said that is ambigu- back needs of the person being inter-
ous or an apparent non sequitur, a clarifica- viewed. Probes are always a combina tion of
tion probe may be useful. Clarification probes verbal and nonverbal cues. Silence at the end
tell the interviewee that you need more in- of a response can indicate as effectively as
formation, a restatement of the answer, or anything else that the interviewer would
more context. like the person to continue. Probes are used
to communicate what the interviewer
You said the program is a "success." What do wants. More detail? Elaboration? Clarity?
you mean by success? Probes, then, provide guidance to the per-
son interviewed. They also provide the in-
I'm not sure I understand what you meant by terviewer with a way to maintain control of
that. Would you elaborate, please? the flow of the interview, a subject we now
turn to.
I want to make sure I understand what you're
saying. I think it would help me if you could
say some more about that. Process Feedback
During the Interview
A clarification probe should be used natu- Previous sections have emphasized the im-
rally and gently. It is best for the interviewer portance of careful wording so that inter-
to convey the notion that the failure to un- view questions are clear. Clear wording
derstand is the fault of the interviewer and concerns the content of the interview. This
not a failure by the person being intersection emphasizes feedback about how the
viewed. The interviewer does not want to process is going.
make the respondent feel inarticulate, stu- A good interview feels like a connection
pid, or muddled. After one or two attempts has been established in which communica-
at achieving clarification, it is usually best to tion is flowing two ways. Qualitative re-
leave the topic that is causing conf usion and search interviews differ from interrogations
move on to other questions, perhaps return- or detective-style investigations. The inter-
ing to that topic at a later point. viewer has a responsibility to communicate
Another kind of clarifying follow-up clearly what information is desired and why
question is the contrast probe (McCracken that information is important and to let the
1988:35). The purpose of a contrast probe is interviewee know how the interview is pro-
to "give respondents something to push off gressing.
against" by asking, "How does * compare to An interview is an interaction. Kvale
y?" This is used to help define the bound- (1996) has emphasized this point by calling
aries of a response. How does this experi- qualitative research interviews "Inter-
ence/feeling/action/term compare to some Views" to highlight that an interchange oc-
other experience/feeling/action/term? curs and a temporary interdependence is
created. The interviewer provides stimuli to understandand that's exactly why I wanted
generate a reaction. That reaction from the to interview you.
interviewee, however, is also a stimulus to
which the interviewer responds. You, as the You can get clues about what kind of re-
interviewer, must maintain awareness of inforcement is appropriate by watching the
how the interview is flowing, how the in- interviewee. When verbal and nonverbal
terviewee is reacting to questions, and behaviors indicate someone is really strug-
what kinds of feedback are appropriate gling with a question, going mentally deep
and helpful to maintain the flow of com- within, working hard trying to form an an-
munication. swer, after the response it can be helpful for
the interviewer to say, "I know that was a
Support and difficult question and I really appreciate
your working with it because what you said
Recognition Responses
was very meaningful and came out very
clearly."
A common mistake among novices is fail- At other times, you may perceive that
ing to provide reinforcement and feedback. only a surface or shallow answer has been
This means letting the interviewee know provided. It may then be appropriate to say
from time to time that the purpose of the in- something like the following: "I don't want
terview is being fulfilled. Words of thanks, to let that question go by without asking you
support, and even praise will help make the to think about it just a little bit more, because
interviewee feel that the interview process is I feel youVe really given some important de-
worthwhile and support ongoing rapport. tail and insights on the other questions and
I'd like to get more of your reflections about
Your comments about program weaknesses this question."
are particularly helpful, I think, because Iden- In essence, the interviewer, through feed-
tification of the kind of weaknesses you de- back, is "trainmg" the interviewee to pro-
scribe can really help in making changes in the vide high-quality and relevant responses.
program.
or Maintaining Control
and Enhancing the
It's really helpful to get such a clear statement
Quality of Responses
of what the program is like. That's just the
kind of thing we're trying to get at.
Time is precious in an interview. Long-
or winded responses, irrelevant remarks, and
digressions reduce the amount of time avail-
We're about halfway through the interview
able to focus on criticai questions. These
now and from my point of view, it's going very
problems exacerbate when the interviewer
well. You've been telling me some really im-
fails to maintain a reasonable degree of con-
portant things. How's it going for you?
trol over the process. Control is facilitated
or by (1) knowing what you want to find out,
(2) asking focused questions to get relevant
I really appreciate your willingness to express answers, (3) listening attentively to assess
your feelings about that. You're helping me the quality and relevance of responses, and
(4) giving appropriate verbal and nonverbal sion. Describe for me what the room looks like,
feedback to the person being interviewed. where the trainees are, where you are, and tell
Knowing what you want to find out me what I would see and hear if I were right
means being able to recognize and distin- there in that session. What would I see you do-
guish appropriate from inappropriate re- ing? What would I hear you saying? What
sponses. It is not enough just to ask the right would I see the trainees doing? What would I
questions. The interviewer must listen care- hear the trainees saying? Take me into a session
fully to make sure that the responses re- so that I can actually experience it.
ceived provide answers to the questions that
are asked. Consider the following exchange: It is the interviewer's responsibility to
work with the person being interviewed to
Question: What happens in a typical inter- facilitate the desired kind of responses. At
viewer training session that you lead? times, it may be necessary to give very direct
feedback about the kind of information that
Answer: I try to be sensitive to where each
has been received and the kind that is de-
person is at with interviewing. I try to
sired.
make sure that I am able to touch base
with each person so that I can find out
Interviewer: I understand what you try to do
how they're responding to their training,
during a training sessionwhat you hope to
to get some notion of how each person is
accomplish and stimulate. Now I'd like you to
doing.
describe to me what you actually do, not what
Question: How do you begin a session, a you expect, but what I would actually see hap-
training session? pening if I was present at the session.
Answer: I believe it's important to begin

It's not enough to simply ask a well-
with enthusiasm, to generate some excite-
formed and carefully focused initial ques-
ment about interviewing.
tion. Neither is it enough to have a well-
planned interview with appropriate basic
In this interaction, the interviewer is ask- questions. The interviewer must listen ac-
ing descriptive, behavioral questions. The tively and carefully to responses to make
responses, however, are about beliefs and sure that the interview is working. Tve seen
hopes. The answers do not actually describe many well-written interviews that have re-
what happened. Rather, they describe what sulted in largely useless data because the in-
the interviewee thinks ought to happen. terviewer did not listen carefully and thus
Since the interviewer wants behavioral data, did not recognize that the responses were
it is necessary to first recognize that the re- not providing the information needed. The
sponses are not providing the kind of data first responsibility, then, in maintaining con-
desired, and then to ask appropriate ques- trol of the interview is knowing what kind of
tions that will lead to behavioral responses, data you are looking for and managing the
something like this: interview so as to get quality responses.
Giving appropriate feedback to the inter-
Interviewer: Okay, you try to establish contact viewee is essential in pacing an interview
with each person and generate enthusiasm at and maintaining control of the interview
the beginning. What would help me now is to process. Head nodding, taking notes,
have you actually take me into a training ses- "uh-huhs," and silent probes (remaining
quiet when a person stops talking to let them ing requested and to establish a framework
know you're waiting for more) are ali signals and context that makes it possible to collect
to the person being interviewed that re- the right kind of information.
sponses are on the right track. These tech- Information that helps the interviewee
niques encourage greater depth in re- understand the purpose of the overall inter-
sponses, but you also need skill and view and the relationship of particular ques-
techniques to stop a highly verbal respon- tions to that overall purpose is important in-
dent who gets off the track. The first step in formation that goes beyond simply asking
stopping the long-winded respondent is to questions. While the reason for asking a par-
cease giving the usual cues that encourage ticular question may be absolutely clear to
talking: stop nodding the head, interject a the interviewer, don't assume it's clear to the
new question as soon as the respondent respondent. You communicate respect for
pauses for breath, stop taking notes, or call persons being interviewed by giving them
attention to the fact that you've stopped tak- the courtesy of explaining why questions are
ing notes by flipping the page of the writing being asked. Understanding the purpose of
pad and sitting back, waiting. When these a question will increase the motiva tion of the
nonverbal cues don't work, you simply have interviewee to respond openly and in detail.
to interrupt the long-winded respondent. The overall purpose of the interview is
conveyed in an opening statement. Specific
Let me stop you here for a moment. I want to questions within the interview should have
make sure I fully understand something you a connection to that overall purpose. (We'll
said earlier. (Then ask a question aimed at get- deal later with issues of informed consent
ting the response more targeted.) and protection of human subjects in relation
to opening statements of purpose. The focus
or here is on communicating purpose to im-
prove responses. Later we'll review the ethi-
Let me ask you to stop for a moment because cal issues related to informing interviewees
some of what you're talking about now I want about the study's purpose.) While the open-
to get later in the interview. First I need to find ing statement at the beginning of an inter-
out from y o u . . . . view provides an overview about the pur-
pose of the interview, it will still be
appropriate and important to explain the
Interviewers are sometimes concerned
purpose of particular questions at strategic
that it is impolite to interrupt an inter-
points throughout the interview. Here are
viewee. It certainly can be awkward, but
some examples from evaluation interviews.
when done with respect and sensitivity, the
interruption can actually help the interview.
It is both patronizing and disrespectful to let This next set of questions is about the program
the respondent run on when no attention is staff. The staff has told us that they don't really
being paid to what is said. It is respectful of get a chance to find out how people in the pro-
both the person being interviewed and the gram feel about what they do, so this part of
interviewer to make good use of the short the interview is aimed at giving them some di-
time available to talk. It is the responsibility rect feedback. But as we agreed at the begin-
of the interviewer to help the interviewee ning, the staff won't know who said what.
understand what kind of information is be- Your responses will remain confidential.
or I had driven a long way to get this inter-

view. The fieldwork was tightly scheduled
This next set of questions asks your back-
and I knew that I would not get another shot
ground and experiences. The purpose of these
at this farmer, even if he later had a change of
questions is to help us find out how people
heart. And I didn't figure it would help
with varying backgrounds have reacted to the
much to explain that I wasn't from the gov-
program.
ernment. Instead, to try to salvage the situa-
tion, I took my one-shot question, a question
The One-Shot Question
stimulated by his demeanor and overt hos-
Informal, conversational interviewing tility.
typically takes place as a natural part of
fieldwork. It is opportunistic and often un- I'm sorry I caught you at a bad time. But as
scheduled. An opportunity arises to talk long as I'm here, let me ask you just one quick
with someone and the interview is under question, then I'll be on my way. Is there any-
way. More structured and scheduled inter- thing you want to tell the b astards in S t. Paul?
viewing takes place by way of formal ap-
pointments and evaluation site visits. Yet, He hesitated for just a moment, grinned,
the best-laid plans for scheduled interviews and then launched into a tirade that turned
can go awry. You arrive at the appointed into a full, two-hour interview. 1 never got
time and place only to find that the person to out of the truck, but I was able to cover the
be interviewed is unwilling to cooperate or entire interview (though without ever refer-
needs to run off to take care of some unex- ring to or taking out the written interview
pected problem. When faced with such a sit- schedule). At the end of this conversational
uation, it is helpful to have a single, one-shot interview, which had fully satisfied my data
question in mind to salvage at least some- collection needs, he said, "Well, I've enjoyed
thing. This one-sJtot question is the one you talkin' with you, and I'm sorry about
ask if you are only going to get a few minutes refusin' to fill out your form. I just don't
with the interviewee. want to do a survey tonight."
For an agricultural extension needs as- I told him I understood and asked him if I
sessment, I was interviewing farmers in ru- could use what he had told me as long as he
ral Minnesota. The farmers in the area were wasn't identified. He readily agreed, having
economically distressed and many felt alien- already signed the consent form when we
ated from politicians and professionals. I ar- set up the appointment. I thanked him for
rived at a farm for a scheduled interview, but the conversation. My scheduled, structured
the farmer refused to cooperate. At first, he interview had become an informal, conver-
refused to even come out of the barn to call sational interview developed from a
off the dogs surrounding my truck. Finally, last-ditch, one-shot question.
he appeared and said, Here's a different example. The story is
told of a young ethnographer studying a vil-
I don't want to talk to you tonight. I know I lage that had previously been categorized in
said I would, but the wife and I had a tiff and anthropological studies as aggressive and
I'm tired. I've always helped with your gov- war oriented. He sat outside the school at the
ernment surveys. I fill out ali the forms the end of the day and asked each boy who came
government sends. But I'm tired of it. No out a one-shot question that must have
more. I don't want to talk. seemed either very stupid or very Euro-
pean, or both. His question: "What do men

do?" The responses he obtained overwhelm- ilCHfl VIUCH TgrHNiniJF?-
ingly referred to farming and fishing, and al- Fiv.i !.!i i : -V.: i::!!' ':! ' .ii'!!'! !'! : i i . ' i =1" r ,!! -.; i :i!
most none to warfare. In one hour, he had a h' 'i I i . ! ! 7 , i ii =>'! f ii !' i.K ,'i.l ,-,tf|.I.'! |>
totally different view of the society than that :: j.VI - I .'I V ! ! ' I ! ' . , I "'I; ! I ! ; I O , * !:; I:1 I:I v i I,:; &>
portrayed by previous researchers.

Mmm^^âmamm
V I , - - ! ' ii!!tf!!::'
!':= = V ' : = V I ! " ! I . ' ! ! . ' " ! V I / S L R , Y P
The Final or Closing Question r ! . ;..! iv

' ! i, i vi =
' ih'' i! i!':lii-=!s lj!! 11
"-.! I" :/!:.',!=!:, I ; I I:' : - : ; ; , M : i'1!!!' ,
In the spirit of emergent interviewing, nn|n r,\ h> -Jr, Vi .(:,!.: :i rkii' i:>.!'.:: t t t C l I M
open-ended interviewing, it's important in ! " i|"iV: !".'!' I V! v ! !!!'.- i ! K i,!,!i !!I:!.'M!.!!
formal interviews to provide an opportunity i;!,'" VI !-v: 1' 11vi ' iJ i. i! i'i i;! i:ir'i' i; i: ! I,!
for the interviewee to have the final say: l l i l i i i i
"That covers the things I wanted to ask.
Anything you care to add?" I've gotten some
my richest data from this question with in-
is appropriate for ali situations, and no par-
terviewees taking me in directions it had
ticular way of wording questions will al-
never occurred to me to pursue. Or try this:
ways work. The specific interview situation,
"What should I have asked you that I didn't
the needs of the interviewee, and the per-
think to ask?"
sonal style of the interviewer ali come to-
gether to create a unique situation for each
Beyond Technique interview. Therein lies the challenge of qual-
itative interviewing.
We've been looking with some care at dif- Maintaining focus on gathering informa-
ferent kinds of questions in an effort to pol- tion that is useful, relevant, and appropriate
ish interviewing technique and increase requires concentration, practice, and the
question precision. Below I'll offer sugges- ability to separate that which is foolish
tions about the mechanics of managing data from that which is important. In his great
collection, things like recording the data and novel Don Quixote, Cervantes describes a
taking notes. Before moving on, though, it scene in which Sancho is rebuked by Don
may be helpful to stand back and remember Quixote for trying to impress his cousin by
the larger purpose of qualitative inquiry so repeating deeply philosophical questions
that we don't become overly technique ori- and answers that he has heard from other
ented. You're trying to understand a per- people, ali the while trying to make the
son^ world and worldview. That's why you cousin think that these philosophical dis-
ask focused questions in a sensitive manner. courses were Sancho's own insights.
You're hoping to elicit relevant answers that
are useful in understanding the inter- "That question and answer," said Don Qui-
viewee^ perspective. That's basically what xote, "are not yours, Sancho. You have heard
interviewing is ali about. them from someone else."
This chapter offers ideas about how to do "Whist, sir," answered Sancho, "if I start
quality interviews, but, ultimately, no recipe questioning and answering, I shan't be done
can prescribe the single right way of inter- till tomorrow moming. Yes, for if it's just a
viewing. No single correct format exists that matter of asking idiotic questions and giving
silly replies, I needn't go begging help from

the neighbors." &i=i:'i;:i:i:" i:i!:VI.!riY-lj iiiVisirh J:;:i:;:;;:
J V ^ I T M ^ .
"You have said more than you know, JT.1 iY!-ii:i'v:i plri! rshi-liil lill.S irvi^ti';-;^:::-
Sancho/' said Don Quixote, "for there are ilJh i'!!':Y:iMi':ii!r!!!. !i:( Vri^-ji:^ :
some people who tire themselves out learning !i!1!::!'ri ;:'V" T! hld-jisiJHi-i-"[i^rsîîi 'iî.^
and proving things that, once learned and LJRMIHI-I;. J ! : ! . ! ! IS; =
: i !;-I'i> r=L!:ii ! i' :;=!.r!"i!vJ:'I:!.'":
proved, don't matter a straw as far as the mind c-s<' : i liiii-rîiiM-iViiîli 'îOy.
or memory is concerned." (Cervantes 1964: ' i:!!ii Ijiij.:rt.iij.îhlftrHijiSifi^îiiiTi/n ;:|ii|"-ii:j:i.i-i;-ri
682) ftlMfil^îilXlTi^îiiitiii ::i!;i.!i:|i M: fj UV ri:!!j:ji = "
Regardless of which interview strategy is I" i i:j\ .IriftiY&^-iV^-ii-

usedthe informal conversational inter-
view, the interview guide approach, or a [ : : " i r i : \ t i ) v i i i : i
standardized open-ended interviewthe f llflHiYijiiOîfj :.M;!;V'tI^VH; |i j
wording of questions will affect the nature
and quality of responses received. So will
careful management of the interview pro- i| ::iii'i:!iii:'i..j:!i-iiii;-i|,iii:iiji! hijfl;;j.'j!i;|;-i;:;j: iVi : !f!:j;." -
cess. Constant attention to both content and
process, with both informed by the purpose
of the interview, will reduce the extent to
which, in Cervantes's words, researchers
and evaluators "tire themselves out learning is said in one place with what is said in an-
and proving things that, once learned and other place, and integrating what different
proved, don't matter a straw as far as the people have said. These processes occur pri-
mind or memory is concerned." marily during the analysis phase after the
data are collected. During the interviewing
process itselfthat is, during the data col-
& Mechanics of Gathering lection phasethe purpose of each inter-
view is to record as fully and fairly as
Interview Data
possible that particular interviewee's per-
Recording the Data spective. Some method for recording the
verbatim responses of people being inter-
No matter what style of interviewing you viewed is therefore essential.
use and no matter how carefully you word As a good hammer is essential to fine car-
questions, it ali comes to naught if you fail to pentry, a good tape recorder is indispensable
capture the actual words of the person being to fine fieldwork. Tape recorders do not
interviewed. The raw data of interviews are "tune out" conversations, change what has
the actual quotations spoken by interview- been said because of interpretation (either
ees. Nothing can substitute for these data: conscious or unconscious), or record words
the actual things said by real people. That's more slowly than they are spoken. (Tape re-
the prize sought by the qualitative inquirer. corders, do, however, break down and mal-
Data interpretation and analysis involve function.) Obviously, a researcher doing
making sense out of what people have said, conversational interviews as part of covert
looking for patterns, putting together what fieldwork does not walk around with a tape
recorder. However, most interviews are ar- But such verbatim note taking has be-
ranged in such a way that tape recorders are come the exception now that most people
appropriate if properly explained to the in- are familiar and comfortable with tape re-
terviewee: corders. More than just increasing the accu-
racy of data collection, using a tape recorder
I'd like to tape record what you say so I don't permits the interviewer to be more attentive
miss any of it. I don't want to take the chance to the interviewee. If you tried to write down
of relying on my notes and maybe missing every word said, you'd have a difficult time
something that you say or inadvertently responding appropriately to interviewee
changing your words somehow. So, if you needs and cues. Ironically, verbatim note tak-
don't mind, I'd very much like to use the re- ing can interfere with listening attentively.
corder. If at any time during the interview you The interviewer can get so focused on note
would like to turn the tape recorder off, ali you taking that the person speaking gets only
have to do is press this button on the micro- secondary attention. Every interview is also
phone, and the recorder will stop. an observation, and having one's eyes fixed
on a notepad is hardly conducive to careful
Exhibit 7.4 lists a set of tips for getting observation. In short, the interactive nature
high-quality recordings and transcriptions. of in-depth interviewing can be seriously af-
These tips were prepared by transcribers fected by an attempt to take verbatim notes.
who had worked on hundreds of hours of in- Lofland (1971) has made this point force-
terviews and estimated that 20% of the tapes fully:
given to them, usually by graduate students,
were so badly recorded as to be impossible One's full attention must be focused upon the
to transcribe accuratelyor at ali. These tips interviewee. One must be thinking about
will also help if, as is often recommended, probing for further explication or clarification
you do either ali or some of your own tran- of what he is now saying; formulating probes
scriptions as a way of more deeply immers- linking up current talk with what he has al-
ing yourself in the data as a first step during ready said; thinking ahead to putting in a new
analysis. question that has now arisen and was not
When it is not possible to use a tape re- taken account of in the standing guide (plus
corder because of some sensitive situation, making a note at that moment so one will not
interviewee request, or tape recorder mal- forget the question); and attending to the in-
fune tion, notes must become much more terviewee in a manner that communicates to
thorough and comprehensive. It becomes him that you are indeed listening. Ali of this is
criticai to gather actual quotations. When hard enough simply in itself. Add to that the
the interviewee has said something that problem of writing it downeven if one takes
seems particularly important or insightful, it shorthand in an expert fashionand one can
may be necessary to say, 'Tm afraid I need to see that the process of note-taking in the inter-
stop you at this point so that I can get down view decreases one's interviewing capacity.
exactly what you said because I don't want Therefore, if conceivably possible, tape record;
to lose that particular quote. Let me read then one can interview. (p. 89)
back to you what I have and make sure it is
exactly what you said." This point empha- So if verbatim note taking is neither desir-
sizes again the importance of capturing able nor really possible, what kind of notes
what people say in their own words. are taken during a tape-recorded interview?
Tips for Tape-Recording Interviews: How to Keep

E X H I B I T 7.4 Transcribers Sane
1. Equipment
a. Use an eiectrica outlet and externai microphone whenever possible; they're more
reliable.
b. If you use batteries check them regularly and carry spares.
c. The recorder should be clean and in good condition-check it before going to an
interview.
d. Use good-quality tapes of 60 minutes or less; longer tapes are more likely to stretch or
break when transcribed.
e. Take along extra cassette tapes.
2. Before the interview

a. Choose a place for the interview that's quiet and free from interruptions.
b. Place the microphone close to the respondent, then speak ioud enough so that ques-
tions can be heard; most important, though, is hearing the responses.
c. Set the recorder on a stable surface.
d. Test the recording system.
3. During the interview

a. Speak clearly and not too fast-the respondent will then be more likely to do the same.
b. Ask the respondent to speak up if his or her voice starts to soften.
c. Run a test with the respondent: Then rewind and listen so the respondent can hear
whether he or she is speaking distinctly. Whether a problem is mechanical or personal,
correct it before continuing.
d. Don't rustle papers, cups, bottles, etc., near the microphone.
e. Turn off the recorder during extended side conversatons, breaks, or interruptions.
f. Watch for tape breakage and tangling.
g. Repeat the test if a tape change is necessary.
i. At end of interview, say, "This is the end of interview with
4. After the interview

a. Listen to the start, middle, and end of the tape; list proper names and unfamiliar or
unusual terminology to help the transcriber.
b. Label tapes and return them to appropriate containers.
c. Keep tapes and recorder in good condition; do not touch tapes or expose them to
extreme temperatures.
d. Give transcribers reasonable time to do a good job.
SOURCE: Prepared by transcribers at the Minnesota Center for Social Research, Universty of Minnesota.
Taking Notes During Interviews along this line include (1) using quotation
marks only to indicate full and actual quota-
The use of the tape recorder does not tions; (2) developing some mechanism for
eliminate the need for taking notes, but indicating interpretations, thoughts, or
does allow you to concentrate on taking ideas that may come to mind during the in-
strategic and focus ed notes, rather than terview, for example, the use of brackets to
attempting verbatim notes. Notes can serve set off one's own ideas from those of the in-
at least four purposes: terviewee; and (3) keeping track of ques-
tions asked as well as answers received.
Questions provide the context for interpret-
1. Notes taken during the interview can
ing answers.
help the interviewer formulate new
Note taking serves functions beyond the
questions as the interview moves along,
obvious one of taking notes. Note taking
particularly where it may be appropri-
helps pace the interview by providing non-
ate to check out something said earlier.
verbal cues about what's important, provid-
2. Looking over field notes before tran- ing feedback to the interviewee about what
scripts are done helps make sure the in- kinds of things are especially "notewor-
quiry is unfolding in the hoped-for di- thy"literally. Conversely, the failure to
rection and can stimulate early insights take notes may indicate to the respondent
that may be relevant to pursue in subse- that nothing of particular importance is be-
quent interviews while still in the ing said. And don't start making out your
fieldthe emergent nature of qualita- shopping list while someone is droning on
tive inquiry. endlessly. TheyTl think you're taking notes,
3. Taking notes about what is said will fa- enchanted, and will keep on talking. The
cilitate later analysis, including locating point is that taking notes affects the inter-
important quotations from the tape it- view process.
self.
After the Interview
4. Notes are a backup in the event the re-
corder has malfunctioned or, as I've had The period after an interview or observa-
happen, a tape is erased inadvertently tion is criticai to the rigor and validity of
during transcription. qualitative inquiry. This is a time for guaran-
teeing the quality of the data.
When a tape recorder is being used dur- Immediately after a recorded interview,
ing the interview, notes will consist primar- check the tape to make sure it was function-
ily of key phrases, lists of major points made ing properly. If, for some reason, a malfunc-
by the respondent, and key terms or words tion occurred, you should immediately
shown in quotation marks that capture the make extensive notes of everything that can
interviewee's own language. It is enor- be remembered. Even if the tape functioned
mously useful to develop some system of properly, the interviewer should go over the
abbreviations and informal shorthand to fa- interview notes to make certain that they
cilitate taking notes, for example, in an inter- make sense, to uncover areas of ambiguity
view on leadership, write "L" instead of the or uncertainty. If you find things that don't
full word. Some important conventions quite make sense, as soon possible, you
should check back with the interviewee for ration, and evaluation. Where a team is
clarification. This can often be done over the working together, the whole team needs to
telephone. In my experience, people who are meet regularly to share observations and de-
interviewed appreciate such a follow-up be- brief together. This is the beginning of analy-
cause it indicates the seriousness with which sis, because, while the situation and data are
the interviewer is taking their responses. fresh, insights can emerge that might other-
Guessing the meaning of a response is unac- wise have been lost. Ideas and interpreta-
ceptable; if there is no way of following up tions that emerge following an interview or
the comments with the respondent, then observation should be written down and
those areas of vagueness and uncertainty clearly marked as emergent, field-based in-
simply become missing data. sights to be further reviewed later.
The immediate postinterview review is a I sometimes think about the time after an
time to record details about the setting and interview as a period for postpartum reflec-
your observations about the interview. tion, a time to consider what has been re-
Where did the interview occur? Under what vealed or what has been birthed. In
conditions? How did the interviewee react 18th-century Europe, the quamt phrase "in
to questions? How well do you think you an interesting condition" became the gentile
did asking questions? How was the rapport? way of referring to an expectant mother in
Answers to these questions establish a "polite company." The coming together of
context for interpreting and making sense of an interviewer and an interviewee makes for
the interview later. Reflect on the quality of "an interesting condition." The interviewer
information received. Did you find out what is certainly expectant, as may be the inter-
you really wanted to find out in the interviewee. What emerged? What was created?
view? If not, what was the problem? Poorly Did it go OK? Is some form of triage neces-
worded questions? Wrong topics? Poor rap- sary? As soon as a child is bom, a few basic
port? Reflect on these issues and make notes observations are made and tests are per-
on the interview process while the experi- formed to make sure that everything is ali
ence is still fresh in your minds. These pro- right. That's what you're doing right after an
cess notes will inform the methodological interviewmaking sure everything came
section of your research report, evaluation, out OK.
or dissertation. Such an analogy may be a stretch for
This period after an interview or obser- thinking about a postinterview debrief, but
vation is a criticai time of reflection and interviews are precious to those who hope to
elabora tion. It is a time of quality control to turn them into dissertations, contributions
guarantee that the data obtained will be use- to knowledge, and evaluation findings. It's
ful, reliable, and authentic. This kind of worth managing the interview process to al-
postinterview ritual requires discipline. In- low time to make observations about, reflect
terviewing and observing can be exhaust- on, and learn from each interview.
ing, so much so that it is easy to forgo this Up to this point we've been focusing on
time of reflection and elaboration, put it off, techniques to enhance the quality of the
or neglect it altogether. To do so is to seri- standard one-on-one interview. We turn
ously undermine the rigor of qualitative in- now to some important variations in inter-
quiry. Interviews and observations should viewing, such as think-aloud protocols, fo-
be scheduled so that sufficient time is avail- cus group interviews, and cross-cultural in-
able afterward for data clarification, elabo- terviewing.
^ Special Applications processes and strategies they think were en-

and Issues gaged while solving problems. Wilson was
able to pinpoint the cognitive challenges
Think-Aloud that confronted students as they tried to de-
Protocol Interviewing rive the acceleration of a particle moving in
various directions and angles with respect to
Protocol analysis or, more literally, the a particular reference frame.
think-aloud protocol approach, aims to elicit The basic strategy of think-aloud proto-
the inner thoughts or cognitive processes cols involves getting people who are doing
that illuminate what's going on in a person's something to verbalize their thoughts and
head during the performance of a task, for feelings as they do whatever they're doing.
example, painting or solving a problem. The This can take some "training" of partici-
point is to undertake interviewing as close to pants to get them used to verbalizing what
the action as possible. While someone en- are usually only internai dialogues with
gages in an activity, the interviewer asks themselves.
questions and probes to get the person to
talk about what the person is thinking as he Focus Group Interviews
or she does the task. In teaching rounds at
hospitais, snior physicians do a version of A focus group interview is an interview
this when they talk aloud about how they're with a small group of people on a specific
engaging in a diagnosis while medicai stu- topic. Groups are typically 6 to 10 people
dents listen, presumably learning the ex- with similar backgrounds who participate in
perts' thinking processes by hearing them in the interview for one to two hours. In a given
action. For details of the think-aloud proto- study, a series of different focus groups will
col method, see Pressley and Afflerbach be conducted to get a variety of perspectives
(1995) and Ericsson and Sirnon (1993). and increase confidence in whatever pat-
Wilson (2000) used a protocol research terns emerge. Focus group interviewing was
design in a doctoral dissertation that investi- developed in recognition that many con-
gated student understanding and problem sumer decisions are made in a social context,
solving in college physics. Twenty students often growing out of discussions with other
in individual 45-minute sessions were vid- people. Thus, market researchers began us-
eotaped and asked to talk aloud as they tried ing focus groups in the 1950s as a way of
to solve three introductory physics prob- stimulating the consumer group process of
lems of moderate difficulty involving New- decision making to gather more accurate in-
ton's second law. This involved a concurrent formation about consumer product prefer-
rather than retrospective approach, because ences (Higginbotham and Cox 1979). On the
students were engaged in thinking aloud academic side, sociologist Robert K. Merton
and problem solving concurrently, as op- and associates wrote the seminal work on re-
posed to explaining their thinking and rea- search-oriented focus group interviews in
soning retrospectively after solving the 1956: The Pocused Interview (Merton, Riske,
problems. Wilson notes that concurrent de- and Kendall 1956).
signs are generally considered more reliable The focus group interview is, first and
because the verbal data and protocols that foremost, an interview. It is not a problem-
are generated do not depend on subjects' solving session. It is not a decision-making
short-term memory recall of the cognitive group. It is not primarily a discussion,
though direct interactions among partici- Krueger recommends that teams of two con-
pants often occur. It is an interview. The twist duct the groups so that one person can focus
is that, unlike a series of one-on-one inter- on facilitating the group while the other
views, in a focus group participants get to takes detailed notes and deals with mechan-
hear each other 's responses and to make ad- ics like as tape recorders, cameras, and any
ditional comments beyond their own origi- special needs that arise, for example, some-
nal responses as they hear what other people one needing to leave early or becoming
have to say. However, participants need not overwrought. Even when the interview is
agree with each other or reach any kind of recorded, good notes help in sorting out
consensus. Nor is it necessary for people to who said what when the tape recording is
disagree. The object is to get high-quality transcribed.
data in a social context where people can Focus group interviews have several ad-.
consider their own views in the context of vantages for qualitative inquiry.
the views of others.
Focus group expert Richard Krueger o Data collection is cost-effective. In one
(1994) explains that a focus group should be hour, you can gather information from
"carefully planned" to obtain perceptions eight people instead of only one, signifi-
"on a defined area of interestin a permissive, cantly increasing sample size. "Focus
nonthreatening environment. It is con- group interviews are widely accepted
ducted . . . by a skilled interviewer. The dis- within marketing research because they
cussion is comfortable and often enjoyable produce believable results at a reason-
for participants as they share their ideas and able cost" (Krueger 1994:8).
perceptions. Group members influence each H Interactions among participants en-
other by responding to ideas and comments hance data quality. Participants tend to
in the discussion" (p. 6). Krueger prefers the provide checks and balances on each
term moderator to interviewer because other, which weeds out false or extreme
views (Krueger and Casey 2000).
this term [moderator] highlights a specific func-
o The extent to which there is a relatively
tion of the interviewerthat of modera ting or
consistent, shared view or great diver-
guiding the discussion. The term interviewer
sity of views can be quickly assessed.
tends to convey a more limited impression of
two-way communication between an inter-
0 Focus groups tend to be enjoyable to par-
viewer and an interviewee. By contrast, the fo- ticipants, drawing on human tendencies
cus group affords the opportunity for multiple as social animais.
interactions not only between the interviewer
and respondent but among ali participants in Focus groups, like ali forms of data collec-
the group. The focus group is not a collection tion, also have limitations.
of simultaneous individual interviews, but
rather a group discussion where the conversa- a The number of questions that can be
tion flows because of the nurturing of the asked is greatly restricted in the group
moderator. (p. 100) setting.
The available response time for any par-
The combination of moderating and in- ticular individual is restrained in order
terviewing is sufficiently complex that to hear from everyone. A rule of thumb:
Focus? No focus.
Just interview each other.
Whatever comes to mind.
Postmodern unfocused group interview
With eight people and an hour for the and more complex when participants
group, plan to ask no more than 10 major have prior established relationships.
questions.
o Controversial and highly personal is-
Facilitating and conducting a focus sues are poor topics for focus groups
group interview require considerable (Kaplowitz 2000).
group process skill beyond simply ask-
ing questions. The moderator must man- o Confidentiality cannot be assured in fo-
age the interview so that it's not cus groups. Indeed, in market research,
dominated by one or two people and so focus groups are often videotaped so
that those participants who tend not to that marketers can view them and see for
be highly verbal are able to share their themselves the emotional intensity of
views. people's responses.
Those who realize that their viewpoint is
H "The focus group is beneficiai for identi-
a minority perspective may not be in-
fication of major themes but not so much
clined to speak up and risk negative reac-
for the micro-analysis of subtle differ-
tions.
ences" (Krueger 1994:x).
n Focus groups appear to work best when
people in the group, though sharing sim- Compared with most qualitative field-
ilar backgrounds, are strangers to each work approaches, focus groups typically
other. The dynamics are quite different have the disadvantage of taking place
outside of the natural settings where so- when their views of a policy may be of inter-
cial interactions normally occur (Madriz est to planners, organizers, and developers.
2000:836). Focus groups are used with special targeted
populations in action research, for example,
As these strengths and limitations sug- with program dropouts or community lead-
gest, the power of focus groups resides in ers. Organizational development consul-
their being focused. The topics are nar- tants make widespread use of focus groups
rowly focused, usually seeking reactions to to identify major motifs of an organization's
something (a product, program, or shared culture.
experience) rather than exploring complex Private sector market research firms have
life issues with depth and detail. The groups developed substantial technology to sup-
are focused by being formed homoge- port focus groups. Krueger (1994), who
neously. The facilitation is focused, keeping works mainly with public sector, education,
responses on target. Interactions among par- and nonprofit groups, thinks that
ticipants are focused, staying on topic. Use
of time must be focused, because the time fancy facilities and technological devices are
passes quickly. Despite some of the limita- generally overrated. The private sector mar-
tions introduced by the necessity of sharp fo- ket research environment often cites these
cus, applications of focus groups are wide- electronic and physical features as bench-
spread and growing (Krueger and Casey marks of quality. The true benchmark is the
2000; Madriz 2000; Fontana and Frey 2000; quality of the discussion, which can easily
Academy for Educational Development erode when participants are overly fascinated,
1989; Morgan 1988). Focus groups reinain a annoyed, or distracted by such devices as
staple of market research where reactions to one-way mirrors, television cameras, and
new or existing products can be explored. knobs and buttons. (p. x)
Focus groups have come to play an impor-
tant role in quality management efforts At the other end of the societal continuum
where consumer feedback about services from focus groups used for corporate mar-
and programs is desired. The feedback from keting are community groups using focus
focus groups is typically more specific, groups as a form of community-wide docu-
meaningful, and animated than what can be mentation and, sometimes, organizing. Fo-
obtained from individually filled out con- cus groups have entered into the repertoire
sumer questionnaires and surveys. Focus of techniques for qualitative researchers and
groups are conducted as part of a needs as- evaluators involved in participatory studies
sessment process with both potential client with coresearchers. For community re-
groups and professionals who know the search, collaborative action research, and
needs of client groups. Focus groups are be- participatory evaluations, local people who
ing used with client and staff groups in pro- are not professional researchers are being
gram evaluation to identify a program's successfully trained and supported to do fo-
strengths, weaknesses, and needed im- cus groups (Krueger and King 1997).
provements. Focus groups can be used at the Because the focus group "is a collectiv-
end of a program, or even months after pro- istic rather than an individualistic research
gram completion, to gather perceptions method," focus groups have also emerged
about outcomes and impacts. Key commu- as a collaborative and empowering ap-
nity people can be interviewed in groups proach in feminist research (Madriz 2000:
836). Sociologist and feminist researcher allybad. Bad. Bad. Bad." She then shared her
Esther Madriz (2000) explains: story. Soon others were telling similar sto-
ries, of ten commenting that they had no idea
Focus groups allow access to research partici- so many other people were having the same
pants who may find one-on-one, face-to-face kind of problems. Several also commented
interaction "scary" or "intimidating." By creat the end of the interview that they would
ating multiple lines of communication, the have been unlikely to share their stories with
group interview offers participants . . . a safe me in a one-on-one interview because they
environment where they can share ideas, be- would have felt individually vulnerable, but
liefs, and attitudes in the company of people they drew confidence and a sense of safety
from the same socioeconomic, ethnic, and and camaraderie from being part of the in-
gender backgrounds.... terview group.
For years, the voices of women of color On the other hand, Kaplowitz (2000)
have been silenced in most research projects. studied whether sensitive topics were more
Focus groups may facilitate women of color or less likely to be discussed in focus groups
"writing culture together" by exposing not versus individual interviews. Ninety-seven
orily the layers of oppression that have sup- year-round residents from the Chelem La-
pressed these women's expressions, but the goon region in Yucatan, Mxico, partici-
forms of resistance that they use every day to pated in one of 12 focus groups or 19 individ-
deal with such oppressions. In this regard, I ar- ual in-depth interviews. A professional
gue that focus groups can be an important ele- moderator used the same interview guide to
ment in the advancement of an agenda of get reactions to a shared mangrove ecosys-
social justice for women, because they can tem. The 31 sessions generated more than
serve to expose and validate women's every- 500 pages of transcripts, which were coded
day experiences of subjugation and their indi- for the incidence of discussions of sensitive
vidual and collective survival and resistance topics. The findings showed that the indi-
strategies. (pp. 835-36) vidual interviews were 18 times more likely
to address socially sensitive discussion top-
I experienced firsthand the potential of ics than the focus groups. In addition, the
focus groups to provide safety in numbers study found the two qualitative methods,
for people in vulnerable situations. I con- focus groups and individual interviews, to
ducted focus groups among low-income re- be complementary to each other, each yield-
cipients of legal aid as one technique in an ing somewhat different information.
evaluation of services provided to people in Computer-based Internet interactions
a large public housing project. As the inter- have created new forms of focus groups.
view opened, participants, who came from Walston and Lissitz (2000) evaluated the fea-
different sections of the project and did not sibility and effectiveness of computer-medi-
know each other, were reserved and cau- ated focus groups. They compared the reac-
tious about commenting on problems they tions of computer-mediated and face-to-face
were experiencing. As one woman shared in participants in focus groups discussing aca-
vague terms a history of problems she had demic dishonesty and found that the com-
had in getting needed repairs, another puter-mediated environment appeared to
woman jumped in and supported her, say- reduce members' anxiety about what the
ing, "I know exactly what you're talking moderator thought of them, making it easier
about and who you're talking about. It's re- for them to share embarrassing information.
As focus groups have become more group of women to meet me on an individual

widely used in needs assessments, organiza- basis and asked if they could meet me during
tional development, and evaluation re- their breaks from classes, I was surprised and
search, new approaches have emerged that uncomfortable with the loud silence that en-
adapt techniques to those purposes. sued....
Rossman and Rallis (1998:135), for example, When I faced similar questions from an-
have done focus groups with young chil- other group of women who also appeared to
dren. The Focus Group Kit (Morgan 1997a, resist my appeals to meet with them alone, I
1997b; Krueger 1997a, 1997b, 1997c; Krue- realized that I had arrogantly encroached into
ger and King 1997) and The Handbook for Fo- their intimate, everyday rituais of friend-
cus Group Research (Greenbaum 1997) are ship Knowing well that without collecting
helpful resources for variations on focus data from these possibly recalcitrant subjects, I
groups. had no project, I reluctantly changed my plans
The next section looks at group inter- and agreed to accept their demands. I began
views more broadly. talking to them in groups first, and gradually,
more than 30 women agreed to meet me in in-
Group Interviews dividual sessions. La ter, I disco vered that they
preferred to respond to me as a group first be-
Not ali group interviews are of the focus cause they were wary about the kinds of ques-
group variety. During fieldwork, unstruc- tions I planned to ask about their sexuality and
tured conversational interviews may occur romance reading. The more public nature of
in groups that are not at ali focused. In evalu- group discussions meant that it was a safe
ating a community leadership program, space where I might hesitate to ask intrusive
much of the most important information and personal questions....
came from talking informally with groups of Group interviews in which women spoke
people during breaks from the formal train- about love, courtship, and heterosexual rela-
ing. During the fieldwork for the wilderness tions in Western romance fiction became op-
education program described extensively in portunities to debate, contradict, and affirm
the last chapter, informal group interviews their opinions about a range of gendered so-
became a mainstay of data collection, some- cial issues in ndia such as sexual harassment
times with just 2 or 3 people, and sometimes of women in public places, stigmas associated
in dinner groups as large as 10. with single women, expectations of women to
Parameswaran (2001) found important be domestic, pressures on married women to
differences in the data she could gather in obey elders in husbands' families, and the
group versus individual interviews during merits of arranged versus choice/love mar-
her fieldwork in ndia. Moreover, she found riages. . . . In contrast to these collective ses-
that she had to do group interviews with the sions where young women's discussions pri-
young women before she could interview marily revolved around gender discrimina-
them one-on-one. She was studying the tion toward women as a group, in individual
reading of Western romance novis among interviews, many women were much more
female college students in ndia. She reports: talkative about restrictions on their sexuality,
and several women shared their frustrations
To my suiprise, several young women did not with immediate, everyday problems pertain-
seem happy or willing to spend time with me ing to family members' control over their
alone right away. When I requested the first movements. (Parameswaran 2001: 84-86) 1
Parameswaran's work illustrates well the was informally when walking somewhere
different kinds of data that can be collected or doing something with an individual. On
from groups versus individuais, with both several occasions, I tried scheduling and
kinds of data being important in long-term conducting individual interviews only to
or intensive fieldwork. I had similar experi- have many other people present when I ar-
ences in Burkina Faso and Tanzania where rived. Such cross-cultural differences in val-
villagers much preferred group interviews uing individual versus group interactions
for formal and structured interactions, and provide a nice segue to the next section on
the only way to get individual interviews cross-cultural interviewing.
13. Cross-Cultural Interviewing
C ulture and place demand our attention not because our concepts of
them are definitive or authoritative, but because they are fragile and
fraught with dispute.
Jody Berland (1997:9)
Cross-cultural inquiries add layers of The smooth professional consensus about the
complexity to the already-coinplex interac- limits and potentials of culture does not easily
tions of an interview. The possibility for mis- emerge when new struggles, conversa tions,
understandings are increased significantly and alliances are forming across once-formi-
as documented in materiais and training dable geopolitical and even linguistic bound-
schemes aimed at cross-cultural sensitiza- aries while old colonial hierarchies, spacings,
tion (e.g., Brislin et al. 1986; Stewart 1985; and "structures of feeling" continue to shape
Casse and Deol 1985; Harris and Moran the social meanings of events in landscapes
1979). Ironically, economic and cultural newly produced or remade by economic glob-
globalization, far from reducing the likeli- alization. For many scholars now, questions of
hood of misunderstandings, may simply identity and community are framed not only
make miscommunications more nuanced by issues of race, class, and gender but by a
and harder to detect because of false as- deeply political concern with place, cultural
sumptions about shared meanings. Whiting memory, and the variable terms of the schol-
(1990) tellingly explores cross-cultural dif- ars' access to an "international" space of de-
ferences in his book You Gotta Have Wa, on bate dominated not only by Western pre-
how Americans and Japanese play seem- occupations but by the English language.
ingly the same game, baseball, quite differ- (p. 319)
entlyand then uses those differences as
entres into the two cultures. Frow and Mor- Ethnographic interviewing has always
ris (2000) have captured the challenges of been inherently cross-cultural but has the
"cross-cultural" engagement in the new mil- advantage of being grounded in long-term
lennium, when the very notion of "culture" relationships and in-depth participant ob-
is in flux: servation (Tedlock 2000). In this section, we
focus on the problematic short-term studies tives, often very difficult to evaluate because
for theses or student exchange projects and of their vagueness. In Sweden, however,
brief evaluation site visits sponsored by in- policies were articulated and even legislated
temational development agencies and phil- at such a levei of specificity that they resem-
anthropic foundations. In the latter case, bled programmatic prescriptions more than
teams of a few Westerners are flown into a the vague policies that typically emanate
developing country for a week to a month to from the legislative process in the United
assess a project, often with counterparts States.
from the local culture. These rapid apprais- The situation becomes more precarious
als revolve around cross-cultural inter- when a transia tor or interpreter must be
viewing and are more vulnerable to misin- used. Special and very precise training of
terpretations and miscommunications than translators is criticai. Translators need to un-
traditional, long-term anthropological field- derstand what, precisely, you want them to
work. Examples of the potential prob- ask and that you will need full and complete
lems, presented in the sections that follow, translation of responses as verbatim as pos-
will, I hope, help sensitize students, short- sible. Interpreters often want to be helpful
term site visitors, and evaluators to the pre- by summarizing and explaining responses.
cariousness of cross-cultural interviewing. This contaminates the interviewee's actual
As Rubin and Rubin (1995) have noted: response with the interpreteis explanation
to such an extent that you can no longer be
You don't have to be a woman to interview sure whose perceptions you havethe in-
women, or a sumo wrestler to interview sumo terpreter's or the interviewee's.
wrestlers. But if you are going to cross social Some words and ideas simply can't be
gaps and go where you are ignorant, you have translated directly. People who regularly
to recognize and deal with cultural barriers to use the language come to know the unique
communication. And you have to accept that cultural meaning of special terms. One of
how you are seen by the person being inter- my favorites from the Caribbean is liming,
viewed will affect what is said. (p. 39) meaning something like hanging out, just
being, doing nothingguilt free. In inter-
views for a Caribbean program evaluation, a
Language Differences number of participants said they were just
"liming" in the program. That was not, how-
The data from interviews are words. It is ever, meant as criticism, for liming is a
tricky enough to be sure what a person highly desirable state of being, at least to
means when using a common language, but participants. Funders viewed the situation
words can take on a very different meaning somewhat differently.
in other cultures. In Sweden, I participated Rheingold (2000) has published a whole
in an international conference discussing book on "untranslatable words and phrases"
policy evaluations. The conference was con- with special meanings in other cultures. Be-
ducted in English, but I was there two days, low are four examples that are especially rel-
much of the time confused, before I came to evant to evaluators.
understand that their use of the term policy
corresponded to my American use of the Schlimmbesserung (German)a so-called
term program. I interpreted policies from an improvement that makes things
American context, to be fairly general direc- worse
biga peula (Kiriwina, New Guinea)po- ordinate. Any number of topics may be in-
tentially disruptive, unredeemable sensitive to ask or indelicate if brought up by
true statements strangers, for example, family matters, polit-
animater (French)a word of respect for a ical views, who owns what, how people
person who can communicate difficult came to be in certain positions, and sources
concepts to general audiences of income.
ta (Chinese)to understand things and Interviewing farmers for an agricultural
thus take them lightly extension project in Central America be-
came nearly impossible to do because, for
In addition to the possibility of misunder- many, their primary source of income came
standings, there may be the danger of con- from growing illegal crops. In an African
tracting culturally specific disease, includ- dictatorship our needs assessment team
ing for some what the Chinese call koro found that we could not ask about "local"
"the hysterical belief that one's penis is leadership because the country could have
shrinking" (Rheingold 1988:59). only one leader. Anyone taking on or being
Attention to language differences cross- given the designation "local leader" would
nationally can, it is hoped, make us more have been endangered. Interviewees can be
sensitive to barriers to understanding that endangered by insensitive and inappropri-
can arise even among those who speak the ate questions, so can naive interviewers. I
same language. Joyce Walker undertook a know of a case where an American female
collaborative study with 18 women who had student was rape d following an evening in-
written to each other annually for 25 years, terview in a foreign country because the
1968 to 1993. She involved them actively in young man interpreted her questions about
the study, including having them confirm local sexual customs and his own dating ex-
the authenticity of her findings. In reacting periences as an invitation to have sex.
to the study, one participant reacted to the As noted in the previous section on group
research language used: "Why call us a co- interviews, different norms govern cross-
hort? There must be something bettera cultural interactions. I remember going to an
group, maybe?" (Walker 1996:10). African village to interview the chief and
finding the whole village assembled. Fol-
Differing Norms and Values lowing a brief welcoming ceremony, I asked
if we could begin the interview. I expected a
The high esteem in which science is hei d private, one-on-one interview. He expected
has made it culturally acceptable in Western to perform in front of and involve the whole
countries to conduct interviews on virtually village. It took me a while to understand
any subject in the name of scholarly inquiry this, during which time I kept asking to go
and the common good. Such is not the case somewhere else so we could begin the inter-
worldwide. Researchers cannot siinply pre- view. He did not share my concern about
sume that they have the right to ask intru- and preference for privacy. What I expected
sive questions. Many topics discussed freely to be an individual interview soon became a
in Western societies are taboo in other parts whole-village group dialogue.
of the world. I have experienced cultures In many cultures, it is a breach of etiquette
where it was simply inappropriate to ask for an unknown man to ask to meet alone
questions of a subordinate about a super- with a woman. Even a female interviewer
may need the permission of a husband, Appropriately, Salmen's book, published by

brother, or parent to interview a village the World Bank, is called Listen to the People
woman. A female colleague created a great and advocates qualitative methods for inter-
commotion, and placed a woman in jeop- national project evaluation of development
ardy, by pursuing a personal interview with- efforts.
out permission from the male headman. Interviewers are not in the field to judge
As difficult as cross-cultural interviewing or change values and norms. Researchers
may be, it is still far superior to standardized are there to understand the perspectives of
questionnaires for collecting data from others. Getting valid, reliable, meaningful,
nonliterate villagers. Salmen (1987) de- and usable information in cross-cultural en-
scribed a major water project undertaken by vironments requires special sensitivity to
the World Bank based on a needs assessment and respect for differences. For additional
survey. The project was a failure because the discussion of cross-cultural research and
local people ended up opposing the ap- evaluation, see Patton (1985) and Lonner
proach used. His reflection on the project's and Berry (1986). For examples of doctoral
failure included a comparison of survey and dissertations based entirely on cross-cul-
qualitative methods. tural qualitative interviewing, see McClure
(1989) and Sandmann (1989).
Although it is difficult to reconstruct the One final observation on international
events and motivation that led to the rejection and cross-cultural evaluations may help em-
there is little question that a failure of ade- phasize the value of such experiences.
quate communication between project offi- Connor (1985) found that doing interna-
cials and potential beneficiaries was at least tional evaluations made him more sensitive
partly responsible. The municipality's project and effective in his domestic evaluation
preparation team had coriducted a house-to- work. The heightened sensitivity we expect
house survey in Guasmo Norte before the out- to need in exotic, cross-cultural settings can
set of the project, primarily to gather basic so- serve us well in our own cultures. Sensitivity
cioeconomic data such as family size, to and respect for other people's values,
employment and income. The project itself, norms, and worldviews is as needed at
however, was not mentioned at this early home as abroad.
stage. On the basis of this survey, World Bank
and local officials had decided that standpipes
would be more affordable to the people than L Beyond Standard
household connections. It now appears, from Interviewing: Creative
hindsight, that the questionnaire survey Qualitative Modes of Inquiry
method failed to elicit the people's negative at-
titude toward standpipes, their own criterion Thus far, this discussion of interviewing has
of affordability, or the opposition of their lead- focused on asking. However, insights into
ers who may have played on the negative feel- the lives and worlds of others can be elicited
ings of the people to undermine acceptance of in many other ways.
the project. Qualitative interviews and open Projection techniques are widely used in
discussions would very likely have revealed psychological assessment to gather informa-
people's preferences and the political climate tion from people. The best-knownprojective
far better than did the preconstructed ques- test is probably the Rorschach. The general
tionnaire. (Salmen 1987:37) principie involved is to have people react to
something other than a questionan ink- responds to 10 index cards, each listing an
blot, picture, drawing, photo, abstract paint- idea, concept, or emotion:
ing, film, story, cartoon, or whatever is rele-
vant. This approach is especially effective in
interviewing children, but it can be helpful
with people of any age. I found, for example, ANGRY
when doing follow-up interviews two years
after completion of a program that some
photographs of the program site and a few
group activities greatly enhanced recall.
Students can be interviewed about work SUCCESS
they have produced. In the wilderness pro-
gram evaluation, we interviewed partici-
pants about entries they shared from their
journals. Walker (1996) used letters ex-
changed between friends as the basis for her SAD
study of a generation of American women.
Holbrook (1996) contrasted official welfare
case records with a welfare recipienfs jour-
nals to display two completely different con-
structions of reality. Hamon (1996) used MOVED, TOUCHED
proverbs, stories, and tales as a starting
point for her inquiry into Bahamian family
life. Rettig, Tam, and Magistad (1996) ex-
tracted quotes from transcripts of public
hearings on child support guidelines as a ba- CHANGE
sis for their fieldwork. Laura Palmer (1988)
used objects left in memory of loved ones
and friends at the Vietnam Veterans Memo-
rial in Washington, D.C., as the basis for her
inquiry and later interviews. An ethno- ANXIOUS, NERVOUS
musicologist will interview people as they
listen and react to recorded music. The pos-
sibilities for creative interviewing stretch out
before us like an ocean teeming with myriad
possibilities, some already known, many STRONG STAND,
more waiting to be discovered or created. CONVICTION
Robert Kegan and colleagues have had
success basing interviews on reactions to 10
words in what they call the "subject-object
interview... In order to understand how the
interviewee organizes interpersonal and TO RN
intrapersonal experiencing, real-life situa-
tions are elicited from a series of ten uniform
probes" (Lahey et ai. n.d.). The interviewee
ences (McClure 1989). Interviewees can be

invited to identify incidents they consider
LOST SOMETHING "criticai" after which a dialogue ensues to
determine what made the incident criticai.
As these many examples illustrate, quali-
tative inquiry need not be confined to inter-
view protocols and taking field notes. Re-
IMPORTANT TO ME searchers and evaluators have considerable
freedom to creatively adapt qualitative
methods to specific situations and purposes
using anything that comes to mindand
Reactions to these words provide data for worksas a way to enter into the world and
the interviewer to explore the interviewee's worldview of others. Not only are there
underlying epistemology or "principie of many variations in what stimuli to use and
meaning-coherence" based on Kegan's how to elicit responses, creative variations
work The Evolving Self (1982). The sub- also exist for who conducts interviews.
ject-object interview is a complex and so-
phisticated methodology that requires ex- Participant Interview Chain
tensive training for proper application and
theoretical interpretation. For my purposes, As a participant observer in the wilder-
the point is that a lengthy and comprehen- ness training program for adult educators, I
sive interview interaction can be based on was involved in (1) documenting the kinds
reaction to 10 deceptively simple ideas pre- of experiences program participants were
sented on index cards rather than fully having and (2) collecting information about
framed questions. the effects of those experiences on the partic-
The subject-object interview methodol- ipants and their work situations. In short,
ogy illustrates another basis for interview- the purpose of the evaluation was to provide
ing: including writing as part of the inter- formative insights that could be used to help
view. Prior to interviewing the research understand the personal, professional, and
participants about the 10 ideas, they are institutional outcomes of intense wilderness
given 15 to 20 minutes to jot down things on experiences for these adult educators. But
the index cards. They subsequently choose the two of us doing the evaluation didn't
which cards to talk about and can use their have suficient time and resources to track
jottrngs to facilitate their verbal responses. everyone, 40 people, in depth. Therefore, we
Such an approach gives interviewees a began discussing with the program staff
chance to think through some things before ways in which the participants might be-
responding verbally. come involved in the data collection effort to
Another substitute for straight questions meet both program and evaluation needs.
in interviewing is to ask for explanations of The staff liked the idea of involving partici-
or reactions to criticai incidents. These can pants thereby introducing them to observa-
be selected by the interviewer based on pre- tion and interviewing as ways of expanding
vious fieldwork or previous interviews. The their own horizons and deepening their per-
interviewer describes the criticai incident to ceptions.
get the interviewee's perspective on what it The participants' backpacking field expe-
means and how it relates to other experi- rience was organized in two groups. We
used this fact to design a data collection ap- the program or burdening participants. But
proach that would fit with the program- before those limits are reached, a consider-
matic needs for sharing information be- able amount of useful information can be
tween the two groups. Participants were collected by involving program participants
paired to interview each other. At the very in the actual data collection process. I have
beginning of the first trip, before people since used similar participant interview
knew each other, ali of the participants were pairs in a number of program evaluations
given a short, open-ended interview of 10 with good results. The trick is to integrate
questions. They were told that each of them, the data collection into the program.
as part of their project participation, was to
have responsibility for documenting the ex- Data Collection by Program Staf
periences of their pair-mate throughout the
year. They were given a little bit of interview Program staff constitutes another re-
training, given a lot of encouragement about source for data collection that is often over-
probing, and told to record responses fully, looked. Involving program staff in data col-
thereby taking responsibility for helping to lection raises objections about staff
build this community record of individual subjectivity, data contamina tion, loss of con-
experiences. They were then sent off in pairs fidentiality, the vested interests of staff in
and given two hours to complete the inter- particular kinds of outcomes, and the threat
views with each other, recording the re- that staff members can pose to clients or stu-
sponses by hand. dents from whom they are collecting the
At the end of the 10-day experience, when data. Balancing these objections are the
the separate groups came back together, the things that can be gained from staff involve-
same pairs of participants, consisting of one ment in data collection: greater staff commit-
person from each group, were again given ment to the evaluation, increased staff reflec-
an interview outline and sent off to inter- tivity, enhanced understanding of the data
view each other about their respective expe- collection process that comes from training
riences. This served the program need for staff in data collection procedures, increased
sharing of information and an evaluation understanding by staff of program partici-
need for the collection of information. The pants' perceptions, increased data validity
trade-off, of course, was that with the mini- because of staff rapport with participants,
mal interview training given the partici- and cost savings in data collection.
pants and the impossibility of carefully su- One of my first evaluation experiences in-
pervising, controlling, and standardizing volved studying a program to train teachers
the data collection, the results were of vari- in open education at the University of North
able quality. This mode of data collection Dakota. Faculty were interested in evaluat-
also meant that confidentiality was minimal ing that program, but there were almost no
and certain kinds of information might not resources available for a formal evaluation.
be shared. But we gathered a great deal more Certainly not enough funds existed to bring
data than we could have obtained if we had in an externai evaluation team to design the
had to do ali the interviews ourselves. study, collect data, and analyze the results.
Limitations certainly exist as to how far The main means of data collection consisted
one can push client involvement in data col- of in-depth interviews with student teachers
lection and analysis without interfering in in 24 different schools and classrooms
throughout North Dakota and structured in- learnings of students and staff who partici-
terviews with 300 parents who had children pated in the data collection. That experi-
in those classrooms. The only evaluation ential impact was more powerful than the
monies available would barely pay for the formal findings of the study, an example of
transportation and the actual mechanical "evaluation process use" (Patton 1997a)
costs of data collection. Staff and students at using an evaluation process for participant
the university agreed to do the interviews as and organizational learning.
an educational experience. I developed By the way, had the interviewers been
structured interview forins for both the paid at the going commercial rate, the data
teacher and parent interviews and trained collection could have cost at least $30,000 in
ali of the interviewers in a full-day session. personnel expenses. As it was, there were no
Interviewers were assigned to geographical personnel costs and a considerable human
reas making sure that no one collected data contribution was made to the university
from their own student teachers. The inter- program by both students and staff.
views were tape recorded and transcribed. I Such participatory action research re-
did follow-up interviews with a 5% sample mains contioversial. As Kemmis and McTag-
as a check on the validity and reliability of gart (2000) noted in their extensive review of
the student and staff data. participatory approaches:
After data collection, seminars were orga-
nized for staff and students to share their In most action research, including participa-
personal perceptions based on their inter- tory action research, the researchers make sac-
view experiences. Their stories rela ted con- rifices in methodological and technical rigor
sider able impact on both staff and students. in exchange for more immediate gains in face
One outcome was the increased respect both validity: whether the evidence they collect
staff and students had for the parents. They makes sense to them in their context. For this
found the parents to be perceptive, knowl- reason, we sometimes characterize participa-
edgeable, caring, and deeply interested in tory action research as "low-tech" research: It
the education of their children. Prior to the sacrifices in methodological sophistication in
interviewing, many of the interviewers had order to generate timely evidence that can be
held fairly negative and derogatory images used and further developed in a real-time pro-
of North Dakota parents. The systematic in- cess of transformation (of practices, practitio-
terviewing had put them in a situation ners, and practice settings). (p. 591)
where they were forced to listen to what par-
ents had to say, rather than telling parents Whether some loss of methodological so-
what they (as educators) thought about phistication is merited depends on the pri-
things, and in learning to listen they had mary purpose of the inquiry and the pri-
learned a great deal. The formal analysis of mary intended users of the results. For an
the data yielded some interesting findings example of this trade-off, see Exhibit 7.5,
that were used to make some changes in the which describes a project where former
program and the data provided a source of prostitutes were trained to facilitate focus
case materiais that were adapted for us in groups with women leaving prostitution.
training future program participants, but it Participatory research will have lower credi-
is likely that the major and most lasting im- bility among externai audiences, especially
pact of the evaluation came from the among scholars who make rigor their pri-
Training Nonresearchers as Focus Group Interviewers:

EXHIBIT 7.5 Women Leaving Prostitution
Rainbow Research studied the feasibility Our interviewers were proud of their con-
of deveoping a transitional housing program tribution. At projecfs end they requested cer-
for prostituted women. To assist us we re- tificates acknowledging the training they had
cruited 5 women who had been prostituted, received and the interviews successfuily per-
trained them in focus group facilitation and formed. And, because they had performed
had them do our interviews with women leav- well, we were pleased to oblige. In the simula-
ing prostitution. For them the experience was tions they critiqued our interview guide, iead-
empowering and transformational.They were ing us to editthe language, content, order and
excited about learning a new skill, pleased to length, introduce new questions and drop
be paid for this work and thought it rewarding others. Clearly they had rapport with their
that it might benefit prostituted women. Es- peers based on shared discourse and experi-
pecially thrilling for them during the inter- ence, allowing them to gather information
views was the validation and encouragement others without the experience of prostitution
they received from their peers for the work would have been hard-pressed to secure. This
they were doing. Our work together also had was apparent in the reliability of our data.
its light moments. During a group simulation, Comparing across the interviews responses to
our interviewers loudly and provocatively the same items were highly consistent. For ali
bantered with one another as they might have concerned it was a positive experience, with
on the street. findings that most definitely shaped our finai
recommendations.
SOURCE: By Barry B. Cohen, executive director of Rainbow Research, inc., Minneapolis, Minnesota. Personal
communication, 2000. Used by permission.
mary criterion for judging quality. Partici- achieve. For participants, the point of collect-
pants involved in improving their work or ing compelling evidence is to achieve these
lives, however, lean toward pragmatism goals, or, more precisely, to avoid subverting
where what is useful determines zuhat is true. them intentionally or unintentionally by their
As Kemmis and McTaggart (2000) conclude, action. Evidence sufficient for this kind of "re-
ality checking" can often be low-tech (in terms
of research methods and techniques) or im-
The inevitabilityfor participantsof having pressionistic (from the perspective of an out-
to live with the consequences of transforma- sider who lacks the contextual knowledge that
tion provides a very concrete "reality check" the insider draws on in interpreting this evi-
on the quality of their transforma tive work, in dence). But it may still be "high-fidelity" evi-
terms of whether their practices are more dence from the perspective of understanding
efficaciousness, their understandings are the nature and consequences of particular in-
clearer, and the settings in which they practice terventions in transformations made by par-
are more rational, just, and productive of the ticipants, in their contextwhere they are
kinds of consequences they are intended to privileged observers. (p. 592)
Interactive Group We propose that clinicai researchers investi-

Interviewing gate questions emerging from the clinicai ex-
and Dialogues perience with the clinicai participants, pay
attention to and reveal any underlying values
and assumptions, and direct results toward
The involvement of program staff or cli-
clinicai participants and policy makers. This
ents as colleagues or coresearchers in action
refocuses the gaze of clinicai research onto the
research and program evaluation changes
clinicai experience and redefines its bound-
the relationship between evaluators and
aries so as to answer three questions: Whose
staff, making it interactive and cooperative
question is it? Are hidden assumptions of the
rather than one-sided and antagonistic. Wil-
clinicai world revealed? For whom are the re-
liam Tikunoff (1980) used an "interactive
search results intended? . . . Patients and clini-
research" approach in educational research
cians are invited to explore their own and/or
and development projects. He found that
each other's questions and concerns with
putting teachers, researchers, and trainer/
whatever methods are necessary. Clinicai re-
developers together as a team increased
searchers share ownership of the research
both the meaningfulness and the validity of
with clinicai participants, thus undermining
the findings because teacher cooperation
the patriarchal bias of the dominant paradigm
with and understanding of the research
and opening its assumptions to investiga tion.
made the research less intrusive, thus reduc-
This is the situated knowledge... where space
ing rather than increasing reactivity. Their
is created to find a larger, more inclusive vi-
discussions were a form o f group interviews
sion of clinicai research. (p. 616)
in which they ali asked each other questions.
The problem of how research subjects or
program clients will react to staff involve-
ment in an evaluation, particularly involve- Creativity and Data Quality:
ment in data collection, needs careful scru- Qualitative Bricolage
tiny and consideration in each situation in
which it is attempted. Reactivity is a poten- No definitive list of creative interviewing
tial problem in both conventional and or inquiry approaches can or should be con-
nonconventional designs. Breaches of confi- structed. Such a list would be a contradiction
dence and/or reactivity-biased data cannot in terms. Creative approaches are those that
be justified in the name of creativity. On the are situationally responsive and appropri-
other hand, as Tikunoffs experiences indi- ate, credible to primary intended users, and
cate, interactive designs may increase the effective in opening up new understand-
validity of data and reduce reactivity by ings. The approaches just reviewed are devi-
making evaluation more visible and open, ations from traditional research practice.
thereby making participants or clients less Each idea is subject to misuse and abuse if
resistant or suspicious. applied without regard for ways in which
These approaches can reframe inquiry the quality of data collected canbe affected. I
from a duality (interviewer-interviewee) to have not discussed such threats and possible
a dialogue in which ali are co-inquirers. errors in depth because I believe it is impos-
Miller and Crabtree (2000) advocate such a sible to identify in the abstract and in ad-
collaborative approach even in the usually vance ali the trade-offs involved in balanc-
closed and hierarchical world of medicai ing concerns for accuracy, utility, feasibility,
and clinicai research: and propriety. For example, having program
staff do client interviews in an outcomes ali trades or a kind of professional do-in-

evaluation could (a) seriously reduce the va- yourself person" (p. 17).
lidity and reliablity of the data, (b) substan-
tially increase the validity and reliability of The qualitative researcher as bricoleur or
the data, or (c) have no measurable effect on maker of quilts uses the aesthetic and material
data quality. The nature and degree of effect tools of his or her craft, deploying whatever
would depend on staff relationships with strategies, methods, or empirical materiais are
clients, how staff were assigned to clients for at hand. If new tools or techniques have to be
interviewing, the kinds of questions asked, invented, or pieced together, then the re-
the training of the interviewers, attitudes of searcher will do this. (Denzin and Lincoln
clients toward the program, the purposes to 2000b :4)
be served by the evaluation, the environ-
mental turbulence of the program, and so Interdisciplinary scholar and artist Jose
on. Program staff might make better or Cedillos has adopted the identity of Brico-
worse interviewers than externai evaluation leur and explained in personal terms how he
researchers depending on these and other came to the "bricolage arts" as his creative
factors. An evaluator must grapple with method of inquiry.
these kinds of data quality questions for ali
designs, particularly nontraditional ap- I learned naive Bricolage in the garbage
proaches. dumps, our backs hemmed in by la playa, the
Practical, but creative, data collection nutrient edge of the Pacific Ocean. How could
consists of using whatever resources are it be otherwise to a bunch of field-working na-
available to do the best job possible. Con- tive Mestizos at the disposable edge of indus-
straints always exist and do what con- trial culture? I learned economic Bricolage
straints doconstrain. Our ability to think from my landsleit on La Frontera, the border be-
of altematives is limited. Resources are al- tween Mxico in the U.S. The bilingual, bi-eco-
ways limited. This means data collection nomic estuaries of Tijuana, Mexicali and the
will be imperfect, so dissenters from re- other border towns produced the entrepre-
search and evaluation findings who want to neurial dynamism of the immigrant who
attack a study's methods can always find works off the land and joins odd forms to-
some grounds for doing so. A major reason gether to make a workable whole, usually by
for actively involving intended evaluation dent of sheer e f f o r t . . . .
users in methods decisions is to deal with Bricolage means to combine odds and
weaknesses and consider trade-off threats to ends, fragments, in making something. The
data quality before data are collected. By stra- word comes from the French bricoleur, who
tegically calculating threats to utility, as well traveled the countryside using odds and ends,
as threats to validity and authenticity, it is materiais at hand, to perform fix-it work. The
possible to make practical decisions about power to employ the Bricolage creatively
the strengths of creative and nonconven- rested in perception, being able to spot materi-
tional data collection procedures (Patton ais nested in the environment and available.
1987a, 1997a). The Bricolage begins by disentangling percep-
The creative, adaptive inquirer using dition from information and cognition. The trick
verse techniques may be thought of as a "bri- is to see what is there, not just our brains
coleur." The term comes from Levi-Strauss speckled onto the environment. Seeing what is
(1966), who defined a bricoleur as a "jack of there releases information, rather like how, af-
ter a moment of stillness, an ecology begins to 13. Specialized and Targeted

chirp and quiver again. (Cedillos forthcoming) Interview Approaches
Bricolage, by taking broken things as its pri- This chapter has reviewed general princi-
mary resource, is a trash methodology for de- pies of and approaches to in-depth qualita-
ploying unbridled optimism in a perceptual tive interviewing. However, most research-
world; a search technology for the aesthetic ers, evaluators, and practitioners specialize
potential, the creativity edge everywhere. in working with and studying specific tar-
Bricolage operates in the pixeling surface of get populations using finely honed inter-
the dumps glistening like diamond fields viewing techniques. Interviewing "elites"
streaked through with veins of gold and silver, or "experts" often requires an interactive
and this brings one to see for oneself that the style.
world is indeed layered like this. (Cedillos
1998:18-19)2 Elites respond well to inquiries about broad
topics and to intelligent, provocative,
Creativity begins with being open to new open-ended questions that allow them the
possibilities, the bricolage of combining old freedom to use their knowledge and imagina-
things in new ways, including alternative tion. In working with elites, great demands are
and emergent forms of data collection, placed on the ability of the interviewer, who
transformed observer-observed relations, must establish competence by displaying a
and reframed interviewer-interviewee in- thorough knowledge of the topic or, lacking
terconnections. Naturalistic inquiry calls for such knowledge, by projecting an accurate
ongoing openness to whatever emerges in conceptualization of the problem through
the field and during interviews. This open- shrewd questioning. (Rossman and Rallis
ness means avoiding forcing new possibili- 1998:134)
ties into old molds. The admonition to re-
main open and creative applies throughout Robert Coles (1990) became adept at in-
naturalistic inquiry, from design through terviewing children, as have Guerrero-
data collection and into analysis. Failure to Manalo (1999), Graue and Walsh (1998),
remain open and creative can lead to the er- Holmes (1998), and Greig and Taylor (1998).
ror made by a traveler who came across a Rita Arditti (1999) developed special cultur-
peacock for the first time, a story told by ally and politically sensitive approaches for
Halcolm. gaining access to and interviewing grand-
A traveler to a new land came across a mothers who se children were among "the
peacock. Having never seen this kind of bird disappeared" during Argentina's military
before, he took it for a genetic freak. Taking regime. Guerrero (1999a, 1999b) and col-
pity on the poor bird, which he was sure leagues have developed special participa-
could not survive for long in such deviant tory approaches for interviewing women in
form, he set about to correct nature's error. developing countries. Judith Arcana (1981,
He trimmed the long, colorful feathers, cut 1983) drew on her own experiences as a
back the beak, and dyed the bird black. mother to become expert at interviewing 180
"There now," he said, with pride in a job mothers for two books about the experi-
well-done, "you now look more like a stan- ences of mothering (one about mothers and
dard guinea hen." daughters, one about mothers and sons). At
the other end of the continuum from Cole's articulating the meanings of the discourses we
delightful stories of childhood innocence are identified in the informants' accounts. Their
Jane Gilgun's haunting interviews with way of thinking was for the most part outside
male sexual offenders and Angela Browne's our frames of reference. As we struggled
intensive interviews with women incarcer- through these interpretive processes, we
ated in a maximum security prison. Gilgun made notes of our r e s p o n s e s Most compel-
(1991,1994,1995,1996,1999) has conducted ling to us [was the men believing they werel
hundreds of hours of interviews with men entitled to take what they wanted and of de-
who have perpetrated violent sex offenses fining persons and situations as they wished
against women and children, most of them . . . , to suit themselves. Overall, the discourses
having multiple victims. She learned to es- they invoked served oppressive hegemonic
tablish relationships with these men ends. We also found that the men experienced
through repeated life history interviews, but chills, thrills, and intense emotional gratifica-
did so without pretending to condone their tion as they imposed their wills on smaller,
actions and sometimes challenging their physically weaker persons. (Gilgun and
portrayals. Two examples offer a sense of the McLeod 1999:175)
challenges for someone undertaking such
work through long hours and horrific de-
tails. One man, engaged tobe married at the The life work of Angela Browne illus-
time of his arrest, confessed to seven rapes; trates a similar commitment to in-depth, life
Gilgun interviewed him for a total of 14 history interviewing with people who are
hours over 12 different interviews, includ- isolated from the mainstream and whose ex-
ing detailed descriptions of his sexual vio- periences are little understood by the gen-
lence. Another man moles ted more than 20 eral culture. After her groundbreaking 1987
b oy s; at the time of his arrest he was married, study of women who kill violent partners in
sexually active with his wife, and a stepfa- self-defense (When Battered Women Kill),
ther of two boys; she obtained 20 hours of Browne began gathering life history narra-
tape over 11 interviews. These cases were tives from women incarcerated in a maxi-
particularly intriguing as purposeful sam- mum security prison. These interviews, con-
ples because both men were White, coliege ducted in a small room off a tunnel in the
graduates in their early 30s who were em- middle of the facility and six hours in length,
ployed as managers with major supervisory included women with life time histories of
responsibilities and who came from upper- trauma, much of it at the hands of family
middle-class, two-parent, never-divorced members in early childhood. Some had wit-
families whose fathers held executive posi- nessed brutal homicides; others were serv-
tions and whose mothers were profession- ing time for crimes of violence they had com-
als. Gilgun worked with an associate in tran- mitted. Their stories were painful to tell and
scribing and interpreting the interviews. to hear. Interviews often were so emotion-
ally draining that Browne came away ex-
hausted, sometimes needing to debrief on
The data were so emotionally evocative that both the impact of what she had heard and
we spent a great deal of time working through dynamics of the interviewing process. On
our personal responses. Almost two years several occasions, we had lengthy phone
went by before we found we had any facility in conversations immediately following inter-
views while she was still within the prison becomes the arena for navigating the empir-
walls. This kind of extreme interviewing ical, aesthetic, and ethical dimensions of au-
takes unusual skill, dedication, and thentic and compelling narrative. (Lawrence-
self-knowledge, coupled with a keen inter- Lightfoot and Davis 1997:xv)
est in the dynamics of human interaction.
Although Browne retumed home drained
By explicitly combining "art and science,"
after each full week of conducting these
and making portraiture a negotiated co-
daylong interviews, her enthusiasm for the
creation between the social scientist and the
task and her appreciation of respondents'
person being depicted, portraiture as a
strength and lucidity never dimmed. As this
method of qualitative inquiry has been sub-
is being written, Browne is writing a book
ject to attack for being too much art and too
about early exposure to violence as a part of
little science (English 2000).
women's pathway to prison, based on the
life stories of the women she interviewed. Holstein and Gubrium (1995) have con-
The works of Gilgun and Browne illus- ceptualized what they call "the active inter-
trate the intensity, commitment, long hours, view." Taking a constructionist perspective,
and hard work involved with certain they emphasize that an interview is a social
in-depth and life history approaches to in- interaction with interviewers and inter-
terviewing. Robert Atkinson (1998) estab- viewees sharing in constructing a story and
lished the Center for the Study of Lives at the its meanings; that is, both are participants in
University of Southern Maine in 1988 to cap- the meaning-making process. Their work re-
ture life stories and further develop methods minds us that one's theoretical orientation
for the "life story interview." Other method- (Chapter 3) can have concrete methodologi-
ological contributions along these lines in- cal implications inhow one thinks about and
clude Cole and Knowles's (2000) Doing Re- engages in data collection.
flexive Life History Research, Denzin's (1989a) A different kind of challenge concems
Interpretive Biography, and The Art and Science how to manage the tremendous amount of
of Portraiture by Sara Lawrence-Lightfoot data that can be, and usually are, collected
and Jessica Hoffman Da vis (1997): during in-depth interviewing. Grant Mc-
Cracken (1988) has contributed ways of
bringing focus to "the long interview":
Portraiture is a method of qualitative research
that blurs the boundaries of aesthetics and em-
piricism in an effort to capture the complexity, [The long interview] is a sharply focused,
dynamics, and subtlety of human experience rapid, highly intensive interview process that
and organizational life. Portraitists seek to re- seeks to diminish the indeterminacy and re-
cord and interpret the perspectives and ex- dundancy that attends more unstructured re-
perience of the people they are studying, doc- search processes. The long interview calls for
umenting their voices and their visionstheir special kinds of preparation and structure, in-
authority, knowledge, and wisdom. The cluding the use of an open-ended question-
drawing of the portrait is placed in social and naire, so that the investigator can maximize
cultural context and shaped through dialogue the value of the time spent with the respon-
between the portraitist and the subject, each dent In other words, the long interview is
one negotiating the discourse and shaping the designed to give the investigator a highly effi-
evolving image. The relationship between the cient, productive, "streain-line" instrument of
two is rich with meaning and resonance and inquiry. (p. 7)
Qualitative inquiry includes methods or more of thoughtfully reflecting on an ex-

adapted for particular disciplines such as perience, a program, or one's life can be
psychology (Kopala and Suzuki 1999), as change inducing; 10,15, or 20 hours of life
well as subspecializations such as health history interviewing can be transformative
psychology (Murray and Chamberlain or not. Therein lies the rub. Neither you
1999), humanistic psychology (Moustakas nor the interviewee can know, in advance,
1990b, 1994, 1995, 1997), and methods of and sometimes even after the fact, what im-
"transpersonal inquiry" that emphasize us- pact an interviewing experience will have
ing intuition, empathy, and self-awareness or has had.
(Braud and Anderson 1998). Specialized ap- The purpose of a research interview is
proaches to qualitative inquiry have been first and foremost to gather data, not change
developed in organizational research (Van people. Earlier, in the section on neutrality, I
Maanen 1998; Lee 1998; Symon and Cassell asserted that an interviewer is not a judge.
1998), social work (Padgett 1998), family Neither is a research interviewer a therapist.
studies (Sussman and Gilgun 1996), health Staying focused on the purpose of the inter-
research (Grbich 1998; Morse and Field view is criticai to gathering high-quality
1995), nursing (Morse 1991), aging research data. Still, there will be many temptations
(Gubrium and Sankar 1993), and cultural to stray from that purpose. It is common for
studies (McGuigan 1998; Alasuutari 1995), interviewees to ask for advice, approval, or
just to cite the range of examples. confirmation. Yielding to these tempta-
Applications and methods of qualitative tions, the interviewer may become the
inquiry, especially interviewing techniques intervieweeanswering more questions
for specially targeted populations and spe- than are asked.
cialized disciplinary approaches, continue On the other hand, the interviewer, in es-
to evolve as interest in qualitative methods tablishing rapport, is not a cold slab of gran-
grows exponentially (a metaphoric rather iteunresponsive to the human issues, in-
than statistical estimation). As applications cluding great suffering and pain, that may
and techniques have proliferated, so have unfold during an interview. In a major farm-
concerns about the ethical challenges of ing systems needs assessment project to de-
qualitative inquiry, our next topic. velop agricultural extension programs for
distressed farm families during the farm cri-
sis of the mid-1980s, I was part of a team of
Ethical Challenges in 10 interviewers (working in pairs) who in-
Qualitative Interviewing terviewed 50 farm families. Many of these
families were in great pain. They were losing
Interviews are interventions. They affect their farms. Their children had left for the
people. A good interview lays open city. Their marriages were under stress. The
thoughts, feelings, knowledge, and experi- two-hour interviews traced their family his-
ence, not only to the interviewer but also to tory, their farm situation, their community
the interviewee. The process of being taken relationships, and their hopes for the future.
through a directed, reflective process affects Sometimes questions would lead to hus-
the persons being interviewed and leaves band-wife conflict. The interviews would
them knowing things about themselves that open old wounds, lead to second-guessing
they didn't knowor least were not fully decisions made long ago, or brmg forth
aware ofbefore the interview. Two hours painful memories of dreams never fulfilled.
People often asked for advicewhat to do viewees, a fact depended on by journalists.

about their finances, their children, govern- Indeed, it seems at times that the very thing
ment subsidy programs, even their mar- someone is determined not to say is the first
riages. But we were not there to give advice. thing they tell, just to release the psychologi-
Our task was to get information about needs cal pressure of secrecy or deceit.
that might, or might not, lead to new pro- I repeat, people in interviews will tell you
grams of assistance. Could we do more than things they never intended to tell. Inter-
just ask our questions and leave? Yet, as reviews can become confessions, particularly
searchers, could we justify in any way inter- under the promise of confidentiality. But be-
vening? Yet again, our interviews were al- ware that promise. Social scientists can be
ready an intervention. Such are the ethical summoned to testify in court. We do not
dilemmas that derive from the power of in- have the legal protection that clergy and
terviews. lawyers have. In addition, some information
What we decided to do in the farm family must be reported to the police, for example,
interviews was leave each family a packet of evidence of child abuse. Thus, the power of
information about resources and programs interviewing can put the interviewees at risk.
of assistance, everything from agricultural The interviewer needs to have an ethical
referrals to financial and family counseling. framework for dealing with such issues.
To avoid having to decide which families re- There are also direct impacts on inter-
ally needed such assistance, we left the in- viewers. The previous section described the
formation with ali familiesseparate and wrenching interviews conducted by Jane
identical packages for both husband and Gilgun, with male sex offenders, and Angela
wife. When interviewees asked for advice Browne, with incarcerated women, and the
during the interview, we could tell them that physical and emotional toll of those inter-
we would leave them referral information at views on them as interviewers xposed for
the end of the interview. hours on end to horrendous details of vio-
While interviews may be intrusive in re- lence and abuse. In a family sexual abuse
opening old wounds, they can also be heal- project (Patton 1991), the fieldwork director
ing. In doing follow-up interviews with found that interviewers needed to be exten-
families who had experienced child sexual sively debriefed, sometimes in support
abuse, our team found that most mothers groups together, to help them process and
appreciated the opportunity to tell their sto- deal with the things they heard. They could
ries, vent their rage against the system, and take in only so much without having some
share their feelings with a neutral but inter- release, some safety valve for their own
ested listener. Our interviews with elderly building anger and grief. Middle-class inter-
residents participating in a program to help viewers going into poor areas may be
them stay in their homes and avoid nursing shocked and depressed by what they hear
home institutionalization typically lasted and see. It is not enough to do preparatory
much longer than planned because the el- training before such interviewing. Inter-
derly interviewees longed to have company viewers may need debriefing and their
and talk. When interviewees are open and observations and feelings can become part
willing to talk, the power of interviewing of the data on team projects.
poses new risks. People will tell you things These examples are meant to illus trate the
they never intended to tell you. This can be power of interviewing and why it is impor-
true even with reluctant or hostile inter- tant to anticipate and deal with the ethical
dimensions of qualitative inquiry. Because require making long and elaborate speeches.
qualitative methods are highly personal Statements of purpose should be simple,
and interpersonal, because naturalistic in- straightforward, and understandable. Long
quiry takes the researcher into the real statements about what the interview is go-
world where people live and work, and being to be like and how it will be used, when
cause in-depth interviewing opens up such statements are made at the beginning
what is inside peoplequalitative inquiry of the interview, are usually either boring or
may be more intrusive and involve greater produce anxiety. The interviewee will find
reactivity than surveys, tests, and other out soon enough what kinds of questions are
quantitative approaches. going to be asked and, from the nature of the
Exhibit 7.6 presents a checklist of ethical questions, will make judgments about the
issues as a starting point in thinking through likely use of such information. The basic
ethical issues in design, data collection, and messages to be communicated in the open-
analysis. The next section elaborates some ing statement are (1) the information is im-
issues of special concem. (For more compre- portant, (2) the reasons for that importance,
hensive discussions of ethics in qualitative and (3) the willingness of the interviewer to
inquiry, see Christians 2000; Kvale 1996: expiam the purpose of the interview out of
109-23; Rubin and Rubin 1995:93-105; Punch respect for the interviewee. Here's an exam-
1986, 1997; for ethics in evaluation and apple of an opening interview statement from
plied research, see Newman and Brown an evaluation study:
1996; Joint Committee on Standards for Edu-
cational Evaluation 1994; Kimmel 1988.) I'm a program evaluator brought in to help
improve this program. As someone who has
Informed Consent been in the program, you are m a unique posi-
tion to describe what the program does and
and Confidentiality
how it affects people. And that's what the in-
terview is about: your experiences with the
Informed consent protocols and opening
program and your thoughts about your expe-
statements in interviews typically cover the
riences.
foliowing issues:
The answers from ali the people we inter-
view, and we're interviewing about 25 people,
What is the purpose of collecting the in-
will be combined for our report. Nothing you
formation?
say will ever be identified with you person-
Who is the information for? How will itbe ally. As we go through the interview, if you
used? have any questions about why I'm asking
What will be asked in the interview? something, please feel free to ask. Or if there's
How will responses be handled, includ- anything you don't want to answer, just say
ing confidentiality? so. The purpose of the interview is to get your
What risks and/or benefits are involved insights into how the program operates and
for person being interviewed? how it affects people.
Any questions before we begin?
The interviewer often provides this infor-
mation in advance of the interview and then This may seem straightforward enough, but
again at the beginning of the interview. Pro- dealing with real people in the real world, ali
viding such information does not, however, kinds of complications can arise. Moreover,
Fm so glad I caught you in. Let me reassure you that Fm

not trying to sell you anything. Fm doing my dissertation
on the lived experience of living and I just need 4 or 5
hours of your time to answer a few hundred questions. If
you'11 just sign this informed consent form, we can begin.
Uninformed consent seeker

Conversational interviewing poses spe- New Directions in Informed

cial informed consent problems: Consent: Confidentiality Versus
People Owning Their Own Stories
Whipping out an informed consent statement
and asking for a signature can be awkward at Confidentiality norms are also being
best. To the extent that interviews are an exten- challenged by new directions in qualitative
sion of a conversation and part of a relation- inquiry. Traditionally, researchers have been
ship, the legality and formality of a consent advised to disguise the locations of their
form may be puzzling to your conversational fieldwork and change the names of respon-
partner or disruptive to the research. On the dents, usually giving them pseudonyms, as
one hand, you may be offering conversational a way of protecting their identities. The pre-
partners anonymity and confidentiality, and sumption has been that the privacy of re-
on the other asking them to sign a legal form search subjects should always be protected.
saying they are participating in the study. This remains the dominant presumption, as
How can they later deny they spoke to you well it should. It is being challenged, how-
which they may need to do to protect them- ever, by participants in research who insist
selvesif you possess a signed form saying on "owning their own stories." Some politi-
they were willing to participate in the study? cally active groups take pride in their identi-
(Rubin and Rubin 1995:95) ties and refuse to be involved in research
that disguises who they are. Some programs
that arm at empowering participants em-
Rubin and Rubin go on to wam that "in-
phasize that participants own their stories
stitutional review boards are not geared for
and should insist on using their real names. I
qualitative research. Often the qualitative
encountered women in a program helping
researcher cannot come up with the detailed
them overcome a history of violence and
proposal the board demands" (p. 96). They
abuse who were combating the stigma of
then distinguish importantly between legal
their past by telling their stories and attach-
compliance with human subjects protection
ing their real names to their stories as part of
requirements and conscientious ethical be-
healing, empowerment, and pride. Does the
havior:
researcher, in such cases, have the right to
impose confidentiality against the wishes of
You cannot achieve ethical research by follow- those involved? Is it patronizing and disem-
ing a set of preestablished procedures that will powering for a university-based human
always be correct. Yet, the reqnirement to behave subjects committee to insist that these
ethically is just as strong in qualitative inter- women are incapable of understanding the
vieioing as in other types of research on humans risks involved if they choose to turn down
mnybe even stronger. You must build ethical an offer of confidentiality? On the other
routines into your work. You should carefully hand, by identifying themselves they give
study codes of ethics and cases of unethical be- up not only their own privacy but perhaps
havior to sensitize yourself to situations in that of their children, other family members,
which ethical commitments become particu- and current or former partners.
larly salient. Throughout your research, keep A doctoral student studying a local
thinking and judging what are your ethical ob- church worked out an elaborate consent
ligations. (Rubin and Rubin 1995:96) form in which the entire congregation de-
pseudonyms or using their real identities in

their case studies. Some chose to be identi-
i 'fiilbtknnvt Qft>AtBOHZIHC!J'
fied; some didn't. A study of organizational
IT ;; :ij-;'':;! jTiiiftiir Yr.Vn^Vi:! . LV:.'X'- leaders offered the same option. In ali of
I . i i = ii!:r.lilri:! w'Ji!! fiW^- these cases, the research participants also
.;.; "i!l!!;vptniii!"^k^d iri!!" A',jj'ili!Dl',iiVl!f had the right to review and approve the final
:v!!|;;!:i.l-. Il iJrii^,'!^ fovi jaiVi!.;";;i' hajM!if.'' . versions of their case studies and transcripts
: ! !
Pftil iiJhhV- YI.ÎT':!;:!'. li! !'!i:'ii':i!i'i JItL i%|- before they were made public. In cases of
^^T^rai-' th X '!! : jprantfrLiw.':!; ^H collaborative inquiry where the researcher
: I.HIIIITI!I=;=TJ I A I F ! L Y L = = I - I R I A - E W E -
:
! FITO-N.HW works with "coresearchers" and data collec-
CA M ii-n? YJ.'; V ;;V.:T R tion involves more of a dialogue than an in-
. SrttiW:[ : k ! l' iriiU:jiYi^ z if terview, the coresearchers may also become
-.!-.Afe^Vria.ast-evn, as :i;.Viii; hnT-.^ &&::= i ^ ; 1 . . . coauthors as they choose to identify them-
.tiikiVppF'OX ffffftttt. Jtfr-Ui I it? ;.:I.!-: . selves and share in publication. These are ex-
: .j!:!!:;" i'J. ' amples of how the norms about confidenti-
= - V.:j:Vi i' ii. ii;|':;::i'il;!!;;i ^ îviA^ xvti"f /i ality are changing and being challenged as
! : l :
1 ..; ;:= nE WMS} pxitfetci \ tension has emerged between the important
1 ; sciif-S8 n i- J:'i i!i'. iii Lh!- .:i1 i^vîiîl-f;: ethic of protecting people's privacy and, in
;,i:,!ir!!:i!|:iiTi^ L&: J&rtfi ftt l ^ j j-; some cases, their desire to own their story.
.;,!!:*! Ishs. J M tervk w -IT-îI^'; fe:; Informed consent, in this regard, does not
: .jjr t ;i:i Liri MiT:!!' Vi: U h. ^sf^lsl -Is -ST3-I*!; automatically mean confidentiality In-
Hl 1 i IliUlÛ^lH^li [ Utiltl liîiti! (Iri [llTiftil^ftU formed consent can mean that participants
llllliiii!!! understand the risks and benefits of having
their real names reported and choose to do
j;; ; il-i Vn::-^;!:'!:!; h! tir j:!:' fi^j ri:::: so. Protection of human subjects properly
'.: tiVi!1!:!' ::|i.::i-l:a I!::j1 rhin i!;ji':r insists on informed consent. That does not
: : .;;:r.=!:i:M!.!.:il;1:!-!.;:-:! Ii:l brifril; 'ib"^nisUiui:*|=:O^S; now automatically mean confidentiality, as
V : Ji!,' :i-:" HJ; I:!.! HI !;,' ;V! iVil^r^-i;' ^; these examples illustrate.
,: ; =:! i!ti! bn:!" \P- U '7^1!^j i;=Ti=:; iVl i! i ii n^Mll ffM!?'
:!a! .!::;:I:^ iiifhiv
; ii|:'i;!:j:i! iW-iîVMjvV; !;;': i| iViiiii: Y^!' T. !: . Reciprocity: Should
r... i:; i Interviewees Be
;:.. vMrn ;hn!!:nmi Ml!! l i .! h-iM I-li
.'l! iri:!! i!i!. ': Compensated? If So, How?
: ':jiiiwivfii rjjpjjJT i;!:.'!' i/i jVi ^jhlV:!; b
: ; :;. riiilnWi n !: MU"! kiWiirM!^ HVIVI ! BIIYR i-i:i:!!'
:
The issues of whether and how to com-
': i illl^b":!:: p. ^.H:Jl; p M::';!:; :!! i ii-i/i!^V! 1 OJ p -
pensate interviewees involve questions of
both ethics and data quality. Will payment,
even of small amounts, affect people's re-
sponses, increasing acquiescence or, alterna-
cided whether to let itself be identified in his tively, enhancing the incentive to respond
dissertation. Individual church members thoughtfully and honestly? Is it somehow
also had the option of using their real names better to appeal to people on the basis of the
or choosing pseudonyms. Another student contribution they can make to knowledge or,
studying alternative health practitioners of- in the case of evaluation, improving the pro-
fered them the option of confidentiality and gram, instead of appealing to their pecuni-
ary interest? Modest payments in surveys the payment comes from a public agency,
can increase response rates to ensure an as our County Attorney has pointed out
adequate sample size. Does the same apply in the past. Consequently, when we
to depth interviewing and focus groups? "pay" for participation, we use incen-
The interviewer is usually getting paid. tives other than cash, e.g., vouchers or
Shouldn't the time of interviewees be re- gift certificates donated by local com-
spected, especially the time of low-income mercial vendors such as a discount store.
people, by offering compensation? What al- These seem to be as effective.
ternatives are there to cash for compensating H If you are detaining a person with a
interviewees? In Western capitalist societies, face-to-face interview, and it isn't a
issues of compensation are arising more and friendly conversation, rather it is a busi-
more both because people in economically ness exercise, it is only appropriate to
disadvantaged communities are reacting to offer to pay the prospective respondent
being overstudied and undervalued and be- for their time and effort. This should
cause private sector marketing firms rou- not preclude, and it would certainly
tinely compensate focus group participants, help, to explain the importance of their
so this practice has spread to the public and contribution.
nonprofit sectors. At the time of this writing,
a lively discussion of these issues took place
on EvalTalk, the American Evaluation Asso- John Reed, owner of TecMRKT Works in
ciation Internet listserv. Here are a few post- Arlington, Virginia, has had extensive expe-
ings. rience on this issue and offered the following
observations on EvalTalk.
I believe in paying people, particularly

in areas of human services. I am thinking We have paid and not paid incentives for focus
of parenting and teen programs where it groups for low-income folks as well as profes-
can be very difficult to get participation sionals and corporate CEOs. The bottom line
in interviews. If their input is valuable, I is that in most cases the incentive doesn't
believe you should put your money make a lot of difference in terms of participa-
where your mouth is. However, I would tion rates, especially if you have well-trained
always make it very clear to the respon- interviewers and well-designed data collec-
dent that, although they are being paid tion procedures.
for their time, they are NOT being paid One of my concems is that we are moving
for their responses and should be as can- in a direction in which it is assumed (with very
did and forthright as possible. little substantive foundation) that people will
only respond if given incentives. Our studies
One inner city project offered parent par-
tell us that one of the most frequently cited rea-
ticipants in a focus group vouchers to
sons why people participate is for the "com-
buy books for their kids, which for some
munity good." Sometimes, I think the
low-income parents proved to be their
incentives are really more for the peace of
first experience owning books rather
mind of the project and evaluation managers
than always borrowing them.
than for increasing participation in programs
H Cash payment for participation in inter- or research. I am concerned that the research
views is considered income and is there- corrununity may go in the direction of substi-
fore taxable. This can create problems if tuting incentives for good methods and obtain
poorer results for havmg done so. asking low-income people to come to you and
We have stopped using incentives for pro- they will have out-of-pocket expenses for
fessional and corporate focus groups and in- transportation, child care, etc. In these circum-
terviews except in very special circumstances. stances, the incentive is clearly justified to help
This hasn't affected participation rates. We people cope with a situation in which other re-
have found that in many instances the incen- sources may not be available. In one recent
tive creates a dilemma for professionals. Some case, we felt we had to use incentives in a
professionals feel that they are participating in low-income neighborhood because the neigh-
activities as a representative of their compa- borhood had been surveyed extensively by
res and that they are getting paid for engag- various firms many of whom had offered in-
ing in such activities as part of their salary. We centives. We were told residents had come to
have had people refuse the incentive because expect to be paid for providing information. In
they really don't have any mechanism for the end, given the methods we used, it wasn't
turning it over to the company, so they don't clear that we really needed to supply the in-
know what to do with it and are bothered by centives other than for the reason the client
the ethics of taking the incentive. Some take it thought we needed to.
and put it in the office party fund and still oth- A host of issues emerge with respect to
ers just pocket it. Some of the professionals paying for interviews in low-income commu-
with whom we work charge several hundred nities. We have run across those who want to
dollars an hour. In these cases, any meaningful count every last dime given to low-income
incentive for travei and participation in a fo- residents, including incentives, as part of
cus group is likely to be in excess of $1,000. It is low-income residents' incomes. There are also
not clear what a $100 incentive means in this potentially security issues with respect to in-
case. We have come to the position that if we terviewers carrying cash or checks, or the
are doing something that professionals feel is problem of negotiating checks.
important and we are doing it in a professional I am not against incentives. In some in-
manner, then they will participate. If not, then stances, for example, small well-chosen incen-
we may not be doing something that is per- tives can make a significant difference. My
ceived as very useful and we may want to ex- plea here is that colleagues not fali into the
amine our own methods and participation in trap of using incentives as a crutch but that
the project. I think this position is easily ex- they constantly examine and reexamine the
tended to other communities. If you are doing whole issue of incentives and not simply as-
something that is perceived to be of value and sume that they are either needed and/or effec-
you do it in a professional and respectful way, tive. (Reed 2000) 3
people will respond.
When we have paid people (lo w-income or Alternatives to cash can instill a deeper
professionals), we haven't seen any evidence sense of reciprocity. In doing family history
that people feel that they are being paid to say interviews, I found that giving families a
what we want to hear. Agam, we think this is copy of the interview was much appreciated
an issue of balancing the issues, good question and increased the depth of responses be-
design, being credible with the respondents, cause they were speaking not just to me, the
and giving credible assurances that respon- interviewer, but to their grandchildren and
dents' responses are confidential. great-grandchildren in telling the family's
With respect to low-income populations, story. In one project in rural areas, we carried
incentives make really good sense if you are a tape duplicator in the truck and made cop-
ies for them instantly at the end of the in- possible. Herb and Irene Rubin (1995) tell of
terview. Providing complete transcripts of interviewing an administrator in Thailand
interviews can also be attractive to partici- and learning that two months after their
pants. In an early childhood parenting pro- fieldwork, he committed suicide, "leaving
gram where data collection included video- us wondering if our encouraging him to talk
taping parents playing with their children, about his problems may have made them
copies of the videotapes were prized by par- more salient to him" (p. 98).
ents. The basic principie informing these ex- In deciding how hard to push for infor-
changes is reciprocity. Participants in re- mation, the interviewer must balance the
search provide us with something of great value of a potential response against the po-
value, their stories and their perspectives on tential distress for the respondent. This re-
their world. We show that we value what quires sensitivity, but it is not a burden the
they give us by offering something in ex- interviewer need take on alone. When I see
change. that someone is struggling for an answer,
seems hesitant or unsure, or I simply know
that an area of inquiry may be painful or un-
How Hard Should Interviewers coinfortable, I prefer to make the inter-
Push for Sensitive Information? viewee a partner in the decision about how
deeply to pursue the matter. I would say
Skillful interviewers can get people to something like: "I realize this is a difficult
talk about things they may later regre t hav- thing to talk about. Sometimes people feel
ing revealed. Or sharing revelations in an in- better talking about something like this and,
terview may unburden people, letting them of course, sometimes they don't. You decide
get something off their chest. Since one can't how much is comfortable for you to share. If
know for sure, interviewers are often faced you do tell me what happened and how you
with an ethical challenge concerning how feel, and later you wish you hadn't, I prom-
hard to push for sensitive information, a ise to delete it from the interview. Okay? Ob-
matter in which the interviewer has a con- viously I'm very interested in what hap-
flict of interest since the interviewer's predi- pened, so please tell me what you're
lection is likely to be to push for as much as comfortable telling me."
Be Careful. Ifs Dangerous Out There.
n our teaching and publications we tend to sell students a smooth,

almost idealized, model of the research process as neat, tidy, and
unproblematic.... Perhaps we should be more open and honest about the
actual pains and perils of conducting research in order to prepare and
forewarn aspiring researchers.
Maurice Punch (1986:13-14)
In the television show Hill Street Blues, the there." The same warning applies to qualita-
sergeant ended each daily briefing of police tive researchers doing fieldwork and
officers by saying, "Let's be careful out interviewing: "Be careful. It's dangerous out
there." It's important to protect those who City. Through in-depth interviews, graduate
honor us with their stories by participating student Mario Brajuha gathered detailed in-
in our studies. It's also important to protect formation from people who worked and ate
yourself. at the restaurant, information about their
I was once interviewing a young man at a lives and their views about others involved
coffee shop for a recidivisrn study when an- with the restaurant. He made the usual
other man showed up, an exchange took promise of confidentiality. In the midst of his
place, and I realized I had been used as a fieldwork, the restaurant was burned and
cover for a drug purchase. In doing straight- the police suspected arson. Learning of his
forward outeomes evaluation studies, I have fieldwork, they subpoenaed his interview
discovered illegal and unethical activities notes. He decided to honor his promises of
that I would have preferred not to have confidentiality and ended up going to jail
stumbled across. When our team did the rather than turning over his notes. This case,
needs assessment of distressed farm families which dragged on for years disrupting his
in rural Minnesota, we took the precaution graduate studies and his life, reaffirmed that
of alerting the sheriffs' offices in the counties researchers lack the protection that clergy
where we would be interviewing in case any and lawyers have when subpoenas are in-
problems arose. One sheriff called back and volved, promises of confidentiality notwith-
said that a scam had been detected in the standing. (For details, see Hallowell 1985;
county that involved a couple in a pickup Brajuha and Hallowell 1986.)
truck soliciting home improvement work It helps to think about potential risks and
and then absconding with the down pay- dangers prior to gathering data, but Brajuha
ment. Since we were interviewing in couple could not have anticipated the arson. Antici-
teams and driving pickup trucks, the sheriff, pation, planning, and ethical reflection in
after assuring himself of the legitimacy of advance only take you so far. As Maurice
our work, offered to provide us with a letter Punch (1986) has observed, sounding very
of introduetion, an offer we gratefully ac- much like he is talking from experience:
cepted. "How to cope with a loaded revolver
I supervised a dissertation that involved dropped in your lap is something you have
interviews with young male prostitutes. We to resolve on the spot, however much you
made sure to clear that study with the local may have anticipated it in prior training"
police and public prosecutors and to get the (p. 13).
men's agreement that promises of confiden- Be careful. It's dangerous out there.
tiality would be respected given the poten-
tial contribution of the findings to reducing f. Personal Reflections
both prostitution and the spread of AIDS. on Interviewing
This, by the way, was a clear case where it
would have been inappropriate to pay the Though there are dangers, there are also re-
interviewees. Instead of cash, the reciprocity wards.
incentive the student offered was the results I find interviewing people invigorating
of a personality instrument he administered. and stimulatingthe opportunity for a
One of the more famous cases of what short period of time to enter another per-
seemed like straightforward fieldwork that son's world. If participant observation
became dangerous involved dissertation re- means "walk a mile in my shoes," in-depth
search on the culture of a bistro in New York interviewing means "walk a mile in my
head." New worlds are opened up to the in- At this pomt the man decided that the
terviewer on these journeys. king's responses were rather shallow and un-
I'm personally convinced that to be a interesting, and he went on his way, crushed
good interviewer you must like doing it. and cynical. Soon afterward, an owl flew into
This means being interested in what people the tree next to the King of the Monkeys.
have to say. You must yourself believe that "What was that man doing here?" the owl
the thoughts and experiences of the people asked.
being interviewed are worth knowing. In "Oh, he was only another silly human,"
short, you must have the utmost respect for said the King of the Monkeys. "He asked a
these persons who are willing to share with bunch of simple and meaningless questions,
you some of their time to help you under- so I gave him simple and meaningless an-
stand their world. There is a Sufi story that swers."
describes what happens when the inter-
viewer loses this basic sensitivity to and re- Not ali interviews are interesting and not
spect for the person being interviewed. ali interviews go well. Certainly, there are
uncooperative respondents, people who are
paranoid, respondents who seem overly
An Interview With the sensitive and easily einbarrassed, aggres-
King of the Monkeys sive and hostile interviewees, timid people,
and the endlessly verbose who go on at great
A man once spent years learning the language lengfh about very little. When an interview
of monkeys so that he could personally inter- is going badly, it is easy to call forth one of
view the King of the Monkeys. Having com- these stereotypes to explain how the inter-
pleted his studies, he set out on his interviewee is ruining the interview. Such blam-
viewing adventure. In the course of searching ing of the victim (the interviewee), however,
for the king, he talked with a number of mon- does little to improve the quality of the data.
key underlings. He found that the monkeys he Nor does it improve interviewing skills.
spoke with were generally, to his mind, nei- I prefer to believe that there is a way to un-
ther very interesting nor very clever. He began lock the internai perspective of every inter-
to doubt whether he could learn very much viewee. My challenge and responsibility as
from the King of the Monkeys after ali. an interviewer involve finding the appropri-
Finally, he located the king and arranged ate and effective interviewing style and
for an interview. Because of his doubts, how- question format for a particular respondent.
ever, he decided to begin with a few basic It is my responsibility as the interviewer to
questions before moving on to the deeper, establish an interview climate that facilitates
meaning-of-life questions that had become his open responses. When an interview goes
obsession. badly, as it sometimes does even after ali
"What is a tree?" he asked. these years, I look first at my own shortcom-
"It is what it is," replied the King of the ings and miscalculations, not the shortcom-
Monkeys. "We swing through trees to move ings of the interviewee. Thafs how, over the
through the jungle." years, Fve gotten better and come to value
"And what is the purpose of the banana?" reflexivity, not just as an intellectual concept
"Purpose? Why, to eat." but as a personal and professional commit-
"How do animais find pleasure?" ment to learning and engaging people with
"By doing things they enjoy." respect.
H a l c o l m cm C7kvferv/iewmc|
Ask.
Listen and record.
Ask.
Listen and record.
Asking involves a grave responsibility.
Listening is a privilege.
Researchers, listen and observe. Remember that your questions will be studied
by those you study. Evaluators, listen and observe. Remember that you shall be
evaluated by your questions.
To ask is to seek entry into another's world. Therefore, ask respectfully and
with sincerity. Do not waste questions on trivia and tricks, for the value of the an-
swering gift you receive will be a reflection of the value of your question.
Blessed are the skilled questioners, for they shall be given mountains of words
to ascend.
Blessed are the wise questioners, for they shall unlock hidden corridors of
knowledge.
Blessed are the listening questioners, for they shall gain perspective.
From Halcolm/s Beautitides
. Notes 2. Copyright 1998 Jose Hilrio Cedillos.

Used with permission.
1. Radhika Parameswaran, Q ualita tive Inquiry 3. Reprinted with permission of John H.
(7,1 February), pp. 69-103, copyright 2001 by Reed, Owner, TecMRKT Works, Arlington, Vir-
Sage Publications. Reprinted with permission. gnia.
APPENDIX 7.1
3. J. 13.
Sample of a Detailed Interview Guide
Guidelines for the Descriptive Interview
Spirit ofthe guidelines. This set of guidelines is not a checklist. If it were, it

would defeat the basic strategy of the studywhich is to make full use of the
observations and thoughts of the teacher and other team members. The
guidelines are intended as an index of topics that should be discussed over
the course of the year.
Organization and use of the guidelines. The guidelines are divided into
three broad categories of topics for discussion:
I. Salient Observations
II. General Behavior Topics
III. Language and Reading Topics
This corresponds roughly to the organization of each interview, though

notnecessarily in the sequence given above. That is, each interview will begin
with the teachers salient impressions derived from observationwhat the
teacher thinks is important to report about the child, Depending on what
those impressions are, the interviewer will pick up on related topics within
the guidelines. For example, if the teacher begins the interview with a de-
scription of some interesting work the child has done, the interviewer might
pick up his/her end of the conversa tion by asking questions about the child' s
method of work. After exploring other related topics that seem pertinent to
the sense of the discussion, the interviewer would then move on to talk about
some topics in another category. If, on the other hand, the teacher 's salient im-
pressions were mainly concerned with reading, the interviewer would move
directly to related topics or reading and eventually wind backwards into ear-
lier topics on the outline.
Teachers should strive to be as descriptive as possible throughout the in-
terview, and interviewers should strive to facilitate description by asking for
concrete instances and examples.
Coverage of topics in the guidelines. No one interview could possibly aspire

to cover ali topics in the guidelines. Throughout the course of five interviews
over the year, however, we willbe able to obtain information relevant to each
topic.
Some topics (e.g., Physical/Gestural Characteristics) may only.be discussed

once, assuming the child does not change. Other topics (e.g., Activities and
Reading Competence) will undoubtedly be touched on at every interview to
update the child's documentary record. Again, it is the judgment of teacher and
interviewer alike that will determine the most relevant topics of discussion for
any given interview.
I. Teacher's Salient Observations About Child's Functioning
Basically cover impressions gained through teacher's own observations of

the children during the normal course of instruction.
Where appropriate include

comments about continuities/changes/fluctuations
comments about child's work samples
Organization of the Day (first interview only)

any changes in organization (subsequent interviews)
II. General Behavior Topics
A. Physical/Gestural Characteristics
typical posture, bearing
pace of movement
forcefulness/impact of physical presence
gestural characteristics
eye contact
voice qualities (e.g., loud, soft, fluent, halting)
voice tone/inflection
B. Affective Expression
characteristic disposition and how expressed
how is anger expressed, controlled?
how is affection expressed?
general levei of energy
C. Relationships
how does child relate to (fit in with) the whole class?
what social situations does child seek in work/play?
do other children seek out child?
relationship to adults
does approach/interaction vary in different settings? at different
times?
D. Activities
what does child do in classrooms when there is an opportunity to
choose?
breadth and depth of activities
what are unusual activities for the child to engage in?
what are things child has never engaged/attempted in
the classroom?
E. Method of Working
how does child organize self for work?
how does child carry through on work?
does child seek feedback about work? when? from whom?
does child ask for help with work? when? from whom?
does child use help that is offered? how?
evidence that child "knows what he or she knows"
evidence that child can gauge own capabilities
how does child demonstrate capabilities?
F. Summary of Progress in School-Related Work (other than reading)

differential/ even progress
unusual accomplishments, activities
unusual difficulties, blockings
(The remaining parts of the guide are omitted because of length.)
SOURCE: My thanks to Ann Bussis and Ted Chittenden of the Educational Testing Service (ETS) for
permission to include this guide, See Bussis, Chittenden, and Amarei (1973).
NOXE: This example of the guide approach to interviewing makes it clear that a great deal of prepa-
ration, effort, and concentration are required of the interviewer in using the guide. The interviewer
must be thoroughly familiar with the details of the outline so that the interview flows smoothly. Af-
ter any one interview session, the interviewer would compare the data ac tually obtained in the inter-
view to the data desired as specified in the guide in order to begin planning for the next interview.
APPENDIX 7.2
^ m. s
Examples of Standardized Open-Ended Interviews
Note. The edited interviews below were used in evaluation of an Outward

Bound program for the disabled. Outward Bound is an organization that uses
the wilderness as an experiential education mdium. This particular program
consisted of a 10-day experience in the Boundary Waters Canoe Area of Minne-
sota. The group consisted of half able-bodied participants and half disabled par-
ticipants including paraplegics; persons with cerebral palsy, epilepsy, or other
developmental disabilities; blind or deaf participants; and, on one occasion, a
quadriplegic. The first interview was conducted at the beginning of the pro-
gram, the second interview was used at the end of the ten-day experience, and
the third interview took place six months later. To save space, many of the probes
and elaboration questions have been deleted and space for writing notes has
been eliminated. The overall thrust and format of the interviews have, however,
been retained.
Precourse Interview: Minnesota Outward

Bound School Course for the Disabled
This interview is being conducted before the course as part of an evalua-

tion process to help us plan future courses. You have received a consent form
to sign, which indicates your consent to this interview. The interview will be
recorded.
1. First, we'd be interested in knowing how you became involved in the

course. How did you find out about it?
a. What about the course appealed to you?
b. What previous experiences have you had in the outdoors?
2. Some people have difficulty deciding to participate in an Outward

Bound course, and others decide fairly easily. What kind of decision
process did you go through in thinking about whether or not to
participate?
a. What particular things were you concerned about?
b. What is happening in your life right now that stimulated your
decision to take the course?
3. Now that you've made the decision to go on the course, how do you
feel about it?
a. How would you describe your feelings right now?
b. What lingering doubts or concerns do you have?

4. What are your expectations about how the course will affect you
personally?
a. What changes in yourself do you hope will result from the
experience?
b. What do you hope to get out of the experience?
5. During the course you'll be with the same group of people for an
extended period of time. What feelings do you have about being part
of a group like that for nine full days?
a. Based on your past experience with groups, how do you see
yourself fitting into your group at Outward Bound?
ForDisabled
6. One of the things we're interested in understanding better as a result of
these courses is the everyday experience of disabled people. Some of the
things we are interested in are
a. How does your disability affect the types of activities you engage
in?
b. What are the things that you don't do that you wish you could do?
c. How does your disability affect the kinds of people you associate
with? (Clarification:) Some people find that their disability means
that they associate mainly with other disabled persons. Others find
that their disability does not affect their contacts with people. What
has your experience been along these lines?
d. Sometimes people with disabilities find that their participation in
groups is limited. What has been your experience in this regard?
For Able-Bodied
6. One of the things we're interested in understanding better as a result
of these courses is feelings able-bodied people have about being with
disabled folks. What kinds of experiences with disabled people have
you had in the past?
a. What do you personally feel you get out of working with disabled
people?
b. In what ways do you find yourself being different from your usual
self when you're with disabled people?
c. What role do you expect to play with disabled people on the
Outward Bound course? (Clarification:) Are there any particular
things you expect to have to do?
d. As you think about your participation in this course, what
particular feelings do you have about being part of an outdoor
course with disabled people?
7. About half of the participants on the course are disabled people and
about half are people without disabilities. How would you expect
your relationship with the disabled people to be different from your
relationship with course participants who are not disabled?
8. We'd like to know something about how you typically face new
situations. Some people kind of like to jump into new situations,
whether or not some risk may be involved. Other people are more
cautious about entering situations until they know more about them.
Between these two, how would you describe yourself?
9. Okay, you've been very helpful. Are there other thoughts or feelings
you'd like to share with us to help us understand how you're seeing
the course right now. Anything at ali you'd like to add?
Postcourse Interview
We're conducting this interview right at the end of your course with Minne-
sota Outward Bound. We hope this will help us better understand what you've
experienced so that we can improve future courses. You have signed a form giv-
ing your consent for material from this interview to be used in a written evalua-
tion of the course. This interview is being tape-recorded.
1. To what extent was the course what you expected it to be?

a. How was it different from what you expected?
b. To what extent did the things you were concerned about before the
course come true?
b-1. Which things came true?
b-2. Which didn't come true?
2. How did the course affect you personally?

a. What changes in yourself do you see or feel as a result of the
course?
b. What would you say you got out of the experience?
3. During the last nine days you've been with the same group of people
constantly. What kind of feelings do you have about having been a part
of the same group for that time?
a. What feelings do you have about the group?
b. What role do you feel you played in the group?
c. How was your experience with this group different from your
experiences with other groups?
d. How did the group affect you?
e. How did you affect the group?
f. In what ways did you relate differently to the able-bodied and
disabled people in your group?
4. What is it about the course that makes it have the effects it has? What
happens on the course that makes a difference?
a. What do you see as the important parts of the course, that make an
Outward Bound course what it is?
b. What was the high point of the course for you?
c. What was the low point?
5. How do you think this course will affect you when you return to your
home?
a. Which of the things you experienced this week will carry over to
your normal life?
b. What plans do you have to change anything or do anything
differently as a result of this course?
For Disabled
6. We asked you before the course about your experience of being
disabled. What are your feelings about what it's like to be disabled
now?
a. How did your disability affect the type of activities you engaged
in on the course? (Clarification:) What things didn't you do
because of your disability?
b. How was your participation in the group affected by your
disability?
ForAble-Bodied
6. We asked you before the course your feelings about being with
disabled people. As a result of the experiences of the last nine days,
how have your feelings about disabled people changed?
a. How have your feelings about yourself in relation to disabled
persons changed?
b. What did you personally get out of being/working with disabled
people on this course?
c. What role did you play with the disabled people?
d. How was this role different from the role you usually play with
disabled people?
7. Before the course we asked you how you typically faced a variety of
new situations. During the last nine days you have faced a variety of
new situations. How would you describe yourself in terms of how you
approached these new experiences?
a. How was this different from the way you usually approach things?
b. How do you think this experience will affect how you approach
new situations in the future?
8. Suppose you were being asked by a government agency whether

or not they should sponsor a course like this. What would you say?
a. What arguments would you give to support your opinion?
9. Okay, you've been very helpful. We'd be very interested in any

other feelings and thoughts you'd like to share with us to help us
understand your experience of the course and how it affected you.
Six-Month Follow-Up Interview
This interview is being conducted about six months after your Outward
Bound course to help us better understand what participants experience so
that we can improve future courses.
1. Looking back on your Outward Bound experience, we'd like to ask

you to begin by describing for me what you see as the main compo-
nents of the course. What makes an Outward Bound course what it is?
a. What do you remember as the highlight of the course for you?
b. What was the low point?
2. How did the course affect you personally?

a. What kinds of changes in yourself do you see or feel as a result of
your participation in the course?
b. What would you say you got out of the experience?
3. For nine days you were with the same group of people; how has your
experience with the Outward Bound group affected your involvement
with groups since then?
For Disabled
(Check previous responses before interview. If person's attitude
appears to have changed, ask if they perceive a change in attitude.)
4. We asked you before the course to tell us what it's like to be disabled.
What are your feelings now about what it's like to be disabled?
a. How does your disability affect the types of activities you engage
in? (Clarification:) What are some of the things you don't do
because you're disabled?
b. How does your disability affect the kinds of people you associate
with? (Clarification:) Some people find that their disability means
they associate mainly with other disabled persons. Other people
with disabilities find that their disability m no way limits their
contacts with people. What has been your experience?
c. As a result of your participation in Outward Bound, how do you
believe you've changed the way you handle your disability?
Qualitative Intervieiving|J,451
For Able-Bodied
4. We asked you before the course to tell us what it's like to work with the
disabled. What are your feelings now about what it's like to work with
the disabled?
a. What do you personally feel you get out of working with disabled
persons?
b. In what ways do you find yourself being different from your usual
self when you are with disabled people?
c. As you think about your participation in the course, what
particular feelings do you have about having been part of a course
with disabled people?
5. About half of the people on the course were disabled people and about
half were people without disabilities. To what extend did you find your-
self acting differently with disabled people compared to the way you
acted with able-bodied participants?
6. Before this course we asked you how you typically face new
situations. For example, some people kind of like to jump into new
situations even if some risks are involved. Other people are more
cautious, etc. How would you describe yourself along these lines
right now?
a. To what extent, if at ali, has the way you have approached new situ-
ations since the course been a result of your Outward Bound expe-
rience?
7. Have there been any ways in which the Outward Bound course
affected you that we haven't discussed? (If yes:) How? Would you elab-
orate on that?
a. What things that you experienced during that week carried over to
your life since the course?
b. What plans have you made, if any, to change anything or do
anything differently as a result of the course?
8. Suppose you were being asked by a government agency whether or not

they should support a course like this. What would you say?
a. Who shouldn't take a course like this?
9. Okay, you've been very helpful. Any other thoughts or feelings you
might share with us to help us understand your reactions to the course
and how it affected you?
a. Anything at ali you'd like to add?
PART 3
Analysis, Interpretation,
and Reporting
Halcolm will tell you this:
"Because you can name something does not mean you understand it.
Because you understand it does not mean it can be named."
And this:
"What you do not see you cannot describe.
What you cannot describe you cannot interpret.
But because you can describe something does not mean you can interpret it."
And yet this:

"The riddle about the sound ofone hand clapping arosefrom watching the first
decision maker reading the first evaluation report."
And finally this:

"Where the sun shines, there too is shadow.
Be illumined by the light of knowledge no less than by its shadow."
m. 429
CJ
Qualitative Analysis
and Interpretation
X k e Cômplete ; A n a l y s i s
The moment you cease observing, pack your bags, and leave the field, you
will get a remarkably clear insight about that one criticai activity you should
have observed . . . but didn't.
The moment you turn off the tape recordei; say goodbye, and leave the inter-
view, it will become immediately clear to you what perfect question you should
have asked to tie the whole thing together . . . but didn't.
The moment you begin analysis it will become perfectly clear to you that
you're missing the most important pieces of information and that without those
pieces of information there is absolutely no hope of making any sense out of
what you have.
Know, then, this:
The complete analysis isn't.
Analysis finally makes clear what would have been most important to study,
if only we had known beforehand.
Evaluation reports finally make clear to decision makers what they had really
wanted to know, but couldn't articulate at the time.
Analysis brings moments of terror that nothmg sensible will emerge and
times of exhilaration from the certainty of having discovered ultimate truth. In
between are long periods of hard work, deep thinking, and weight-lifting vol-
umes of material.
From Halcolm's Iron Laws of Evaluation Research
(3. 431
432 !}. ANALYSIS, INTERPRETATION, AND REPORTING
The Challenge art of case study research. Van Maanen

(1988) emphasizes the storytelling motifs of
qualitative writing in his ethnographic book
Qualitative analysis transforms data into
on telling tales. Golden-Biddle and Locke
findings. No formula exists for that transfor-
(1997) make story the central theme in their
mation. Guidance, yes. But no recipe. Direc-
book Composing Qualitative Research. Corrine
tion can and will be offered, but the final des-
Glesne (1999), a researcher and a poet, be-
tination remains unique for each inquirer,
gins with the story analogy, describing qual-
known only whenand ifarrived at.
itative analysis as "finding your story," then
Medieval alchemy aimed to transmute
later represents the process as "improvising
base metais into gold. Modern alchemy aims
a song of the world." Lawrence-Lightfoot
to transform raw data into knowledge, the
coin of the information age. Rarity in- and Davis (1997) call to mind "portraits" in
creases value. Fine qualitative analysis re- naming their form of qualitative analysis
mains rare and difficultand therefore The Art and Science of Portraiture. Brady
valuable. (2000) explores "anthropological poetics."
Metaphors abound. Analysis begins dur- Janesick (2000) evokes dance in "The Chore-
ing a larval stage that, if fully developed, ography of Qualitative Research Design,"
metamorphoses from caterpillar-like begin- which suggests that, for warming up, we
nings into the splendor of the mature butter- may need "stretching exercises" (Janesick
fly. Or this: The inquirer acts as catalyst on 1998). Hunt and Benford (1997) call to mind
raw data, generating an interaction that theater as they use "dramaturgy" to exam-
synthesizes new substance born alive from ine qualitative inquiry. Richardson (2000b)
the catalytic conversion. Or this: Findings reminds us that qualitative analysis and
emerge like an artistic mural created from writing involve us not just in making
collage-like pieces that make sense in new sense of the world but also in making sense
ways when seen and understood as part of a of our relationship to the world and there-
greater whole. fore in discovering things about ourselves
Consider the patterns and themes run- even as we discover things about some phe-
ning through these metaphors. Transforma- nomenon of interest. In this complex and
tion. Transmutation. Conversion. Synthesis. multifaceted analytical integration of dis-
Whole from parts. Sense-making. Such mo- ciplined science, creative artistry, and per-
tifs run through quaHtative analysis like sonal reflexivity, we mold interviews, obser-
golden threads in a royal garment. They dec- vations, documents, and field notes into
orate the garment and enhance its quality, findings.
but they may also distract attention from the The challenge of qualitative analysis lies
basic cloth that gives the garment its in making sense of massive amounts of data.
strength and shapethe skill, knowledge, This involves reducing the volume of raw
experience, creativity, diligence, and work information, sifting trivia from significance,
of the garment maker. No abstract processes identifying significant patterns, and con-
of analysis, no matter how eloquently structing a framework for communicating
named and finely described, can substitute the essence of what the data reveal. The
for the skill, knowledge, experience, creativ- problem is that "we have few agreed-on can-
ity, diligence, and work of the of the qualita- ons for qualitative data analysis, in the sense
tive analyst. Thus, Stake (1995) writes of the of shared ground rules for drawing conclu-
Qualitative Analysis and Interpreta tion !J, 433
Depth and Detail in

Qualitative Analysis
Transcriplions
Cf
sions and verifying their sturdiness" (Miles cially helpful, as in the Miles and Huberman
and Huberman 1984: 16). There are no for- (1994) sourcebook. But guidelines, proce-
mulas for determining significance. No dural suggestions, and exemplars are not
ways exist of perfectly replicating the re- rules. Applying guidelines requires judg-
searcher^ analytical thought processes. No ment and creativity. Because each qualita-
straightforward tests can be applied for reli- tive study is unique, the analytical approach
ability and validity. In short, no absolute used will be unique. Because qualitative in-
rules exist except perhaps this: Do your very quiry depends, at every stage, on the skills,
best with your full intellect to fairly repre- training, insights, and capabilities of the in-
sent the data and communicate what the quirer, qualitative analysis ultimately de-
data reveal given the purpose of the study. pends on the analytical intellect and style of
Appendix 9.1, "A Documenter's Perspec- the analyst. The human factor is the great
tive," at the end of the next chapter takes you strength and the fundamental weakness of
inside the experience of one novice analyst qualitative inquiry and analysisa scien-
as she tries to make sense of the voluminous tific two-edged sword.
data she had gathered from observations The first chapter presented several exam-
and interviews. ples of qualitative findings:
Guidelines for analyzing qualitative data
can be found in abundance, and studying H Patterns in women's ways of knowing
examples of qualitative analysis canbe espe- (Belenky et al. 1986)
Eight characteristics of organizational work in which analysts must observe their

excellence (Peters and Waterman 1982) own processes even as they are doing the
Seven habits of highly effective people analysis. The final obligation of analysis is to
(Covey 1990) analyze and report on the analytical process
as part of the report of actual findings. The
a Case studies illuminating why battered
extent of such reporting will depend on the
women kill (Browne 1987) purpose of the study.
B Three primary processes that contribute
to the development of a relationship:
Being-In, Being-For, and Being-With Purpose as Context
(Moustakas 1995)
Purpose guides analysis. Chapter 5 pre-
Paradigm motifs in tribe-centered initia- sented a typology of inquiry purposes: basic
tions compared with modem youth- research, applied research, summative eval-
centered coming-of-age celebrations uation research, formative evaluation, and
(Patton 1999a) action research. These varying purposes af-
Case examples illustrating the diversity fect analysis because they in volve different
of experiences and outeomes in an adult norms and expectations for what will be
literacy program (Patton and Stockdill concluded and how it will be presented.
1987) Basic qualitative research is typically re-
ported through a scholarly monograph or
n Teachers' reactions to a Kalamazoo
published article with primary attention to
school accountability system (Perrone
the contribution of the research to social sci-
and Patton 1976)
ence theory. The theoretical framework
o Observation of a parent education class within which the study is conducted will
to illuminate the parent-staff interactive heavily shape the analysis. As Chapter 3
process made clear, the theoretical framework for an
ethnographic study will differ from that for
Reviewing these examples of qualitative ethnomethodology, heuristics, or herme-
findings from the first chapter will ground neuties.
this discussion of analytical processes in Applied qualitative research may have a
samples of the real fruit of qualitative in- more or less scholarly orientation depend-
quiry. And this chapter will add many more ing on primary audience. If the primary au-
examples. dience is scholars, then applied research will
The strategies, guidelines, and ideas for be judged by the standards of basic research,
analysis offered here are meant to be sugges- namely, research rigor and contribution to
tive and facilitating rather than confining or theory. If the primary audience is policy-
exhaustive. In actually doing analysis, you makers, the relevance, clarity, utility, and ap-
will have to adapt what is presented here to plicability of the findings will become most
fit your specific situation and study. How- important.
ever analysis is done, analysts have an obli- For scholarly qualitative research, the pub-
gation to monitor and report their own ana- lished literature on the topic being studied
lytical procedures and processes as fully helps bring focus to a particular study.
and truthfully as possible. This means that Scholarship involves an ongoing dialogue
qualitative analysis is a new stage of field- with colleagues about particular questions
of interest within the scholarly community. evaluation users for the results. At the point
The analytical focus, therefore, derives in of beginning formal analysis, the evaluator
part from what one has learned that will will have a much better perspective on what
make a contribution to the literature in a kinds of questions can be answered with the
field of inquiry. That literature will likely data that have been collected. It pays to
have contributed to the initial design of the check out which questions should take pri-
study (impHcitly or explicitly), so itis appro- ority in the final report and to suggest new
priate to revisit that literature to help focus possibilities that may have emerged during
the analysis. fieldwork.
Focus in evaluation research should derive Summative evaluations will be judged by
from questions generated at the very begin- the extent to which they contribute to mak-
ning of the evaluation process, ideally ing decisions about a program or interven-
through interactions with primary intended tion, usually decisions about overall effec-
users of the findings. Too many times evalu- tiveness, continuation, expansion, and/or
ators go through painstaking care, even ag- replication in other sites. A full report pre-
ony, in the process of working with primary senting data, interpretations, and recom-
stakeholders to clearly conceptualize and fo- mendations is required. In contrast, forma-
cus evaluation questions before data collective evaluations, conducted for program
tion begins. But, then, once the data are col- improvement, may not even generate a writ-
lected and analysis begins, they never look ten report. Findings may be reported pri-
back over their notes to review and renew marily orally. Summary observations may
their clarity on the central issues in the eval- be listed in outline form or an executive
uation. It is not enough to count on remem- summary may be written, but the timelines
bering what the evaluation questions were. for formative feedback and the high costs of
The early negotiations around the purpose formal report writing may make a full, writ-
of an evaluation usually involve important ten report unnecessary. Staff and funders
nuances. To reestablish those nuances for the often want the insights of an experienced
purpose of helping focus the analysis, it is outsider who can interview program partici-
important to review notes on decisions that pants effectively, observe what goes on in
were made during the conceptual part of the the program, and provide helpful feedback.
evaluation. (This assumes, of course, that The methods are qualitative, the purpose is
the evaluator has treated the conceptual practical, and the analysis is done through-
phase of the evaluation as a field experience out fieldwork; no written report is expected
and has kept detailed notes about the negoti- beyond a final outline of observations and
ations that went on and the decisions that implications. Academic theory takes second
were made.) place to understanding the program's the-
In addition, it may be worth reopening ory of action as actually practiced and im-
discussions with intended evaluation users plemented. In addition, foimative feedback
to make sure that the original focus of the to program staff may be ongoing rather than
evaluation remains relevant. This accom- simply at the end of the study. However, in
plishes two things. First, it allows the evalu- some situations, funders may request a care-
ator to make sure that the analysis will focus fully documented, fully developed, and for-
on needed information. Second, it prepares mally written formative report. The nature
of formative reporting, then, is dictated by course of fieldwork, ideas about directions

user needs rather than scholarly norms. for analysis will occur. Patterns take shape.
Action research reporting also varies a Possible themes spring to mind. Hypotheses
great deal. In much action research, the pro- emerge that inform subsequent fieldwork.
cess is the product, so no report will be pro- While earlier stages of fieldwork tend to be
duced for outside consumption. On the generative and emergent, foliowing wher-
other hand, some action research efforts are ever the data lead, later stages bring closure
undertaken to test organizational or com- by moving toward confirmatory data collec-
munity development theory and therefore tiondeepening insights into and confirm-
require fairly scholarly reports and publica- ing (or disconfirming) patterns that seem to
tions. Action research undertaken by a have appeared.
group of people to solve a specific problem Ideas for making sense of the data that
may involve the group sharing the analysis emerge while still in the field constitute the
process to generate a mutually understood beginning of analysis; they are part of the re-
and acceptable solution with no permanent, cord of field notes. Sometimes insights
written record of the analysis. emerge almost serendipitously. When I was
Students writing dissertations will typi- interviewing recipients of MacArthur Foun-
cally be expected to follow very formal and dation Fellowships, I happened to interview
explicit analytical procedures to produce a several people in major professional and
scholarly monograph with careful attention personal transitions followed by several in
to methodological rigor. Graduate students quite stable situations. This happenstance of
will be expected to report in detail on ali as- how interviews were scheduled suggested a
pects of methodology, usually in a separate major distinction that became important in
chapter, including thorough discussion of the final analysisdistinguishing the im-
analytical procedures, problems, and limita- pact of the fellowships on recipients in tran-
tions. sition from those in more stable situations, at
The point here is that the rigor, duration, least comparatively.
and procedures of analysis will vary de- Recordmg and tracking analytical in-
pending on the study's purpose and audi- sights that occur during data collection are
ence. Likewise, the reporting format will part of fieldwork and the beginning of quali-
vary. First and foremost, then, analysis de- tative analysis. I've heard graduate students
pends on clarity about purpose (as do ali instructed to repress ali analytical thoughts
other aspects of the study). while in the field and concentrate on data
collection. Such advice ignores the emergent
When Does Analysis Begin? nature of qualitative designs and the power
of field-based analytical insights. Certainly,
Research texts typically make a hard- this can be overdone. Too much focus on
and-fast distinction between data collection analysis while fieldwork is still going on can
and analysis. For data collection based on interfere with the openness of naturalistic
surveys, standardized tests, and experimen- inquiry, which is its strength. Rushing to
tal designs, the lines between data collection premature conclusions should be avoided.
and analysis are clear. But the fluid and But repressing analytical insights may mean
emergent nature of naturalistic inquiry losing them forever, for there's no guarantee
makes the distinction between data gather- they'11 return. And repressing in-the-field
ing and analysis far less absolute. In the insights removes the opportunity to deepen
data collection that would test the authentic- ducted several follow-up and clarifying in-
ity of those insights while still in the field terviews with my two key mformants. Each
and fails to acknowledge the confirmatory time that I thought, at last, fieldwork was
possibilities of the closing stages of feld- over and I could just concentrate on writing,
work. In the MacArthur Fellowship study, I I came to a point where I simply could not
added transitional cases to the sample near continue without more data collection. Such
the end of the interviewing to better under- can be the integrative, iterative, and syner-
stand the varieties of transi tions fellows gistic processes of data collection and analy-
were experiencingan in-the-field form of sis in qualitative inquiry.
emergent, purposeful sampling driven by A final caveat, however: Perfectionism
field-based analysis. Such overlapping of breeds imperfections. Often additional field-
data collection and analysis improves both work isn't possible, so gaps and unresolved
the quality of data collected and the quality ambiguities are noted as part of the final
of the analysis so long as the fieldworker report. Dissertation and publication dead-
takes care not to allow these initial interpre- lines may also obviate additional confirma-
ta tions to overly confine analytical possibili- tory fieldwork. And no amount of addi-
ties. Indeed, instead of focusing additional tional fieldwork can, or should, be used to
data collection entirely on confirming pre- force the vagaries of the real world into
liminary field hypotheses, the inquiry hard-and-fast conclusions or categories.
should become particularly sensitive to Such perfectionist and forced analysis ulti-
looking for alternative explanations and mately undermines the authenticity of in-
patterns that would invalidate initial inductive, qualitative analysis. Finding pat-
sights. terns is one result of analysis. Finding
In essence, when data collection has for- vagaries, uncertainties, and ambiguities is
mally ended and it is time to begin the final another.
analysis, the investigator has two primary
sources to draw from in organizing the anal-
ysis: (1) the questions that were generated
Thick Description
during the conceptual and design phases of Thick, rich description provides the founda-
the study, prior to fieldwork, and (2) analytic tion for qualitative analysis and reporting.
insights and interpretations that emerged Good description takes the reader into the
during data collection. setting being described. In his classic Street
Even then, once analysis and writing are Comer Society, William Foote Whyte (1943)
under way, fieldwork may not be over. On took us to the "slum" neighborhood where
occasion, gaps or ambiguities found during he did his fieldwork and introduced us to
analysis cry out for more data collection, so, the characters there, as did Elliot Liebow in
where possible, interviewees may be recon- Tally's Comer (1967), a description of the
tacted to clarify or deepen responses, or new lives of unemployed Black men in Washing-
observations are made to enrich descrip- ton, D.C., during the 1960s. In Constance
tions. While writing the Grand Can- Curry's (1995) oral history of school integra-
yon-based book that describes modern male tion in Drew, Mississippi, in the 1960s, she
coming-of-age issues (Patton 1999a), I re- tells the story of African American mother
turned to the Grand Canyon four times to Mae Bertha Carter and her seven children as
deepen my understanding of Canyon geol- they faced day-to-day and night-to-night
ogy and add descriptive depth, and I con- threats and terror from resistant, angry
Whites. Through in-depth case study de- ing the findings, answering "why" ques-
scriptions, Angela Browne (1987) helps us tions, attaching significance to particular re-
experience and understand the isolation sults, and putting patterns into an analytic
and fear of being a battered woman whose framework. It is tempting to rush into the
life is controlled by a rage-filled, violent creative work of interpreting the data before
man. Through detailed description and rich doing the detailed, hard work of putting to-
quotations, Alan Peshkin (1986) showed gether coherent answers to major descrip-
readers the "total world of a fundamentalist tive questions. But description comes first.
Christian school" as Erving Goffman (1961) Several options exist for organizing and
had done earlier for other "total institu- reporting descriptive findings. Exhibit 8.1
tions," closed worlds such as prisons, army presents several alternatives, dependmg on
camps, boarding schools, nursing homes, whether the primary organizing motif cen-
and mental hospitais. Howard Becker (1953, ters on telling the story of what occurred,
1985) described how one learns to become a presenting case studies, or illuminating an
marijuana user in such detail that you al- analytical framework.
most get the scent of the smoke from his These are not mutually exclusive or ex-
writing. haustive ways of organizing and reporting
These classic qualitative studies share the qualitative data. Different parts of a report
capacity to open up a world to the reader may use different reporting approaches. The
through rich, detailed, and concrete descrip- point is that one must have some initial
tions of people and places"thick descrip- framework for organizing and managing
tion" (Geertz 1973; Denzin 2001)in such a the voluminous data collected during field-
way that we can understand the phenome- work.
non studied and draw our own interpreta- For example, where variations in the ex-
tions about meanings and significance. periences of individuais are the primary fo-
Description forms the bedrock of ali qual- cus of the study, it is appropriate to begin by
itative reporting, whether for scholarly in- writing a case study using ali the data for
quiry, as in the examples above, or for pro- each person. Only then are cross-case analy-
gram evaluation. For evaluation studies, sis and comparative analysis done. For ex-
basic descriptive questions include the fol- ample, if one has studied 10 juvenile delin-
lowing: What are the stated goals of the pro- quents, the analysis would begin by doing a
gram (including different goals reported by case description of each juvenile before do-
different stakeholders)? What are the pri- ing cross-case analysis. On the other hand, if
mary activities of the program? How do the focus is on a criminal justice program
people get into the program? What is the serving juveniles, the analysis might begin
program setting like? What happens to peo- with description of variations in answers to
ple in the program? What are the effects of common questions, for example, what were
the program on participants? Thick evalua- patterns of major program experiences,
tion descriptions take those who need to use what did they like, what did they dislike,
the evaluation findings into the experience how did they think they had changed, and
and outcomes of the program. so forth.
A basic tenet of research admonishes Likewise in analyzing interviews, the an-
careful separation of description from inter- alyst has the option of beginning with case
pretation. Interpretation involves explain- analysis or cross-case analysis. Beginning
E X H I B I T 8.1 Options for Organizlng and Reporting Qualitative Data
Storytelling Approaches
Chronology Describe what happened chronologically, over time, telling the story from
and history beginning to end. This focuses on some development over time to portray
the life of a person, the history of an organization or community, or the story
of a family
Flashback Start at the end, then work backward to describe how the ending emerged.
For example, in an evaluation study, a participant case study might begin
with the outcome realized (or unrealized) and then present the chronology
or story that illuminates that outcome.
Case Study Approaches
People If individuais or groups are the primary unit of analysis, then case studies
of people or groups may be the focus for case studies. In Respect, Sara
Lawrence-Lightfoot (2000) illustrates different forms of respect through case
studies of people who manifest those different forms in the way they live
their lives.
Criticai Criticai incidents or major events can constitute self-contained descriptive
incidents units of analysis, often presented in order of importance rather than in
sequence of occurrence. McClure (1989) reported a case study of a university
through the criticai incidents that shaped it.
Various Describe various places, sites, settings, or locations (doing case studies of
settings each) before doing cross-setting pattern analysis. In an evaluation of multi-
national efforts to preserve ancient buildings, we reported on cases in
Japan, England, and Indonsia before drawing cross-cultural conclusions.
Analytical Framework Approaches
Processes Qualitative data may be organized to describe important processes. For
example, an evaluation of a program may describe recruitment processes,
socializa tion processes, decision-making and communication processes, and
so on. Distinguishing important processes becomes the analytical framework
for organizing qualitative descriptions.
Issues An analysis can be organized to illuminate key issues, often the equivalent
of the primary evaluation questions, for example, variations in how
participants changed as a result of the program. In a study of leadership
training, we organized the qualitative report around such key issues as
conflict management, negotiation skills, enhancing creativity, and effective
communicationsali important training issues.
Questions Responses to interviews can be organized question by question, especially
where a standardized interviewing format was used. For example, if an
evaluation includes questions about perceived strengths and perceived
weaknesses, responses to these questions would be grouped together.
Sensitizing Where sensitizing concepts such as "leadership" versus "followership" have
concepts played an important preordinate role in guiding fieldwork, the data can be
organized and described through those sensitizing concepts.
with case analysis means writing a case by topics from the guide, but the relevant
study for each person interviewed or each data won't be found in the same place in
unit studied (e.g., each criticai event, each each interview. An interview guide, if it has
group, or each program location). Beginning been carefully conceived, actually consti-
with cross-case analysis means grouping to- tutes a descriptive analytical framework for
gether answers from different people to analysis.
common questions, or analyzing different A qualitative study will often include
perspectives on central issues. If a standard- both kinds of analysisindividual cases
ized open-ended interview has been used, it and cross-case analysesbut one has to be-
is fairly easy to do cross-case or cross-inter- gin somewhere. Trying to do both individ-
view analysis for each question in the inter- ual case studies and cross-case analysis at
view. With an interview guide approach, an- the same time will likely lead to confusion.
swers from different people can be grouped
3. Organizing the Data
i t wasn't curiosity that killed the cat.

K ^ s It was trying to make sense of ali the data curiosity generated.
Halcolm
The data generated by qualitative meth- interview will yield 10 to 15 single-spaced

ods are voluminous. I have found no way of pages of text; 10 two-hour interviews will
preparing students for the sheer mass of in- yield roughly 200 to 300 pages of transcripts.
formation they will find themselves con- Getting organized for analysis begins
fronted with when data collection has with an inventory of what you have. Are the
ended. Sitting down to make sense out of field notes complete? Are there any parts
pages of interviews and whole files of field that you put off to write later and never got
notes canbe overwhelming. Organizing and to but need to be finished, even at this late
analyzing a mountain of narrative can se em date, before beginning analysis? Are there
like an impossible task. any glaring holes in the data that can still be
How big a mountain? Consider a study of filled by collecting additional data before the
community and scientist perceptions of HTV analysis begins? Are ali the data properly la-
vaccine trials in the United States done by beled with a notation system that will make
the Centers for Disease Control. In a large, retrieval manageable (dates, places, inter-
complex, multisite effort called Project viewee identifying information, etc.)? Are
LinCS: Linking Communities and Scientists, interview transcriptions complete? Get a
the study's 313 interviews generated more sense of the data; check out the quality of the
than 10,000 pages of transcribed text from information you have collected. Get a sense
238 participants on a range of topics of the whole.
(MacQueen and Milstein 1999). Now that's The problem of incomplete data is illus-
an extreme case, but, on average, a one-hour trated by the experience of a student who
had conducted 30 in-depth pre- and post- transcriptions, or at least checking them by
interviews with participants in a special pro- listening to the tapes as you read them, can
gram. The transcription process took several be quite different from just working off tran-
weeks. She made copies of three transcripts scripts done by someone else.
and brought them to our seminar for assis-
tance in doing the analysis. As I read the in- Protecting Data
terviews, I got a terrible sinking feeling in
my stomach. While other students were go- Thomas Carlyle lent the only copy of his
ing over the transcriptions, I pulled her handwritten manuscript on the history of
aside and asked her what instructions she the French Revolution, his master work, to
had given the typist. It was clear from read- philosopher J. S. Mill, who lent it to a Mrs.
ing just a few pages that she did not have ver- Taylor. Mrs. Taylor's illiterate housekeeper
batim transcriptionsthe essential raw data thought it was waste paper and burned it.
for qualitative analysis. The language in Carlyle behaved with nobility and stoicism,
each interview was the same. The sentence and immediately set about rewriting the
structures were the same. The answers were book. It was published in 1837 to criticai ac-
grammatically correct. People in natural claim and Consolidated Carlyle's reputation
conversations simply do not talk that way. as one of the foremost men of letters of his
The grammar in natural conversations co- day. WeTl never know how the acclaimed
mes out atrocious when transcribed. Sen- version compared with the original or what
tences hang incomplete, interrupted by new else Carlyle might have written in the year
thoughts before the first sentence was com- lost after the fireplace calamity.
pleted. Without the knowledge of this stu- So, it is prudent to make back-up copies of
dent, and certainly without her permission, ali your data, putting one master copy a way
the typist had decided to summarize the someplace secure for safekeeping. Indeed, if
participants' responses because "so much of data collection has gone on over any longpe-
what they said was just rambling on and on riod, it is wise to make copies of the data as
about nothing." Ali of the interviews had to they are collected, being certain to put one
be transcribed again before analysis could copy in a safe place where it will not be dis-
begin. turbed and cannot be lost or burned. The
Earlier I discussed the transition between data you've collected are unique and pre-
fieldwork and analysis. Transcribing offers cious. The exact observations you've made,
another point of transition between data col- the exact words people have spoken m inter-
lection and analysis as part of data manage- viewsthese can never be recaptured in
ment and preparation. Doing ali or some of precisely the same way, even if new observa-
your own interview transcriptions (instead tions are undertaken and new interviews are
of having them done by a transcriber), for conducted. Moreover, you've likely make
example, provides an opportunity to get im- promises about protecting confidentiality,
mersed m the data, an experience that usu- so you have an obligation to take care of the
ally generates emergent insights. Typing data. Field notes and interviews should be
and organizinghandwritten field notes offer treated as the valuable material they are.
another opportunity to immerse yourself in Protect them.
the data in the transition between fieldwork Beyond Thomas Carlyle's cautionary
and full analysis, a chance to get a feel for the tale, my advice in this regard comes from
cumulative data as a whole. Doing your own two more recent disasters. I was at the Uni-
versity of Wisconsin when antiwar protest- field notes. But the qualitative analyst doing
ers bombed a physics building, destroying content analysis must still decide what
the life work of several professors. I also had things go together to form a pattern, what
a psychology doctoral student who carried constitutes a theme, what to name it, and
his dissertation work, including ali the raw what meanings to extract from case studies.
data, in the trunk of his car. An angry patient The human being, not the software, must
from a mental health clinic with whom he decide how to frame a case study, how much
was working firebombed his car, destroying and what to include, and how to tell the
ali of his work. Tragic stories of lost research, story. Still, computers can play a role in
while rare, occur just often enough to re- qualitative analysis as they do in statistical
mind us about the wisdom of an ounce of analysis.
prevention.
Once a copy is put away for safekeeping, I Quantitative programs revolutionized that re-
like to have one hard copy handy through- search by making it possible to crunch our
out the analysis, one copy for writing on, numbers, more accurately, more quickly, and
and one or more copies for cutting and past- in more ways Much of the tedious, boring,
ing. A great deal of the work of qualitative mistake-prone data manipulation has been re-
analysis involves creative cutting and past- moved. This makes it possible to spend more
ing of the data, even if done on a computer, time investigating the meaning of their data.
as is now common, rather than by hand. Un- In a similar way, QDA [qualitative data
der no circumstances should one yield to the analysis] programs improve our work by re-
temptation to begin cutting and pasting the mo ving drudgery in inanaging qualitative
master copy. The master copy or computer data. Copying, highlighting, cross-referenc-
file remains a key resource for locating mate- ing, cutting and pasting transcripts and field
riais and maintaining the context for the raw notes, covering floors with index cards, mak-
data. ing multiple copies, sorting and resorting card
piles, and finding misplaced cards have never
been the highlights of qualitative research. It
!J. Computer-Assisted makes at least as much sense for us to use qual-
Qualitative Data itative programs for tedious tasks as itdoes for
Management and Analysis those people down the hall to stop hand-cal-
culating gammas. (Durkin 1997:93)
Computers and software are tools that assist
analysis. Software doesn't really analyze The analysis of qualitative data involves
qualitative data. Qualitative software pro- creativity, intellectual discipline, analytical
grams facilitate data storage, coding, re- rigor, and a great deal of hard work. Com-
trieval, coinparing, and linkingbut hu- puter programs can facilitate the work of
man beings do the analysis. Software has analysis, but they can't provide the creativ-
eased significantly the old drudgery of ity and intelligence that make each qualita-
manually locating a particular coded parative analysis unique. Moreover, since new
graph. Analysis programs speed up the pro- software is being constantly developed and
cesses of locating coded themes, grouping upgraded, this book can do no more than
data together in categories, and comparing provide some general guidance about how
passages in transcripts or incidents from to undertake computer-assisted analysis.
Most of this chapter wiil focus on the human eral criticai ways in which qualitative analy-
thinking processes involved in analysis sis software varies (Fielding 1995):
rather than the mechanical data manage-
ment challenges that computers help solve. a How you enter your data (typing di~
For an excellent review of computer soft- rectly, imported from word processing,
ware in relation to various theoretical and scanning; flexible or fixed forinatting)
practical issues in qualitative analysis, see
a Storage differences (internai vs. externai
Fielding and Lee (1998).
databases)
Exhibit 8.2 presents examples of major
computer-assisted qualitative data analysis Coding varia tions (on-screen coding vs.
software (CAQDAS). Whatbeganas distinct assign the codes first)
software approaches have become more H Differences in ease of organizing, reorga-
standardized as the various packages con- nizing, and relabeling codes
verged to offer similar functions, though
H Variations in whether memos and anno-
sometimes with different names for the
tations can be attached to codes (espe-
same functions. They ali facilitate marking
cially important for team analysis)
text, building codebooks, indexing, catego-
rizing, creating memos, and displaying mul- o Data-linking mechanisms and ease vary
tiple text entries side-by-side. Import and (connecting different data sources or
export capabilities vary. Some support segments during analysis)
teamwork and multiple users more than
Ease of navigating and browsing
others. Graphics and matrix capabilities
vary but are becoming increasingly sophisti- o Ease, speed, and process of search and
cated. Ali take time to learn to use effec- retrieval
tively. The greater the volume of data to be Important display variations (e.g., with
analyzed, the more helpful these software and without context)
programs are. Moreover, knowing which
Tracking details (recording what you've
software program you will use before data
done for review)
collection will help you collect and enter
data in the way that works best for a particu-
lar program. Qualitative discussion groups on the
Fielding (1995, 2000), who has followed Internet regularly discuss, rate, compare,
qualitative software as diligently as anyone, and debate the strengths and weaknesses of
distinguishes three basic types of qualitative different software programs (see Exhibit 8.3
analysis software: text retrievers, code-and- for examples of such groups). While prefer-
retrieve packages, and theory-builders. He ences vary, these discussions usually end
advises that packages vary substantially with consensus that any of the major pro-
and that one must use care in picking the grams will satisfy the needs of most qualita-
right software for a particular set of analysis tive researchers. Increasingly, distinctions
challenges, for example, whether you'11 be depend on "feel," "style," and "ease of use"
engaged in individual or team analysis. In- matters of individual tastemore than
deed, he cautions that you need to know differences in function. Still, differences ex-
something about qualitative analysis before ist and new developments can be expected
choosing a package. He has identified sev- to solve existing limitations.
Examples of Software Programs for Qualitative

EXHIBIT 8.2 Analysis
AnSWR (Analysis Software for Word-Based Records):

(freeware from CDCa) www.cdc.gov/hiv/software/answr.htm
ATLAS.ti www.atlasti.de/
C-I-SAID www.scolari.com
(Code-A-Text)
CDC EZ-Text www.cdc.gov/hiv/software/ez-text.htm
(freeware from CDC)
Ethnograph www.qualisresearch.com
HyperRESEARCH www.researchware.com
QCA www.nwu.edu/IPR/publications/qca.htmi
(Qualitative Comparative Analysis)
QSR NVivo www.qsr-software.com
QSR NUDHST www.qsr-software.com
(Non-numerical Unstructured Data With Indexing, Searching, and Theorizing)
TextSmart www.spss.com
winMAX www.scoiari.com
Computer-assisted qualitative data anaysis software (CAQDAS) continues to develop rapidly. See
www.scolari.com for the latest versions, prices, and links to home pages of software companies. Most
major software marketers have their own Web sites for support and will provide demo disks that allow a
user to learn and compare functions. A number of organizations worldwide have developed training
workshops to teach use of CAQDAS. These are often posted on qualitative listservs (see Chapter 1,
Exhibit 1.5, for a resource list). Software comparisons are also frequently discussed on the Qualitative
Internet listservs, Exhibit 8.3 in this chapter lists Internet listserv resources for analysis, including soft-
ware support discussion lists.
a. Centers for Disease Control, U.S. government: if these URLs have changed, go to the CDC home page
(www.cdc.gov). From there, a simpie text search should turn up the software.
Data management is a black box in virtually extent to which the database elements can be
ali qualitative software, hidden from view and separated from the software program with
difficult to access. Programs differ m the spe- their linkages intact (the export capability).
cific elements in the underlying database (the (MacQueen and Milstein 1999:30)
design), the way these elements are config-
ured (the architecture), the mechanics of how One special challenge involves better in-
the user works with the database on-screen terfaces between programs. To help solve
(the graphical user interface, or GUI), and the this problem, MacQueen and Milstein (1999)
Internet Resources and E-mail Discussion Groups

E X H I B I T 8.3 (listservs) on Qualitative Analysis
1. [email protected]: A list on qualitative analysis computer programs; to

subscribe, send this message to [email protected]: join qua-software ourname.
2. [email protected]: Topics on the text analysis, text management, and theory-building
program ATLAS/ti; to subscribe, send a one-line message to [email protected]: SUB
ATLAS-Tl yourfirstname yourlastname your institution.
3. [email protected] (Qualitative Solutions and Research), for the qualitative analysis
programs NUDHST and Nvivo. To subscribe, send a message to maiing-list-request@qsr.
com.au with the words SUBSCRIBE QSR-FORUM in the main body of the t e x t If you have
any problems, send e-mail to [email protected].
4. [email protected]: Visual Communications Discussion List; to subscribe, send
this message to [email protected]: subscribe viscom ourname.
5. [email protected]: Discussion of analysis, methodology, and ethics in online
research, including sociology, anthropology, and other related disciplines.
6. Online articles about CAQDAS: http://caqdas.soc.surrey.ac.uk/news.htm.
NOTE: Thanksto Judith Pressle, Aderhold Distinguished Professor, Social Foundationsof Education, University
of Gergia, for list subscription details. These sites and subscription details may change, and this list is not ex-
haustive.This list ismeantto besuggestive ofthe qualitative analysis resources available through the Internet.
See Chapter 1, Exhibit 1.5, and Chapter3, Exhibit 3.7, for additional qualitative resources through the Internet.
have proposed "a systems approach to qual- tion of a finding or " a n s w e r " in qualitative
itative data management and analysis" that analysis are
focus es on the data elements that are com-
(1) characteristics of the sources where infor-
monly found in a wide range of qualitative
mation is sought, (2) primary information or
approaches in hopes of encouraging the de-
objects collected from the sources, (3) second-
velopment of common protocol s for import-
ary information or objects created to aid in the
ing and exporting data between software
interpretation of primary objects, and (4) char-
programs. They note that "with a common
acteristics of the coders who construct the sec-
foundation, qualitative researchers could
ondary objects. (MacQueen and Milstein
work with multiple programs without pen-
1999:31)
alty" (p. 30). Their proposal centers on a da-
tabase whose elements correspond to the Their approach has been encapsulated as
fundamental types of information associ- the foundation for a public domain software
ated with qualitative research and the pro- program called AnSWR: Analysis Software
cesses driving the generation of that infor- for Word-Based Records, sponsored by the
mation. These four fundamental types of Centers for Disease Control (CDC) (see Ex-
information that contribute to the construc- hibit 8.2). From a database management per-
spective, they divide coding activities into For more detailed discussion of com-
two categories: segmenting activities and puter-assisted qualitative data management
metadata activities. and analysis, especially graphics and dis-
play capabilities, see Ryan and Bernard
Segmenting activities: Any analy tic actions that (2000), Fielding and Lee (1998), and Gahan
can be directly mapped onto text or other digi- and Hannibal (1998). In considering whether
tized objects are classified here as segment- to use software to assist in analysis, keep in
ing activities. Examples include defining the mind that this is partly a matter of individ-
boundaries of a narrative passage or segment, ual style, comfort with computers, amount
applying codes to a segment, using tags or of data to be analyzed, and personal prefer-
other marks to identify points in an object, and ence. Computer analysis is not necessary
creating hyper links between segments or and can interfere with the analytic process
points in an object. for those who aren't comfortable spending
Metadata activities: Metadata activities en- long hours in front of a screen. Some self-de-
tail the creation of data about data; here, we scribed "concrete" types like to get a physi-
extend the meaning of data to encompass ali cal feel for the data that isn't possible with a
symbolic representations of information and computer. Participants on a qualitative
meaning. Prompted by meaning discerned in listserv posted these responses to a thread
the primary objects, the coder generates on software analysis:
metadata in the form of codes, comments,
The best advice I ever received about coding
ineinos, and annotations, as well as graphical
was to read the data I collected over and over
summaries of the interpreted objects (e.g., dia-
and over. The more I interacted with the data,
grains, networks, clusters, and maps) capable
the more patterns and categories began to
of showing the multidimensional structure of
"jump out" at me. I never even bothered to use
coding patterns.
the software program I installed on the com-
Segmenting and metadata activities take
puter because I found it much easier to code it
place in an iterative fashion, with feedback be-
by hand.
tween the two elements. For example, a typi-
cal sequence of coder activities may include I found that hand-coding was easier and more
the highlighting or bracketing of a chunk of productive than using a computer program.
text containing semantically related terms For me, actually seeing the data in concrete
(segmenting), the creation of a code to de- form was vital in recognizing emerging
scribe the cultural significance of the chunk of themes. I actually printed multiple copies of
text (metadata), the establishment of a link be- data and cut it into individual "chunks," color
tween the code and the chunk in the data base coding as I went along, and actually physi-
(segmenting), the creation of a memo describ- cally manipulating the data by grouping
ing related concepts described in the literature chunks by apparent themes, filing in colored
(metadata), the establishment of a link be- folders, etc. This technique was especially use-
tween the memo and the chunk in the data ful when data seemed to fit more than one
base (segmenting), and incorporation of the theme and facilitated merging of my initial
code into a diagram describing conceptual impressions as themes solidified. Messy, but
links among related codes (metadata). This vital for us concrete people.
complex process is the primary focus of most
qualitative approaches. (MacQueen and So although software analysis has be-
Milstein 1999:35-36) come common and many swear by itit can
offer leaps in productivity for those adept at classrooms are a more meaningful unit of
itit is not a requisite for qualitative in- analysis and shift to case studies of class-
quiry. We tum now to how to think through rooms instead of schools, or add case studies
and actually analyze qualitative data with of particular teachers or students. Contrari-
emphasis on where the real work takes wise, one could begin by sampling class-
placein your head. rooms and end up doing case studies on
schools. This illustrates the criticai impor-
tance of thinking carefully about the ques-
S Case Studies tion "What is a case?" (Ragin and Becker
1992).
Case study is not a methodological choice but The case study approach to qualitative
a choice of what is to be studied We could analysis constitutes a specific way of collect-
study it analytically or holistically, entirely by ing, organizing, and analyzing data; in that
repeated measures or hermeneutically, organ- sense it represents an analysis process. The
ically or culturally, and by mixed meth- purpose is to gather comprehensive, sys-
odsbut we concentrate, at least for the time tematic, and in-depth information about
being, on the case. (Stake 2000:435) each case of interest. The analysis process re-
sults in a product: a case study. Thus, the term
Case analysis involves organizing the case study can refer to either the process of
data by specific cases for in-depth study and analysis or the product of analysis, or both.
comparison. Well-constructed case studies Case studies may be layered or nested.
are holistic and context sensitive, two of the For example, in evaluation, a single program
primary strategic themes of qualitative in- may be a case study. However, within that
quiry discussed in Chapter 2. Cases can be single-program case {N = 1), one may do
individuais, groups, neighborhoods, pro- case studies of several participants. In such
grams, organiza tions, cultures, regions, or an approach, the analysis would begin with
nation-states. "In an ethnographic case the individual case studies; then the
study, there is exactly one unit of analysis cross-case pattern analysis of the individual
the community or village or tribe" (Ber- cases might be part of the data for the pro-
nard 1995:35-36). Cases can also be criticai gram case study. Likewise, if a national or
incidents, stages in the life of a person or pro- state program conssts of several project
gram, or anything that can be defined as a sites, the analysis may consist of three layers
"specific, unique, bounded system" (Stake of case studies: individual participant case
2000:436). Cases are units of analysis. What studies at project sites combined to make up
constitutes a case, or unit of analysis, is usu- project site case studies, project site case
ally determined during the design stage and studies combined to make up state program
becomes the basis for purposeful sampling case studies, and state programs combined
in qualitative inquiry (see Chapter 5, espe- to make up a national program case study.
cially Exhibit 5.5). Sometimes, however, new Exhibit 8.4 shows this layered case study ap-
units of analysis, or cases, emerge during proach.
fieldwork or from the analysis af ter data col- This kind of layering recognizes that you
lection. For example, one might have sam- can always build larger case units out of
pled schools as the unit of analysis, expect- smaller ones; that is, you can always com-
ing to do case studies of three schools, and bine studies of individuais into studies of a
then, reviewing the fieldwork, decide that program, but if you only have program-
IfcMLJsKM Case Study: Layers of Possible Analysis
Case Study
Single National Program
levei data, you can't disaggregate it to con- Collect data about individuais, for exam-
struct individual cases. pie, rather than about households. If you are
interested in issues of production and con-
Remember this rule: No matter what you are sumption (things that make sense at the
studying, always collect data on the lowest household levei), you can always package
levei unit of analysis possible your data about individuais into data about
households during analysis. . . . You can al- levei, case data can include program docu-
ways aggregate data collected on individuais, ments, statistical profiles, program reports
but you can never disaggregate data collected and proposals, interviews with program
on groups. (Bernard 1995:37) participants and staff, observations of the
program, and program histories.
Though a scholarly or evaluation project
may consist of several cases and include From Data to Case Study
cross-case comparisons, the analysfs first
and foremost responsibility consists of do- Once the raw case data have been accu-
ing justice to each individual case. Ali else mulated, the researcher may write a case re-
depends on that. cord. The case record pulls together and or-
ganizes the voluminous case data into a
Ultimately, we may be interested in a general
comprehensive, primary resource package.
phenomenon or a population of cases more
The case record includes ali the major infor-
than in the individual case. And we cannot un-
mation that will be used in doing the final
derstand this case without knowing about
case analysis and writing the case study. In-
other cases. But while we are studying it, our
formation is edited, redundancies are sorted
meager resources are concentrated on trying
out, parts are fitted together, and the case re-
to understand its complexities. For the while,
cord is organized for ready access chrono-
we probably will not study comparison cases.
logically and/or topically. The case record
We may simultaneously carry on more than
must be complete but manageable; it should
one case study, but each case study is a concen-
include ali the information needed for sub-
trated inquiry into a single case. (Stake
sequent analysis, but it is organized at a levei
2000:436)
beyond that of the raw case data.
Case data consist of ali the information A case record should make no concessions to
one has about each case: interview data, ob- the reader in terms of interest or communica-
servations, the documentary data (e.g., pro- tion. It is a condensation of the case data aspir-
gram records or files, newspaper clippings), ing to the condition that no interpreter
impressions and statements of others about requires to appeal behind it to the raw data to
the case, and contextual informationin ef- sustain an interpretation. Of course, this crite-
fect, ali the information one has accumu- rion cannot be fully met: some case records
lated about each particular case goes into will be better than others. The case record of a
that case study. These diverse sources make school attempts a portrayal through the orga-
up the raw data for case analysis and can nization of data alone, and a portrayal without
amount to a large accumulation of material. theoretical aspirations. (Stenhouse 1977:19)
For individual people, case data can include
interviews with the person and those who The case record is used to construct a case
know her or him, clinicai records, back- study appropriate for sharing with an in-
ground and statistical information about the tended audience, for example, scholars,
person, a life history profile, things the per- policymakers, program decision makers, or
son has produced (diaries, photos, writings, practitioners. The tone, length, form, struc-
paintings, etc.), and personality or other test ture, and format of the final case presenta-
results (yes, quantitative data can be part of tion depend on audience and study pur-
a qualitative case study). At the program pose. The final case study is what will be
The Process of Constructing Case Studies
Step 1 Assemble the raw case data.

These data consist of ali the information collected about the person,
program, organization, or setting for which a case study is to be
written.
Step 2 Construct a case record.
(optional; depends This is a condensation of the raw case data organized, classified, and
on complexity of edited into a manageable and accessible file.
data and case)
Step 3 Write a final case studv narra tive.
The case study is a readable, descriptive picture of or story about a
person, program, organization, and so for th, making accessible to
the reader ali the information necessary to understand the case in ali
its uniqueness. The case story can be told chronologically or pre-
sented thematically (sometimes both). The case study offers a holis-
tic portrayal, presented with any context necessary for understand-
ing the case.
communicated in a publication or report. lowing the reader to understand the case as a

The full report may include several case unique, holistic entity. At a later point in
studies that are then compared and con- analysis, it is possible to compare and con-
trasted, but the basic unit of analysis of such trast cases, but initially each case must be
a comparative study remains the distinct represented and understood as an idiosyn-
cases and the credibility of the overall find- cratic manifestation of the phenomenon of
ings will depend on the quality of the indi- interest. A case study should be sufficiently
vidual case studies. Exhibit 8.5 shows this detailed and comprehensive to illuminate
sequence of moving from raw case data to the focus of inquiry without becommg bor-
the written case study. The second ing and laden with trivia. A skillfully crafted
stepconverting the raw data to a case re- case reads like a fine weaving. And that, of
cord before writing the actual case study course, is the trick. How to do the weaving?
is optional. A case record is constructed How to tell the story? How to decide what
only when a great deal of unedited raw data stays in the final case presentation and what
from interviews, observations, and docu- gets deleted along the way? Elmore Leonard
ments must be edited and organized before (2001:7), the author of Glitz and other popu-
writing the final case study. In many studies, lar detective thrillers, was once asked how
the analyst will work directly and selectively he managed to keep the action in his books
from raw data to write the final case study. moving so quickly. He said, "I leave out the
The case study should take the reader into parts that people skip." Not bad advice for
the case situation and experiencea per- writing an engaging case study.
son^ life, a group's life, or a program/s life. In doing biographical or life history case
Each case study in a report stands alone, al- studies, Denzin (1989b) has found particular
value in identifying what he calls "epipha- working hardstill ended up in a "cata-

nies""existentially problematic moments strophic failure" (Levy 2001).
in the lives of individuais" (p. 129). Studying such examples is one of the best
ways to learn how to write case studies. The
It is possible to identify four major structures, Thick Description section, earlier in this
or types of existentially problematic moments, chapter, cited a number of case studies that
or epiphanies, in the lives of individuais. First, have become classics in the genre. Chapter 1
there are those moments that are major and presented case vignettes of individuais in an
touch every fabric of a person's life. Their ef- adult literacy program. Chapter 4 (Exhibit
fects are immediate and long term. Second, 4.2) presented highlights of a participant
there are those epiphanies that represent erup- case study used to illuminate a Vietnamese
tions, or reactions, to events that have been go- woman's experience in an employment
ing on for a long period of time. Third are training program; in addition to describing
those events that are minor yet symbolically what a job placement meant to her, the case
representative of major problematic moments was constructed to illuminate such hard-
in a relationship. Fourth, and fmally, are those to-measure outcomes as "understanding the
episodes whose effects are immediate, but American workplace culture" and "speak-
their meanings are only given later, m retro- ing up for oneself," learnings that can be crit-
spection, and in the relving of the event. I give icai to long-term job success for an emigrant.
the following names to these four structures of Another example of a full individual case
problematic experience: (1) the major epiph- study is presented as Appendix 8.2 at the
any, (2) the cumulative epiphany, (3) the end of this chapter. Originally prepared for
illuminative, minor epiphany, and (4) the re- an evaluation report that included several
lived epiphany. (Of course, any epiphany can participant case studies, it tells the story of
be relived and given new retrospective mean- one person's experiences in a career educa-
ing.) These four types may, of course, build tion program. This case represents an exem-
upon one another. A given event may, at dif- plar of how multiple sources of information
ferent phases in a person's or relationship's can be brought together to offer a compre-
life, be first, major, then minor, and then later hensive picture of a person's experience, in
relived. A cumulative epiphany will, of this instance, a studenfs changing involve-
course, erupt into a major event in a person's ment in the program and changing attitudes
life. (Denzin 1989b:129) and behaviors over time. The case data for
each student in the evaluation study in-
Programs, organizations, and communities cluded
have parallel types of epiphanies, though
they're usually called criticai incidents, cri- (a) observations of selected students at
ses, transitions, or organizational lessons employer sites three times during the
learned. For a classic example of an organi- year,
zational development case study in the busi- (b) interviews three times per year with
ness school tradition, see the analysis of the the students' employer-instructors at
Nut Island sewage treatment plant in the time of observation,
Quincy, Massachusetts, the complex story of
how an outstanding teamhighly compe- (c) parent interviews once a year,
tent, deeply committed to excellence, fo- (d) in-depth student interviews four
cused on the organizational mission, and times a year,
(e) informal discussions with program dix 8.2) exemplifies how an individual case
staff, study can be prepared and presented.
(f) a review of student projects and The same rigorous process would apply
other documents, and to case study data at the group or program
levei. For excellent examples of case studies
(g) 23 records from the files of each stu- in education, see Brizuela et al. (2000), Stake,
dent (including employer evalua- Bresler, and Mabry (1991), Perrone (1985),
tions of students, student products, and Alkin, Daillak, and White (1979); for
test scores, and staff progress evalua- family research see Sussman and Gilgun
tions of students). (1996); for international development see
Salmen (1987) and Searle (1985); in govern-
ment accountability see Kloman (1979); and
A set of guide questions was prepared for
for a detailed example of conducting and
analyzing and reviewing each source presenting an evaluation case study, see
(Fehrenbacher, Owens, and Haehnn 1976: Hbert (1986).
7-8). Information from ali of these sources
How one compares and contrasts cases
was rntegrated to produce a highly readable will depend on the purpose of the study and
narrative that could be used by decision how cases were sampled. As discussed in
makers and funders to better understand Chapter 5, criticai cases, extreme cases, typi-
what it was like to be in the program cal cases, and heterogeneous cases serve dif-
(Owens, Haehnn, and Fehrenbacher 1987). ferent purposes. Other excellent resources
The evaluation staff of the Northwest Re- for qualitative case analysis include Stake
gional Educational Laboratory took great (1995), Merriain (1997), Yin (1994), Hamel
pains to carefully validate the information in (1993), and the U.S. General Accounting Of-
the case studies. Different sources of infor- fice (1987). To pursue case studies as stories
mation were used to cross-validate findings, that build on and display the elements of
pattems, and conclusions. Two evaluators good storytelling, see Glesne (1999).
reviewed the material in each case study to Once case studies have been written, the
independently make judgments and inter- analytic strategies described in the remain-
pretations about the content and meaning of der of this chapter can be used to fur ther an-
the material in the case. In addition, an exter- alyze, compare, and interpret the cases to
nai evaluator reviewed the raw data to check generate cross-case themes, pattems and
for biases or unwarranted conclusions. Stu- findings.
dents were asked to read their own case
studies and comment on the accuracy of fact
and interpreta tion in the study. Finally, to
guarantee the read ability of the case studies,
=L Pattern, Theme,
a newspaper journalist was employed to
and Content Analysis
help organize and edit the final versions.
Such a rigorous case study approach in- The ability to use thematic analysis appears
creases the confidence of readers that the to involve a number of underlying abilities, or
cases are accurate and comprehensive. Both competencies. One competency can be called
in its content and the process by which it was pattern recognition. It is the ability to see pat-
constructed, the Northwest Lab case study tems in seemingly random information.
presented at the end of this chapter (Appen- (Boyatzis 1998:7)
No precise or agreed-on terms describe Inductive and Deductive

varieties and processes of qualitative analy- Qualitative Analyses
sis. Content analysis, for example, some-
times refers to searching text for recurring Francis Bacon is known for his emphasis on in-
words or themes. For example, a speech by a duction, the use of direct observation to confirm
politician might be analyzed to see what ideas and the linking together of observed facts to
phrases or concepts predominate, or form theories or explana tions ofhow natural phe-
speeches of two politicians might be com- nomenon zuork. Bacon correctly never told us
pared to see how many times and in what how to get ideas or how to accomplish the
contexts they used a phrase such as "global linkage of empirical facts. Those activities re-
economy" or "family values." Content anal- main essentially humanisticyou think hard.
ysis usually refers to analyzing text (inter- (Bernard 2000:12)
view transcripts, diaries, or documents)
rather than observation-based field notes. Bacon (1561-1626) is recognized as one of
More generally, however, content analysis the founders of scientific thinking, but he
is used to refer to any qualitative data re- also has been awarded "the dubious honor
duction and sense-making effort that takes of being the first martyr of empiricism" (Ber-
a volume of qualitative material and at- nard 2000:12). Still pondering the uni verse
tempts to identify core consistencies and at age 65, he got an idea one day while driv-
meanings. Case studies, for example, can be ing his carriage in the snow in a farming area
content analyzed. north of London. It occurred to him that cold
The core meanings found through con- might delay the biological process of putre-
tent analysis are often called patterns or faction, so he stopped, purchased a hen from
themes. Alternatively, the process of search- a farmer, killed it on the spot, and stuffed it
ing for patterns or themes may be distin- with snow. His idea worked. The snow did
guished, respectively, as pattem analysis or delay the rotting process, but he caught
theme analysis. I'm asked frequently about bronchitis and died a month later. As I noted
the difference between a pattem and a in Chapter 6, fieldwork can be risky. En-
theme. There's no hard-and-fast distinction. gaging in analysis, on the other hand, is sel-
The term pattem usually refers to a descrip- dom life threatening, though you do risk be-
tive finding, for example, "Almost ali partic- ing dsputed and sometimes ridiculed by
ipants repor ted feeling fear when they those who arrive at contrary conclusions.
rappelled down the cliff," while a theme Inductive analysis involves discovering pat-
takes a more categorical or topical form: Fear. terns, themes, and categories in one's data.
Putting these terms together, a report on a Findings emerge out of the data, through the
wilderness education study might state: analysfs interactions with the data, in con-
trast to deductive analysis where the data are
The content analysis revealed a pattem of par- analyzed according to an existing frame-
ticipants reporting being afraid when rap- work. Qualitative analysis is typically in-
pelling down cliffs and running river rapids; ductive in the early stages, especially when
many also initially experienced the group pro- developing a codebook for content analysis
cess of sharing personal feelings as evoking or figuring out possible categories, patterns,
some fear. Those patterns make "Dealing with and themes. This is often called "open cod-
fear" a major theme of the wilderness educa- ing" (Strauss and Corbin 1998:223) to em-
tion program experience. phasize the importance of being open to the
data. "Grounded theory" (Glaser and work developed by someone else (e.g., test-
Strauss 1967) emphasizes becoming im- ing Piagefs developmental theory on case
mersed in the databeing groundedso that studies of children). After or alongside this
embedded meanings and relationships can deductive phase of analysis, the researcher
emerge. The French would say of such an strives to look at the data afresh for undis-
immersionprocess: Jem'enracine. "Irootmy- covered patterns and emergent understand-
self." The analyst becomes implanted in the ings (inductive analysis). IT1 discuss both
data. The resulting analysis grows out of grounded theory and analytic deduction at
that groundedness. greater length later in this chapter.
Once patterns, themes, and/or categories Because, as identified and discussed in
have been established through inductive Chapter 2, inductive analysis is one of the
analysis, the final, confirmatory stage of primary characteristics of qualitative in-
qualitative analysis may be deductive in quiry, we'11 focus on strategies for thinking
testing and affirming the authenticity and and working inductively. There are two dis-
appropriateness of the inductive content tinct ways of analyzing qualitative data in-
analysis, including carefully examining deductively. First, the analyst can identify, de-
viate cases or data that don't fit the catego- fine, and elucidate the categories developed
ries developed. Generating theoretical prop- and articulated by the people studied to fo-
ositions or formal hypotheses after cus analysis. Second, the analyst may also
inductively identifying categories is consid- become aware of categories or patterns for
ered deductive analysis by grounded theo- which the people studied did not have labels
rists Strauss and Corbin (1998): "Anytime or terms, and the analyst develops terms to
that a researcher derives hypotheses from describe these inductively generated catego-
data, because it involves interpretation, we ries. Each of these approaches is described
consider that to be a deductive process" below.
(p. 22). Grounded theorizing, then, involves
both inductive and deductive processes: "At
the heart of theorizing lies the interplay of Indigenous Concepts
making inductions (deriving concepts, their and Practices
properties, and dimensions from data) and
deductions (hypothesizing about the rela- A good place to begin inductive analysis
tionships between concepts)" (Strauss and is to inventory and define key phrases,
Corbin 1998:22). terms, and practices that are special to the
Analytic induction, in contrast to people in the setting studied. What are the
grounded theory, begins with an analysfs indigenous categories that the people inter-
deduced propositions or theory-derived hy- viewed have created to make sense of their
potheses and "is a procedure for verifying world? What are practices they engage in
theories and propositions based on qualita- that can be understood only within their
tive data" (Taylor and Bogdan 1984:127). worldview? Anthropologists call this emic
Sometimes, as with analytic induction, qual- analysis and distinguish it from etic analysis,
itative analysis is first deductive or quasi- which refers to labels imposed by the re-
deductive and then inductive as when, for searcher. (For more on this distinction and
example, the analyst begins by examining its origins, see Chapter 6, which discusses
the data in terms of theory-derived sensitiz- emic and etic perspectives m fieldwork.)
ing concepts or applying a theoretical frame- "Identifying the categories and terms used
by informants themselves is also called in stand the thoughts of a people, the whole
vivo coding" (Bemard and Ryan 1998:608). analysis of experience must be based on
Consider the practice among traditional their concepts, not ours" (p. 314).
Dani women of amputating a finger joint In an intervention program, certain terms
when a rela tive dies. The Dani people live in may emerge or be created by participants to
the lush Baliem Valley of Irian lava, Indon- capture some essence of the program. In the
sia^ most remote province on the western wilderness education program I evaluated,
half of New Guinea. The joint is removed to the idea of "detoxification" became a power-
honor and placate ancestral ghosts. Mission- ful way for participants to share meaning
aries have fought the practice as sinful and about what being in the wilderness together
the government has banned it as barbaric, meant (Patton 1999a:49-52). In the Carib-
but many traditional women still practice it. bean Extension Project evaluation, the term
liming had special meaning to the partici-
Some women in Dani villages have only four pants. Not really transiatable, it essentially
stubs and a thumb on each hand. In tribute to means passing time, hanging out, doing
her dead mother and brothers, Soroba, 38, has nothing, shooting the breezebut doing so
had the tops of six of her fingers amputated. agreeably, without guilt, stress, or a sense
"The first time was the worst/' she said. "The that one ought to be doing something more
pain was so bad, I thought I would die. But it's productive with one's time. Liming has posi-
worth it to honor my family." (Sims 2001:6) tive, desirable connotations because of its
social, group meaningpeople just enjoy-
Analyzing such an indigenous practice being being together with nothing that has to
gins by understanding it from the perspec- be accomplished. Given that uniquely Ca-
tive of its practitioners, within the indige- ribbean term, what does it mean when par-
nous context, in the words of the local peo- ticipants describe what happened in a train-
ple, in their language, within their world- ing session or mstructional field trip as
view. primarily "liming"? How much liming
could acceptably be built into training for
According to this view, cultural behavior participant satisfaction and still get some-
should always be studied and categorized in thing done? How much programmatic lim-
terms of the inside viewthe actors' defini- ing was acceptable? These became key for-
tionof human events. That is, the units of mative evaluation issues.
conceptualization in anthropological theories In evaluating a leadership training pro-
should be "discovered" by analyzing the cog- gram, we gathered extensive data on what
nitive processes of the people studied rather participants and staff meant by the term
than "imposed" from cross-cultural (hence, leadership. Pretraining and posttraining exer-
ethnocentric) classifications of behavior. cises involved participants in writing a para-
(Peito and Peito 1978:54) graph on leadership; the writing was part of
the program curriculum, not designed for
Anthropologists, working cross-cultur- evaluation, but the results provided useful
ally, have long emphasized the importance qualitative evaluation data. There were
of preserving and reporting the indigenous small group discussions on leadership. The
categories of people studied. Franz Boas training included lectures and group discus-
(1943) was a major influence in this direc- sions on leadership, which we observed. We
tion: "If it is our serious purpose to under- participated in and took notes on informal
discussions about leadership. Because the He used the idea of victimization to tie to-
very idea of "leadership" was central to the gether the foowing quotes from police
program, it was essential to capture varia- officers:
tions in what participants meant when they
talked about leadership. The results showed As a police oficer and as an individual I think
that ongoing confusion about what leader- I have lost the ability to feel and to empathize
ship meant was one of the problematic is- with people. I had a little girl that was run over
sues in the program. Leadership was an by a bus and her mother was there and she had
indigenous concept in that staff and partici- her little bookbag. It was really sad at the time
pants throughout the training experience but I remember feeling absolutely nothing. It
used it extensively, but it was also a sensitiz- was like a mannequin on the street instead of
ing concept because we knew going into the some little girl. I really wanted to be able to cry
fieldwork that it would be an important no- about it and I really wanted to have some feel-
tion to study. ings about it, but I couldn't. It's a little fright-
ening for me to be so callous and I have been
Sensitizing Concepts unable to relax.
In contrast to purely indigenous con- I am paying a price by always being on edge

cepts, sensitizing concepts refer to catego- and by being alone. I have become isolated
ries that the analystbrings to the data. Expe- from old friends. We are different. I feel sepa-
rienced observers often use sensitizing rate from people, different, out of step. It be-
concepts to orient fieldwork, an approach comes easier to just be with other police
discussed in Chapter 6. These sensitizing officers because they have the same basic un-
concepts have their origins in social science derstanding of my environment, we speak the
theory, the research literature, or evaluation same language. The terminology is crude.
issues identified at the beginning of a study. When I started I didn't want to get into any
Sensitizing concepts give the analyst "a gen- words like "scumbags" and "scrotes," but it so
eral sense of reference" and provide "direc- aptly describes these people.
tions along which to look" (Blumer 1969:
148). Using sensitizing concepts involves ex- I have become isolated from who I was be-
amining how the concept is manifest and cause I have seen many things I wish I had not
given meaning in a particular setting or seen. It's frustrating to see things that other
among a particular group of people. people don't see, won't see, can't see. I wish
Conroy (1987) used the sensitizing con- sometimes, I didn't see the thmgs. I need to be
cept "victimization" to study police officers. assertive, but don't like it. I have to put on my
Innocent citizens are frequently thought of police mask to do that. But now it is getting
as the victims of police brutality or indiffer- harder and harder to take that mask off. I take
ence. Conroy turned the idea of victim my workhome with me. I don'twantmy work
around and looked at what it would mean to to invade my personal life but I'm finding I
study police officers as victims of the experi- need to be alone more and more. I need time to
ences of law enforcement. He found the sen- recharge my batteries. I don't like to be alone,
sitizing concept of victimization helpful in but m u s t (Conroy 1987:52)
understanding the isolation, lack of inter-
personal affect, cynicism, repressed anger, Two additional points are worth making
and sadness observed among police officers. about these quotations. First, by presenting
the actual data on which the analysis is the world into parts along a continuum. They
based, the readers are able to make their own differ from taxonomies, which completely
determination of whether the concept "vic- classify a phenomenon through mutually ex-
timization" helps in making sense of the clusive and exhaustive categories, like the
data. By presenting respondents in their biological system for classifying species.
own words and reporting the actual data Typologies, in contrast, are built on ideal-
that was the basis of his interpretation, types or illustrative endpoints rather than a
Conroy invites readers to make their own complete and discrete set of categories. Well-
analysis and interpretation. The analysfs known and widely used sociological typolo-
constructs should not dominate the analysis, gies include Redfield's folk-urban continuum
but rather should facilitate the reader's un- (gemeinschaft/gesellschaft) and Von Wiese's
derstanding of the world under study. and Becker's sacred-secular continuum (for
Second, these three quotations illustrate details, see Vidich and Lyman 2000:52). So-
the power of qualitative data. The point of ciologists classically distinguish ascribed
analysis is not simply to find a concept or la- from achieved characteristics. Psychologists
bel to neatly tie together the data. What is distinguish degrees of mental illness (neuro-
important is understanding the people stud- ses to psychoses). Political scientists classify
ied. Concepts are never a substitute for di- governmental systems along a democratic-
rect experience with the descriptive data. authoritarian continuum. Economists distin-
What people actually say and the descrip- guish laissez-faire from centrally planned
tions of events observed remain the es- economic systems. Systems analysts distin-
sence of qualitative inquiry. The analytical guish open from closed systems. In ali of
process is meant to organize and elucidate these cases, however, the distmctions in-
telling the story of the data. Indeed, the volves matters of degree and interpretation
skilled analyst is able to get out of the way of rather than absolute distinctions. Ali of
the data to let the data tell their own story. these examples have emerged from social
The analyst uses concepts to help make science theory and represent theory-based
sense of and present the data, but not to the typologies constructed by analysts. WeTl ex-
point of straining or forcing the analysis. The amine that approach in greater depth in a
reader can usually tell when the analyst is moment. First, however, let's look at identi-
more interested in proving the applicability fying indigenous typologies as a form of
and validity of a concept than in letting the qualitative analysis.
data reveal the perspectives of the people in- Illuminating indigenous typologies re-
terviewed and the intricacies of the world quires an analysis of the continua and dis-
studied. tinctions used by people in a setting to break
Having suggested how singular concepts up the complexity of reality into distinguish-
can bring focus to inductive analysis, the able parts. The language of a group of people
next levei of analysis, constructing typolo- reveals what is important to them in that
gies, moves us into a somewhat more com- they name something to separate and distin-
plex analytical strategy. guish it from other things with other names.
Once these labels have been identified from
Indigenous Typologies an analysis of what people have said during
fieldwork, the next step is to identify the at-
Typologies are classification systems made tributes or characteristics that distinguish
up of categories that divide some aspect of one thing from another. In describing this
kind of analysis, Charles Frake (1962) used derlines and chronics were treated. Many
the example of a hamburger. Hamburgers teachers, for example, refused even to at-
can vary a great deal in how they are cooked tempt to deal with chronics. They consid-
(rare to well-done) or what is added to them ered it a waste of their time. Students, it
(pickles, mustard, ketchup, lettuce), and turned out, knew what labels were applied
they are still called hamburgers. However, to them and how to manipulate these labels
when a piece of cheese is added to the meat, to get more or less attention from teachers.
it becomes a cheeseburger. The task for the Students who wanted to be left alone called
analyst is to discover what it is that separates themseives "chronics" and reinforced their
"hamburger" from "cheeseburger," that is, "chronic" image with teachers. Students
to discern and report "how people construe who wanted to graduate, even if only barely
their world of experience from the way they and with minimal school attendance, culti-
talk about it" (Frake 1962:74). va ted an image as "borderline."
An analysis example of this kind comes Another example of an indigenous
from a formative evaluation aimed at reduc- typology emerged in the wilderness educa-
ing the dropout rate among high school stu- tion program I evaluated. During the second
dents. In observations and interviews at the year of the project, one subgroup's members
targeted high school, it became important to started calling themselves the "turtles."
understand the ways in which teachers cate- They contrasted themselves to the "truck-
gorized students. With regard to problems ers." On the surface, these labels were aimed
of truancy, absenteeism, tardiness, and skip- at distinguishing different styles of hiking
ping class, the teachers had come to label and backpacking, one slow and one fast. Be-
students as either "chronics" or "border- neath the surface, however, the terms came
lines." One teacher described the chronics as to represent different approaches to the wil-
"the ones who are out of school ali the time derness and different styles of experience in
and everything you do to get them in doesn't relation to the wilderness and the program.
work." Another teacher said, "You can al- Groups, cultures, organizations, and fam-
ways pick them out, the chronics. They're ilies develop their own language systems to
usually the same kids." The borderlines, on emphasize distinctions they consider im-
the other hand, "skip a few classes, waiting portant. Every program gives rise to a spe-
for a response, and when it comes they cial vocabulary that staff and participants
shape up. They're not so different from your use to differentiate types of activities, kinds
typical jnior high student, but when they of participants, styles of participation, and
see the chronics getting away with it, they variously valued outeomes. These indige-
get more brazen in their actions." Another nous typologies provide clues to analysts
teacher said, "Borderlines are gone a lot but that the phenomena to which the labels refer
not constantly like the chronics." are important to the people in the setting and
Not ali teachers used precisely the same that to fully understand the setting it is nec-
criteria to distinguish chronics from border- essary to understand those terms and their
lines, but ali teachers used these labels in implications.
talking about students. To understand the
program activities directed at reducing high Analyst-Constructed Typologies
school dropouts and the differential impact
of the program on students, it became im- Once indigenous concepts, typologies,
portant to observe differences in how bor- and themes have been surfaced, the analyst
BEHffl WrtMUG -'Miiiv::^ :-H:.iLMa a n u OH*ULr WAftZ;) AJi/il..
=;!!:!! rpi tf D15 !:i! I: h I :=!;: m !' 1..I ==

.= :lllll
::== = 11:- || j) y | j i ji| 11 jj i I [ | j|| i| |i 1111 ]!
I-IVI,'.:-!:Jro:; ^ t m {oR:-I.T/MI::; u! po*:/=!S: v\V\. :;!'."!'.! .!!: W"!! .!!!!!.!'! !i'O!""'!O.!-! W"!:M
:;:; nctfn: hîPT :vi:-V; rf:u! .
/i ! i:-i fri::' !o nMiVITI' e,iIVI.-':;i h =; C!;-:ibry a;!:T! ==Jir: VM! i=:\=-i;'r or
.-: i 'V g i x \ wi? o r i? : !.; n? s vi iV! iVi'!!!!'!:' iV' ir'1!! i i ! \ J n:=!' ,;i ;i
.-:I ,';::!. !RI'I I:-:!: I'
1 ::!:,::S j||iSrti|ii.t.'fLSlrtilT iWi-vir. s-iin
, l , î:!:=='
IV! r I - 1 : .V I : : . =: .; A W x ; N . I R U r M-!. '-
1
i'!'. Wd i i^V^.t-:. !:'.M" 'Si! "i" ! ; ' " ^ !!
I=p.::it:- RFIITV I A I; ;':: s-:="s v. =:J: I : I ! : I , ! : I J . ' ! : ! . ' I ! J ' ' >:I' !:'I:! :
1 !
>: !'!!! EUi? !:u: "i- cwkriE !::i;:n:!'!;:! = -! ir.
:. 11:. i I- !;>:'.; i/i'i . V l i iV .j i.Yi .!: iVft! 11 i
! ' " d ! ' :! "l 1!;-:!
i i S r!-:iYHY !: hiiS'':; VM:'! w r w w i w .-n-aj.-
ri Iv .-i Vi :,'i: .-ri "V !!tfr/d.V,'!=T = - !: i=s "= ' 1 => J.
itHiilViW! i'i': 'li.i.:.:^ i-W ii'-:' iii"!!. i.'jv' n-n=;v
T'it NI.1!.! '! "V.'i i" ! i!':ii "".vi :".!i i.-! r;::i:; r-q drti^ftr !;ii;V'':- jVis:.vr!-]!1 iVii: '!:!!! :':,!: i.:
1 "i I f.'i l!"' i\ Til! '! I! :! i 'Ai :i ifcj!.!>::: i^ 'i,'i '-a I;= !'!C.!!; iii!!=!!!.'!! : .:'!;!:N; !>;'.:::= I -;: ::; ; - ;
IV!.': .i " ! =': ::; I i i i! i JL lis: i-.: ils !ii-!'.!> s.=.si i: i' -.11. .1 l i Oi& I\ & >!:& & . ! ;: - ! ! 1 O V I V K : : ! : ; ; ' H * !'
!
i!' Iii:' I-K::l'!. .ii!ii.ivi!-i O !v.\ '.i!' ii"':: .'VLICIIIIK-I'!'! V !'!.';!.:; I F C : ! K ' - S !:I :V-::::!;!I;R
ATHI >1! \A' I.. IV/i1 i''iI i -i- il jYi fhiv fi^vi-i ,!': ! s ; : ' iTr. V v ^" W !
i'!:!it!'lr l!Vi:':i.r ! jii i^Tl h;: :; py d , ^ li.':! :! diWfi flSTK J t :!'!:!!:!'.'':;:.!: in-s iW^-n. -y
r. II i v."i i i I'i:/.' .'-i!. I -I : IV.Y IVi : u! i>. i.iîifei;! hi!.'i! iVi !!:I'::! ir? S - i ^ !;-,'!! |.: wir:!!.;
!
i! 11 i.! 'v:t I- S y : I- n'i :fi!'! ==i: = i-.cd l hi :i j::!i h:iT !"!-: i" !:' ! Vi M i i'" r iVi iii i/. i iV I
I:..I . . . . i i ,
<II i i.n'ii'i n 1"i"i i " i v 111
1 - - Jj rw.x :s j!i'?.r :I nV:;-:1 : J i I.i' l- rj-h u-.j
!:1 S.:iii .1 - R A - I I P L! ! " I ! . I I ' " S T I ! . I"I/. "ii !!I Vi d i i |1:V!!.'!'::- JOXL, !,!
IVi.;":', i<i'J;r,.!R,r.! h" n:> j i; !",'IV.: I.V " ! V , i i: !=!
moves to a different task of induction Chapter 1 presented my ideal-typical com-

looking for patterns, categories, and themes parison of coming-of-age paradigms that
for which the analyst can construct a contrasts tribal initiation themes with con-
typology to further elucidate findings. Such temporary coming-of-age themes (Patton
constructions must be done with consider- 1999a). A series of patterns is distilled into
able care to avoid creating things that are not contrasting themes that create alternative
really in the data. The advice of biological ideal-types. The notion of "ideal-types"
theorist John Maynard Smith (2000) is infor- makes it explicit that the analyst has con-
mative in this regard: Seek models of the structed and interpreted something that su-
world that make sense and whose conse- persedes purely descriptive analysis.
quences can be worked out, for "to replace a In creating analyst-constructed typolo-
world you do not understand by a model of gies through inductive analysis, you take on
a world you do not understand is no ad- the task of identifying and making explicit
vance" (p. 46). patterns that appear to exist but remain un-
Constructing ideal-types or alternative perceived by the people studied. The danger
paradigms is one simple form of presenting is that analyst-constructed typologies im-
qualitative comparisons. Exhibit 1.3 in pose a world of meariing on the participants
that better reflects the observer's world than not really sure that s/he is going to find
the world under study. One way of testing anything interesting in this particular ex-
analyst-constructed typologies is to present hibit hall. Occasionally the Nomad
them to people whose world is being ana- stops, but it does not appear that the no-
lyzed to find out if the constructions make madic visitor finds any one thing in the
sense to them. hall more interesting than any other
thing.
The best and most stringent test of observer
a The Cafeteria Type: This is the interested
constructions is their recognizability to the
visitor who wants to get interested in
participants themselves. When participants
something, and so the entire museum
themselves say, "yes, that is there, I'd simply
and the hall itself are treated as a cafete-
never noticed it before," the observer can be
ria. Thus, the person walks along, hop-
reasonably confident that he has tapped into
ing to find something of interest, hoping
extant patterns of participation. (Lofland
to "put something on his or her tray" and
1971:34)
stopping from time to time in the hall.
While it appears that there is something
Exhibit 8.6, using the problem of classify- in the hall that spontaneously sparks the
ing people's ancestry, shows what can hap- person's interest, we perceive this visitor
pen when indigenous and official construc- has a predilection to becoming inter-
tions conflict, a matter of some consequence ested, and the exhibit provides the many
to those affected. things from which to choose.
A good example of an analyst-generated
typology comes from an evaluation of the o The V.I.P.Very Interested Person: This
National Museum of Natural History, visitor comes into the hall with some
Smithsonian Institution, done by Robert L. prior interest in the content area. This
Wolf and Barbara L. Tymitz (1978). This has person may not have come specifically to
become a classic in the museum studies the hall, but once there, the hall serves to
field. They conducted a naturalistic inquiry remind the V.I.P.'s that they were, in fact,
of viewers' reactions to the "Ice Age Mam- interested in something in that hall be-
mals and Emergence of Man" exhibit. From forehand. The V.I.P. goes through the hall
their observations, they identified four dif- much more carefully, much slower,
ferent kinds of visitors to the exhibit. These much more critically that is, he or she
descriptions are progressive in that each moves from point to point, stops, exam-
new category identifies a person more seri- ines aspects of the hall with a greater de-
ous about the exhibit hall. gree of scrutiny and care. (Wolf and
Tymitz 1978:10-11)
The Commuter: This is the person who
merely uses the hall as a vehicle to get This typology of types of visitors became
from the entry point to the exit point important in the full evaluation because it
The Nomad: This is a casual visitor, a permitted analysis of different kinds of mu-
person who is wandering through the seum experiences. Moreover, the evaluators
hall, apparently open to becoming inter- recommended that when conducting inter-
ested in something. The Nomad is not reviews to get museum visitors' reactions to
ally sure why he or she is in the hall and exhibits, the interview results should be dif-
E X H I B I T 8.6 Qualitative Analysis of Ancestry at the U.S. Census
To count different kinds of peoplethe job "I went into places where there were
of the Census Bureau-you need categories to young people playing games, went into res-
count them in. The longform of the 2000 cen- taurants, and places where older people gath-
sus, given to 1 in 6 households, asked an ered," says Ms. McKenney She paid a visit
open-ended, fiil-in-the-blank question about to Assyrian neighborhoods in Chicago, where
"ancestry." Analysts then coded the responses a large concentration of Assyrian Americans
into categories, 1 of 604 categories, up from live. At a local community center and later
467 in 1980. The government doesn't ask that day at the Assyrian restaurant next door,
about religion, so if people respond that they community leaders presented their case for
are Jewish, they don't get their ancestry keeping the ancestry code the same. Over the
counted. However, those who write in that same period, she visited Detroit to look into
they are Amish or Mennonite do get counted the Chaldean matter....
because those are considered cultural catego- " found that many of the people, espe-
ries. cially theyoungerpeople,viewed itasan eth-
Ethnic minorities that cross national nic group, not a religion," says Ms. McKenney.
boundaries, such as French and Spanish She and Mr. Reed (Census Bureau ancestry re-
Basques, and groups affected by geopoiitica search expert) concurred that enough differ-
change, such as Czechs and Slovaks or groups ences existed that the Chaldeans could po-
within the former Yugoslavia, are counted in tentiallyquafyasa separate ancestry group.
distinct categories. The Census Bureau, fol- In a conference call between interested
lowing advice from the U.S. State Depart- parties, a compromise was struck. Assyrians
ment, differentiates Taiwanese Americans and Chaldeans would remain under a single
from Chinese Americans, a matter of political ancestry code, but the name would no longer
sensitivity. be Assyrian, it would be Assyrian/Chaldean/
Can Assyrians and Chaldeans be lumped SyriacSyriac being the name of the Aramaic
together? When the Census Bureau an- dialect that Assyrians and Chaldeans speak.
nounced that it would combine thetwo inthe "There was a meeting of the minds between
same ancestry code, an Assyrian group sued ali the representatives, and basically it was a
over the issue, but lost the lawsuit. Assyrian unified decision to say that we're going to go
Americans trace their roots to a biblicai-era under the same name," says the Chaldean
empire covering much of what is now Iraq and Federation's Mr. Yono. (Kulish 2001:1)
believe that Chaldeans are a separate reli-
gious subgroup. A fieldworker for the Census
Bureau did fieldwork on the issue.
ferentially valued depending on the type of "Museum Encounters of the First, Second,
person being interviewedcommuter, no- and Third Kind," a take-off on the popular
mad, cafeteria type, or VIP. science fiction movie Close Encounters ofthe
A different typology was developed to Third Kind, which referred to direct human
distinguish how visitors learn in a museum: contact with visitors from outer space.
o Museum Encounters of the First Kind: This s Skidders: Most often women, typically
encounter occurs in halls that use dis- in their 30s, grew up middle or upper
play cases as the primary approach to class but "skidded" into homelessness as
specimen presentation. Essentially, the divorced or separated parents.
visitor is a passive observer to the "ob- 0 Street people: Mostly men, often veter-
jects of interest." Interaction is visual and ans, rarely married; highly visible net
may occur only at the awareness levei. and know how to use the resources of the
The visitor is probably not provoked to street.
think or consider ideas beyond the visual
display. a Wingnuts: People with severe mental
problems, occasionally due to longterm
alcoholism, a visible subgroup.
n Museum Encounters of the Second Kind:
This encounter occurs in halls that em- Transitory zuorkers: People with job
ploy a variety of approaches to engage skills and a history of full-time work who
the visitor's attention and/or learning. travei from town to town, staying
The visitor has several choices to become months or years in a place, and then
active in his/her participation. . . . The heading off to greener pastures.
visitor is likely to perceive, question,
compare, hypothesize, etc. Categories of how homeless people spend
their time:
Museum Encounters ofthe ThirdKind: This Hanging out
encounter occurs in halls that invite high
Getting by
leveis of visitor participation. Such an
encounter invites the visitor to observe Getting ahead
phenomena in process, to create, to ques-
tion the experts, to contribute, etc. Inter- As these examples illustrate, the first pur-
action is personalized and within the pose of typologies is to distinguish aspects
control of the visitor. (Wolf and Tymitz of an observed pattern or phenomenon de-
1978:39) scriptively. Once identified and distin-
guished, these types can later be used to
Here's a sample of a quite different classi- make interpretations and they can be related
fication scheme, this one developed from to other observations to draw conclusions,
fieldwork by sociologist Rob Rosenthal but the first purpose is description based on
(1994) as "a map of the terrain" of the home- an inductive analysis of the patterns that ap-
less. pear in the data.
13. The Intellectual and Mechanical Work of Analysis
Coding Data, Finding Patterns, Labeling Themes,

and Developing Category Systems
^"~^lassification is Ariadne's clue through the labyrinth of nature.
George Sand, Nouvelles Lettres d'un Voyageur, 1869

Thus far, I've provided lots of examples of at organizing the data into topics and files.
the fruit of qualitative inquiry: patterns, Coming up with topics is like constructing
themes, categories, and typologies. Let's an index for a book or labels for a file system:
back up now to consider how you recognize You look at what is there and give it a name,
patterns in qualitative data and turn those a label. The copy on which these topics and
patterns into meaningful categories and labels are written becomes the indexed copy
themes. This chapter could have started of the field notes or interviews. Exhibit 8.7
with this section, but I think it's helpful to shows a sampling of codes from the field
understand what kinds of findings can be note margins of the evaluation of the wilder-
generated from qualitative analysis before ness education program I described in the
delving very deeply into the mechanics, es- chapter on observation.
pecially because the mechanics vary greatly The shorthand codes are written directly
and are undertaken differently by analysts on the relevant data passages, either in the
in different disciplines and working from di- margins or with an attached tab on the rele-
vergent frameworks. That said, some guid- vant page. Many passages will illustrate
ance can be offered. more than one theme or pattern. The first
Raw field notes and verbatim transcripts reading through the data is aimed at devel-
constitute the undigested complexity of re- oping the coding categories or classification
ality. Simplifying and making sense out of system. Then a new reading is done to actu-
that complexity constitutes the challenge of ally start the formal coding in a systematic
content analysis. Developing some man- way. Several readings of the data may be
ageable classification or coding scheme is necessary before field notes or interviews
the first step of analysis. Without classifica- canbe completely indexed and coded. Some
tion there is chos and confusion. Content people find it helpful to use colored high-
analysis, then, involves identifying, coding, lighting penscolor coding different idea or
categorizing, classifying, and labeling the concepts. Using self-adhesive colored dots
primary patterns in the data. This essentially or Post-it notes offers another option. Some
means analyzing the core content of inter- use a color printer to print out transcripts in
views and observations to determine whafs different colors to make it easy to track the
significant. In explaining the process, TH de- source of a quote when cutting and pasting
scribe it as done traditionally, which is with- different quotes into a theme.
out software, to highlight the thinking and
mechanics involved. Software programs If sensing a pattem or "occurrence" can be
provide different tools and formats for cod- called seeing, then the encoding of it can be
ing, but the principies of the analytical pro- called seeing as. That is, you first make the ob-
cess are the same whether doing it manually servation that something important or notable
or with the assistance of a computer pro- is occurring, and then you classify or describe
gram. it [T]he seeing as provides us with a link be-
I begin by reading through ali of my field tween a new or emergent pattem and any and
notes or interviews and making comments ali patterns that we have observed and consid-
in the margins or even attaching pieces of ered previously. It also provides a link to any
paper or Post-it notes that contain my no- and ali patterns that others have observed and
tions about what I can do with the different considered previously through reading. (Boy-
parts of the data. This constitutes the first cut atzis 1998:4)
464 LEI. ANALYSIS, INTERPRETATION, AND REPORTING
First-Cut Coding Examples:

E X H I B I T 8.7 Sample Codes From the Field Note Margins
Code Ps Re Prog (meaning: participants' reactions to the program)

Code Ps Re Ps (participants' reactions to other participants)
Code Ob PP (observations of participants' interactions)
Code Ob SS (observations of staff s interactions)
Code Ob SP (observations of staff/participant interactions)
Code Phil (statements about program philosophy)
Code Prc {examples of program processes)
Code P/outs (effects of program on participants/outcomes)
Code S - G (subgroup formations)
Code GPrc (group process)
Code C! (conflicts)
Code C-PP (conflicts among participants)
Code C-SP (conflicts between staff and participants)
Code C - S S (conflicts among staff)
NOTE: P = participants, S = staff. These codes are from the field note margins of the evaluation of the wilder-
ness education program described in the chapter on observation.The shorthand codes (abbreviations) are writ-
ten in the margins directly on the reevantdata passages or quotations.The ful labels in parentheses are the
designations for separate files that contain ali simlarly coded passages.
Where more than one person is working provide easy access to the data by any of the
on the analysis, it is helpful to have each per- student or faculty researchers. Had only one
son (or small teams for large projects) de- nvestigator been intending to use the data,
velop the coding scheme independently, such an elaborate classification scheme
then compare and discuss similarities and would not have been necessary. However, to
differences. Important msights can emerge provide access to several students for differ-
from the different ways in which two people ent purposes, every paragraph in every in-
look at the same set of data, a form of analyti- terview was coded using a systematic and
cal triangulation. comprehensive coding scheme made up of
Often an elaborate classification system 15 general categories with subcategories.
emerges during coding, particularly in large Portions of the codebook used to code the
projects where a formal scheme must be de- utilization of evaluation data appear in Ap-
veloped that can be used by several trained pendix 8.1 at the end of this chapter as an ex-
coders. In the study of evaluation use that ample of one kind of qualitative analysis
is the basis for Utilization-Focused Evaluation codebook. This codebook was developed
(Patton 1997a), graduate students in the from four sources: (a) the standardized
evaluation program at the University of open-ended questions used in interviewing,
Minnesota conducted lengthy interviews (b) review of the utilization literature for
with 60 project officers, evaluators, and fed- ideas to be examined and hypotheses to be
eral decision makers. We developed a com- reviewed, (c) our initial inventory review of
prehensive classification system that would the interviews in which two of us read ali the
data arid added categories for coding, and written, he had gone back to the drawing
(d) a few additional categories added during board and was working on a more usable
coding when passages didn't fit well in the coding framework to capture and code the
available categories. stories of program participants in a stan-
Every interview was coded twice by two dardized framework, an approach to be
independent coders. Each individual code, called "journey mapping."
including redundancies, was entered into Classifying and coding qualitative data
our qualitative analysis database so that we produce a framework for organizing and de-
could retrieve ali passages (data) on any scribing what has been collected during
subject included in the classification fieldwork. (For published examples of cod-
scheme, with brief descriptions of the con- ing schemes, see Bernard 1998:325-28,
tent of those passages. The analyst could 387-89, 491-92, 624; Bernard 2000:447-50;
then go directly to the full passages and Boyatzis 1998; Strauss and Corbm 1998;
complete interviews from which passages Miles and Huberman 1994.) This descriptive
were extracted to keep quotations in context. phase of analysis builds a foundation for the
In addition, the computer analysis permit- interpretative phase when meanings are ex-
ted easy cross-classification and cross-com- tracted from the data, comparisons are
parison of passages for more complex analy- made, creative frameworks for interpreta-
ses across interviews. tion are constructed, conclusions are drawn,
Some such elaborate coding system is significance is determined, and, in some
routine for very rigorous analysis of a large cases, theory is generated.
amount of data. Complex coding systems
with multiple coders categorizing every
paragraph in every interview constitute a la- Convergence and Divergence
bor-intensive form of coding, one that in Coding and Classifying
would not be used for small-scale formative
evaluation or action research projects. How- In developing codes and categories, a
ever, where data are going to be used by sev- qualitative analyst must first deal with the
eral people, or where data are going to be challenge of convergence (Guba 1978)figur-
used over a long period of time, including ing out what things fit together. Begin by
additions to the data set over time, such a looking for recurring regularities in the data.
comprehensive and computerized system These regularities reveal patterns that can be
can be well worth the time and effort re- sorted into categories. Categories should
quired. then be judged by two criteria: internai homo-
Kibel (1999) developed a very sophisti- geneity and externai heterogeneity. The first
cated and comprehensive system for coding criterion concerns the extent to which the
stories of successful outcomes attainment data that belong in a certain category hold
that he called "results mapping." His system together or "dovetail" in a meaningful way.
permitted converting individualized stories The second criterion concerns the extent to
into standardized categories that permitted which differences among categories are bold
aggregation, comparison, and even quantifi- and clear. "The existence of a large number
cation. However, it required intensive train- of unassignable or overlapping data items is
ing to use and proved too cumbersome and good evidence of some basic fault in the cat-
demanding for most human services and egory system" (Guba 1978:53). The analyst
educational programs. As this was being then works back and forth between the data
and the classification system to verify the 4. The set should be credible to the persons
meaningfulness and accuracy of the catego- who provided the information which
ries and the placement of data in categories. the set is presumed to assimilate. . . .
If several different possible classification Who is in a better position to judge
systems emerge or are developed, some pri- whether the categories appropriately re-
orities must be established to determine flect their issues and concerns than the
which are more important and illuminative. people themselves? (Guba 1978:56-57)
Prioritizing is done according to the utility,
salience, credibility, uniqueness, heuristic After analyzing for convergence, the mir-
value, and feasibility of the classification ror analytical strategy involves examining
schemes. Finally, the category system or set divergence. By this Guba means the analyst
of categories is tested for completeness. must "flesh out" the patterns or categories.
This is done by processes of extension
1. The set should have internai and exter- (building on items of information already
nai plausibility, a property that might be known), bridging (making connections
termed "integratability." Viewed inter- among different items), and surfacing (pro-
nally, the individual categories should posing new information that ought to fit and
appear to be consistent; viewed exter- then verifying its existence). The analyst
nally, the set of categories should seem brings closure to the process when sources
to comprise a whole picture.... of information have been exhausted, when
sets of categories have been satura ted so that
2. The set should be reasonably inclusive new sources lead to redundancy, when clear
of the data and information that do exist. regularities have emerged that feel inte-
This feature is partly tested by the ab- grated, and when the analysis begins to
sence of unassignable cases, but can be "overextend" beyond the boundaries of the
further tested by reference to the prob- issues and concerns guiding the analysis.
lem that the inquirer is investigating or Divergence also includes careful and
by the mandate given the evaluator by thoughtful examination of data that doesn't
his client/sponsor. If the set of catego- seem to fit including deviant cases that don't
ries did not appear to be sufficient, on fit the dominant identified patterns.
logical grounds, to cover the facets of the This sequence, convergence then diver-
problem or mandate, the set is probably gence, should notbe followed mechanically,
incomplete. linearly, or rigidly. The processes of qualita-
tive analysis involve both techriical and cre-
3. The set should be reproducible by an- ative dimensions. As noted early in this
other competent judge. . . . The second chapter, no abstract processes of analysis, no
observer ought to be able to verify that matter how eloquently named and finely de-
(a) the categories make sense in view of scribed, can substitute for the skill, knowl-
the data which are available, and (b) the edge, experience, creativity, diligence, and
data have been appropriately arranged work of the qualitative analyst. "The task of
in the category system.... The category converting field notes and observations
system auditor may be called upon to at- about issues and concerns into systematic
test that the category system "fits" the categories is a difficult one. No infallible
data and that the data have been prop- procedure exists for performing it" (Guba
erly "fitted into" it. 1978:53).
Determining
Substantive Significance ihTERQOJLMI SIGNIFICA MCE
"F aiir ii.vr : Y i f i IV-ili I:'L n'::i! , ; ' T : ! : ; ! : ! : ' ; ! V ^ I '
: !
:
: :
In lieu of statistical significance, qualita- sfg.iFO-*8**-'* i--ri'!: lTt-rt i:-i;-fi.. . i Vi" iV:',"! :
tive findings are judged by their substantive .|i m;!:;i ;:":. snlV! siJ$fi r|.' :VV vrv :! -i IT
significance. The analyst makes an argument i i v i - vr."r ii! 11rh fn -.}:.ii { n\Vi i.!
for substantive significance in presenting ' I - P1 .V !.':! I'!; :'}] 1" M". " i:\! :i i.Vi I.V li f-
findings and conclusions, but readers and ;]!V;.:-I, ;i !=i;!.':i:! ri s:=v.' !!=:! r' i.VJ ! v: !'
users of the analysis will make their own =."".!
. ! .'n.vrVi . fiv. <i!i:\iln:V!jri hMsii!'l";!iH! dJr'-
value judgments about significance. In de- SrI- ;-.-= ; ; : ,
S -.==;J; "r.-iV vi I r V : " V .VIILI-
termining substantive significance, the ana- v"!.!Jri." .{iViVi"!:!' :"i";;j r.'i r rS-V:';? iijn,^r.'ii:iJ J:i!'
1 !
lyst addresses these kinds of questions: i il li-:: !'!=: i:n | ' i\\ 1;-Y";|.' j Vn-iVn'! i ,;i!!!.'i:j'
How solid, coherent, and consistent is

the evidence in support of the findings?
(Triangulation, for example, can be used
in determining the strength of evidence they should take seriously the responses of
in support of a finding.) those who were studied or participated in
the inquiry; and third, the researcher or eval-
0 To what extent and in what ways do the
uator should consider the responses and re-
findings increase and deepen under-
actions of those who read and review the
standing of the phenomenon studied
results. Where ali threeanalyst, those
(Verstehen)?
studied, and reviewersagree, one has con-
To what extent are the findings consis- sensual validation of the substantive signifi-
tent with other knowledge? (A finding cance of the findings. Where disagreements
supported by and supportive of other emerge, which is the more usual case, you
work has confirmatory significance. A get a more interesting life and the joys of de-
finding that breaks new ground has dis- bate.
covery or innovative significance.) Determining substantive significance can
To what extent are the findings useful for involve the making of the qualitative ana-
some intended purpose (e.g., contribut- lyst's equivalent of Type I and Type II errors
ing to theory, informing policy, summa- from statistics: The analyst may decide that
tive or formative evaluation, or problem something is not significant when in fact it is,
solving in action research)? or, conversely, the analyst may attribute sig-
nificance to something that is meaningless.
The qualitative analysfs effort at un- A story illustrates this problem of making
covering patterns, themes, and categories judgments about what is really significant.
includes using both creative and criticai fac- Halcolm was approached by a woman
ulties in making carefully considered judg- who handed him something. Without hesi-
ments about what is really significant and tation, Halcolm returned the object to the
meaningful in the data. Since qualitative an- woman. The many young disciples who fol-
alysts do not have statistical tests to tell them lowed Halcolm to learn his wisdom began
when an observation or pattern is signifi- arguing among themselves about the special
cant, they must rely first on their own intelli- meaning of this interchange. A variety of in-
gence, experience, and judgment; second, terpretations were offered.
When Halcolm heard of the argument tant/change agents worked with teachers in
among his young followers, he called them the school to help them develop approaches
together and asked each one to report on the to the dropout probiem. Observations of the
significance of what they had observed. program and interviews with the teachers
They offered a variety of interpretations. gave rise to two dimensions. The first di-
When they had finished he said, "The real mension distinguished teachers' beliefs about
purpose of the exchange was to enable me to what kind ofprogrammatic intervention was ef-
show you that you are not yet sufficiently fective with dropouts, that is, whether they
masters of observation to know when you primarily favored maintenance (caretaking
have witnessed a rneaningless interaction." or warehousing of kids to just keep the
schools running), rehabilitation efforts
(helping kids with their problems), or pun-
S Logical Analysis ishment (no longer letting them get away
While working inductively, the analyst is with the infractions they had been commit-
looking for emergent patterns in the data. ting in the past). Teachers' behaviors toward
These patterns, as noted in preceding sec- dropouts could be conceptualized along a
tions, can be represented as dimensions, continuum from taking direct responsibility
categories, classifcation schemes, themes, for doing something about the probiem, at
and categories. Once some dimensions have one end, to shifting responsibility to others
been constructed, using either participant- at the opposite end. Exhibit 8.8 shows what
generated constructions or analyst-gener- happens when these two dimensions are
ated constructions, it is sometimes useful to crossed. Six cells are created, each of which
cross-classify different dimensions to gen- represents a different kind of teacher role in
erate new insights about how the data can response to the program.
be organized and to look for patterns that The evaluator analyst working with these
may not have been immediately obvious in data had been struggling in the inductive
the initial, inductive analysis. Creating analysis to find the patterns that would ex-
cross-classification matrices is an exercise in press the different kinds of teacher roles
logic. manifested in the program. He had tried
The logical process involves creating po- several constructions, but none of them
tential categories by crossing one dimension quite seemed to work. The labels he came up
or typology with another, and then working with were not true to the data. When he de-
back and forth between the data and one's scribed to me the other dimensions he had
logical constructions, filling in the resulting generated, I suggested that he cross them, as
matrix. This logical system will create a new shown in Exhibit 8.8. When he did, he said
typology, ali parts of which may or may not that "the whole thing immediately fell into
actually be represented in the data. Thus, the place." Working back and forth between the
analyst moves back and forth between the matrix and the data, he generated a full de-
logical construction and the actual data in a scriptive analysis of diverse and conflicting
search for meaningful patterns. teacher roles.
In the high school dropout program de- The description of teacher roles served
scribed earlier, the focus of the program was several purposes. First, it gave teachers a
reducing absenteeism, skipping of classes, mirror image of their own behaviors and at-
and tardiness. An externai team of consul- titudes. It could thus be used to help teach-
An Empirical Typology of Teacher Roles in

E X H I B I T 8.8 Dealing With High School Dropouts
Behaviors Toward Dropouts

Shifting Responsibility
Taking Responsibility > to Others
Rehabilitation
A Referral Agent:
Counselor/Friend:
Refer them to other

Help kids directly
helping agencies
Traffic Cop: Ostrich:

Maintenance
(caretaking) Just keep them moving Ignore the situation
5
no through the system and hope someone else
does something
Old-Fashioned
Complainer:
Schoolmaster:
Somebody shouid remove
Punishment Make them feel
the problem kids
the consequences
ers make more explicit their own under- description of teacher roles provided in-
standing of roles. Second, it could be used by sights into the nature of the dropout
the externai team of consultants to more problem. Having identified the various
carefully gear their programmatic efforts to- roles, the evaluator analyst had a responsi-
ward different kinds of teachers who were bility to report on the distribution of roles in
acting out the different roles. The matrix this school and the observed consequences
makes it clear that an omnibus strategy for of that distribution.
helping teachers establish a program that One must be careful about this kind of
would reduce dropouts would not work in logical analysis. It is easy for a matrix to be-
this school; teachers manifesting different gin to manipulate the data as the analyst is
roles would need to be approached and tempted to force data into categories created
worked with in different ways. Third, the by the cross-classification to fill out the ma-
trix and make it work. Logical analysis to occurrence of a particular event, and then
generate new sensitizing concepts must be works back in time in an effort to reconstruct
tested out and confirmed by the actual data. the events (causes) that produced the event
Such logically derived sensitizing con- (consequence) in question. (pp. 109-10)
cepts provide conceptual hypotheses to
test. Levin-Rozalis (2000), following Ameri- Famous fictional detective Sherlock
can philosopher Charles Sanders Pierce of Holmes relied on abduction more than de-
the pragmatic school of thought, suggests la- duction or induction, at least according to a
beling the logical generation and discovery review by William Sanders (1976) of
of hypotheses and findings abduction to dis- Holmes's analytical thinking in The Sociolo-
tinguish such logical analysis from data- gist as Detective. We've already suggested
based inductive analysis and theory-derived that the qualitative analyst is part scientist
deductive analysis. and part artist. Why not add the qualitative
Denzin (1978b) has explained abduction analyst as detective? The empty cell of a logi-
in qualitative analysis as a combination of cally derived matrix (the cell created by
inductive and deductive thinking with logi- crossing two dimensions for which no name
cal underpinnings: or label immediately occurs) creates an in-
tersection of a possible consequence and an-
Naturalists inspect and organize behavior tecedent thatbegs for abductive exploration
specimens in ways which they hope will per- and explanation. Each such intersection of
mit them to progressively reveal and better consequence and antecedent sensitizes the
understand the underlying problematic fea- analyst to the possibility of a category of ac-
tures of the social world under study. They tivity or behavior that either has been over-
seek to ask the question or set of questions looked in the data or is logically a possibility
which will make that world or social organiza- in the setting but has not been manifested.
tion understandable. They do not approach The latter cases are important to note be-
that world with a rigd set of preconceived hy- cause their importance derives from the fact
potheses. They are initially directed toward an that they did not occur. The next section will
interest in the routine and taken-for-granted look in detail at a process/outcomes matrix
features of that world. They ask how it is that ripe with abductive possibihties.
the persons in question know about produc- Nick Smith (1980) used a matrix to draw
ing orderly patterns of interaction and mean- important distinctions among different
ing. . . . They do not use a full-fledged kinds of evaluation use by asking if "tech-
deductive hypothetical scheme in thinking niques of effective evaluation utilization dif-
and developing propositions. Nor are they fer with regard to audience or entity stud-
fully inductive, letting the so-called "facts" ied." His matrix crossed a programs/
speak for themselves. Facts do not speak for policies dimension (what can be studied?)
themselves. They must be interpreted. Previ- with a program managers/policymakers
ously developed deductive models seldom distinction (who is to be aided?) to show dif-
conform with empirical data that are gath- ferent kinds of utilization in each case. Ex-
ered. The method of abduction combines the hibit 8.9 shows a matrix for mapping stake-
deductive and inductive models of proposi- holders' stakes in a program or policy. This
tion development and theory construction. It matrix can be used to guide data collection
canbe defined as workingfrom consequence back as well as analysis. Later this chapter pres-
to cause or antecedent. The observer records the ents a process/outcomes matrix for crossing
nity, district), 10 components of the state-

www orcairai m\.:m* wide project (planning, goal-setting, . . .
; : student involvement), and 10 factors af-
itfwVa !iM!h!:xii'!:!!!j' (MQi n!.!; ?o !:!!; ii:!!!! fecting utilization (personal factor, political
iiii'" rwMm ihwvti M \ i : i ^n-jVn-i . factors, . . . ). Exhibit 8.10 again illustrates
I JFIJ -II WPY
! R Y N ^ I I . N Y N ; ; ! I: ; ..::.: O T Y
! FY. matrix thinking for both data organization
fji^V&i i!i.j ;:I !:'!!J|!!.V:; ; T ! ! V I L E F D ! i=!MI! S F V I L -
:
:
; and analytical/conceptual purposes.
iii.Pjii.' !jii.nii J?. .Tri! m Î.L-TÎRIITII D
: ; Miles and Huberman (1994) have pro-
if i F&T! !:-i I:tvXi: i:;. W
; :!.!;=!!: .V; li!.i !"
! vided a rich source of ideas and illustrations
îfeiNVi!: ij=:'! : iVlS:!; =:=; n;:|X! !!! KJ of how to use matrices in qualitative analy-
:;=:!fidtwi -ii' wtZ fO MXrO fff nj;: !i:i:v:=! sis. They include examples of a time-ordered
=:- " ;=S:-::;:.==: & . ; * ' : " ! : : ! ; .
=
5 p : t s V^WS! matrix, role-ordered matrix, role-by-time
MO jV!;;!;!Ji;:i;:il :|:!m-Nrig:,!
, ; ; : ! rVSt i|j- h'M;
! matrix, role-by-group matrix, conceptually
;:.:i!i1 !-fir-ilt;i;:i!! !'h!'':T :ITi:! ri lrr vi ^ V T ! ; clustered matrix, site dynamics matrix, and
predictor-outcome matrix, among others.
T R A I ;J:S I: H:' . P I S F R L I I R N ' M R W - D 1 K ; N,
Their Sourcebook provides a variety of ideas
irh S:'i iji::i !)!!/:!! -fc EIW5" .!;>:! 1!: p 1"/
for analytical approaches to qualitative data
iiv.ir Ji ii i::; * {\s'.gjjs&Eflate carais. :>r- "
including a variety of concept mapping and
Y-n-r-e ii' :=i if!!! iY^ r-YV J:!' C î piX?'!: !:! h;i Yn:!;'
! : :
visual display techniques.
i.= ii: .i::r .li::i !l|iM!yi':i' j^l*!;.!;;]:'^!- =::;: iArfrS;
Other ways of graphing and mapping
. J;H; :: i bii- :::;: ttt Lhiii i' vk\ v\ U i'iY t) t-p'IIVS-
findings include concept mapping and cog-
:;.!:i':ViVi.'virt-::,:!.'! :!,=].' -iViiriDm|,'.',;! rrTi/illlfVl!!"'^'!
nitive mapping. For a detailed discussion of
liJi ir :-.! !:=. ihjJiYihnY,;! iV !: i:liii!!:i;i:=!;|! îllf;:/
concept mapping as a way of visually dis-
. 'iY-ijiJr af :'!:i i:i:Vn:;i i\ ::i.!l"ttii JY;^-I^-TY:;^.:
play ing data to facilitate analy tic clarity and
NIYIFCL .;:=. ::;:-'"|:!::RNLI .!;:= iV! - K I M L I : : ^ S F L L ! " ; |
1
depicting relationships in a network or sys-

ircm-j- ! i1 := :!i iV i p-^îjn^-IViiiYli;:^ î'; ; j
tem, see Trochim (1989). For an example of
I! VI I:!E'RI;R : T ! ! . F N !YI;TT! : J J R R J ^ H I ^ I Y - U Y ^ F H V . : |
! :
:
cognitive mapping as a way of displaying

pjrf3 hi;;l"! ir^^^v-JJ,'!;;; Îjiiii !i.i.ViVii-ii 1:1; \ j
qualitative results showing the "structure
finrfeTTE 6;kv:V!':'!.1 :"!.!!.:' IJ-h:-!5!:: . !
:
and content of decision schemas" among s-

nior managers, see Clarke and Mackaness
(2001).
program processes with program outcomes

as a qualitative analysis framework.
A Process/Outcomes Matrix
To study how schools used planning and The linkage between processes and out-
evaluation processes, Campbell (1983) de- comes constitutes such a fundamental issue
veloped a 500-cell matrix (Exhibit 8.10) that in many program evaluations that it pro-
begins (but just begins) to reach the outer vides a particularly good focus for illustrat-
limits of what one can do in three-dimen- ing qualitative matrix analysis. As dis-
sional space. Campbell used this matrix to cussed in Chapter 4, qualitative methods
guide data collection and analysis in study- can be particularly appropriate for evalua-
ing how the mandated, statewide education where program processes, impacts, or
tional planning, evaluation, and reporting both are largely unspecified or difficult to
system in Minnesota was used. She exam- measure. This can be the case because the
ined 5 leveis of use (high school,... commu- outcomes are meant to be individualized;
Mapping Stakeholders' Stakes
Estimate of Various Stakeholders' Inclination Toward the Program
How high are the stakes for

Favorable Neutra! or Unknown Antagonistic
various primary stakeholders?
High
Moderate
Low
SOURCE: Patton (1997a:344).

NOTE: Construct illustrative case studies for each cell based on fieldwork.
sometimes the program is simply uncertain linkages, patterns, themes, experiences, con-
what its outcomes will be; and, in many pro- tent, or actual activities that help us
grams, neither processes nor impacts have understand the relationships between pro-
been carefully articula ted. Under such con- cesses and outcomes. Such relationships
ditions, one purpose of the evaluation may may have been identified by participants
be to illuminate program processes, pro- themselves during interviews or discovered
gram impacts, and the lihkages between the by the evaluator in analyzing the data. In ei-
two. This task can be facilita ted by con- ther case, the process/outcomes matrix be-
structing a process/outcomes matrix to or- comes a way of organizing, thinking about,
ganize the data. and presenting the qualitative connections
Exhibit 8.11 (p. 474) shows how such a between program implementation dimen-
matrix can be constructed. Major program sions and program impacts.
processes or identified implementation An example will help make the notion of
components are listed along the left side. the process/outcomes matrix more con-
Types or leveis of outcomes are listed across crete. Suppose we have been evaluating a ju-
the top. The category systems for program venile justice program that places delin-
processes and outcomes are developed from quent youth in foster homes. We have
the data in the same way that other typolo- visited several foster homes, observed what
gies are constructed (see previous sections). the home environments are like, and inter-
The cross-classification of any process with viewed the juveniles, the foster home par-
any outcome produces a cell in the matrix; ents, and the probation officers. A regularly
for example, the first cell in Exhibit 8.11 is recurring process theme concerns the impor-
created by the intersection of process 1 with tance of "letting kids learn to make their
outcome a. The information that goes in cell own decisions." A regularly recurring out-
la (or any other cell in the matrix) describes comes theme involves "keeping the kids
Conceptual Guide for Data Collection and Analysis:

EXHIBIT 8.10 Utilization of Planning, Evaluation, and Reporting
C. FACTQRS INFLUENCING UTILIZATION
^////Ay/
j
& c T <P ** ? O^
<o- b- A' <t>- <- ^
1. Planning
2, Goal-setting
cn 3 Professional
LU evaluation
Q.
LL Consumer
o evaluation
p
z Reporting
LU
z Community
o R
D- involvement
Q_
5
O Student
7.
O involvement
o Teacher
CD involvement
o Administrative
a- involvement
School Board
10. involvement
straight" (reduced recidivism). By crossing decisions juveniles actually make in the pro-
the program process ("kids making their gram, the decision makers to whom our
own decisions") with the program outcome findings are reported can make their own
("keeping kids straight"), we create a data judgments about the strength or weakness
analysis question: What actual decisions do of the linkage between this program process
juveniles make that are supposed to lead to and the desired outcome. Moreover, once
reduced recidivism? We then carefully re- the process/outeomes descriptive analysis
view our field notes and interview quota- of linkages has been completed, the evalua-
tions looking for data that help us under- tor is at liberty to offer interpretations and
stand how people in the program have judgments about the nature and quality of
answered this question based on their actual this process/outeomes connection.
behaviors and practices. By describing what
Matrix of Linkages Between Program Processes

EXHIBIT 8.11 and Impacts
Types or Leveis of Program Outcomes
a b c d
LINKAGES EXPRESSED AS THEMES,

PATTERNS, QUOTATIONS, PROGRAM
Program
Processes
or
Implementa tion
Components
CONTENT OR ACTUAL ACTIVITIES
SOURCE: Campbell (1983).
An Analysis Example: Recognizing tion of how these concepts can be used in

Processes, Outcomes, and Linkages qualitative analysis. How does one recog-
in Qualitative Data nize a program process? Learning to identify
and label program processes is a criticai
evaluation skill. This sensitizing notion of
Because of the centrality of the sensitizing "process" is a way of talking about the com-
concepts "program process" and "program mon action that cuts across program activi-
outcome" in evaluation research, it may be ties, observed interactions, and program
helpful to provide a more detailed descrip- content. The example I shall use involves
data from the wilderness education pro- nuance of the process. The qualitative ana-
gram I evaluated and discussed throughout lyst works back and forth between the data
the observations chapter (Chapter 6). That (field notes and interviews) and his or her
program, titled the Southwest Field Training conception of what it is that needs to be ex-
Project, used the wilderness as a training pressed to find the most fitting language to
arena for professional educators in the phi- describe the process. What language do peo-
losophy and methods of experiential educa- ple in the program use to describe what
tion by engaging those educators in their those activities and experiences have in
own experiential learning process. Partici- common? What language comes closest to
pants went from their normal urban envi- capturing the essence of this particular pro-
ronments into the wilderness for 10 days at a cess? What levei of generality or specificity
time, spending at least one day and night will be most useful in separating out this
completely alone in some wilderness spot particular set of things from other things?
"on solo." At times, while backpacking, the How do program participants and staff react
group was asked to walk silently so as not to to the different terms that could be used to
be distracted from the wilderness sounds describe the process?
and images by conversation. In group dis- It's not unusual during analysis to go
cussions, participants were asked to talk through several different phrases before fi-
about what they had observed about the wil- nally settling on exact language that will go
derness and how they felt about being in the into a final report. In the Southwest Field
wilderness. Participants were also asked to Training Project, we began with the concept
write about the wilderness environment in label "Experiencing the wilderness." How-
journals. What do these different activities have ever, after several revisions, we finally de-
in common, and how can that commonality be ex~ scribed the process as "developing sensitiv-
pressed? ity to the environment" because this broader
We begin with several different ways of label permitted us to include discussions
abstracting and labeling the underlying pro- and activities that were aimed at helping
cess: participants understand how they were af-
fected by and acted in their normal institu-
a Experiencing the wilderness tional environments. "Experiencing the wil-
derness" became a specific subprocess that
Learning about the wilderness
was part of the more global process of "de-
Appreciating the wilderness veloping sensitivity to the environment."
Immersion in the environment Program participants and staff played a ma-
jor role in determining the final phrasing
0 Developing awareness of the
and description of this process.
environment
Below are other processes identified as
D Becoming conscious of the wilderness important in the implementation of the pro-
Developing sensitivity to the gram:
environment
H Encountering and managing stress
Any of these phrases, each of which consists
a Sharing in group settings
of some verb form (experiencing, learning,
developing, and so on) and some noun form H Examining professional activities,
(wilderness, environment), captures some needs, and commitments
476 lJ. ANALYSIS, INTERPRETATION, AND REPORTING
n Assuming responsibility for quotations) are used to support and explain

articulating personal needs inductive thematic analysis.
Deductively, the evaluator analyst may
n Exchanging professional ideas and
draw from outcomes identified in similar
resources
programs or from goal statements found in
H Formally monitoring experiences, program proposals, brochures, and plan-
processes, changes, and impacts ning documents that were used to guide
data collection.
Logically (or abductively), constructing a
As you struggie with finding the right
process/outcomes matrix can suggest addi-
language to communicate themes, patterns,
tional possibilities. That is, where data on
and processes, keep in mind that there is no
both program processes and participant out-
absolutely "right" way of stating what
comes have been sorted, analysis can be
emerges from the analysis. There are only
deepened by organizing the data through a
more and less useful ways of expressing
logical scheme that links program processes
what the data reveal.
to participant outcomes. Such a logically de-
Identifying and conceptualizing program
rived scheme was used to organize the data
outcomes and impacts can involve induc-
in the Southwest Field Training Project.
tion, deduction, and/or logical analysis. ln~
First, a classification scheme that described
ductively, the evaluator analyst looks for
different types of outcomes was conceptual-
changes in participants, expressions of
ized:
change, program ideology about outcomes
and impacts, and ways that people in the
(a) changes in skills,
program make distinctions between "those
who are getting it" and "those who aren't (b) changes in attitudes,
getting it" (where it is the desired outcome). (c) changes in feelings,
In highly individualized programs, the
(d) changes in behaviors, and
statements about change that emerge from
program participants and staff may be (e) changes in knowledge.
global. Such outcomes as "personal growth,"
increased "awareness," and "insight into These general themes provided the reader of
self" are difficult to operationalize and stan- the report with examples of and insights into
dardize. That is precisely the reason qualita- the kinds of changes that were occurring and
tive methods are particularly appropriate how those changes that were perceived by
for capturing and evaluating such out- participants to be related to specific program
comes. The task for the evaluator analyst, processes. I emphasize that the process/out-
then, is to describe what actually happens to comes matrix is merely an organizing tool;
people in the program and what they say the data from participants themselves and
about what happens to them. Appendix 8.3 from field observations provide the actual
at the end of this chapter presents portions of linkages between processes and outcomes.
the report describing the effects on partici- What was the relationship between the
pants of their experiences in the wilderness program process of "developing sensitivity
education program. The data come from in- to the environment" and these individual-
depth, open-ended interviews. This report level outcomes? Space permits only a few
excerpt shows how descriptive data (direct examples from the data.
Skills: "Are you kidding? I learned how to the effects of the fires on the vegetation, where
survive without the comforts of civilization. I the river comes from and where it goes,"
learned how to read the terrain ahead and
pace myself. I learned how to carry a heavy A different way of thinking about orga-
load. I learned how to stay dry when it's rain- nizing data around outcomes was to think of
ing. I learned how to tie a knot so that it different leveis of impact: effects at the indi-
doesn't come apart when pressure is applied. vidual levei, effects on the group, and effects
You think those are metaphors for skills I need on the institutions from which participants
in my work? You're damn right they are." came into the program. The staff hoped to
have impacts at ali of these leveis. Thus, it
Altitudes: "I think it's important to pay at-
also was possible to organize the data by
tention to the space you're in. I don't want to
looking at what themes emerged when pro-
just keep going through my life oblivious to
gram processes were crossed with leveis of
what's around me and how it affects me and
impact. How did "developing sensitivity to
how I affect it."
the environment" affect individuais? How
Feelings: "Being out here, especially on solo, did the process of "developing sensitivity to
has given me confidence. I know I can handle a the environment" affect the group? What
lot of things I didn't think I could handle." was the effect of "developing sensitivity to
the environment" on the institutions to
Behaviors: "I use my senses in a different way which participants returned after their wil-
out here. In the city you get so you don't pay derness experiences? The process/outcomes
much attention to the noise and the sounds. matrix thus becoines a way of asking ques-
But listening out here I've also begun to listen tions of the data, an additional source of fo-
more back there. I touch more things too, just cus in looking for themes and patterns in
to experience the different textures." hundreds of pages of field notes and inter-
view transcriptions.
Knozvledge: "I know about how this place
was formed, its history, the rock formations,
!3. Interpreting Findings
imply observing and interviewing do not ensure that the research is

qualitative; the qualitative researcher must also interpret the beliefs and
behaviors of participants.
Valerie J. Janesick (2000:387)
Interpreting for Meaning this mean? What does this tell me about the
nature of the phenomenon of interest? In
Qualitative interpretation begins with asking these questions, the analyst works
elucidating meanings. The analyst examines back and forth between the data or story (the
a story, a case study, a set of interviews, or a evidence) and his or her own perspective
collection of field notes and asks, What does and understandings to make sense of the ev-
idence. Both the evidence and the perspec- stand on their own as pure description of ex-
tive brought to bear on the evidence need to perience or be analyzed for connections
be elucidated in this choreography in between the psychological, sociological, cul-
searching of meaning. Alternative interpre- tural, political, and dramaturgic dimensions
tations are tried and tested against the data. of human experience to reveal larger mean-
For example, when we analyzed fol- ings. Much of the analytical focus in narra-
low-up interviews with participants who tive studies concerns the nature of interpre-
had gone through intensive community tation (Denzin 1989a, 1989b, 1997b). How to
leadership training, we found a variety of interpret stories and, more specifically, the
expressions of uncertainty about what they texts that tell the stories is at the heart of nar-
should do with their training. In the final rative analysis (Lieblich, Tuval-Mashiach,
day of a six-day retreat, after learning how to and Zilber 1998). Meaning-making also co-
assess community needs, work with diverse mes from comparing stories and cases and
groups, communicate clearly, empower can take the form of inquiring into and inter-
people to action, and plan for change, they preting causes, consequences, and relation-
were cautioned to go easy in transitioning ships.
back to their communities and take their
time in building community connections be-
fore taking action. What program staff Comparisons, Causes,
meant as a last-day warning about not re- Consequences, and Relationships
turning to the community as a buli in a china
shop and charging ahead destructively had, Thus far, this chapter has emphasized the
in fact, paralyzed the participants and made tasks of organization, description, and link-
them afraid to take any action at ali. The pro- ing. Even the matrix analyses just discussed
gram, which intended to poise participants were aimed at organizing and describing the
for action, had inadvertently left graduates themes, patterns, activities, and content of a
in "action paralysis" for fear of making mis- study rather than elucidating causai link-
takes. That meaning, "action paralysis," ages between processes and outcomes. To
emerged from the data analysis through in- the extent that you are describing the causai
terpretation. No one used that specific linkages suggested by and believed in by
phase. Rather, we interpreted that as the es- those you've interviewed, you haven't
sence of what interviewees were reporting crossed the line from description into causai
through a haze of uncertainties, ambiguities, interpretation. And, indeed, much qualita-
worried musings, and wait-and-see-before- tive inquiry stops with the presentation of
acting reflections. case data and cross-case descriptive com-
Narrative analysis (see Chapter 3) has fo- parisons aimed at enhancing understanding
cused specifically on how to interpret sto- rather than explaining "why." Stake (1995)
ries, life history narratives, historical mem- has emphasized that "explana tions are in-
oirs, and creative nonfiction to reveal tended to promote understanding and un-
cultural and social patterns through the lens derstanding is sometimes expressed in
of individual experiences. This "biographi- terms of explanationbut the two aims are
cal turn in social science" (Chamberlayne, epistemologically quite different. . . , a dif-
Bornat, and Wengraf 2000) or "narrative ference important to us, the difference be-
turn" in qualitative inquiry (Bochner 2001) tween case studies seeking to identify cause
honors people's stories as data that can and effect relationships and those seeking
Comparative
Pattern
Analysis
C O M P A R I N G A P P L E S AND O R A N G E S
understanding of human experience" (p. 38). clearly qualified as what they are: interpre-
Appreciating and respecting this distinc- tation and hypothesizing.
tion, once case studies have been written A researcher who has lived in a commu-
and descriptive typologies have been devel- nity for an extensive period of time will
oped and supported, the tasks of organiza- likely have insights into why things happen
tion and description are largely complete as they do there. A qualitative analyst who
and it is appropriate, if desired, to move on has spent hours interviewing people will
to making comparisons and considering likely come away from the analysis with
causes, consequences, and relationships. possible explanations for how the phenome-
Statements about which things appear to non of interest takes the f orms and has the ef-
lead to other things, for example, which as- fects it does. The evaluator who has studied
pects of a program produce certain effects, a program, lived with the data from the field,
and how processes lead to outcomes are nat- and reflected at length about the patterns
ural areas for interpretation and hypothesiz- and themes that run through the data is in as
ing. When careful study of the data gives rise good a position as any one else at that point
to ideas about causai linkages, there is no to speculate about meanings, make conjec-
reason to deny those interested in the tures about significance, and offer hypothe-
study's results the benefit of those insights. ses about relationships. Moreover, if deci-
What is important is that such statements be sion makers and evaluation users have
asked for such informationand in my ex- long as the researcher owns the interpreta-
perience they virtually always welcome tion and makes clear the difference between
these kinds of analysesthere is no reason description and interpretation.
not to share insights with them to help them Schlechty and Noblit (1982) concluded
think about their own causai presupposi- that an interpretation may take one of three
tions and hypotheses and to explore what forms:
the data do and do not support in the way of
interconnec tions and potential causai rela- Making the obvious obvious
tionships.
Making the obvious dubious
Lofland's (1971) musings are helpful in
clarifying the role of causai speculation in Making the hidden obvious
qualitative analysis. He argued that the
strong suit of the qualitative researcher is the This captures rather succmctly what re-
ability "to provide an orderly description of search colleagues, policymakers, and evalu-
rich, descriptive detail" (p. 59); the consideration stakeholders expect: (1) Confirm what
ation of causes and consequences using we know that is supported by data, (2) dis-
qualitative data should be a "tentative, qual- abuse us of misconceptions, and (3) illumi-
ified, and subsidiary task" (p. 62). nate important things that we didn't know
but should know. Accomplish these three
It is perfectly appropriate that one be curious things and those interested m the findings
about causes, so long as one recognizes that can take it from there.
whatever account or explana tions he develops A particular limitation as one moves mto
is conjecture. In more legitimacy-conferring the arena of interpretations about causes,
terms, such conjectures are called hypotheses consequences, and relationships concerns
or theories. It is proper to devote a portion of our capacity to escape simplistic linear mod-
one's report to conjectured causes of varia- eling. We fali back on the linear assumptions
tions so long as one clearly labels his conjec- of much quantitative analysis and begin to
tures, hypotheses or theories as being that. specify isolated independent and depend-
(Lofland 1971:62) ent variables that are mechanically linked
together out of context. In contrast, the chal-
Interpretation, by definition, mvolves go- lenge of qualitative inquiry involves por-
ing beyond the descriptive data. Interpreta- traying a holistic picture of what the phenom-
tion means attaching significance to what enon, setting, or program is like and
was found, making sense of findings, offer- struggling to understand the fundamental
ing explanations, drawing conclusions, ex- nature of a particular set of activities and
trapolating lessons, making inferences, con- people in a specific context. "Particularization
sidering meanings, and otherwise imposing is an important aim, coming to know the
order on an unruly but surely patterned particularity of the case" (Stake 1995:39).
world. The rigors of interpretation and Simple statements of linear relationships
bringing data to bear on explanations in- may be more distorting than illuminating.
clude dealing with rival explanations, ac- The ongoing challenge, paradox, and di-
counting for disconfirming cases, and ac- lemma of qualitative analysis engage us m
counting for data irregularities as part of constantly moving back and forth between
testing the viability of an interpretation. Ali the phenomenon of interest and our abstrac-
of this is expectedand appropriateas tions of that phenomenon, between the de-
scriptions of what has occurred and our in- among and label the different meanings of
terpreta tions of those descriptions, between the situation expressed by the characters ob-
the complexity of reality and our simplifica- served in the story, then write a statement of
tions of those complexities, between the the form: These things and these things
circularities and interdependencies of hu- came together to create . Don't try to
man activity and our need for linear, ordered decide that one approach is right and the
statements of cause-effect. other is wrong; simply try to experience and
Gregory Bateson traced at least part of the understand the two approaches. Here's the
source of our struggle to the ways we have case data, otherwise known as a story.
been taught to think about things. We are
told that a noun is the "name of a person, Walking one evening along a deserted road,
place, or thing." We are told that a verb is an Mulla Nasrudin saw a troop of horsemen
"action word." These kinds of definitions, coming towards him. His imagina tion started
Bateson argues, were the beginning of teach- to work; he imagined himself captured and
ing us that "the way to define something is sold as a slave, or robbed by the oncoming
by what it supposedly is in itselfnot by its horsemen, or conscripted into the army.
rela tions to other things." Fearing for his safety, Nasrudin bolted,
climbed a wall into a graveyard, and lay down
Today ali that should be changed. Children in an open tomb.
could be told a noun is a word having a certain Puzzled at this strange behavior the men
relationship to a predicate. A verb has a cer- honest travelerspursued Nasrudin to see
tain relationship to a noun, its subject, and so if they could help him. They found him
on. Relationship could now be used as a basis stretched out in the grave, tense and quiver-
for definition, and any child could then see ing.
that there is something wrong with the sen- "What are you doing in that grave? We saw
tence, " 'Go' is a v e r b . " . . . We could have been you run away and see that you are in a state of
told something about the pattern which con- great anxiety and fear. Can we help you?"
nects: that ali communication necessitates Seeing the men up close Nasrudin realized
context, and that without context there is no that they were honest travelers who were gen-
meaning. (Bateson 1978:13) uinely interested in his welfare. He didn't
want to offend them or embarrass himself by
Without belaboring this point about the telling them how he had misperceived them,
difference between linear causai analysis (x so Nasrudin simply sat up in the grave and
causes y) and a holistic perspective that de- said, "You ask what I'm doing in this grave. If
scribes the interdependence and interrelat- you must know, I can tell you only this: I am
edness of complex phenomena, I would sim- here because of you, and you are here because
ply offer the reader a Sufi story. I suggest of me." (adapted from Shah 1972:16)
trying to analyze the data represented by the
story in two ways. First, try to isolate specific
variables that are important in the story, de-
ciding which are the independent and
13. Theory-Based
which the dependent variables, and then Analysis Approaches
write a statement of the form: These things
caused this thing. Then read the story again. Thus far, this chapter has been looking at ge-
For the second analysis, try to distinguish neric approaches to qualitative analysis.
The next sections examine how certain theo- fore the analysis. While Chapters 3 and 4
retical and philosophical perspectives affect were presented early in this book to help re-
analysis. Every perspective presented in searchers and evaluators select frameworks
Chapter 3 on theoretical orientations has to guide their inquiry, those chapters also
implications for analysis in that the funda- offer frameworks for analyzing data. The
mental premises articulated in a theoretical two sections that follow contrast two of the
framework or philosophy are meant to in- major theory-oriented analytical approaches
form how one makes sense of the world. discussed in Chapter 3, but this time focus-
Likewise, the various applications in Chap- ing on analysis. The two contrasting ap-
ter 4 affect analysis in that they shape the proaches are phenomenological analysis
questions that guide the inquiry and there- and grounded theory.
Phenomenological Analysis
P henomenology asks for the very nature of a phenomenon, for that

which makes a some-"thing" what it isand without which it could not
be what it is.
Max Van Manen (1990:10)
Phenomenological analysis seeks to tive experience as reported." Harper (2000:

grasp and elucidate the meaning, structure, 727) talks of looking at images through "the
and essence of the lived experience of a phe- phenomenological mode," that is, from the
nomenon for a person or group of people. perspective of the self: "from the phenom-
Before I present the steps of one particular enological perspective, photographs ex-
approach to phenomenological analysis, it press the artistic, emotional, or experiential
is important to note that phenomenology intent of the photographer." Add to this con-
has taken on a number of meanings, has a fusion of terminology the difficulty of distin-
number of forms, and encompasses varying guishing phenomenological philosophy
traditions including transcendental phe- from phenomenological methods and phe-
nomenology, existential phenomenology, nomenological analysis, ali of which adds to
and hermeneutic phenomenology (Schwandt tensions and contradictions in qualitative in-
2001). Moustakas (1994:13) further distin- quiry (Gergen and Gergen 2000).
guishes empirical phenomenological from
transcendental phenomenology. Gubrium
and Holstein (2000:488) add the label "social The use of the term phenomenology in con-
phenomenology." Van Manen (1990) prefers temporary versions of qualitative inquiry in
"hermeneutical phenomenological reflec- North America tends to reflect a subjectivist,
tion." Sonnemann (1954:344) introduced the existentialist, and non-critical emphasis not
term "phenomenography" to label phe- present in the Continental tradition repre-
nomenological investigation aimed at "a sented in the work of Husserl and Heidegger.
descriptive recording of immediate subjec- The latter viewed the phenomenological pro-
studying everyday experience from the point

A PttKfflWENOQR^Pm' OF of view of the subject, and it shuns criticai
DUit MPICAl fttfECTI0N evaluation of forms of social life. (Schwandt

;
;: . : 2001:192)
!! PjÊjftr UiijTynl inl.yil i'-i.b rt~
hHl i T - . J v t y k u v i i nr-r JMVIrisy.:ri|j= These distinctions and variations in use
- i i l T O ^ ^ i ^ ^ T ^ . !l!1 ii! >!' !!:!='=!' r
make it relatively meaningless to describe
' l ^ ^ R - i i h S ^ r ftj. =!"!j1=!X!; .Vi;:!Y|;i;-i;ri;=:S " n r ' "
"phenomenological analysis" as if it consti-
tuted a single approach or perspective. I
' Fihff faiifih:;?-;ipfrr'ffri))1 n"- have chosen to include here the phenom-
!'!':: fiV; fofe! !"i:i;i: hi M;!!'n:;i i!: i Jjr- enological approach to analysis taken by
. rtmA M i TMi l ^î i^ ^ rhM lij Mi sj s^lr b f " . Clark Moustakas and Bruce Douglass of
: |p. ii-rJ. !=!:;! m v w t M w o ^ i ffrp j i * ^ : The Union Institute Graduate College
: !.';= js-Ws-el^ p\fafa-st^TK'tf rafa 'M-" (Cincinnati, Ohio) and the Center for Hu-
manistic Studies (Detroit, Michigan). More
than most approaches, they focus on the
* !'!:! p a - s x s A ' m ^ J ^ l m !>jj: ji;!;1:-;;- .. analytical process itself (Douglass and
Iri jji' i !; c ri \k ;i H.pu |j h !:: =!:; j;|.iv d: ;":; b l;i Moustakas / 1985). Moreover, the extensive
rji.ii ;' -^..- writings of Moustakas on phenomenology
* N.IN.V !;':!:::lj=i:. !=!!;:;ii=;;!|rjia 6i:'i : ;!T. : (1961,1988,1990b, 1994,1995) are readily ac-
: h:!i! i: Y?) 1=!: ij :;::! nyjsTt&ru L :;; !:i-!J!!nj!:!j:;!::i * :.jcessible and highly readable. Finally, they
. \ i!:!>::; fc in ij- ;=:!:!:; i=i: i:1;i; !?|:=i;r= j:h!i: m l.j?l!:!;: are esteemed colleagues whose work I
!.h a i:fa-wd !'i!! : \\X- ;!::ii!.i:;i In :!;m! ?"! % pknow, ; appreciate, and, no small point when
=!= ;: LUs ir!I!J::IJ,=i:.vi IJ:'='! jiii::i : A. ; : I dealing with phenomenology, I think I un-
: i.i:i.i! li !i'i i:<:-i!'ri /i^-i!: !:.. m[B*'rrp Irj ; ;i|!.i \i l. derstand. They have developed an outline of
pr; r : ! Idiri i.V 1 :=M i 'i. VjJi;-!!!:!::;:!j::i!.! d.-: .. phenomenological analysis that they use in
rrAi i:i l;:i i ii i i=;! ri Kiiiiijii- fjkStl i=i! j;l?: / /graduate
* seminars. Much of this section is
' ! t ! : - ;;n i !:vi i>i
!, ih hi iii i!1'i;.i::i hi- ly. iti i:.!.i.i =.based on their work and that of their stu-
:
!
ijr! 81 v. ::: i r! A! r\ Lh; . ftfrti . Vnîî:!::; * jf.* . dents. Before presenting the steps and pro-
!;h ri:j ij i'|! ii rd iii ii i i : . ^ ' . : .. ; . . . : ! ; . cedures of phenomenological analysis, let's
a- iiviiaP;'!!;li;:.i= I ; Y H ' :::ii. get deeper into the perspective and lan-
. ;;.!!ji"j"rs |1||:I;;:i:!j;;i.in;;;j:i b i / [ \ i shjIi\ guage.
i:!n:=n!::i !i. A . : : .:
Husserl's transcendental phenomenology is
intimately bound up in the concept of
intentionality. In Aristotelian philosophy the
ject, so to speak, as an effort to get beneath or term intention indicates the orientation of the
behind subjective experience to reveal the mind to its object; the object exists in the mind
genuine, objective nature of things, and as a in an intentional w a y . . . .
critique of both taken-for-granted ineanings Intentionality refers to consciousness, to the
and subjectivism. Phenomenology, as it is internai experience of being conscious of
commonly discussed in accounts of qualita- something; thus the act of consciousness and
tive research, emphasizes just the opposite: It the object of consciousness are intentionally
aims to identify and describe the subjective ex- related. Included in understanding of con-
periences of respondents. It is a matter of sciousness are important background factors
such as stirrmgs of pleasure, shapings of judg- tion of the intentional processes themselves.
ment, or incipient wishes.
Knowledge of intentionality requires that Summarizing the challenges of intention-
we be present to ourselves and things in the ality, the following processes stand out:
world, that we recognize that self and world
are inseparable components of m e a n i n g . . . . 1. Explicating the sense in which our experi-
Consider the experience of joy on witness- ences are directed;
ing a beautiful landscape. The landscape is the
matter. The landscape is also the object of the 2. Discerning the features of consciousness
intentional act, for example, its perception in that are essential for the individuation of
consciousness. The matter enables the land- objects (real or imaginary) that are before
scape to become manifest as an object rather us in consciousness (Noema);
than merely exist in consciousness.
The interpretive form is the perception that 3. Explicating how beliefs about such ob-
enables the landscape to appear; thus the jects (real or imaginary) may be acquired,
landscape is self-given; my perception creates how it is that we are experiencing what
it and enables it to exist in my consciousness. we are experiencing (Noesis); and
The objectifying quality is the actuality of the
landscape's existence, as such, while the 4. Integra ting the noematic and noetic cor-
non-objectifying quality is a joyful feeling relates of intentionality into meanings
evoked in me by the landscape. and essences of experience. (Moustakas
Every intentionality is composed of a 1994:28-32)
nomea and noesis. The nomea is not the real ob-
ject but the phenomenon, not the tree but the
If those are the challenges, what are the
appearance of the tree. The object that appears
steps for meeting them? The first step in
in perception varies m terms of when it is per-
phenomenological analysis is called epoche.
ceived, from what angle, with what back-
ground of experience, with what orienta tion
of wishing, willing, or judging, always from Epoche is a Greek word meaning to refrain
the vantage point of the perceiving individual. from judgment, to abstain from or stay away
. . . The tree is out there present in time and from the everyday, ordinary way of perceiving
space while the perception of the tree is in con- things. In a natural attitude we hold knowl-
sciousness. . . . edge judgmentally; we presuppose that what
Every intentional experience is also no- we perceive in nature is actually there and re-
etic.... mains there as we perceive it. In contrast,
In considering the nomea-noesis corre- Epoche requires a new way of looking at
late . . . , the "perceived as such" is the nomea; things, a way that requires that we learn to see
the "perfect self-evidence" is the noesis. Their what stands before our eyes, what we can dis-
relationship constitutes the intentionality of tinguish and d e s c r i b e . . . .
consciousness. For every nomea, there is a In the Epoche, the everyday understand-
noesis; for every noesis, there is a nomea. On ings, judgments, and knowings are set aside,
the noematic side is the uncovering and expli- and the phenomena are revisited, visually, na-
cation, the unfolding and becoming distinct, ively, in a wide-open sense, from the vantage
the clearmg of what is actually presented in point of a pure or transcendental ego.
consciousness. On the noetic side is an explica- (Moustakas 1994:33)
In taking on the perspective of epoche, the of epoche epitomizes the data-based, eviden-
researcher looks inside to becorne aware of tial, and empirical (vs. empiricist) research
personal bias, to eliminate personal in- orientation of phenomenology.
volvement with the subject material, that is, Foliowing epoche, the second step is
eliminate, or at least gain clarity about, pre- phenomenological reduction. In this analyt-
conceptions. Rigor is reinforced by a "phe- ical process, the researcher "brackets out"
nomenological attitude shift" accomplished the world and presuppositions to identify
through epoche. the data in pure form, uncontaminated by
extraneous intrusions.
The researcher examines the phenomenon by
attaining an attitudinal shift. This shift is Bracketing is Husserl's (1913) term. In brack-
known as the phenomenological attitude. This eting, the researcher holds the phenomenon
attitude consists of a different way of looking up for serious mspection. It is taken out of the
at the investigated experience. By moving be- world where it occurs. It is taken apart and dis-
yond the natural attitude or the more prosaic sected. Its elements and essential structures
way phenomena are imbued with meaning, are uncovered, defined, and analyzed. It is
experience gains a deeper meaning. This takes treated as a text or a document; that is, as an in-
place by gaining access to the constituent ele- stance of the phenomenon that is being stud-
ments of the phenomenon and leads to a de- ied. It is not interpreted in terms of the
scription of the unique qualities and compo- standard meanings given to it by the existing
nents that make this phenomenon what it is. In literature. Those preconceptions, which were
attaining this shift to the phenomenological isolated in the deconstruction phase, are sus-
attitude, epoche is a primary and necessary pended and put aside during bracketing. In
phenomenological procedure. bracketing, the subject matter is confronted, as
Epoche is a process that the researcher en- much as possible, on its own terms. Bracketing
gages in to remove, or at least become aware involves the following steps:
of, prejudices, viewpoints or assuinptions re-
garding the phenomenon under invstigation. 1. Loca te within the personal experience, or
Epoche helps enable the researcher to investi- self-story, key phrases and statements
gate the phenomenon from a fresh and open that speak directly to the phenomenon in
viewpoint without prejudginent or imposing question.
meaning too soon. This suspension of judg-
ment is criticai in phenomenological investi- 2. Interpret the meanings of these phrases,
gation and requires the setting aside of the reas an informed reader.
searcher^ personal viewpoint in order to see
the experience for itself. (Katz 1987:36-37) 3. Obtain the subject's interpreta tions of
these phrases, if possible.
According to Ihde (1979), epoche requires
that looking precede judgment and that 4. Inspect these meanings for what they re-
judgment of what is "real" or "most real" be veal about the essential, recurring fea-
suspended until ali the evidence (or at least tures of the phenomenon being studied.
sufficient evidence) is in (p. 36). As such,
epoche is an ongoing analytical process 5. Offer a tentative statement, or definition,
rather than a single fixed event. The process of the phenomenon in terms of the essen-
tiai recurring features identified in step 4. ence" (Moustakas 1994:144). In summary,

(Denzin 1989b:55-56) the primary steps of the Moustakas tran-
scendental phenomenological model are
Once the data are bracketed, ali aspects of epoche, phenomenological reduction, imagi-
the data are treated with equal value, that is, native variation, and synthesis of texture
the data are "horizontalized." The data are and structure. Other detailed analytical
spread out for examination, with ali ele- techniques are used within each of these
ments and perspectives having equal stages (see Moustakas 1994:180-81).
weight. The data are then organized into Heuristic inquiri/ (Moustakas 1990b) in-
meaningful clusters. Then the analyst un- volves a somewhat different analytical pro-
dertakes a delimitation process whereby ir- cess. The heuristic process of phenom-
relevant, repetitive, or overlapping data are enological inquiry is a highly personal pro-
eliminated. The researcher then identifies cess. Moustakas describes five basic phases
the invariant themes within the data in order in the heuristic process of phenomenolog-
to perform an "imaginative variation" on ical analysis: immersion, incubation, illumi-
each theme. Douglass has described this as nation, explication, and creative synthesis.
"moving around the statue" to see the same Immersion is the stage of steeping oneself
object from differing views. Through imagi- in ali that is, of contacting the texture, tone,
native variation, the researcher develops en- mood, range, and content of the experience.
hanced or expanded versions of the invari- This state "requires my full presence, to sa-
ant themes. vor, appreciate, smell, touch, taste, feel, know
Using these enhanced or expanded ver- without concrete goal or purpose" (Mous-
sions of the invariant themes, the researcher takas 1981:56). The researcher's total life and
moves to the textural portrayal of each being are centered in the experience. He or
themea description of an experience that she becomes totally involved in the world of
doesn't contain that experience (i.e., the feel- the experience, questioning, mediating,
ings of vulnerability expressed by rape vic- dialoging, daydreaming, and indwelling.
tims). The textural portrayal is an abstrac- The second state, incubation, is a time of
tion of the experience that provides content "quiet contemplation" where the researcher
and illustration, but not yet essence. waits, allowing space for awareness, intu-
Phenomenological analysis then involves itive or tacit insights, and understanding. In
a "structural description" that contains the the incubation stage, the researcher deliber-
"bons" of the experience for the whole ately withdraws, permitting meaning and
group of people studied, "a way of under- awareness to awaken in their own time. One
standing hoio the coresearchers as a group "must permit the glimmerings and awaken-
experience what they experience" (Mousta- ings to form, allow the birth of understand-
kas 1994:142). In the structural synthesis, the ing to take place in its own readiness and
phenomenologist looks beneath the affect completeness" (Moustakas 1981:50). This
inherent in the experience to deeper mean- stage leads the way toward a clear and pro-
ings for the individuais who, together, make found awareness of the experience and its
up the group. meanings.
The final step requires "an integration of In the phase of illumination, expanding
the composite textual and composite struc- awareness and deepening meaning bring
tural descriptions, providing a synthesis of new clarity of knowing. Criticai textures and
the meanings and essences of the experi- structures are revealed so that the experi-
ence is known in ali of its essential parame- disciplined analysis is to elucidate the es-
ters. The experience takes on a vividness and sence of experience of a phenomenon for an
understanding grows. Themes and pattems individual or group. The analytical vocabu-
emerge, forming clusters and parallels. New lary of phenomenological analysis is ini-
life and new visions appear along with new tially alien, and potentially alienating, until
discoveries. the researcher becomes immersed in the ho-
In the explication phase, other dimensions listic perspective, rigorous discipline, and
of meanings are added. This phase involves paradigmatic parameters of phenomenol-
a full unfolding of the experience. Through ogy. As much as anything this outline re-
focusing, self-dialogue, and reflection, the veals the difficulty of defining and sequenc-
experience is depicted and further delin- ing the internai intellectual processes
eated. New connections are made through involved in qualitative analysis more gener-
further explorations into universal elements ally.
and primary themes of the experience. The
heuristic analyst refines emergent pattems Grounded Theory
and discovered relationships.
Theory denotes a set of zuell-developed categories
It is an organiza tion of the data for oneself, a (e.g., themes, concepts) that are systematically in~
clarifica tion of pattems for oneself, a concep- terrelated through statements of relationship to
tualization of concrete subjective experience form a theoretical framework that explains some
for oneself, an integra tion of generic meanings relevant social, psychological, educational, nurs-
for oneself, and a refinement of ali these re- ing, or other phenomenon. The statements of rela-
sults for oneself. (Craig 1978:52) tionship explain who, what, when, where, why,
how, and with what consequences an event
What emerges is a depiction of the experi- occurs. Once concepts are related through
ence and a portrayal of the individuais who statements of relationship into an explanatory
participated in the study. The researcher is theoretical framework, the research findings
ready now to communicate findings in a cre- move beyond conceptual ordering to theory.
ative and meaningful way. Creative synthesis . . . A theory usually is more than a set of find-
is the bringing together of the pieces that ings; it offers an explanation about phenom-
have emerged into a total experience, show- ena. (Strauss and Corbin 1998:22)
ing pattems and relationships. This phase
points the way for new perspectives and Chapter 3 provided an overview of
meanings, a new vision of the experience. grounded theory in the context of other the-
The fundamental richness of the experience oretical perspectives such as ethnography,
and the experiencing participants is cap- constructivism, phenomenology, and her-
tured and communicated in a personal and meneutics. Norman K. Denzin, coeditor of
creative way. In heuristic analysis, the in- the Handbook of Qualitative Research and the
sights and experiences of the analyst are pri- joumal Qualitative Inquiry, has called
mary, including drawing on "tacit" knowl- grounded theory "the most influential para-
edge that is deeply internai (Polanyi 1967). digm for qualitative research in the social
These brief outlines of phenomenolog- sciences today" (1997a:18). As I noted in
ical and heuristic analysis can do no more Chapter 3, grounded theory has opened the
than hint at the in-depth living with the data door to qualitative inquiry in many tradi-
that is intended. The purpose of this kind of tional academic social science and education
Things jist ain't been he same 'round here

since that re-search dude did those inter-views.
-d
u
e
ra
"O
is
ra
co
3
a
"S
X,
u
is
Heuristic inquiry reactivity
departments, especially as a basis for doc- in the attention paid to objectivity in

toral disser tations, in part, I believe, because grounded theory.
of its overt emphasis on the importance of
and specific procedures for generating the- Fortunately, over the years, researchers have
ory. In addition, I suspect its popularity learned that a state of complete objectivity is
(Glaser 2000) may owe much to the fact that impossible and that in every piece of re-
it unabashedly admonishes the researcher to searchquantitative or qualitativethere is
strive for "objectivity." The postmodern at- an element of subjectivity. What is important
tack on objectivity has found its way into is to recognize that subjectivity is an issue and
qualitative inquiry through constructivism, researchers should take appropriate measures
hermeneutic interpretivism, and the empha- to minimize its intrusion into their analyses.
sis on subjective experience in phenomenol- . . . Over the years, we have wrestled with the
ogy. Those social scientists and academics problem of objectivity and have developed
who find some value in the methods of qual- some techniques to increase our awareness
itative inquiry, namely, in-depth interview- and help us control intrusion of bias into anal-
ing and observation, but who eschew the ysis while retaming sensitivity to what is be-
philosophical underpinnings of construc- ing said in the data. (Strauss and Corbin
tivism and interpretivism can find comfort 1998:43)
Thinking comparatively is one such tech- expert in the substantive area. . . . And if an
nique. incident comes his way that is new he can
humbly through constant comparisons mod-
Theoretical comparisons are tools (a list of proper- ify his theory to integrate a new property of a
ties) for looking at something somewhat objectively category....
rather than naming or classifying without a thor- Grounded theory methodology leaves
ough examination of the object at the property and nothing to chance by giving you rules for ev-
dimensional leveis. If the properties are evident ery stage on what to do and what to do next. If
within the data, then we do not need to rely on the reader skips any of these steps and rules,
these tools. However, because details are not the theory will not be as worthy as it could be.
always evident to the "naked" eye, and be- The typical falling out of the package is to
cause we (as human beings) are so fallible in yield to the thrill of developing a few new, cap-
our interpretations despite ali atteinpts to "de- turing categories and then yielding to use
construct" an event, incident, or interview, them m unending conceptual description and
there are times when this is not so easy and we incident tripping rather than analysis by con-
have to stand back and ask, "What is this?" In stant comparisons. (Glaser 2001:12)
asking this question, we begin, even if uncon-
sciously, to draw on properties from what we In their book on techniques and proce-
do know to make comparisons. (Strauss and dures for developing grounded theory,
Corbin 1998:80-81) Strauss and Corbin (1998:13) emphasize that
analysis is the interplay between researchers
In addition to coinfort with striving for and data, so what grounded theory offers as
objectivity, grounded theory emphasizes a framework is a set of "coding procedures"
systematic rigor and thoroughness from ini- to "help provide some standardization and
tial design, through data collection and anal- rigor" to the analytical process. Grounded
ysis, culminating in theory generation. theory is meant to "build theory rather than
test theory." It strives to "provide research-
By systematic, I still mean systematic every ers with analytical tools for handling masses
step of the way; every stage done systemati- of raw data." It seeks to help qualitative ana-
cally so the reader knows exactly the process lysts "consider alternative meanings of phe-
by which the published theory was generated. nomena." It emphasizes being "systematic
The bounty of adhering to the whole and creative simultaneously." Finally, it elu-
grounded theory method from data collection cida tes "the concepts that are the building
through the stages to writing, using the con- blocks of theory." Grounded theory opera tes
stant comparative method, shows how well from a correspondence perspective in that it
grounded theory fits, works and is relevant. aims to generate explanatory propositions
Grounded theory produces a core category that correspond to real-world phenomena.
and continually resolves a main concern, and The characteristics of a grounded theorist,
through sorting the core category organizes they posit, are these:
the integration of the theory.... Grounded the-
ory is a package, a lock-step method that starts 1. The ability to step back and critically an-
the researcher from a "know nothing" to later alyze situations
become a theorist with a publication and with
a theory that accounts for most of the action in 2. The ability to recognize the tendency to-
a substantive area. The researcher becomes an ward bias
3. The bility to think abstractly Microanalysis: "The detailed line-by-line

analysis necessary at the beginning of a study
4. The ability to be flexible and open to to generate initial categories (with their prop-
helpful criticism erties and dimensions) and to suggest rela-
tionships among categories; a combination of
5. Sensitivity to the words and actions of open and axial coding" (p. 57).
respondents
Theoretical sampling: "Sampling on the basis
of the emerging concepts, with the aim being
6. A sense of absorption and devotion to to explore the dimensional range or varied
work process. (Strauss and Corbin conditions along which the properties of con-
1998:7) cepts vary" (p. 73).
Theoretical saturation: "The point in category

Grounded theory begins with basic de-
development at which no new properties, di-
scription, moves to conceptual ordering (orga-
mensions, or relationships emerge during
nizing data into discrete categories "accord-
analysis" (p. 143).
ing to their properties and dimensions and
then using description to elucidate those cat- Range of variability: "The degree to which a
egories," p. 19), and then theorizing ("con- concept varies dimensionally along its proper-
ceiving or intuiting ideasconceptsthen ties, with variation being built into the theory
also formulating them into a logical, system- by sampling for diversity and range of proper-
atic, and explanatory scheme," p. 21). ties" (p. 143).
Open coding: "The analytic process through

In doing our analyses, we conceptualize and which concepts are identified and their prop-
classify events, acts, and outcomes. The catego- erties and dimensions are discovered in data"
ries that emerge, along with their relation- (p. 101).
ships, are the foundations for our developing
theory. This abstracting, reducing, and relat- Axial coding: "The process of relating catego-
ing is what makes the difference between theories to their subcategories, termed 'axial' be-
retical and descriptive coding (or theory building cause coding occurs around the axis of the
and doing description). Doing line-by-line cod- category, linking categories of the levei of
ing through which categories, their proper- properties and dimensions" (p. 123).
ties, and relationships emerge automatically
Relational statements: "We call these initial
takes us beyond description and puts us into a
hunches about how concepts relate 'hypothe-
conceptual mode of analysis. (Strauss and Corbin
ses' because they link two or more concepts,
1998:66)
explaining the what, why, where, and how of
phenomena" (p. 135).
Strauss and Corbin (1998) have defined
terms and processes in ways that are quite As no ted in introducing this section, com-
specific to grounded theory. It is informative parative analysis constitutes a central fea-
to compare the language of grounded theory ture of grounded theory development.
with the language of phenomenological Making theoretical comparisonssystemati-
analysis presented in the previous section. cally and creativelyengages the analyst in
Here's a sampling of important terminology. "raising questions and discovering proper-
ties and dimensions that mightbe in the data Theory bits come from two sources. First,
by increasing researcher sensitivity" (p. 67). they come from generating one concept in a
Theoretical comparisons are one of the tech- study and conjecturing without generating
niques used when doing microscopic analy- the rest of the theory. With the juicy concept,
sis. Such comparisons enable "identification the conjecture sounds grounded, but it is not;
of variations in the patterns to be found in the it is only experiential. Second, theory bits
data. It is not just one form of a category or come from a generated substantive theory. A
pattern in which we are interested but also theory bit emerges in normal talk when it is
how that pattern varies dimensionally, impossible to relate the whole theory. So, a bit
which is discerned through a comparison of with grab is related to the listener. The listener
properties and dimensions under different can then be referred to an article or a report
conditions" (p. 67). Strauss and Corbin that describes the whole theory....
(1998) offer specific techniques to increase Grounded theory is rich in imageric con-
the systematic and rigorous processes of cepts that are easy to apply "on the fly." They
comparison, for example, "the flip-flop tech- are applied intuitively, with no data, with a
nique": feeling of "knowing" as a quick analysis of a
substantive incident or area. They ring true
This indicates that a concept is turned "inside with great credibility. They empower concep-
out" or "upside down" to obtain a different tually and perceptually. They feel theoretically
perspective on the event, object, or actions/incomplete ("Yes, that accounts for it"). They are
teraction. In other words, we look at opposites exciting handles of explanation. They can run
or extremes to bring out significant properties. way ahead of the structural constraints of re-
(p- 94) search. They are simple one or two variable
applications, as opposed to being multivariate
In the course of conducting a grounded and complex. . . . They are quick and easy.
theory analysis, one moves from lower- They invade social and professional conversa-
level concepts to higher-level theorizing: tions as colleagues use them to sound knowl-
e d g e a b l e The danger, of course, is that they
Data go to concepts, and concepts get tran-
might be just plain wrong or irrelevant unless
scended to a core variable, which is the under-
based in a grounded theory. Hopefully, they
lying pattem. Formal theory is on the fourth
get corrected as more data come out. The
levei, but the theory can be boundless as the
grounded theorist should try to fit, correct,
research keeps comparing and trying to figure
and modify them even as they pass his or her
out what is going on and what the latent pat-
lips.
terns are. (Glaser 2000:4)
Unfortunately, theory bits have the ability
Glaser (2000) worries that the popularity to stunt further analysis because they can
of grounded theory has led to a preponder- sound so correct. . . . Multivariate thinking
ance of lower-level theorizing without com- stops in favor of a juicy single variable, a quick
pleting the full job. Too many qualitative an- and sensible explanation. . . . Multivariate
alysts, he warns, are satisfied to stop when thinking can continue these bits to fuller ex-
they've merely generated "theory bits." plana tions. This is the great beneft of trusting
a theory that fits, works, and is relevant as it is
Theory bits are a bit of theory from a substan- continually modified. . . . But a responsible
tive theory that a person will use briefly in a grounded theorist always should finish his or
sentence or so her bit with a statement to the effect that "Of
course, these situations are very complex or figurational approach to cross-case pattern
multivariate, and without more data, I cannot analysis was to retain the strength of holism
tell what is really going on." (Glaser 2000:7-8) embedded in context-rich individual cases
while making possible systematic compari-
As noted throughout this chapter in com- sons of relatively large numbers of cases, for
menting on how to learn qualitative analy- example, 15 to 25, or more. Ragin (2000)
sis, it is crucial to study examples. Bunch draws on fuzzy set theory and calls the re-
(2001) has published a grounded theory sult "diversity-oriented research" because it
study about people living with HIV/AIDS. systematically codes and takes into account
Glaser (1993) and Strauss and Corbin (1997) case variations and uniquenesses as well as
have collected together in edited volumes a commonalities, thereby elucidating both
range of grounded theory exemplars that in- similarities and differences. The analysis in-
clude several studies of health (life after volves constructing a "truth table" in which
heart attacks, emphysema, chronic renal the analyst codes each case for the presence
failure, chronically ill men, tuberculosis, or absence of each attribute of interest
Alzheimer's disease), organizational head- (Fielding and Lee 1998:158-59). The infor-
hunting, abusive relationships, women mation in the truth table displays the differ-
alone in public places, selfhood in women, ent combinations of conditions that produce
prison time, and characteristics of contem- a specific outcome. To deal with the large
porary Japanese society. The journal Grounded number of comparisons needed, QCA is
Theory Review began publication in 2000. done using a software program (Drass and
(See Exhibit 3.7 in Chapter 3 for the Ragin 1992; see Exhibit 8.2).
grounded theory Web site.) Analysts conducting diversity-oriented
research are admonished to assume maxi-
Qualitative Comparative mum causai complexity by considering the
Analysis possibility that no single causai condition
may be either necessary or sufficient to ex-
plain the outcome of interest. Different com-
Another approach that focuses on mak-
binations of causai conditions might pro-
ing comparisons to generate explanations is
duce the observed result, though singular
"qualitative comparative analysis" (QCA)
causes can also be considered, examined,
presented by Charles Ragin (1987, 2000).
and tested. Despite reducing large amounts
Ragin has taken on the problem of making
of data to broad patterns represented in ma-
systematic case comparisons across a num-
trices or some other form of shorthand,
ber of cases. He uses Boolean algebra to facil-
Ragin (1987) stresses repeatedly that these
itate comparisons of large case units such as
representa tions must ultimately be evalu-
nation-states and historical periods, or
ated by the extent to which they enhance un-
macro-social phenomena such as social
derstanding of specific cases. A cause-conse-
movements. His comparative method in-
quence comparative matrix, then, can be
volves representing each case as a combina-
thought of as a map providing guidance
tion of causai and outcome conditions.
through the terrain of multiple cases.
These combinations can be compared with
each other and then logically simplified
through a bottom-up process of paired com- QCA seeks to recover the complexity of partic-
parison. Ragin's aim in developing this con- ular situations by recognizing the conjunc-
tural and context-specific character of causa- carefully done case studies as one of the
tion. Unlike much qualitative analysis, the three primary strategies available for deal-
method forces researchers to select cases and ing with and sorting out rival explanations
variables in a systematic manner. This reduces in generating theory; the other two are ex-
the likelihood that "inconvenient" cases will periment-based inferences and multivariate
be dropped from the analysis or data forced analysis. Analytic induction as a compara-
into inappropriate theoretical m o u l d s . . . . tive case method
QC A clearly has the potential to be used be-
yond the historical and cross-national con- was to be the criticai foundation of a revital-
texts originally envisioned by Ragin. (Fielding ized qualitative sociology. The claim to uni-
and Lee 1998:160,161-62) - versality of the causai generalizations is . . .
derived from the examination of a single case
In cross-cultural research, the challenge studied in light of a preformulated hypothesis
of determining comparable units of analysis that might be reformulated if the hypothesis
has created controversy. For example, when does not fit the facts Discovery of a single
definitions of "family" vary dramatically, negative case is held to disprove the hypothe-
can one really do systematic comparisons? sis and to require its reformulation. (Vidich
Are extended families in nonliterate societ- and Lyman 2000:57)
ies and nuclear families in modem societies
so different that, beyond the obvious surface Over time, those using analytic induction
differences, they cease to be comparable have eliminated the emphasis on discover-
units for generating theory? "The main ing universal causai generalizations and
problem for ethnologists has been to define have instead emphasized it as a strategy for
and develop adequate and equivalent cul- engaging in qualitative inquiry and compar-
tural units for cross-cultural comparison" ative case analysis that includes examining
(De Munck 2000:279). Analytic induction, preconceived hypotheses, that is, without
another comparative approach, which we the pretense of the mental blank slate advo-
tum to now, also depends on defining com- cated in purer forms of phenomenological
parable units of analysis. inquiry and grounded theory.
Analytic Induction In analytic induction, researchers develop hy-

potheses, sometimes rough and general ap-
Analytic induction also involves cross- proximations, prior to entry into the field or, in
case analysis in an effort to seek explana- cases where data already are collected, prior to
tions. Ragin's QCA formalized and moder- data analysis. These hypotheses can be based
ated the logic of analytic induction (Ryan on hunches, assumptions, careful examina-
and Bemard 2000:787),butit was first articu- tion of research and theory, or combinations.
lated as a method of "exhaustive examina - Hypotheses are revised to fit emerging inter-
tion of cases in order to prove universal, pretations of the data over the course of data
causai generalizations" (Peter Manning collection and analysis. Researchers actively
quoted in Vidich and Lyman 2000:57). Nor- seek to disconfirm emerging hypotheses
man Denzin, in his sociological methods through negative case analysis, that is, analy-
classic The Research Act (1978b), identified sis of cases that hold promise for discon-
analytic induction based on comparisons of firming emerging hypotheses and that add
variability to the sample. In this way, the origi- uing the incest when children wanted to stop,
nators of the method sought to examine withholding permission to do ordinary things
enough cases to assure the development of until the children submitted sexually, and let-
universal hypotheses. ting others think the children were lying when
Originally developed to produce universal the incest was disclosed. These perpetrators,
and causai hypotheses, contemporary re- therefore, did not view incest as harmful to
searchers have de-emphasized universality victims, did not reflect on how they used their
and causality and have emphasized instead power and authority to coerce children to co-
the development of descriptive hypotheses operate, and even interpreted their behavior
that identify patterns of behaviors, interac- in many cases as forms of care and romantic
tions and perceptions.... Bogdan and Biklen love. (Gilgun 1995:270)
(1992) have called this approach modified an-
alytic induction. (Gilgun 1995:268-69) Analytic induction reminds us that quali-
tative inquiry can do more than discover
Jane Gilgun used modified analytic in- emergent concepts and generate new theory.
duction in a study of incest perpetrators to A mainstay of science has always been ex-
test hypotheses derived from the literature amining and reexamining and reexamining
on care and justice and to modify them to fit yet again those propositions that have be-
in-depth subjective accounts of incest perpe- come the dominant belief or explanatory
trators. She used the literature-derived con- paradigm within a discipline or group of
cepts to sensitize her throughout the re- practitioners. Modified analytic induction
search while remaining open to discovering provides a name and guidance for undertak-
concepts and hypotheses not accounted for ing such qualitative inquiry and analysis.
in the original formulations. And she did
have new insights:
Special Analytical
Most striking about the perpetrators' accounts Issues and Frameworks
was that almost ali of them defined incest as
love and care. The types of love they expressed
Reflexivity and Voice
ranged from sexual and romantic to care and
concern for the welfare of the children. These
were unanticipated findings. I did not hypoth- In Chapter 2, when presenting the major
esize that perpetrators would view incest as strategic themes of qualitative inquiry, I in-
caring and as romantic love. Rather, I had as- cluded as one of the 12 primary themes that
sumed that incest represented lack of care and, of "voice, perspective, and reflexivity."
implicitly, an inability to love. It did not occur
to me that perpetrators would equate incest The qualitative analyst owns and is reflective
and romance, or even incest and feelings of about her or his own voice and perspective; a
sexualized caring. From previous research, I credible voice conveys authentidty and trust-
did assume that incest perpetrators would worthiness; complete objectivity being impos-
experience profound sexual gratification sible and pure subjectivity undermning
through incest. Ironically, their professed love credibility, the researcher's focus becomes
of whatever type was contradicted by many balanceunderstanding and depicting the
other aspects of their accounts, such as contin- world authentically in ali its complexity while
being self-analytical, politically aware, and re- tions suggest, attention to voice applies not
flexive in consciousness. (see Exhibit 2.1) only to intentionality about the voice of the
analyst but also to intentionality and con-
Analysis and reporting are where these sciousness about whose voices and what
awarenesses come to the fore. Throughout messages are represented in the stories and
analysis and reporting, as indeed through- interviews we report. Qualitative data "can
out ali of qualitative inquiry, questions of re- be used to relay dominant voices or can be
flexivity and voice mustbe asked as part of a appropriated to 'give voice' to otherwise si-
process of engaging the data and extracting lenced groups and individuais" (Coffey and
findings. Triangulated reflexive inquiry in- Atkinson 1996:78). Eminent qualitative soci-
volves three sets of questions (see Exhibit 2.2 ologist Howard Becker (1967) posed this
in Chapter 2): classically as the question of "Whose side are
we on?" Societies, cultures, organizations,
1. Self-reflexivity. What do I know? How programs, and families are stratified. Power,
do I know what I know? What shapes and resources, and status are distributed differ-
has shaped my perspective? How have my entially. How we sample in the field, and
perceptions and my background affected then sample again during analysis in decid-
the data I have collected and my analysis of ing who and what to quote, involves deci-
those data? How do I perceive those I have sions about whose voices will be heard.
studied? With what voice do I share my per- Finally, as we report findings, we need to
spective? (See Chapter 3, discussion of anticipate how what we report will be heard
autoethnography.) What do I do with what I and understood. We need strategies for
have found? These questions challenge the thinking about the nature of the reporter-au-
researcher to also be a learner, to reflect on dience interaction, for example, under-
our "personal epistemologies"the ways standing how "six basic tendencies of hu-
we understand knowledge and the con- man behavior come into play in generating a
struction of knowledge (Rossman and Rallis positive response: reciprocation, consis-
1998:25). tency, social validation, liking, authority and
scarcity" (Cialdini 2001:76). Some writers es-
2. Reflexivity about those studied. How do
chew this responsibility, claiming that they
those studied know what they know? What
write only for themselves. But researchers
shapes and has shaped their worldview?
and evaluators have larger social responsi-
How do they perceive me, the inquirer?
bilities to present their findings for peer re-
Why? How do I know?
view and, in the cases of applied research,
evaluation and action research, to present
3. Reflexivity about audience. How do those
their findings in ways that are understand-
who receive my findings make sense of
able and useful.
what I give them? What perspectives do
they bring to the findings I offer? How do Triangulated reflexive inquiry provides a
they perceive me? How do I perceive them? framework for sorting through these issues
How do these perceptions affect what I re- during analysis and report writingand
port and how I report it? then including in the report how these re-
flections informed your findings. For exam-
Self-awareness, even a certain degree of ples of qualitative writings centered on illu-
self-analysis, has become a requirement of minating issues of reflexivity and voice, see
qualitative inquiry. As these reflexive ques- Hertz (1997).
Collaborative and pose primary preconceptions or predilec-

Participatory Analyses tions. M. W. Morris (2000) found that shared
learning, especially the capacity to deal with
Collaborative and participatory ap- counterfactuals, was reduced when partici-
proaches to qualitative inquiry include pants feared judgment by others, especially
working with nonresearchers and noneval- those in positions of authority.
uators not only in collecting data but also in In analyzing hundreds of open-ended in-
analyzing data. This requires special facilita- terviews with parents who had participated
tion skills to help those involved adopt ana- in early childhood parent education pro-
lytical thinking. Some of the challenges in- grams throughout the state of Minnesota, I
clude the following: facilitated a process of analysis that involved
some 40 program staff. The staff worked in
Deciding how much involvement groups of two and three, each analyzing 10
nonresearchers will have, for example, pre and post paired interviews at a time. No
whether they will simply react and re- staff analyzed interviews with parents from
spond to the researcher's analysis or their own programs. The analysis included
whether they will be involved in the gen- coding interviews with a frainework devel-
erative phase of analysis. Determining oped at the beginning of the study as well as
this can be a shared decision. "In partici- inductive, generative coding in which the
patory research, participants make deci- staff could create their own categories. Fol-
sions rather than function as passive lowing the coding, new and larger groups
subjects" (Rinharz 1992:185). engaged in interpreting the results and
extracting central conclusions. Everyone
0 Creating an environment in which those
worked together in a large center for three
collaborating feel that their perspective
days. I moved among the groups helping re-
is genuinely valued and respected.
solve problems. Not only did we get the data
o Demystifying research. coded, but the process, as is intended in col-
Combining training in how to do analy- laborative and participatory research pro-
sis with the actual work of analysis. cesses, proved to be an enormously stimu-
lating and provocative learning experience
a Managing the difficult mechanics of the
for the staff participants. The process forced
process, especially where several people
them to engage deeply with parents' percep-
are involved.
tions and feedback, as well as to engage each
Developing processes for dealing with other's reactions, biases, and interpreta-
conflicts in interpreta tions (e.g., agree- tions. In that regard, the process also facili-
ing to report multiple interpreta tions). tated communication among diverse staff
Maintaining confidentiality with multi- members from across the state, another in-
ple analysts. tended outcome of the collaborative analy-
sis process. Finally, the process saved thou-
A good example of these challenges con- sands of dollars in research and evaluation
cerns how to help lay analysts deal with costs, while making a staff and program de-
counterintuitive findings and counter- velopment contribution. The results were
factuals, that is, data that don't fit primary intended primarily for internai program im-
patterns, negative cases, and data that op- provement use. As would be expected in
such a nonresearcher analysis process, exter- Construing the meaning of the whole meant
nai stakeholders placed less value on the re- making sense of the parts, and grasping the
sults than did those who participated in the meaning of the parts depended on having
process (Program Evaluation Division 2001; some sense of the w h o l e [T]he hermeneutic
Mueller 1996; Mueller and Fitzpatrick 1998). circle indicates a necessary condition of inter-
pretation, but the circularity of the process is
only temporaryeventually the interpreter
The Hermeneutic can come to something approximating a com-
Circle and Interpretation plete and correct understanding of the mean-
ing of a text in which whole and parts are
related in perfect harmony. Said somewhat
Hermes was messenger to the Greek gods
differently, the interpreter can, in time, get out-
Himself the god of travei, commerce, inven-
side of or escape the hermeneutic circle in dis-
tion, eloquence, cunning, and thievery, he ac-
covering the "true" meaning of the text.
quired very early in his life a reputa ti on for
(Schwandt 2001:112)
being a precocious trickster. (On the day he
was born he stole Apollo's cattle, invented the The method involves playing the strange
lyre, and made a fire.) His duties as messenger and unfamiliar parts of an action, text, or ut-
included conducting the souls of the dead to terance off against the integrity of the action,
Hades, waming Aeneas to go to Italy, where narrative, or utterance as whole until the
he founded the Roman race, and commanding meaning of the strange passages and the
the nymph Calypso to send Odysseus away meaning of the whole are worked out or ac-
on a raft, despite her love for him. With good counted for. (Thus, for example, to under-
reason his name is celebrated in the term "her- stand the meaning of the first few lines of a
meneutics," which refers to the business of in- poem, I must ha ve a grasp of the overall mean-
terpreting. . . . Since we don't have a godly ing of the poem, and vice versa.) In this pro-
messenger available to us, we have to inter- cess of applying the hermeneutic method, the
pret things for ourselves. (Packer and Addi- interpreteis self-understanding and socio-
son 1989:1) historical location neither affects nor is af-

fected by the effort to interpret the meaning of
Hermeneutics focuses on interpreting the text or utterance. In fact, in applying the
something of interest, traditionally a text or method, the interpreter abides by a set of pro-
work of art, but in the larger context of quali- cedural rules that help ensure that the inter-
tative inquiry, it has also come to include in- p r e t e i s historical situation does not distort
terpreting interviews and observed actions. the bid to uncover the actual meaning embed-
The emphasis throughout concerns the na- ded in the text, act, or utterance, thereby help-
ture of interpretation, and various philoso- ing to ensure the objectivity of the interpreta-
phers have approached the matter differ- tion. (Schwandt 2001:114)
ently, some arguing that there is no method
of interpretation per se because everything The circularity and universality of herme-
involves interpretation (Schwandt 2000, neutics (every interpretation is layered in
2001). For our purposes here, the hermeneutic and dependent on other interpretations, like
circle, as an analytical process aimed at en- a series of dolls that fit one inside the other,
hancing understanding, offers a particular and then another and another) pose the
emphasis in qualitative analysis, namely, re- problem for the qualitative analyst of where
lating parts to wholes, and wholes to parts. to begin. How and where do you break into
the hermeneutic circle of interpretation? much as the loss of our human ability to en-
Packer and Addison (1989), in adapting the counter new concerns and uncover fresh
hermeneutic circle as an inquiry approach puzzles. So although hermeneutic inquiry
for psychology, suggest beginning with proceeds from a starting place, a self-con-
"practical understanding": sciously interpretive approach to scientific
investigation does not come to an end at
Practical understanding is not an origin for some final resting place, but works instead
knowledge in the sense of a foundation; it is, to keep discussion open and alive, to keep
instead, the starting place for interpretation. inquiry under way. (p. 35)
Interpretive inquiry begins not from an abso-
lute origin of unquestionable data or totally At a general levei and in a global way,
consistent logic, but at a place delineated by hermeneuties reminds us of the interpre-
our everyday participatory understanding of tive core of qualitative inquiry, the impor-
people and events. We begin there in full tance of context and the dynamic
awareness that this understanding is corrigi- whole-part interrelations of a holistic per-
ble, and that it is partial in the twin senses of spective. At a specific levei and in a
being incomplete and perspectivai. Under- particularistic way, the hermeneutic circle
standing is always moving forward. Practical offers a process for formally engaging in
activty projects itself forward mto the world interpretation.
from its starting place, and shows us the enti-
ties we are home among. This means that nei-
Analyzing
ther commonsense nor scientific knowledge
Institutional Documents
can be traced back to an origin, a foundation.
. . . (p. 23)
The circularity of understanding, then, is Gale Miller (1997) has studied the par-
that we understand in terms of what we al- ticular challenges of "contextualizing or-
ready know. But the circularity is not, ganizational texts." Written documents of
Heidegger argues, a "vicious" one where we ali kinds are pervasive in modem institu-
simply confirm our prejudices, it is an "essen- tions such as hospitais, schools, nursing
tial" one without which there would be no un- homes, police departments, courts, clinics,
derstanding at ali. And the circle is complete; and social welfare agencies. Governments,
there is accommodation as well as assimila- nonprofit agencies, philanthropic organi-
tion. If we are persevering and open, our at- zations, and private institutions produce
tention will be drawn to the projective massive amounts of files and reports.
character of our understanding andin the Miller argues that
backward are, the movements of returnwe
gain an increased appreciation of what the qualitative researchers are uniquely posi-
fore-strueture involves, and where it might tioned to study these texts by analyzing the
best be c h a n g e d . . . . (p. 34). practical social contexts of everyday life
Hermeneutic inquiry is not oriented to- within which they are constructed and used.
ward a grand design. Any final construction Texts are one aspect of the sense-making ac-
that would be a resting point for scientific in- tivities through which we reconstruct, sus-
quiry represents an illusion that must be re- tam, contest and change our senses of social
sisted. If ali knowledge were to be at last reality. They are socially constructed reali-
collected in some gigantic encyclopedia this ties that warrant study in their own right,
would mark not the triumph of science so (P- 77)
Special challenges in analyzing docu- Confrontations between protagonists

ments include the following: and antagonists
Costumes and props
Getting access to documents
Dramaturgical loyalty, which "requires
Understanding how and why the performers to 'act as if they have ac-
documents were produced cepted certain moral obligations'"
Determining the accuracy of (p. 113).
documents
o Linking documents with other sources, Hunt and Benford (1997) argue that
including interviews and observations "dramaturgy might provide a reflexive so-
ciological method":
Deconstructing and demystifying
institutional texts
First, our approach presents a conceptual
framework for understanding research pro-
Miller concludes, "Demystifying institu-
ductions generally and field studies more
tional texts is one way of demystifying insti-
specifically. Dramaturgical method also il-
tutional authority" (p. 91).
luminates common pitfalls in social science
work, implying that researchers might be
Dramaturgical Analysis well-advised to pay particular attention to
the details of impression management as
Dramaturgy is a perspective that uses a theat- well as the problems of securing resources,
rical metaphor to understand social interac- audiences and the like. The third contribu-
tion. The approach takes act to be its central tion is that dramaturgical method furnishes
concept. From a dramaturgical point of view, a vantage point for social scientists to exam-
humans, in a specific social and temporal con- ine their own research productions critically.
text, act to create meaning and demonstrate By equating research with drama, we have
p u r p o s e [Doing this involves] "impression sought to limit the pretentiousness that
management," suggesting that individuais seems endemic to most social science work.
present themselves to others so as to foster and Instead of presenting a window to "reality,"
maintain particular images or fronts. In their a dramaturgical method serves as a constant
performances, individuais construct some im- reminder that researchers are in the business
ages intentionally and provide others inad- of "reality construction." (Hunt and Benford
vertently. (Hunt and Benford 1997:106) 1997:116-17)
Dramaturgy puts the concept of "acting" To appreciate how an interpretive

on center stage at the theater of qualitative framework such as dramaturgical analysis
inquiry. A dramaturgical analysis of human affects interpretation, it helps to compare
interactions employs theatrical sensitizing data and conclusions using different
concepts: frameworks. Martha Feldman (1995) has
done just that by analyzing her study of a
a Scripting university housing office through the
lenses of ethnomethodology (how physi-
Staging
cal realities such as buildings become in-
Dialogue and direction stitutional realities), semiotics (how writ-
Developing dramatis personae ten policies become institutional realities
with real consequences), deconstruction (of You can only find nothing if you
university salaries in relation to hierarchy immerse yourself in nothing.
and power), and dramaturgical analysis You can only find nothing if you go
(how "backstage" events deep within the in- nowhere.
stitution become manifest for targeted audi- Go to real places.
ences). She compares the strengths and
Talk to real people.
weaknesses of each approach, a useful re-
minder that ali frameworks have both Observe real things.
strengths and weaknesses. You will find something.
Indeed, you will find much, for much
is there.
Finding Nothing
You will find the world.
Students beginning dissertations often

ask me, their anxiety palpable and under- M. Synthesizing
standable, "What if I don't find out any- Qualitative Studies
thing?" Bob Stake of responsive evaluation
and case study fame said at his retirement: Synthesizing research to aggregate and sub-
stantiate knowledge has become one of the
Paraphrasing Milton: They also serve who
important challenges of the information
leave the null hypothesis t e n a b l e . . . .
age, especially synthesizing applied re-
It is a sophisticated researcher who beams
search to inform policy making (Cooper
with pride having, with thoroughness and dil-
1998). As qualitative research has become
igence, found nothing there. (Stake 1998:364,
better understood, more widely used, and
with a no d to Michael Scriven for inspiration)
more fully reported, a new opportunity
and a new challengehas emerged: syn-
thesizing qualitative studies. In one sense each
True enough. But in another sense, it's not qualitative study is a case. Synthesis of dif-
possible to find nothing there, at least not in ferent qualitative studies on the same sub-
qualitative inquiry. The case study is there. It ject is a form of cross-case analysis. Such a
may not have led to new insights or con- synthesis is much more than a literature re-
firmed one's predictions, but the description view. Noblit and Hare (1988) describe syn-
of that case at that time and that place is thesizing qualitative studies as "meta-eth-
there. That is much more than nothing. The nography" in which the challenge is to
interview responses and observations are "retain the uniqueness and holism of ac-
there. They, too, may not have led to head- counts even as we synthesize them in the
line-grabbing insights or confirmed some- translations" (p. 7).
one's eminent theory, but the thoughts and For scholarly inquiry, the qualitative syn-
reflections from those people at that time thesis is a way to build theory through in-
and that place are there, recorded and reduction and interpretation. For evaluators, a
ported. That is much more than nothing. qualitative synthesis can identify and ex-
Halcolm will tell you this: trapolate lessons learned. Evaluators can
synthesize lessons from a number of case
You can only find nothing if you stare at a studies to generate generic factors that con-
vacuum. tribute to program effectiveness as, for ex-
ample, Lisbeth Schorr (1988) did for poverty ducted and presented to The McKnight
programs in her review and synthesis Within Foundation showing that these programs
Our Reach: Breaking the Cycle ofDisadvantage. had successfully attained and exceeded in-
The U.S. Agency for International Devel- tended outcomes. But why were they suc-
opment has supported lessons learned syn- cessful? That was the intriguing and com-
thesis studies on such subjects as irriga- plex question on which the synthesis study
tion (Steinberg 1983), rural electrification focused.
(Wasserman and Davenport 1983), food for The synthesis design included fieldwork
peace (Rogers and Wallerstein 1985), educa- (interviews with key players and site visits
tion development efforts (Warren 1984), pri- to each project) as well as extensive review
vate sector development (Bremer et al. of their independent evaluations. I identi-
1985), contraceptive social marketing (Bin- fied common success factors that were man-
nendijk 1986), agriculture and rural devel- ifest in ali three projects. Those were ill-
opment (Johnston et al. 1987), agricultural uminating, but not surprising. The real
policy analysis and planning (Tilney and contribution of the synthesis was in how the
Riordan 1988), and agroforestry (Chew success factors fit together, an unanticipated
1989). In synthesizing separate evaluations pattern that deepened the implications for
to identify lessons learned, evaluators build understanding effective philanthropy.
a store of knowledge for future program de- The 12 success factors common to ali
velopment, more effective program imple- three programs were as folio ws:
mentation, and enlightened policy making.
The sample for synthesis studies usually o High-quality people
consists of case studies with a common fo- Substantial financial resources
cus, for example, elementary education,
health care for the elderly, and so forth. Creative partnerships
However, one can also learn lessons about Leverage
effective human intervention processes H Vision
more generically by synthesizing case stud-
ies on quite different subjects. I synthesized A clear values orientation
three quite different qualitative evaluations a Self-sustaining institutions
conducted for The McKnight Foundation: a
a Long time frames
major family housing effort, a downtown
development endeavor, and a graduate fel- a Flexibility
lowship program for minorities. Before un- Cutting edge foresight
dertaking the synthesis, I knew nothing
Risk taking
about these programs, nor did I approach
them with any particular preconceptions. I Leadership
was not looking for any specific similarities
and none were suggested to me by either While each of these factors provided in-
McKnight or program staff. The results were sight into an important element of effective
intended to provide insights into The philanthropic programming, the unantici-
McKnight Foundation's operating philoso- pated pattern was how these factors fit to-
phy and strategies as exemplified in practice by gether to form a constellation of excellence. I
real operating programs. Independent evalua- found that I couldn't prioritize these factors
tions of each program had already been con- because they worked together in such a way
that no one factor was primary or sufficient; through leverage, careful planning, commu-
rather, each made a criticai contribution to nity involvement, and shared commitments.
an integrated, effectively functioning whole. And doing ali these things in mutually rein-
The lesson that emerged for effective philan- forcing ways. The challenge for effective
thropy was not a series of steps to follow, but philanthropy, then, is putting ali the pieces
rather a mosaic to create; that is, effective and factors together to support integrated,
philanthropy appears to be a process of holistic, and high-impact efforts and re-
matching and integrating elements so that sultsand to do so creatively (Storm and
the pieces fit together in a meaningful and Vitt 2000:115-16).
comprehensive way as a solution to complex As qualitative evaluation and research
problems. This means matchingpeople with prolifera te, the opportunities for and impor-
resources; bringing vision and values to bear tance of synthesizing diverse studies will in-
on problems; and nurturing partnerships crease accordingly.
[I. Reporting Findings
A t one time, one blade of grass is as effective as a sixteen-foot golden

statue of Buddha. At another time, a sixteen-foot golden statute of
Buddha is as effective as a blade of grass.
Wu-Men
It can happen that an overall structure that or-

Some reports are thin as a blade of grass; ganizes a great deal of material happens also
others feel 16 feet thick. Size, of course, is not to leave out some of one's most favorite mate-
the issue. Quality is. But given the volume of rial and small pieces of analysis Unless one
data involved in qualitative inquiry and the decides to write a relatively disconnected re-
challenges of data reduction already dis- port, he must face the hard truth that no over-
cussed, reporting qualitative findings is the all analytic structure is likely to encompass
final step in data reduction and size is a real every small piece of analysis and ali the empir-
constraint, especially when writing in forms ical material that one has on h a n d . . . .
other than research monographs and The underlying philosophical point, per-
book-length studies, such as journal articles haps, is that everything is related to every-
and newsletter summaries. Each step in thing else in a flowing, even organic fashion,
completing a qualitative project presents making coherence and organization a difficult
quality challenges (Morse 1997), but the final and problematic human task. But in order to
step is completing a report so that others can have any kind of understanding, we humans
know what you've learned and how you require that some sort of order be imposed
learned it. This means finding and writing upon that flux. No order fits perfectly. Ali or-
your story (Glesne 1999). It also means deal- der is provisional and partial. Nonetheless,
ing with what Lofland (1971) called the "the understanding requires order, provisional
agony of omitting"deciding what mate- and partial as it may be. It is with that philo-
rial to leave out of the story. sophical view that one can hopefully bring
himself to accept the fact that he cannot write tively significant and providing enough de-
about everything that he has seen (or ana- tail and evidence to illuminate and make
lyzed) and still write something with overall that case.
coherence or overall structure. (Lofland Yet, the description must not be so "thin"
1971:123) as to remove context or meaning. Qualita-
tive analysis, remember, is grounded in
This chapter opened with the reminder "thick description."
that purpose guides analysis. Purpose also
guides report writing and dissemination of A thick description does more than record
findings. The keys to ali writing start with what a person is doing. It goes beyond mere
(1) knowing your audience and (2) knowing fact and surface appearances. It presents de-
what you want to say to thema form of tail, context, emotion, and the webs of social
strategic communications (Weiss 2001). Dis- relationships that join persons to one another.
ser tations have their own formats and re- Thick description evokes emotionality and
quirements (Patton 1996a; Fitzpatrick, self-feelings. It inserts history into experience.
Secrist, and Wright 1998; Rudestam and It establishes the significance of an experience,
Newton 1992). Scholarly journals in various or the sequence of events, for the person or
disciplines and applied research fields have persons in question. In thick description, the
their own standards and norms for what voices, feelings, actions, and meanings of in-
they publish. The best way to learn those is teracting individuais are heard. (Denzin
to read and study them, and study special- 1989b :83)
ized qualitative methods journals such as
Qualitative Inquiry, Field Methods, Symbolic Thick description sets up and makes pos-
Interaction, Journal ofContemporary Ethnogra- sible interpretation. "It contains the neces-
phy, and Grounded Theory Review. Below IT1 sary ingredients for thick interpretation"
discuss evaluation and action research re- (Denzin 1989b:83). By "thick interpretation"
porting. Denzin means, in part, connecting individ-
ual cases to larger public issues and to the
programs that serve as the linkage between
Balance Between Description
individual troubles and public concerns.
and Interpretation
"The perspectives and experiences of those
One of the major decisions that has to be persons who are served by applied pro-
made about what to omit in the process of grams mustbe grasped, interpreted, and un-
data reduction for reporting involves how derstood if solid, effective, applied pro-
much description to include. Description grams are to be put into place" (p. 105).
and quotation provide the foundation of Description is thus balanced by analysis
qualitative reporting. Sufficient description and interpretation. Endless description be-
and direct quotations should be included to comes its own muddle. The purpose of anal-
allow the reader to enter into the situation ysis is to organize the description so that it is
and thoughts of the people represented in manageable. Description provides the skele-
the report. Description should stop short, tal frame for analysis that leads into inter-
however, of becoming trivial and mundane. pretation. An interesting and readable re-
The reader does not have to know every- port provides sufficient description to allow
thing that was done or said. Focus comes the reader to understand the basis for an in-
from having determined what's substan- terpretation, and sufficient interpretation to
allow the reader to appreciate the descrip- qualitative analyst must make judgments
tion. that provide clues for the reader as to the
Details of verification and validation pro- writer's belief about variations m the credi-
cesses (topics of the next chapter) are typi- bility of different findings: When are pat-
cally placed in a separate methods section of tems "clear"? When are they "strongly sup-
a report, but parenthetical remarks through- ported by the data"? When are the patterns
out the text about findings that have been "merely suggestive"? Readers will ulti-
validated can help readers value what they mately make their own decisions and judg-
are reading. For example, if I describe some ments about these matters based on the
program process and then speculate on the evidence you've provided, but your analy-
relationship between that process and client sis-based opinions and speculations deserve
outeomes, I may mention that (1) staff and to be reported and are usually of interest to
clients agreed with this analysis when they readers given that youVe struggled with the
read it, (2) I experienced this linkage person- data and know the data better than anyone
ally in my own participant-observation ex- else.
perience in the program, and (3) this cormec- Appendix 8.3 at the end of this chapter
tion was independently arrived at by two presents portions of a report describing the
analysts looking at the data separately. effects on participants of their experiences in
The analyst should help readers under- the wilderness education program. The data
stand different degrees of significance of come from in-depth, open-ended inter-
various findings, if these exist. Because qual- views. This excerpt illustrates the centrality
itative analysis lacks the parsimonious sta- of quota tions in supporting and explaining
tistical significance tests of statisties, the thematic findings.
Communicating With Metaphors and Analogies
11 perception of truth is the detection of an analogy.
Henry David Thoreau (1817-1862)
The museum study reported earlier in the serves as a form of representation, shaping
discussion of analyst-generated typologies how we perceive the world (Chatterjee 2001;
differentiated different kinds of visitors by Patton 2000; Smith 1981).
using metaphors: the "commuter," the "no- Metaphors and analogies can be power-
mad," the "cafeteria type," and the "V.I.P." f ul ways of connecting with readers of quali-
and an analogy between visitors to Earth tative studies, but some analogies offend
from outer space and visitors to a museum. certain audiences. Thus, metaphors and
In the dropout study, we relied on meta- analogies mustbe selected with some sensi-
phors to depict the different roles we ob- tivity to how those being described would
served teachers playing in interacting with feel and how intended audiences will re-
truants: the "cop," the "old-fashioned school spond. At a meeting of the Midwest Socio-
master/' and the "ostrich." Language not logical Society, distinguished sociologist
only supports communication but also Morris Janowitz was asked to participate in
a panei on the question "What is the cutting mentthe violence has already been done!
edge of sociology?" Janowitz, having writ- How about brooms to sweep away the attic-y
ten extensively on the sociology of the mili- cobwebs of our male/female stereotypes? The
tary, took offense at the "cutting edge" meta- tests and assessment techniques we fre-
phor. He explained: quently use are full of them. How about
knives, forks, and spoons to sample the feast
Paul Russell, the humanist, has prepared a of human diversity in ali its richness and color.
powerful and brilliant sociological study of Where are the techniques that assess the deli-
the literary works of the great wars of the 20th cious-ness of response variety, independence
century which he entitled The Great YJar and of thought, originality, uniqueness? (And lest
Modem Memory. It is a work which ali sociolo- you think those are female metaphors, let me
gists should read. His conclusion is that World do away with that mythat our house every-
War I and World War II, Korea and Vietnam body sweeps and everybody eats!) Our
have militarized our language. I agree and workgroup talked about another metaphor
therefore do not like the question "Where is the cafeteria line versus the smorgasbord
the cutting edge of sociology?" "Cutting banquet of styles of teaching/learning/as-
Edge"is a military term. I am put off by the sessmg. Many new metaphors are needed as
very term cutting edge. Cutting edge, like the we seek clarity in our search for better ways of
parallel term breakthrough, are slogans which evaluating. To deal with diversity is to look for
intellectuals have inherited from the manag- new metaphors. (Hurty 1976)
ers of violence. Even if they apply to the physi-
cal sciences, I do not believe that they apply to Metaphors can be powerful and cie ver
the social sciences, especially sociology, which ways of communicating findings. A great
grows by gradual accretion. (Janowitz deal of meaning can be conveyed in a single
1979:591) phrase with a powerful metaphor. More-
over, developing and using metaphors can
"Strategic planning" has military origins be fun, both for the analyst and for the
and connotations as does "rapid reconnais- reader. It is important, however, to make
sance," a phrase used to describe certain sure that the metaphor serves the data and
short-term, intensive fieldwork efforts (see not vice versa. The creative analyst who
Chapter 4). Some stakeholder groups will finds a powerful metaphor may be tempted
object to such associations. Of particular im- to manipula te the data to fit the metaphor.
portance, in this regard, is avoiding meta- Moreover, because metaphors carry implicit
phors with possible racist and sexist conno- connotations, it is important to make sure
tations, for instance, "It's black and white." that the data fit the most prominent of those
At the Educational Evaluation and Public connotations so that what is communicated
Policy Conference sponsored by the Far is what the analyst wants to communicate.
West Laboratory for Educational Research Finally, one must avoid reifying metaphors
and Development, the women's caucus ex- and acting as if the world were really the
pressed concern about the analogies used in way the metaphor suggests it is.
evaluation and went on to suggest some al-
ternatives. The metaphor is chiefly a tool for revealing
special properties of an object or event. Fre-
To deal with diversity is to look for new meta- quently, theorists forget this and make their
phors. We need no new weapons of assess- metaphors a real entity in the empirical world.
It is legitimate, for example, to say that a social power (and exceed the limitations) of an indi-
system is like an organism, but this does not vidual case. (p. 55)
mean that a social system is an organism.
When metaphors, or concepts, are reified, they This admonition reminds us not to take
lose their explanatory value and become tau- anything for granted or fali into following
tologies. A careful line must be followed in some recipe for writing. Asking yourself,
the use of metaphors, so that they remain a "When ali is said and done, what conclu-
powerful means of illumination. (Denzin sions do I draw from ali this work?" can be a
1978b:46) focusing question that forces you to get at es-
sence. Or, as Wolcott suggests, it can be an
unnecessary and inappropriate burden.
Drawing Conclusions
Or it can be a chance to look to the future.
In his practical monograph Writing Up The Spanish-born philosopher and poet
Qualitative Research, Wolcott (1990) consid- George Santayana concluded thusly when
er s the challenge of how to conclude a quali- he retired from Harvard. Students and col-
tative study. Purpose again rules in an- leagues packed his classroom for his final
swering this question. Scholarly articles, appearance. He gave an inspiring lecture
dissertations, and evaluation reports have and was about to conclude when, in
different norms for drawing conclusions. mid-sentence, he cut the head of a forsythia
But Wolcott goes further by questioning the beginning to blossom in melting snow out-
very idea of conclusions: side the wrndow. He stopped abruptly,
picked up his coat, hat, and gloves, and
Give serious thought to dropping the idea that headed for the door. He turned at the door
your final chapter must lead to a conclusion or and said gently, "Gentlemen, I should not be
that the account must build toward a drama tic able to finish that sentence. I have just dis-
climax. . . . In reporting qualitative work, I covered that I have an appointment with
avoid the term conclusion. I do not want to April."
work toward a grand flourish that might Or as Halcolm would say, Not conclud-
tempt me beyond the boundaries of the mate- ing is its own conclusion.
rial I have been presenting ordetract from the
!zl Special Issues in Evaluation Reporting and an Example
dialectic among several mindsets is essential to good evaluation.
Robert Stake (1998:370)
Feedback and Analysis is still under way or as soon as data collec-

tion is over. Providing preliminary feedback
Evaluation poses special challenges provides an opportunity to reaffirm with in-
when, as is typical, intended users (espe- tended users the final focus of the analysis
cially program staff and administrators) and nurture their interest in findings. Aca-
want preliminary feedback while fieldwork demic social scientists have a tendency to
want to withhold their findings until they back we provided after evaluating the
have polished their presentation. Use of leadership development program described
evaluation findings, however, does not nec- earlier. After six days of intense (and some-
essarily center on the final report, which times tense) participant observation in a re-
should be viewed as one element in a total treat setting, we needed a framework for
utilization process, sometimes a minor ele- providing formative, descriptive feedback
ment, especially in formative evaluation. to program staff m a way that could be heard.
Evaluators who prefer to work diligently We knew that staff were heavily ego-in-
in the solitude of their offices until they can volved in the program and would be very
spring a final report on a waiting world may sensitive to an approach that might appear
find that the world has passed them by. to substitute our concept of the program for
Feedback can inform ongoing thinking theirs. Yet, a majorpurpose of the evaluation
about a program rather than serve only as a was to help them identify and make explicit
one-shot information input for a single deci- their operating assumptions as evidenced in
sion point. However, sessions devoted to re- what actually happened during the six-day
establishing the focus of the evaluation anal- retreat. As our team of three accumulated
ysis and providing initial feedback need to more and more data, debriefing each night
be handled with care. The evaluator will what we were finding, we became increas-
need to explain that analysis of qualitative ingly worried about how to focus feedback.
data involves a painstaking process requir- The probiem was solved the f if th night when
mg long hours of careful work, going over we realized that we could use their frame-
notes, organizing the data, looking for pat- works for describing to them what we were
terns, checking emergent patterns against finding. For example, a major component of
the data, cross-validating data sources and the program was having participants work
findings, and making linkages among the with the Myers-Briggs Type Indicator, an in-
various parts of the data and the emergent strument that measures individual person-
dimensions of the analysis. Thus, any early ality type based on the work of Carl Jung (cf.
discussion of findings can only be prelimi- Berens and Nardi 1999; Myers 1995; Krueger
nary, directed at the most general issues and and Thuesen 1988; Hirsh and Kummerow
the most striking, obvious results. If, in the 1987). The Myers-Briggs Type Indicator
course of conducting the more detailed and gives individuais scores on four bipolar
complete analysis of the data, the evaluator
scales:
finds that statements made or feedback
given during a preliminary session were in-
accurate, evaluation users should be in- (E) Extraversion-Introversion (I)
formed about the discrepancy at once. (S) Sensing-Intuition (N)
(T) Thinking-Feeling (F)
(I) ludgment-Perception (P)
Evaluative Feedback Using
Indigenous Typologies In the feedback session, we began by ask-
ing the six staff members to characterize the
Identifying indigenous typologies as part overall retreat culture using the Myers-
of a program evaluation can facilitate in- Briggs framework. Staff members shared
creased understanding when providing their separate ratings, on which there was
feedback. A good example comes from feed- not consensus, and then we shared our per-
ceptions. We spent the whole morning dis- dowed with $15 million ($10 million from
cussing the data for and implications of each The McKnight Foundation and $5 million
scale as a manifestation of the program's cul- from the state of Florida). The program had
ture. We ended the session by discussing conducted its own evaluations, which
where the staff wanted the program to be on showed they were successfully attaining in-
each dimension. Staff members were able tended outcomes. The question posed to me
to hear what we said, without becoming by The McKnight Foundation decision mak-
defensive, because we used their frame- ers was, What factors explain the high levei
work, a framework they had defined as of success achieved by this program? I ob-
nonjudgmental, facilitative, and develop- served the program's annual conference for
mental. ali 92 doctoral fellows; made site visits; re-
We formatted our presentation to staff us- viewed program records and documents;
ing a distinction between "observations" interviewed a purposeful sample of partici-
and "perceived impacts" that program par- pants, key knowledgeables, and the pro-
ticipants were taught as part of the leader- gram^ executive director; and asked ali
ship training. Observation: "You inter- participants to write responses to some
rupted me in midsentence." Perceived questions. The analysis of ali that data re-
impact: "I felt cut-off and didn't contribute duced to 10 major success factors (which
after that." This simple distinction, aimed at later became part of the synthesis reported
enhancing interpersonal communications, earlier, p. 501):
served as a comfortable, familiar format for
1. Strong leadership through a bold initia-
program staff to receive formative evalua-
tive from The McKnight Foundation
tion feedback. Our report, then, followed
that mobilized educational leaders in
this format. Three of 20 observations from
Florida.
the report are reproduced in Exhibit 8.12.
The criticai point here is that we pre-
2. A sizable amount of money ($15 million)
sented the findings using their categories
able to attract attention and generate
and their frameworks. This greatly facili-
support.
tated the feedback and enhanced the subse-
quent formative, developmental discus- 3. Effective use of leverage at every levei of
sions. Capturing and using indigenous program operation. (McKnight insisted
typologies can be a powerful analytical ap- on major matching funds and use of lo-
proach for making sense of and reporting cal in-kind resources from participating
qualitative data. universities.)
For evaluators, the inductive search for 4. A long-term perspective on and commit-
patterns is guided by the evaluation ques- ment to a sustainable program with cu-
tions identified at the beginning of the study mula tive impact over timein perpetu-
and a focus on how the findings are in- ity. (The program was finally converted
tended to be used by intended users (Patton to an endowment.)
1997a). This utilization focus keeps findings
from becoming too abstract, esoteric, or the- 5. A carefully rnelded public-private part-
oretical. For example, I was asked by The nership.
McKnight Foundation to review The 6. A program based on a vision made real
McKnight Programs in Higher Education in through a carefully designed model that
Florida, a minority fellowship program en- was true to the vision.
BBBI:1>:HBJ Distinquishinq Observations From Perceived Impacts

BBBaaiMMMaMBIMM
I ^M
I MMmamaMM ~r 1
Observo tions Perceived Impacts
1. The retreat setting, away from the 1. There is deep bonding among group members;
world, is ntroverted. there is a sense of the group as separate from
the "real" world, though participants are
expected to engage the "real" world after
the retreat.
2. The retreat is more conceptual and 2. Participants are conceptually stimulated and
abstract in content than fact and exposed to a variety of ideas. Some express
skll oriented. It is primarily intuitive uncertainty about what to do with the ideas
(as opposed to step-by-step and (lack of practical applications).
practical).
3. Retreat culture is heavily affective, 3a. Highly emotional connections are made
feelings oriented, not thinking among participants.
oriented.
b. Participants are sensitized to how they
feel about what they are experiencing,
expicitly encouraged to share feelings.
c. Participants are affirmed as important;
they feel special, cared about, and valued;
it is a safe environment for learning.
d. Participants are not stretched intellectualy;
logica distinctions are not made, key
concepts remain ambiguous. Affirming
participants is clearly more important than
challengng them; harmony is valued over
clarity.
7. Taking the time and effort to carefully 10. Clear accountability and evaluation so that
plan in a process that generated broad- problems could be corrected and accom-
based community and political support plishments could be recognized.
throughout the state.
8. The careful structuring of local board con-

These patterns are straightforward and
trol so that responsibility and owner-
understandable. The themes above answer a
ship resided in Florida among key
focused evaluation question. The report pre-
influentials.
sented data supporting each success factor
9. Taking advantage of the right timing and and explairting in greater detail what each
climate for this kind of program. one meant and how it operated. But the list
Oh good Bill! So, what do you think? Is the

You're reading my report! thick description thick enough?
...ril send it to the print shop for

national distribution.
represents the 10 major patterns in the data. To Write a Report or

There is no presentation of an elegant theory
Not to Write a Report?
or carefully conceptualized typology. These
10 factors were the qualitative evaluation
findings. They answered the intended users' I find in my own work that final reports
primary evaluation question. Such an analy- frequently have less impact than the direct,
sis is an example of practical, utilization- face-to-face interactions I have with primary
focused evaluation. evaluation users to provide them with feed-
back about evaluation findings and to share like, following some basic principies can en-
with them the nature of the data. Making hance the presentation of qualitative evalua-
oral briefings is an increasingly important tion data.
evaluation competence (Hendricks 1982).
Final reports often serve an important dis- Focus
semination function to audiences beyond
immediate decision makers and informa- Even a comprehensive report will have to
tion users, but they are not automatically omit a great deal of information collected by
and necessarily the primary source of infor- the evaluator. Focus is essential. Evaluators
mation for those who are expected to actu- who try to include everything risk losing
ally use evaluation findings. I have done their readers in the sheer volume of the pre-
evaluations that involved no polished, final sentation. To enhance a report's impact, the
report because certain formative situations evaluation should address clearly each ma-
don't justify putting a lot of scarce resources jor evaluation question, that is, present the
into the production of a polished final report descriptive findings, analysis, and interpre-
that will sit on a shelf somewhere. Eyebrows tation of each focused issue together suc-
may be raised when evaluators ask, "Is there cinctly. An evaluation report should be read-
any reason to produce a final, written report able, understandable, and relatively free of
for this evaluation?" But it's a question academic jargon. The data should impress
worth asking, and, in my opinion, the bur- the reader, not the academic training of the
den of proof lies with the evaluation users to evaluator.
justify production of a full report in cases of The advice I find myself repeating most
formative evaluation and informal action re- often to students when they are writing re-
search. ports is, Focus, focus, focus! The agony on the
Normally, of course, a full report will be part of the evaluator of having omitted
produce d. The contents, length, and nature things is matched only by the readers' agony
of the report are partly a matter for negotia- in having to read those things that were not
tion between evaluators and primary users omitted but should have been. (See illustra-
(Patton 1997a). While individual style will tion of utilization-focused reporting in Ex-
and should affect what a final report looks hibit 8.13 [p. 512].)
The Executive Summary and Research Abstract
Th< he executive summary is a fiction.
Robert Stake (1998:370)
The fact that qualitative reports tend to be standing (a preference I share, by the way),
relatively lengthy can be a major problem my pragmatic, living-in-the-real-world side
when busy decision makers do not have the leads me to conclude that evaluators must
time (or, more likely, zoill not take the time) to develop the ability to produce an executive
read a lengthy report. Stake's preference for summary of one or two pages that presents
insisting on telling the whole story notwith- the essential findings, conclusions, and rea-
EXHIBIT 8.13 Utilization-Focused Evaluation Reporting
Unfocused Reporting: Focused Reporting:

Lots of side tracks Parts cohere in addressing priority concerns
of primary intended users
/ \
sons for confidence in the summary. The ex- briefing. Legendary are the stories of having
ecutive summary is a dissemination docu- spent a year of one's life gathering data,
ment, a political instrument, and cannot pouring over it, and writing a rigorous and
benor is it meant to bea full and fair rep- conscientious evaluation report, then en-
resentation of the study. An executive sum- countering some "decision" maker (I use the
mary or abstract should be written in plain term here lightly) who says, "Well, now, I
language, be highly focused, and state the know that you put a lot of work into this. I'm
core findings and conclusions. Keep in anxious to hear ali about what you've
mind, when writing the executive summary learned. I've got about 10 minutes before my
or research abstract, that more people are next appointment."
likely to read the summary than any other Should you turn heel and show him your
document you produce. back side? Not if you want your findings to
make a difference. Use that 10 minutes well!
Carpe Diem Briefings Be prepared to make it count. Carpe diem.
As the hymnbook is to the sound of mu-

sic, the executive summary is to the oral
13. The Creativity of Qualitative Inquiry
/ reativity will dominate our time after the concepts of work and fun
Vto-" have been blurred by technology.
Isaac Asimov (1983:42)

I have commented throughout this book cai mind. The criticai thinker studies details
that the human element in qualitative in- and looks beyond appearances to find out
quiry is both its strength and weaknessits what is really happening. Evaluators are
strength in allowing human insight and ex- trained to be rigorous and unyielding in crit-
perience to blossom into new understand- ically thinking about and analyzing pro-
ings and ways of seeing the world, its poten- grams. Indeed, evaluation is built on the
tial weakness in being so heavily dependent foundation of criticai analysis.
on the inquirer's skills, training, intellect, Criticai thinkers, however, tend not to be
discipline, and creativity. Because the re- very creative. The creative mind generates
searcher is the instrument of qualitative in- new possibilities; the criticai mind analyzes
quiry, the quality of the result depends those possibilities looking for inadequacies
heavily on the qualities of that human being. and imperfections. In summarizing research
Nowhere does this ring more true than in on criticai and creative thinking, Barry An-
analysis. Being an empathic interviewer or derson (1980) warned that the centrality of
astute observer does not necessarily make doubt in criticai thinking can lead to a nar-
one an insightful analystor a creative one. row, skeptical focus that hampers the cre-
Creativity seems to be one of those special ative ability to come up with innovative
human qualities that plays an especially im- linkages or new insights.
portant part in qualitative analysis, interpre-
tation, and reporting. Therefore, I close this The criticai attitude and the creative attitude
chapter with some observations on creativ- seem to be poles apart.. . . On the one hand,
ity in qualitative inquiry. there are those who are always telling you
I opened this chapter by commenting on why ideas won't work but who never seem
qualitative inquiry as both science and art, able to come up with alterna ti ves of their own;
especially qualitative analysis. The scientific and, on the other hand, there are those who are
constantly coming up with ideas but seem un-
part demands systematic and disciplined in-
able to tell good from the bad.
tellectual work, rigorous attention to details
within a holistic context, and a criticai per- There are people in whom both attitudes
spective in questioning emergent patterns are developed to a high degree . . . , but even
even while bringing evidence to bear in sup- these people say they assume only one of these
port of them. The artistic part invites explo- attitudes at a time. When new ideas are
ration, metaphorical flourishes, risk taking, needed, they put on their creative caps, and
insightful sense-making, and creative con- when ideas need to be evaluated, they put on
nection-making. While both science and art their criticai caps. (Anderson 1980:66)
involve criticai analysis and creative expres-
sion, science emphasizes criticai faculties Qualitative inquiry draws onboth criticai
more, especially in analysis, while art en- and creative thinkingboth the science and
courages creativity. The criticai thinker as- art of analysis. But the technical, procedural,
sumes a stance of doubt and skepticism; and scientific side of analysis is easier to
things have to be proven; faulty logic, slip- present and teach. Creativity, while easy to
pery linkages, tautological theories, and un- prescribe, is harder to teach, and perhaps
supported deductions are targets of the criti- harder to learn, but here's some guidance
derived from research and training on cre- 7. Make linkages. Many creative exercises
ative thinking (Kelley and Littman 2001; De include practice in learning how to con-
Bono 1999; Von Oech 1998; Patton 1987a: nect the seemingly unconnected. Matrix
247-48). approaches presented in this chapter
push linkages. Explore linking qualita-
1. Be open, Creativity begins with openness tive and quantitative data.
to multiple possibilities.
8. Trust yourself. Self-doubt short-circuits
2. Generate options. There's always more creative impulses. If you say to yourself,
than one way to think about or do some- 'Tm not creative," you won't be. Trust
thing. the process.
3. Diverge-converge-integrate. Begin by ex- 9. Work at it. Creativity is not ali fun. It

ploring a variety of directions and possi- takes hard work, background research,
bilities before focusing on the details. and mental preparation.
Branch out, go on mental excursions
and brainstorm multiple perspectives
10. Play at it. Creativity is not ali work. It can
before converging on the most promis-
and should be play and fun.
ing.
4. Use multiple stimuli. Creativity training I close this chapter with a practical re-
often includes exposure to many dif- minder that both the science and art of quali-
ferent avenues of expression: drawing, tative analysis are constrained by limited
music, role-playing, story-boarding, meta- time. Some people thrive under intense time
phors, improvisation, playing with toys, pressure and their creativity blossoms. Oth-
and constructing futuristic scenarios. ers don't. The way in which any particular
Synthesizing through triangulation (see analyst combines criticai and creative think-
Chapter 9) promotes creative integra- ing becomes partly a matter of style, partly a
tion of multiple stimuli. function of the situation, and often is de-
pendentonhow much time canbe found to
5. Side-track, zigzag, and circumnavigate.
Creativity is seldom a result of purely
linear and logical induction or deduc- play with creative possibilities. But explor-
tion. The creative person explores back mg possibilities can also become an excuse
and forth, round and about, in and out, for not finishing. There comes a time for
over and under. bringing closure to analysis (or a book chap-
ter) and getting on with other things. Taking
6. Change patterns. Habits, standard oper- too much time to contemplate creative pos-
ating procedures, and patterned think- sibilities may involve certain risks, a point
ing pose barriers to creativity. Become made by the following story (to which you
aware of and change your patterned can apply both your criticai and creative fac-
ways of thinking and behaving. ulties).
X K e . T-âst arvd t k e P u l u l e :
i D e c i d m c j \n W K i c K !D\ve.c,Y\ov\ t o l _ o o I <
A spirit appeared to a man walking along a narrow road. "You may know with
certainty what has happened in the past, or you may know with certainty what
will happen in the future, but you cannot know both. Which do you choose?"
The startled man sat down in the middle of the road to contemplate his choices.
"If I know with certainty what will happen in the future," he reasoned to himself,
"then the future will soon enough become the past and I will also know with cer-
tainty what has happened in the past. On the other hand, it is said that the past is
prologue to the future, so if I know with certainty what has happened in the past I
will know much about what will happen in the future without losing the elements
of surprise and spontaneity."
Deeply lost to the present in the reverie of his calculations about the past and fu-
ture he was unaware of the sound of a truck approaching at great speed. Just as he
came out of his trance to tell the spirit that he had chosen to know with certainty
the future, he looked up and saw the truck bearing down on him, unable to stop its
present momentum.
From Halcolm's Evaluation Parables
APPENDIX 8.1
s m.
Excerpts From a Codebook for Use by Multiple Coders
Characteristics of Program Evaluated

0101 nature or kind of program
0102 program relationship to government hierarchy
0103 funding (source, amount, determination of, etc.)
0104 purpose of program
0105 history of program (duration, changes, termination, etc.)
0106 program effectiveness
Evaluator's Role in Specific Study
0201 evaluator's role in initiation and planning stage
0203 evaluator's role in data collection stage
0204 evaluator's role in final report and dissemination
0205 relationship of evaluator to program (internai/externai)
0206 evaluator's organization (type, size, staff, etc.)
0207 opinions/feelings about role in specific study
0208 evaluator's background
0209 cornments on evaluator, evaluator process
Decision Maker's Role in Specific Study
0301 decision maker's role in initiation and planning stage
0302 decision maker's role in data-collection stage
0303 decision maker's role in final report and dissemination
0304 relationship of decision maker to program
0305 relationship of decision maker to other people or units in government
0306 cornments on decision maker and decision-making process
(opinions, feelings, facts, knowledge, etc.)
Stakeholder Interactions
0501 stakeholder characteristics
0502 interactions during or about initiation of study
0503 interactions during or about design of study
0504 interactions during or about data collection
0505 interactions during or about final report/findings
0506 interactions during or about dissemination
Planning and Initiation Process of This Study (how and who started)
0601 initiator
0602 interested groups or individuais
0603 circumstances surrounding initiation
Purpose of Study (why)
0701 description of purpose
0702 changes in purpose
Political Context
0801 description of political context
0802 effects on study
Expectations for Utilization
0901 description of expectations
0902 holders of expectations
0903 effect of expectations on study
0904 relationship of expectations to specific decisions
0905 reasons for lack of expectations
0906 people mentioned as not having expectations
0907 effect of lack of expectations on study
Data Collection, Analysis, Methodology
1001 methodological quality
1002 methodological appropriateness
1003 factors affecting data collection and methodology
Findings, Final Report
1101 description of findings/recommendations
1102 reception of findings/recommendations
1103 comments on final report (forms, problems, quality)
1104 comments and description of dissemination
Impact of Specific Study
1201 description of impacts on program
1202 description of nonprogram impacts
1203 impact of specific recommendations
Factors and Effects on Utilization
1301 lateness
1302 methodological quality
1303 methodological appropriateness
1304 positive/nega tive findings
1305 surprise findings
1306 central/peripheral objectives
1307 point in life of program
1308 presence/absence of other studies
1309 political factors
1310 interaction with evaluators
1311 resources
1312 most important factor
NOTE: This codebook was for use by multiple coders of interviews with decision makers and
evaluators about their utilization of evaluation research.
APPENDIX 8.2
m. s
Mike: An ILIustrative Case Study
Background: Sitting in a classroom at Metro City High School was difficult

for Mike. In some classes he was way behind. In math he was always the first to
finish a test. "I loved math and could always finish a test in about ten minutes,
but I wasn't doing well in my other classes," Mike explained.
He first heard about Experience-Based Career Education (EBCE) when he
was a sophomore. "I really only went to the assembly to get out of one of the
classes I didn't like," Mike confessed.
But after listening to the EBCE explanation, Mike was quickly sold on the
idea. He not only liked the notion of learning on the job, but also thought the
program might allow him to work at his own speed. The notion of no grades
and no teachers also appealed to him.
Mike took some descriptive materiais home to his parents and they joined
him for an evening session at the EBCE learning center to find out more about
the program. Now, after two years in the program, Mike is a snior and his par-
ents want his younger brother to get into the program.
Early EBCE testing sessions last year verified the inconsistency of Mike's ex-
periences in school. While his reading and language scores were well below the
average scored by a randomly selected group of juniors at his school, he
showed above average abilities in study skills and demonstra ted superior abil-
ity in math.
On a less tangible levei, EBCE staff members early last school year described
Mike as being hyperactive, submissive, lacking in self-confidence and uncon-
cemed about his health and physical appearance when he started the EBCE
program. He was also judged to have severe writing deficiencies. Conse-
quently, Mike's EBCE learning manager devised a learning plan that would
build his communication skills (in both writing and interpersonal relations)
while encouraging him to explore several career possibilities. Mike's job expe-
riences and projects were designed to capitalize on his existing interests and to
broaden them.
First-year EBCE experiences. A typical day for Mike started at 8:00 a.m., just
as m any other high school, but the hours in between varied considerably.
When he first arrived at the EBCE learning center, Mike said he usually spent
some time "fooling around" with the computer before he worked on projects
underway at the center.
On his original application, Mike indicated his career preference would be
computer operator. This led to an opportunity in the EBCE program to further
explore that area and to learn more about the job. During April and May, Mike's
second learning levei experience took place in the computer department of City
Bank Services. He broke up his time there each day into morning and afternoon
blocks, often arriving before his employer instructor did for the morning pe-
riod. Mike usually spent that time going through computer workbooks. When
his employer instructor arrived they went over flow charts together and
worked on computer language.
Mike returned to the high school for lunch and a German class he selected as
a project. EBCE students seldom take classes at the high school but Mike had a
special interest in German since his grandparents speak the language.
Following German class, Mike returned to the learning center for an hour of
work on other learning activities and then went to City Bank. "I often stayed
there until 5:00 p.m.," Mike said, even thoughhigh school hours ended atthree.
Mike's activities and interests widened after that first year in the EBCE pro-
gram but his goal of becommg a computer programmer was reinforced by the
learning experience at City Bank. The start of a new hobbycollection of com-
puter materiaisalso occurred during the time he spent at City Bank. "My em-
ployer instructor gave me some books to read that actually started the collec-
tion," Mike said.
Mike's interests in animais also was enhanced by his EBCE experience. Mike
has always liked animais and his family has owned a horse since he was 12
years old. By picking blueberries Mike was able to save enough to buy his own
colt two years ago. One of Mike's favorite projects during the year related to his
horse. The project was designed to help Mike with Basic Skills and to improve
his criticai thinking skills. Mike read about breeds of horses and how to traia
them. He then joined a 4-H group with hopes of training his horse for show.
Several months later, Mike again focused on animais for another EBCE pro-
ject. This time he used the local zoo as a resource, interviewing the zoo manager
and doing a thorough study of the Alaskan brown bear. Mike also joined an Ex-
plorer Scouting Club of volunteers to help at the zoo on a regular basis. "I really
like working with the bears," Mike reflected. "They were really playful. Did
you know when they rub their hair against the bars it sounds like a violin?"
Evaluation of the zoo project, one of the last Mike completed during the year,
showed much improvement. The learning manager commented to Mike, "You
are getting your projects done faster, and I think you are taking more time than
you did at first to do a better job."
Mike got off to a slow start in the area of Life Skills development. Like some
of his peers, he went through a period described by one of the learning manag-
ers as "freedom shock" when removed from the more rigid structure norinally
experienced in a typical school setting. Mike tended to avoid his responsibility
to the more "academic" side of his learning program. At first, Mike seldom fol-
lowed up on commitments and often did not let the staff know what he was do-
ing. By the end of the year, he had improved remarkably in both of these behav-
ior areas.
Through the weekly writing required in maintaining his journal, Mike dem-
onstrated a significant improvement in written communications, both in terms
of presenting ideas and feelings and in the mechanics of writing. Mike also
544lJ.ANALYSIS, INTERPRETATION, AND REPORTING
noted an interesting change in his behavior. "I used to watch a lot of TV and
never did any reading." At the beginning of the following year, Mike said: "I
read two books last year and have completed eightmore this summer. Now I go
to the book instead of the television" Mike's favorite reading materiais are sci-
ence fiction.
Mike also observed a difference in his attitude about homework. "After go-
ing to school for six hours I wouldn't sit down and do homework. But in the
EBCE program I wasn't sitting m a classroom, so I didn't mind going home with
some more work on my journal or projects."
Mike's personal development was also undergoing change. Much of this
change was attributed to one of his employer instructors, an elementary school
teacher, who told him how important it is in the work world to wash and wear
clean clothes. Both she and the project staff gave Mike much positive reinforce-
ment when his dress improved. That same employer also told Mike that she
was really interested in what he had to say and therefore wanted him to speak
slower so he could be understood.
Mike's school attendance improved while in the EBCE program. During the
year, Mike mis se d only six days. This was better than the average absence for
others in the program, which was found to be 12.3 days missed during the year,
and much improved over his high school attendance.
Like a number of other EBCE students in his class, Mike went out on explora-
tion levei experiences but completed relatively few other program require-
ments during the first three months of the school year. By April, however, he
was simultaneously working on eight different projects and pursuing a learn-
ing experience at City Bank. By the time Mike completed his jnior year he had
finished nine of the required thirteen competencies, explored nine business
sites, completed two learning leveis and carried through on eleven projects.
Two other projects were dropped during the year and one is uncompleted but
could be finished in the commg year.
On a more specific levei, Mike's competencies included transacting business
on a credit basis, maintaining a checking account, designing a comprehensive
insurance program, filing taxes, budgeting, developing physical fitness, learn-
ing to cope with emergency situations, studying public agencies and operating
an automobile.
Mike did not achieve the same levei of success on ali of his job sites. How-
ever, his performance consistently improved throughout the year. Mike criti-
cized the explora tion packages when he started them in the first months of the
program and, although he couldn't pinpoint how, said they could be better. His
own reliance on the questions provided in the package was noted by the EBCE
staff with a comment that he rarely followed up on any cues provided by the
person he interviewed. The packets reflected Mike's disinterest in the explora-
tion portion of EBCE work. They showed little effort and a certain sameness of
remarks about his impressions at the various sites.
Mike explored career possibilities at an automobile dealer, an audiovisual

repair shop, a supermarket, an air control manufacturer, an elementary school,
a housing development corporation, a city public works, a jnior high school
and a bank services company.
Mike's first learning levei experience was at the elementary school. At the
end of three and one-half months the two teachers serving as his employer in-
structors indicated concern about attendance, punctuality, initiative in learn-
ing and amount of supervision needed to see that Mike's time was used con-
structively. Mike did show significant improvement in appropriate dress,
personal grooming and quality of work on assignments.
Reports from the second learning levei experienceat the computer depart-
ment of the bank services companyshowed a marked improvement. The em-
ployer instructor there rated Mike satisfactory in ali aspects and by the time of
the final evaluation gave excellentratings in ten categoriesattendance/punc-
tuality, adhering to time schedules, understanding and accepting responsibil-
ity, observing employer rules, showing interest and enthusiasm, poise and
self-confidence, using initiative in seeking opportunities to learn, using em-
ployer site learning resources, beginning assigned tasks promptly and com-
pleting tasks assigned.
During the latter part of the school year, Mike worked on several projects at
once. He worked on a project on basic electricity and took a course on "Begin-
ning Guitar" for project credit.
To improve his communication skills Mike also worked on an intergroup re-
lations project. This project grew out of an awareness by the staff that Mike
liked other students but seemed to lack social interaction with his peers and the
staff. Reports at the beginning of the year indicated that he appeared depend-
ent and submissive and was an immature conversationalist. In response to
these observations, Mike's learning manager negotiated project objectives and
activities with him that would help improve his communication skills and help
him solve some of his interpersonal problems. At the end of the year Mike
noted a positive change related to his communication skills. "\ can now speak
up in groups," he said.
Mike's unfinished project related to his own experience and interests. He
had moved to the Portland area from Canada ten years previously and fre-
quently returns to see relatives. The project was on immigration laws and regu-
lations in the functional citizenship area. At the same time, it will help Mike im-
prove his grammar and spelling. Since students have the option of completing
a project started during their jnior year when they are a snior, Mike had a
chance to finish the project this year. Of the year Mike said, "It turned out even
better than I thought." Things he liked best about the new experience in EBCE
were working at his own speed, going to a job and having more freedom.
At the end of the year, Mike's tests showed significant increases in both read-
ing and language skills. In the math and study skills areas where he was already
above average, only slight increases were indicated.
Tests on attitudes, given both at the beginning and the end of the year, indi-
cated positive gains in self-reliance, understanding of roles in society, tolerance
for people with differences in background and ideas than his, and openness to
change.
Aspira tions did not change for Mike. He still wants to go into computer pro-
gramming after finishing college. "When I started the year I really didn't know
too much about coinputers. I feel now that I know a lot and want even more to
make it my career."
(The description ofMike's secondyear in EBCE is omitted. We pickup the case study
after the second-year description.)
Mike's views of EBCE. Mike reported that his EBCE experiences, especially
the learning leveis, had improved ali of his basic skills. He felt he had the free-
dom to do the kinds of things he wanted to do while at employer sites. These ex-
periences, according to Mike, have strengthened his vocational choice in the
field he wanted to enter and have caused him to look at educational and train-
ing requirements plus some other alternatives. For instance, Mike tried to enter
the military, figuring it would be a good source of training in the field of com-
puters, but was unable to because of a medicai problem.
By going directly to job sites Mike has gotten a feel for the "real world" of
work. He said his work at computer repair-oriented sites furthered his concep-
tions of the patience necessary when dealing with customers and fine degree of
precision needed in the repair of equipment. He also discovered how a cus-
tomer engineer takes a problem, evaluates it and solves it.
When asked about his work values Mike replied, "I figure if I get the right
job, I'd work at it and try to do my b e s t . . . in fact, I'm sure that even though I
didn't like the job I'd still do more than I was asked to I'd work as hard as I
could." Although he has always been a responsible person, he feels that his ex-
periences in EBCE have made him more trustworthy. Mike also feels that he is
now treated more like an adult because of his own attitudes. In fact, he feels he
understands hiinself a lot more now.
Mike's future plans concern trying to get a job in computer programming at
an automobile dealership or computer services company. He had previously
done some computer work at the automobile dealership in relation to a project
in Explorer Scouts. He also wants more training in computer programming and
has discussed these plans with the student coordinator and an EBCE secretary.
His attitude towards learning is that it may not be fun, but it is important, im-
portant to his future.
When asked in which areas he made less growth than he had hoped to, Mike
responded, "I really made a lot of growth in ali areas." He credits the EBCE pro-
gram for this, finding itmore helpful thanhigh school. It gives you the opportu-
nity to "get out and meet more people and get to be able to communicate better
with people out in the community."
Most of Mike's experiences at the high school were not too personally re-
warding. He did start a geometry class there this year, but had to drop it as he
had started late and could not catch up. Although he got along ali right with the
staff at the high school, in the past he felt the teachers there had a "barrier be-
tween them and the students." The EBCE staff "treat you on a more individual
type circumstance . . . have the time to talk to you." In EBCE you can "work at
your own speed . . . don't have to be in the classroom."
Mike recommends the program to most of his friends, although some of his
friends had already dropped out of school. He stated, "I would have paid to
come into EBCE, I think it's really that good of a program. . . . In fact, I've
learned more in these two years in EBCE than I have m the last four years at the
high school." He did not even ask for reimbursement for travei expenses be-
cause he said he liked the program so much.
The viezvs ofhis parents. When Mike first told his parents about the program
they were concerned about what was going to be involved and whether it was a
good program and educational. When interviewed in March, they felt that
EBCE had helped Mike to be more mature and know where he is going.
Mike's parents said they were well-informed by the EBCE staff in ali areas.
Mike tended to talk to them about his activities in EBCE, while the only thing he
ever talked about at the high school was photography. Mike's career plans have
not really changed since he entered EBCE and his parents have not tried to in-
fluence him, but EBCE has helped him to rule out mechanic and truck driving
as possible careers.
Since beginning the EBCE program his parents have found Mike to be more
mature, dependable and enthusiastic. He also became more reflective and con-
cerned about the future. His writing improved and he read more.
There are no areas where his parents felt that EBCE did not help him and
they rated the EBCE program highly in ali areas.
Test progress measures on Mike. Although Mike showed a great improvement
in almost ali areas of the Comprehensive Test of Basic Skills during the first year
of participation, his scores decline considerably during the second year. Espe-
cially significant were the declines in Mike's arithmetic applications and study
skills scores.
Mike's attitudinal scores ali showed a positive gain over the two-year total
period, but also tended to decline during the second year of participation. On
the semantic differential, Mike scored significantly below the EBCE mean at
FY 75 posttest on the community resources, adults, learning and work scales.
Mike showed continued growth over the two-year period on the work,
self-reliance, communication, role, and trust scales of the Psychosocial Matu-
rity Scale. He was significantly above the EBCE posttest means on the work,
role, and social commitment scales and below average on only the openness to
change scale. The openness to change score also showed a significant decline
over the year.
The staff rated Mike on seven student behaviors. At the beginning of the year
he was significantly above the EBCE mean on "apples knowledge of his/her
own aptitudes, interests, and abilities to potential career interests" and below
the mean on "understands another person's message and feelings." At posttest
time he was still below the EBCE mean on this latter behavior as well as on
"demonstrates willingness to apply Basic Skills to work tasks and to vocational

interests."
Over the course of the two years in the EBCE program Mike's scores on the
Self-Directed Search (SDS) showed little change in pattern, although the num-
ber of interests and competencies did expand. Overall, realistic (R) occupations
decreased and enterprising (E) occupations increased as his code changed from
RCI (where C is conventional and I is investigative occupations) at pretestFY 74
to ICR at pretest FY 75 (a classification which includes computer operators and
equipment repairers) to CEI at posttest FY 75. However, the I was only one
point stronger than the R and the CER classification includes data processing
workers. Thus, Mike's SDS codes appeared very representa tive of his desired
occupational future.
Evaluators' reflections. Mike's dramatic declines in attitudes and basic skill
scores reflect behavior changes which occurred during the second half of his
second year of the program and were detected by a number of people. In Febru-
ary at a student staffing meeting his learning manager reported of Mike that
"no progress is seen in this zone with projects... still elusive... coasting right
now . . . may end up in trouble." The prescription was to "watch himmake
him produce . . . find out where he is." However, at the end of the next to last
zone in mid-May the report was still "the elusive butterfly! (Mike) needs to get
himself in high gear to get everything completed on time!!!" Since the
posttesting was completed before this time, Mike probably coasted through the
posttesting as well.
Other data suggesting his lack of concern and involvement during the sec-
ond half of his snior year was attendance. Although he missed only two days
the first half of the year, he missed thirteen days during the second half.
Mike showed a definite change in some of his personality characteristics
over the two years he spent in the EBCE program. In the beginning of the pro-
gram he was totally lacking in social skills and self-confidence. By the time he
graduated, he had made great strides in his social skills (although there was
still much room for improvement). However, his self-confidence had grown to
the point of overconfidence. Indeed the employer instructor on his last learning
levei spent a good deal of time trying to get Mike to make a realistic appraisal of
his own capabilities.
When interviewed after graduation, Mike was working six evenings a week
at a restaurant where he worked part-time for the last year. He hopes to work
there for about a year, working his way up to cook, and then go to a business
college for a year to study computers.
SOURCE: Fehrenbacher, Owens, and Haehnn (1976). Used by permission of Northwest Regional
Educational Laboratory.
APPENDIX 8.3
Excerpts From an Illustrative Interview Analysis:

Reflections on Outcomes From Participants
in a Wilderness Education Program
Experiences affect people in different ways. This experiential education

truism means that the individual outcomes, impacts, and changes that result
from participation in some set of activities are seldom predictable with any
certainty. Moreover, the meaning and meaningfulness of such changes as do
occur are likely to be highly specific to particular people in particular circum-
stances. While the individualized nature of learning is a fundamental tenet of
experiential education, it is still important to stand back from those individ-
ual experiences in order to look at the patterns of change that cut across the
specifics of person and circumstances. One of the purposes of the evaluation
of the Learninghouse Southwest Field Training Project was to do just thatto
document the experiences of individuais and then to look for the patterns that
help provide an overview of the project and its impacts.
A major method for accomplishing this kind of reflective evaluation was
the conduct of follow-up interviews with the 11 project participants. The first
interviews were conducted at the end of October 1977, three weeks following
the first field conference in the Gila wilderness of New Mxico. The second in-
terviews were conducted during the third week of February, three weeks af-
ter the wilderness experience in the Kofa Mountains of Arizona. The third
and final interviews were conducted in early May following the San Juan
River conference in southern Utah. Ali interviews were conducted by tele-
phone. The average interview took 20 minutes with a range from 15 to 35 min-
utes. Interviews were tape-recorded and transcribed for analysis.
The interviews focus on three central issues: (1) How has your participa-
tion in the Learninghouse Project affected you personally? (2) How has your
participation in the project affected you professionally? (3) How has your
participation in the Learninghouse Project affected your institution?
In the pages that foliow, participant responses to these questions are pre-
sented and analyzed. The major purpose of the analysis was to organize par-
ticipant responses in such a way that overall patterns would become clear.
The emphasis throughout is on letting participants speak for themselves. The
challenge for the evaluators was to present participant responses in a cogent
fashion that integrates the great variety of experiences and impacts recorded
during the interviews.
!3. 525
FCDEF
Personal Change
"How has your participation in the Learninghouse Project affected you per-
sonally? What has been the impact of the project on you as a person?"
Questions about personal change generated more reactions from partici-
pants than subsequent questions about professional and institutional change.
There is an intensity to these responses about individual change that makes it
clear just how significant these experiences were in stimulating personal
growth and development. Participants attempted throughout the interviews to
indica te that they felt differently about themselves as persons because of their
Learninghouse experiences. While such personal changes are often difficult to
articulate, the interviews reflect a variety of personal impacts.
Confidence: A Sense of Self
During the three weeks in the wilderness, participants encountered a num-

ber of opportunities to test themselves. Can I carry a full pack day after day, up-
hill and downhill? Can I make it up that mountain? Do I have anything to con-
tribute to the group? As participants encountered and managed stress, they
learned things about themselves. The result was often an increase in personal
confidence and a greater sense of self.
It's really hard to say that LH did one thing or another. I think increased self-confi-
dence has helped me do some things that I was thinking about doing. And I think
that came, self-confidence came about largely because of the field experiences. I,
right after we got back, I had my annual merit evaluation meeting with my boss,
and at that I requested that I get a, have a change in title or a different title, and an-
other title really is what it amounts to, and that I be given the chance for some other
responsibilities that are outside the area that I work in. I want to get some individ-
ual counseling experience, and up to this point I have been kind of hesitant to ask
for that, but I feel like I have a better sense of what I need to do for myself and that I
have a right to ask for it at least. (Cliff, post-Kofas)
I guess something that has been important to me in the last couple of trips and will
be important in the next one is just the outdoor peace of it. Doing things that per-
haps I'd not been willing to attempt before for whatever reason. And finding I'm
better at it than expected. Before I was afraid. (Charlene, post-Kofas)
The interviews indicate that increased confidence came not only from physi-
cal accomplishments but alsoand especiallyfrom interpersonal accom-
plishments.
After the Kofas I achieved several things that I've been working on for two years.
Basically, the central struggle of the last two years of my life has been to no longer
try to please people. No matter what my own feelings and needs are I try to please
you. And in the past I had done whatever another person wanted me to do in spite
of my own feelings and needs. And to have arrived at a point where I could tend to
my own feelings and take care of what I needed to do for me is by far the most im-
portant victory I've won . . . a major one.
In the Kofas, I amazed my self that I didn't more than temporarily buy into h o w . . . I
was being described.. .when Ididn't recognize myself yet. And that's new for me.
In the past I'd accept others' criticisms of me as if they were indeed describing me
. . . and get sucked into that. And I felt that was an achievement for me to hold onto
my sense of myself in the face of criticisms has long been one of my monsters I've
been struggling with, so to hold onto me is, especially as I did, was definitely an
achievement. (Billie, post-Kofas)
I've been paying a lot of attention to not looking for valida tion from other people.
Just sticking with whatever kinds of feelings I have and not trying to go outside of
m y s e l f . . . and lay myself on a platter for approval. I think the project did have a lot
to do with that, especially this second trip in the Kofas. (Greg, post-Kofas)
I would say the most important thing that happened to me was being able to talk to
other people quite honestly about, I think really about their problems more than
mine. That's very interesting in that I think that I had, I think I had an effect upon
Billie and Charlene both. As a result of that it gave me a lot more confidence and
positive feelings. Do you follow that? Where rather than saying I had this problem
and I talked to somebody and they solved it for me, it was more my helping other
people to feel good about themselves that made me feel more adequate and better
about myself. (Rod, post-Gila)
Another element of confidence concerns the extent to which one believes in

one's own ideasa kind of intellectual confidence.
I think if I take the whole project mto consideration, I think that I've gained a lot of
confidence myself in some of the ideas that I have tried to use, both personally and
let's say professionally. Especially m my teaching aspects, especially teaching at a
woman's college where I think one of our roles is not only to teach women subject
matter, but also to teach them to be more assertive. I think that's a greater compo-
nent of our mission than normally would have it at most colleges. I think that a lot
of the ideas that I had about personal growth and about my own interactions with
people were maybe reinforced by the LH experience, so that I felt more confident
about them, and as a result they have come out more in my dealings with people. I
would say specifically in respect to a sort of a more humanistic approach to things.
(Rod, post-Kofas)
Increased confidence for participants was often an outcome of learning that

they could do somethingnew and difficult. At other times, however, increased
confidence emerged as a result of finding new ways to handle old and difficult
situations, for example, learning how to recognize and manage stress.
A change I've noticed most recently and most strongly is the ability to recognize
stress. And also the ability to recognize that I can do a task without needing to
make it stressful, which is something I didn't know I did. So what I find I wind up
doing, for example, is when I've had a number of things happen during the day
and I begin to feel myself keying up I find myself very willing to say both to close
friends and to people I don't know very well, I can't deal with this that you're
brmging me. Can we talk about it tomorrow? This is an issue that really needs a lot
of time and a lot of attention. I don't want to deal with it today, can we talk later,...
etc. So I'm finding myself really able to do that. And I'm absolutely delighted about
it.
(Whereas before you just piled it on?)
Exactly. I'd pile it and pile it until I wouldn't understand why I was going in cir-
cles. (Charlene, post-Kofas)
Personal ChangeOverview
The personal outcomes cited by Learninghouse participants are ali difficult
to measure. What we have in the interviews are personal perceptions about
personal change. The evidence, in total, indicates that participants felt differ-
ently and, in many cases, behaved differently as a result of their project par-
ticipation. Different participants were affected in different ways and to vary-
ing extents. One participant reported virtually no personal effects from the
experiences.
And as far as the effect it had on me personally, which was the original question,
okay, to be honest with you, to a large degree it had very little effect, and that's not a
dig on the program, because at some point in people's lives 1 think things start to
have smaller effect, but they still have effect. So I think that for me, what it did have
an effect on was tolerance. Because there were a lot of things that occurred on the
trip that I didn't agree with. And still don't agree, but I don't find myself to be vi-
ciously in disagreement any longer, just plainly in disagreement. So it was kind of
like before, I didn't want to listen to the disagreement, or I wanted to listen to it but
resolve it. Now, you know, there's a third option, that I can listen to it, continue to
disagree with it, and not mind continuing to listen to it. (Cory, post-San Juan)
The more common reaction, however, was surprse at just how much per-
sonal change occurred.
My expected outcome was increase the number of contacts in the Southwest, and
every one of my expected outcomes were professional. That, you know, much
more talk about potential innovations in education and directions to go, and you
know, field-based education, what that's about, and I didn't expect at ali, which
may not be realistic on my part, but at least I didn't expect at alithe personal im-
pact. (Charlene, post-Gila)
For others the year's participation in Learnmghouse was among the most
important learning experiences of a lifetime, precisely because the project em-
braced personal as well as professional growth.
l've been involved in institutions and in projects as an educator, let's say, for 20
years. I started out teaching in high school, going to the NSF institutions during the
summertime and I've gone to a lot of Chautauqua things and a lot of conferences,
you know, of various natures. And I really think that this project has by far the
greatest... has had by far the greatest impact on me. And I think that the reason is
that in ali the projects that l've had in the p a s t . . . they've been ali very spedfically
oriented toward one subject or toward o n e . . . more of a, I guess, more of a science,
more of a subject matter orientation to them. Whereas this having a process orien-
ta tion has a longer effect. I mean a lot of the things I learn in these instances is out of
date by now and you keep up with the literature, for example, and ali that and
maybe that stimulates you to keep up . . . but in reality as far as a growth thing on
my part, 1 think on the part of other participants, I think that this has been phenom-
enal. And 1 just think that this is the kind of thing that we should be looking to-
wards funding on any levei, federal, or any levei. (Rod, post-San Juan)
We come now to a transition point in this report. Having reported partici-

pants' perceptions about personal change, we want to report the professional
outcomes of the Learninghouse Project. The probiem is that in the context of
a holistic experience like the Southwest Field Training Project, the personal-
professional distinction becomes arbitrary. A major theme running throughout
discussions during the conferences was the importance of reducing the per-
sonal-professional schism, the desirability of living an integrated life and being
an integrated self. This theme is reflected in the interviews, as many partici-
pants had difficulty responding separately to questions about personal versus
professional change.
Personal/Professional Change
Analytically, there is at least a connotative difference between personal and

professional change. For evaluation purposes, we tried to distinguish one from
the other as follows: personal changes concern the thoughts, feelings, behav-
iors, intentions, and knowledge people have about themselves; professional
changes concern the skills, competences, ideas, techniques, and processes peo-
ple use in their work. There is, however, a middle ground. How does one cate-
gorize changes in thoughts, feelings, and intentions about competences, skills,
and processes? There are changes in the person that affect that person's work.
This section is a tribute to the complexity of human beings in defying the neat
categories of social scientists and evaluators. This section reports changes that,
for lack of a better nomenclature, we have called simply personal/professional
impacts.
The most central and most common impact in this regard concerned changes
in personal perspective that affected fundamental notions about and ap-
proaches to the world of work. The wilderness experiences and accompanying
group processes permitted and/or forced many participants to stand back and
take a look at themselves in relation to their work. The result was a changed
perspective. The following four quotations are from interviews conducted after
the first field conference in the Gila, a time when the contrasts provided by the
first wilderness experience seemed to be felt most intensely.
The trip carne at a real opportune time. I've been on this new job about 4-5 weeks
and was really getting pretty thoroughly mired in it, kind of overwhelmed by it,
and so it came after a particularly hellish week, so in that sense it was just a criticai,
really help fui time to get a way. To feel that I had, to remember that I had some
choices, both in terms of whether I stayed here or went elsewhere, get some per-
spective of what it was I actually wanted to accomplish in higher education rather
than just survivmg to keep my sanity. And it gave me some, it rene wed some of my
ability to think of doing what I wanted to do here at the University, or trying to, that
there were things that were important for me to do rather than just handling the
stuff that poured across my desk. (Heniy, post-Gila)
I think it's helped make me become more creative, and just, and thafs kind of tied
in with the whole idea of the theory of experiential education. And the way we ap-
proached it on these trips. And so for instance I'm talking with my wife the other
night, after I gotLaura's paper that she'd given in Colorado, and I said you oughta
read this because you can go out and teach history and you know, experientially.
Then I gave her an idea of how I would teach frontier history for instance, and I
don't know beans about frontier history. But it was an idea which, then she told an-
other friend about it, and this friend says oh, you can get a grant for that. You know.
So that was just a real vivid example, and I feel like, it's, I've been able to apply, or
be creative in a number of different situations, I think just because I give myself a
certain freedom, I don't know, I can't quite pinpoint what brought it about, but I
just feel more creative in my work. (Cliff, post-San Juan)
You know my biggest problem is I've been trying to save the world, and what I'm
doing is pulling back. Because, perhaps the way I've been going about it has been
wrong or whatever, but at least my motives are clearer and I know much more di-
rectly what I need and what I don't need and so I'm more open but less, yeah, as I
said, I've been in a l e f s save the world kind of thing, now I feel more realistic and
honest. (Charlene, post-Gila)
I've been thinking about myself and my relationship to men and my boss, and es-
pecially to ideas about fear and risk . . . I decided that I needed to become a little
more vsible at the department. After the October experience, I just said I was a bit
more ready to become visible at the department levei. And I volunteered then to
work on developing a department training policy and develop the plan and went
down to the department and talked to the assistant about it and put myself in a con-
sulting role while another person was assigned the actual job of doing it. And I
think that I was ready to make that decision and act on it after I first of ali got clear
that I was working on inale-female relationships. My department has a man, again,
not a terribly easy one to know, so it's a risk for me to go talk with him and yet I did
it. I was relatively comfortable and felt very good and very pleased with myself
that I had done that and I think that's also connected. (Billie, post-Kofas)
The connection between personal changes andprofessional activities was an

important theme throughout the Learninghouse Project. The passages re-
por ted in this section illustrate how that connection took hold in the minds and
lives of project participants. As we turn now to more explicit professional im-
pacts, it is helpful to keep in mind the somewhat artificial and arbitrary nature
of the personal-professional distinction.
(Omitted are sections on changed professional knowledge about experien-
tial education, use of journals, group facilitation skills, individual professional
skills, personal insights regarding work and professional life, and the specific
projects participants undertookprofessionally. Also omitted are sections on in-
stitutional impacts. We pick up the report in the concluding section.)
Final Reflections
Personal change... professional change... institutional change Evalua-

tion categories aim at making sense out of an enormously complex reality. The
reflections by participants throughout the interviews make it clear that most of
them came away from the Learninghouse program feeling changes in them-
selves. Something had touched them. Sometimes it meant a change in perspec-
tive that would show up in completely unexpected ways.
For one thing, I just finished the purchase of my house. First of ali, thafs a new ex-
perience for me. I've never done it before. I've never owned a home and never even
wanted to. It seemed odd to me that my desire to "settle down" or make this type of
commitment to a place occurred just right after the Gila trip. Just sort of one of
those things that I woke up and went, "Wow, I want to stay here. I like this place. I
want to buy it." And I had never in my life lived in a house or a place that I felt that
way about. I thought that was kind of stTange. And I do see that as a function of per-
sonal growth and stability. At least some kind of stability.
Other areas of personal growth: one has been, and this kind of crosses over I
think into the professional areas, and that would be an ability to gain perspective.
Certainly the trips I t h i n k . . . incredibly valuable for gaining perspective on w h a f s
happening in my home situation, my personal life, my professional life . . . the
whole thing. And it has allowed me to focus on some priority types of things for
me. And deal with some issues that I've been kind of dragging on for years and
years and not really wanting to face up with them or deal with them. And I have
been able to move on and move through those kinds of things in the last 6 or 9
months or so to a much greater extent than ever before. (Tom, post-San Juan)
Other participants came a way from the wilderness experiences with a more
concrete orientation that they could apply to work, play, and life.
The thing that I realized as I was trying to make some connections between the
river and raft trip, was that in some ways I can see the parallels of my life being
kind of like our raft trip was, and the rapids, or the thrill ride, and they're a lot of
fun, but it's nice to get out of them for a while and dry off. It's nice sometimes to be
able to just drift along and not worry about things. But a lot of it also is just hard
work. A lot of times I wish I could get out of it and go a different way, and that's
been kind of a nice thing for me to think about and kind of a viewpoint to have
whenever I see things in a lull or in a real high speed pace, that I can say, "Okay, I'm
going to be in this for a while, but I'm going to come out of it and go into something
else." And so that's kind of a metaphor that I use as somewhat of a philosophy or
point of view thafs helpful as I go from day to day. (Cliff, post-San Juan)
A common theme that emerged as participants reflected on their year's in-

volvement with Learninghouse was a new awareness of options, alterna ti ves,
and possibilities.
I would say that if I have one overall comment, the effect of the first week overall, is
to renew my sense of the broader possibilities in my job and in my life. Opens
things to me. I realize that I have a choice to be here and be myself. And since I have
a choice, there are responsibilities. Which is a good feeling. (Henry, post-Gila)
I guess to me what sticks out overall is that the experience was an opportunity for
me to step out of the rest of my life and focus on it and evaluate it, both my personal
life and my work, professional life aspect. (Michael, post-San Juan)
As participants stood back and examined themselves and their work they
seemed to discover a clarity that had previously been missing. Perspective,
awareness, clarity . . . stuff of which personal/professional/insti tu tional
change is made.
I think I had a real opportunity to explore some issues of my own worth with a
group of people who were willing to allow me to explore those. And it may have
come Iater, but it happened then. On the Learninghouse, through the Learning-
house . . . and I think it speeded up the process of growing for me in that way, ac-
cepting my own worth, my own ideas about education, about what I was doing,
and in terms of being a teacher it really aided my discussions of people and my in-
Qualitative Analysis and Interpreta tion!J,557
teractions. It really gave me a lot of focus on what I was doing. I think I would've
muddled around a long time with some issues that I was able to, I think, gain some
clarity on pretty quickly by talking to people who were sharing their experience
and were working towards the same goals, self-directed learning, and experiential
education. (Greg, post-San Juan)
I think what happened is that for me it served as a catalyst for some personal
changes, you know, the personal, institutional, they're ali wound up, bound up to-
gether. I think I was really wrestling with jobs and career and so on. For me the
whole project was a catalyst, a kind of permission to look at things that I hadn't
looked at before. One of the realizations, one of the insights that I had in the process
was, kind of neat on my part, to become concrete, specific in my actions in my life,
no matter whether that was writing that I was doing, or if it was in my job, or what-
ever it was. But to really pay attention to that. I think thafs one of the things that
happened to me. (Peter, post-San Juan)
These statements from interviews do not represent a final assessment of the

impacts of the Learninghouse Southwest Field Training Project. Several partici-
pants resisted the request to make summary statements about the effects and
outcomes of their participation in the program because they didn't want to
force premature closure.
(Can you summarize the overall significance of participation in the project?)

I do want to make a summary, and I don't again It feels like the words aren't
easy and forme being very much a words person, thafs unusual. IFsnotnecessar-
ily that the impact hasn't been in the cognitive areas. There have been some. But
what they've been, where the impact has been absolutely overwhelming is in the
affective areas. Appreciation of other people, appreciation of this kind of educa-
tion. Though I work in it, I haven 't done it before! A real valuing of people, the profes-
sion, of my colleagues in a sense that I never had b e f o r e . . . .
The impact feels like it's been dramatic, and I'm not sure that I can say exactly
how. I'm my w h o l e . . . it ali can be summarized perhaps by saying Fm much more
in control. In a good kind of sense. In accepting risk and being willing to take it; ac-
cepting challenge and being willing to push myself on that; accepting and under-
standing more about working at the edge of my capabilities... what that means to
me. Recognizing very comfortably what I can do and feeling good about that confi-
dence, and recognizing that what I haven't yet done, and feeling okay about trying
it. The whole perception of confidence has changed. (Charlene, post-San Juan)
The Learninghouse program was many thingsthe wilderness, a model of

experiential education, stress, professional developmentbut most of ali, the
project was the people who participated. In response after response partici-
pants talked about the importance of the people to everything that happened.
Because of the dominance of that motif throughout the interviews, we want to
end this report with that highly personal emphasis.
I said before I think that to know some people, that meant a lot to me, people who
were also caring. And people who were also involved, very involved in some is-
sues, philosophical and educational, that were pretty basic not only to education,
but to living. Knowing these people has been really important to me. I f s given me
a kind of continuity and something to hold onto in the midst of a really frustrating,
really difficult situation where I didn't have people where I could get much feed-
back from, or that I could share much thinking about, talking about, and working
with. I f s just kind of basic issues. That kind of continuity is real important to just
my feelings, important to myself. Feeling like I have someplace to g o . . . . Some-
times I feel funny about placing so much emphasis on the people But the people
have really meant a lot to me as far as putting things together for myself. Being able
to have my hands in something that might, that really offers me a way to go. (Greg,
post-San luan)
SOURCE: By Jeanne Campbell and Michael Patton.

Riddles of Qualitative Inquiry
Who Am I?
Gary D. Shank
life and creation. This is ali well and good,
L ately, I have been thinking about rid-

dles. Riddles are one of those things
that we used for millennia to build
inquiry around and then conveniently mis-
but why can't we reclaim the riddle as well?
Each of the following four riddles seeks to
highlight and illuminate some overlooked
laid or trivialized. Riddles were once pow- or covert or murky aspect of a qualitative
erful and heady things. Now we have rid- research skill.1 Since most riddles are in
dles that are nothing but child's word play. verse, I decided to preserve the formfor
Word play was certainly important in rid- these riddles 1 used Petrarchian sonnet
dles, but they were anything but simply structure. (Note: As a reminder of the im-
child's fare. perfect patterns found in the real world, the
We have discarded the riddle in favor of last line of Riddle Four violates the sonnet
the puzzle. Scientists and other empirical rules; instead of abbaabba cdecde it is
inquirers "puzzle" over the meaning of abbaabba cdecdc.)
their data and seek to solve the "puzzles" of
!3. 537
538 tI. ANALYSIS, INTERPRETATION, AND REPORTING
The question is: Can you solve the rid- Of ordinary folk, like some strange creed
dles? Who seek out yet another staged applause.
Riddle Number One What do you say, that I have never said?
What brave new world can you make me
believe?
When I have fears that I have found a place
Are you this calm, or are you filled with
Where I have never chanced to be before
spite?
And where the odds are great, that never-
These ragged thoughts take root, and then
more
my head
Will I again be out there, face to face;
Seeks any path of rest. You may relieve
How then should I begin to set the chase?
My fright, or plunge me deeper in the
When wonder's great and familiarity poor
night.
How then should my tired eyes keep up
the score
Who am I?
When ali things strange are ordinary
grace?
Riddle Number Three
Where is my ear, when eyes run fast
ahead? Suppose your home looks like a subway
What do my fingertips alone reveal? station
What is the pulse and pace of this strange
Where geeks and pimps roll out their
land?
tattered wares
And by whose claim are things mundane
And teenage mothers linger on the stairs,
instead,
Framed once more in hollow
Like some dried tangerine stripped of its consternation.
peel,
Refugees who know both love and
An hourglass sucked dry of ali its sand?
Haitian
Size up easy marks, doled out in pairs
Who am I?
You feel like turning circles into squares
Two moves away from last year's
Riddle Number Two conflagration.
Your hands rest lightly on your chin, How could there be no peace in Paradise?
because Where children and their parents ali excel?
You cannot always find the words you With levees standing high above the flood.
need. How can you rage, if everything is nice?
Life races past our thoughts, both trapped Down here inside the Nineteenth hole
and freed ofHell
Of solid form, like sheets of film and gauze Where school kids lie in puddles of their
Whose shifting shapes cause us to halt and blood?
pause.
We find ourselves belonging to a breed Who am I?
Riddles of Qualitative Inquiri/ S 539
Riddle Number Four But they seek me as much as they are

sought,
I see the rats somewhere inside the cheese. They bind my hands and make me walk
Cheddar, or Brie, or Swiss with ali its the plank
holes?
Rats burrowmg inside, like long-tailed And night is broken down without a shot.
moles
Or ghostly galleons tossed on stormy seas? Who am I?
How do these metaphors lock up and seize
My brain, like glaciers marching from the Answers are at the end of
Poles Chapter 9, page 598.
Or fiery furnaces with red-hot coals
That simultaneously burn and freeze?
Sa Note
Things are themselves, as much as they are
not 1. Riddles composed by Gary D. Shank, au-
I want to put my hand upon their flank thor of Qualitative Research: A Personal Skills Ap-
And with a mighty yank to reel them in. proach (2002). Used by permission.
Enhancing the Quality and
Credibility of Qualitative Analysis
I?nterpretik\c| X r u t k
A young man traveling through a new country heard that a great Mulla, a Sufi
guru with unequaled insight into the mysteries of the world, was also traveling
in that region. The young man was deterinined to become his disciple. He found
his way to the wise man and said, "I wish to place my education in your hands
that I might learn to interpret what I see as I travei through the world."
After six months of traveling from village to village with the great teacher, the
young man was confused and disheartened. He decided to reveal his frustration
to the Mulla.
"For six months I have observed the services you provide to the people along
our route. In one village you tell the hungry that they must work harder in their
fields. In another village you tell the hungry to give up their preoccupation with
food. In yet another village you tell the people to pray for a richer harvest. In
each village the problem is the same, but always your message is different. I can
find no pattern of Truth in your teachings."
The Mulla looked piercingly at the young man.
"Truth? When you came here you did not tell me you wanted to learn Truth.
Truth is like the Buddha. When met on the road it should be killed. If there were
S 541
only one Truth to be applied to ali villages there would be no need of Mullahs to
travei from village to village.
"When you first came to me you said you wanted to 'learn how to interpret'
what you see as you travei through the world. Your confusion is simple. To inter-
pret and to state Truths are two quite different things."
Having finished his story Halcolm smiled at the attentive youths. "Go, my chil-
dren. Seek what you will, do what you must."
From Halcolm/s Evaluation Parables
tL Alternative Criteria for Judging Quality
^ ^ very way of seeing is also a way of not seeing.
David Silverman (2000:825)
It ali depends on criteria. Judging qual- that particular philosophical underpinnings

ity requires criteria. Credibility flows from or theoretical orientations and special pur-
those judgments. Quality and credibility poses for qualitative inquiry will generate
are connected in that judgments of quality different criteria for judging quality and
constitute the foundation for perceptions credibility.
of credibility. In broad terms, I have identified five con-
Diverse approaches to qualitative inquiry trasting sets of criteria for judging the qual-
phenomenology, ethnomethodology, eth- ity of qualitative inquiry from different per-
nography, hermeneutics, symbolic interac- spectives and within different philosophical
tion, heuristics, criticai theory, realism, frameworks. Some of the criteria within
grounded theory, and feminist inquiry, to these frameworks overlap, but even then
name but a fewremind us that issues of subtle differences in nuances of meaning can
quality and credibility intersect with audi- be distinguished. The five contrasting, and
ence and intended inquiry purposes. Re- to some extent competing, sets of criteria
search directed to an audience of indepen- flow from the following:
dent feminist scholars, for example, may be
judged by somewhat different criteria from Traditional scientific research criteria
research addressed to an audience of gov- Social construction and constructivist
ernment economic policy makers. Forma- criteria
tive research or action inquiry for program
Artistic and evocative criteria
improvement involves different purposes
and therefore different criteria of quality Criticai change criteria
compared with summative evaluation Evaluation standards and principies
aimed at making fundamental continuation
decisions about a program or policy. Thus, it Exhibit 9.1 lists the criteria that flow from
is important to acknowledge at the outset each of these perspectives or frameworks.
Enhancing Quality and Credibility lJ. 543
The traditional scientific research criteria are tative inquiry. In this chapter, we are primar-
embedded in and derived from what I dis- ily concerned with how others respond to
cussed in Chapter 3 in the Truth and Reality- our work. With what perspectives and by
Oriented Correspondence Theory section what criteria will our work be judged by
that included postpositivist and realist ap- those who encounter and engage it?
proaches to qualitative inquiry. The social Some of the confusion that people have in
construction and constructivist criteria high- assessing qualitative research stems from
light elements of the detailed discussion of thinking it represents a uniform perspective,
those perspectives in the section by that especially in contrast to quantitative re-
name in Chapter 3. The artistic and evocative search. This makes it hard for them to make
criteria are derived from the Autoethnog- sense of the competing approaches within
raphy and Evocative Forms of Inquiry sec- qualitative inquiry. By understanding the
tion in Chapter 3, especially the criteria sug- criteria that others bring to bear on our
gested by Richardson (2000b) for "creative work, we can anticipate their reactions and
analytic practice ethnography." The four th help them position our intentions and crite-
set of criteria, criticai change criteria, flow ria in relation to their own expectations and
from criticai theory, feminist inquiry, activist criteria. In terms of the reflexive triangu-
research, and participatory research pro- lated inquiry model presented in Chapter 2
cesses aimed at empowerment; these were as Exhibit 2.2, we're dealing here with the in-
discussed in Chapter 3 as Orientational tersection between the inquirer's perspec-
Qualitative Inquiry (done from a particular tive and those receiving the study (the audi-
values-based perspective) and in Chapter 4 ences).
as participatory and collaborative strategies. Different perspectives about such things
The final set of criteria, evaluation standards as truth and the nature of reality constitute
and principies, are from The Standards for Pro- paradigms or worldviews based on
gram Evaluation (Joint Committee 1994) and alternatve epistemologies and ontologies.
"Guiding Principies for Evaluators" (AEA People viewing qualitative findings through
Task Force 1995); they provide the founda- different paradigmatic lenses will react dif-
tion for the extended discussion of qualita- ferently just as we, as researchers and evalu-
tive evaluation applications in Chapter 4. ators, vary in how we think about what we
To some extent, ali of the theoretical, do when we study the world. These differ-
philosophical, and applied orientations re- ences are nicely illustrated by the classic
viewed in Chapters 3 and 4 provide some- story of three baseball umpires who, having
what distinct criteria, or at least priorities retired after a game to a local establishment
and emphases, for what constitutes a quality for the dispensing of reality-distorting but
contribution within those particular per- truth- enhancing libations, are discussing
spectives and concerns. I've chosen these how they call balls and strikes.
five broader sets of criteria to correspond
roughly with major stages in the develop- "I call them as I see them," says the first.
ment of qualitative research (Denzin and
Lincoln 2000b), to capture the primary de- "I call them as they are," says the second.
bates that differentiate qualitative ap-
proaches and, more specifically, to highlight "They ain't nothing until I call them," says
what seem to differentiate reactions to quali- the third.
Alternative Sets of Criteria for Judging the

EXHIBIT 9.1 Quality and Credibility of Qualitative Inquiry
Traditional Scientific Research Criteria

Objectivity of the inquirer (attempts to minimize bias)
Vaiidity of the data
Systematic rigor of fieldwork procedures
Triangulation (consistency of findings across methods and data sources)
Reliability of codings and pattern analyses
Correspondence of findings to reality
Generalizability (externai vaiidity)
Strength of evidence supporting causai hypotheses
Contributions to theory
Social Construction and Constructivist Criteria

Subjectivity acknowledged (discusses and takes into account biases)
Trustworthiness
Authenticity
Triangulation (capturing and respecting multiple perspectives)
Reflexivity
Praxis
Particularity (doing justice to the fntegrity of unique cases)
Enhanced and deepened understanding (Verstehen)
Contributions to dialogue
Artistic and Evocative Criteria

Opens the world to us in some way
Creativity
Aesthetic quality
Interpretive vtalty
Flows from self; embedded in lived experience
As an exercise in distinguishing paradigms,

Traditional Scientific
try matching the three umpires' perspec-
Research Criteria
tives to the frameworks in Exhibit 9.1. (Hint:
Ali four of the other perspectives can be
found within evaluation, so treating the um- One way to increase the credibility and le-
pires as evaluators reduces your matching gitimacy of qualitative inquiry among those
options to the remaining four.) The short sec- who place priority on traditional scientific
tions that follow elaborate the five alterna- research criteria is to emphasize those crite-
tive sets of criteria for judging the quality of ria that have priority within that tradition.
qualitative work. Science has traditionally emphasized objec-
Stimulating
Provocative
Connects with and moves the audience
Voice distinct, expressive
fee/5 "true" or "authentic" or "real"
Criticai Change Criteria

Criticai perspective: Increases consciousness about injustices
Identifies nature and sources of inequalities and injustices
Represents the perspective of the less powerful
Makes visible the ways in which those with more power exercise
and beneflt from power
Engages those with less power respectfully and collaboratively
Bulds the capacity of those involved to take action
Identifies potential change-making strategies
Praxis
Clear historical and values context
Consequential validity
Evaluation Standards and Principies

Utility
Feasibility
Propriety
Accuracy (balance)
Systematic inquiry
Evaluator competence
Integrity/honesty
Respect for people (fairness)
Responsibility to the general public welfare
(taking into account diversity of interests and values]
tivity, so qualitative inquiry within this tra- of pattern and theme analysis. Qualitative
dition emphasizes procedures for minimiz- researchers working in this tradition are
ing investigator bias. Those working within comfortable using the language of "vari-
this tradition will emphasize rigorous and ables" and "hypothesis testing" and striving
systematic data collection procedures, for for causai explana tions and generalizability,
example, cross-checking and cross-validat- especially in combina tion with quantitative
ing sources during fieldwork. In analysis it data. Qualitative approaches that manifest
means, whenever possible, using multiple some or ali of these characteristics include
coders and calculating intercoder consis- grounded theory (Glaser 2000: 200), qual-
tency to establish the validity and reliability itative comparative analysis (Ragin 1987,
nai validity, transferability as an analog to

: vffriNd-ciirrttiA externai validity, dependability as an analog
. jF0H.-J3^ftMWlHE ITiUTIH ; to reliability, and confirmability as an analog
' t t ; .!:dl!Ji!! j'!.;:'i 'Ji.!j fCfl .tl Clf T-VL ll to objectivity." In combination, they viewed
/ jiJhV!:-;^ ^^ TMIS'! : Y;;S! i, .t? fyjMS these criteria as addressing "trustworthi-
. /fl/ikii-iTrt HAN:* :!;.=::!:::M.:; IMCI rs-ate ness (itself a parallel to the term rigor)"
Mfel^^/IM^LLTLH;--l!rl!!;=!!!=:riDFL.YIS: il*=- ' ! (pp. 76-77). They went on to emphasize that
!.lato/RK:'; ^ f u r S;.!.?. ' w in U ivng w.'s naturalistic inquiry should be judged by de-
;/ il/li:: Mil fk.i^MmtYl^ VhiS i'!'!,::i pendability (a systematic process systemati-
h i ; YRIIFECFI': ' K G A - : I V J I N K - H - ^ V : . D - L H I TI
cally followed) and authenticity (reflexive
| ;4;i;!:!ii;' pfe&iwzs ztpititti ii:ii i in" .Vf tr consciousness about one's own perspective,
f. -[I|: I!;=35:i: 'i:i i'.!::ftC1foi1'ate- appreciation for the perspectives of others,
!' Ff EVgKi mtn ipy.ftj i*.1;-!:!!' ^;;::!: ^; and fairness in depicting constructions in
; trijE4-:* \Hni r/!.!.'-:. rs.?.;?. J.3 =!.i:. the values that undergird them). They view
the social world (as opposed to the physical
world) as socially, politically, and psycho-
logically constructed, as are human under-
2000), realists such as Miles and Huberman standings and explanations of the physical
(1994), and some aspects of analytic induc- world. They triangulate to capture and re-
tion (see Chapter 8). Their common aim is to port multiple perspectives rather than seek a
use qualitative methods to describe and ex- singular truth. Constructivists embrace sub-
piam phenomena as accurately and com- jectivity as a pathway deeper into under-
pletely as possible so that their descriptions standing the human dimensions of the
and explanations correspond as closely as world in general as well as whatever specific
possible to the way the world is and actually phenomena they are examining (Peshkin
operates. Government agencies supporting 1985, 1988, 2000a). They're more interested
qualitative research (e.g., the U.S. General in deeply understanding specific cases
Accounting Office, the National Science within a particular context than in hypothe-
Foundation, and the National Institutes of sizing about generalizations and causes
Health) usually operate within this tradi- across time and space. Indeed, they are sus-
tional scientific framework. picious of causai explanations and empirical
generalizations applied to complex human
interactions and cultural systems. They offer
Social Construction and
perspective and encourage dialogue among
Constructivist Criteria
perspectives rather than aiming at singular
Social construction, constructivist, and truth and linear prediction. Social con-
"interpretivist" perspectives have gener- structivists' case studies, findings, and re-
ated new language and concepts to distin- ports are explicitly informed by attention to
guish quality in qualitative research (e.g., praxis and reflexivity, that is, understanding
Glesne 1999:5-6). Lincoln and Guba (1986) how one's own experiences and background
proposed that constructivist inquiry de- affect what one understands and how one
manded different criteria from those inher- acts in the world, including acts of inquiry.
ited from traditional social science. They Guba and Lincoln (1989,1990), Lincoln and
suggested "credibility as an analog to inter- Guba (1986), Smith (1991), Denzin (1997a,
Marry you?
There you go, trying to construct reality again.
A realist views a constructivist proposal.
2001), Neimeyer (1993), and Potter (1996) Artistic and Evocative Criteria
have articulated and work within the tradi-
tion of social constructionism and con- In the last chapter, I discussed qualitative
structivism. (See Chapter 3 for a much analysis as both science and art. Researchers
lengthier discussion of constructionism and and audiences operating from the perspec-
constructivism.) tive of traditional scientific research criteria
emphasize the scientific nature of qualita- expressive voice, experience, and perspec-
tive inquiry. Researchers and audiences that tive may become as central to the work as
view the world through the lens of social depictions of others or the phenomenon of
construction emphasize qualitative inquiry interest. Qualitative inquiry illustrative of
as both science and art, and mix the two mo- this emergent approach includes the works
tifs. That brings us to this third alternative, of Bochner and Ellis (2001), Goodall (2000),
which emphasizes the artistic and evocative Richardson (2000b), Barone (2000), Ellis and
aspects of qualitative inquiry, or what is Bochner (1996, 2000), Glesne (1997), Patton
sometimes called "the narrative turn" in so- (1999a), and Denzin (2000b).
cial science (Bochner 2001). Keep in mind
that these are matters of emphasis drawn Criticai Change Criteria
here to highlight contras ts, and not mutually
exclusive or pur types. Artistic criteria fo- Those engaged in qualitative inquiry as a
cus on aesthetics, creativity, interpretive vi- form of criticai analysis aimed at social and
tality, and expressive voice. Case studies be- political change eschew any pretense of
come literary works. Poetry or performance open-mindedness or objectivity; they take
art may be used to enhance the audience's an activist stance. For example, consequen-
direct experience of the essence that emerges tial validity as a criterion for judging a re-
from analysis. Artistically oriented qualita- search design or instrument makes the so-
tive analysts seek to engage those receiving cial consequences of its use a value basis for
the work, to connect with them, move them, assessing its credibility and utility. Thus,
provoke and stimulate. Creative nonfiction standardized achievement tests are criti-
and fictional forms of representation blur cized because of the discriminatory conse-
the boundaries between what is "real" and quences for minority groups of educational
what has been created to represent the es- decisions made with "culturally biased"
sence of a reality, at least as it is perceived, tests. Consequential validity asks for assess-
without a literal presentation of that per- ments of who benefits and who is harmed by
ceived reality. The results may be called cre- an inquiry, measurement, or method
ative syntheses, ideal-typical case construc- (Messick 1989; Shepard 1993; Brandon,
tions, scientific poetics, or any number of Lindberg, and Wang 1993). As an example of
phrases that suggest the artistic emphasis. the criticai change orientation, criticai theory
(See Exhibit 3.3 in Chapter 3, Varieties of approaches fieldwork and analysis with an
Autoethnography: A Partial Lexicology.) explicit agenda of elucidating power, eco-
Artistic expressions of qualitative analysis nomic, and social inequalities. The "criticai"
strive to provide an experience with the nature of criticai theory flows from a com-
findings where "truth" or "reality" is under- mitment to go beyond just studying society
stood to have a feeling dimension that is every for the sake of increased understanding.
bit as important as the cognitive dimension. Criticai theorists set out to use research to
The performance art of The Vagina Mono- critique society, raise consciousness, and
logues (Ensler 2001), based on interviews change the balance of power in favor of
with women but presented as theater, offers those less powerful. Iniluenced by Marxism,
a prominent example. The audience feels as informed by the presumption of the central-
much as knows the truth of the presentation ity of class conflict in understanding com-
because of the essence it reveals. In the artis- munity and societal structures, and updated
tic tradition, the analysfs interpretive and in the radical struggles of the 1960s, criticai
theory provides both philosophy and meth- Evaluation Association (AEA Task Force
ods for approaching research and evalua- 1995) added the following principies: sys-
tion as fundamental and explicit manifesta- tematic inquiry, evaluator competence, in-
tions of political praxis (connecting theory tegrity/honesty, respect for people (fair-
and action) and as change-oriented forms of ness), and responsibility to the general
engagement. public welfare (taking into account diversity
Likewise, feminist inquiry often includes of interests and values). The complete and
an explicit agenda of bringing about social specific standards and principies are avail-
change (e.g., Benmayor 1991). Liberation re- able through the AEA Web site (see Exhibit
search and empowerment evaluation derive, in 4.9 in Chapter 4).
part, from Paulo Freire's philosophy of In the 1970s, as evaluation was just
praxis and liberation education articulated emerging as a field of professional practice,
in his classics Pedagogy ofthe Oppressed (1970) many evaluators took the position of tradi-
and Education for Criticai Consciousness tional researchers that their responsibility
(1973), still sources of influence and debate was merely to design studies, collect data,
(e.g., Glass 2001). Barone (2000:247) aspires and publish findings; what decision makers
to "emancipatory educational storyshar- did with those findings was not their prob-
ing." Qualitative studies informed by criti- lem. This stance removed from the evaluator
cai change criteria range from largely intel- any responsibility for fostering use and
lectual and research-oriented approaches placed ali the "blame" for nonuse or
that aim to expose injustices to more activist underutilization on decision makers. More-
forms of inquiry that actually engage in over, before the field of evaluation identified
bringing about social change. This category and adopted its own standards, criteria for
can include collaborative and participatory ap- judging evaluations could scarcely be differ-
proaches to fieldwork that are conducted in entiated from criteria for judging research in
ways that build the capacity of those in- the traditional social and behavioral sci-
volved to better understand their own situa- ences, namely, technical quality and meth-
tions, raise consciousness, and support fu- odological rigor. Utility was largely ignored.
ture action aimed at political change. Methods decisions dominated the evalua-
Examples of a range of criticai change ap- tion design process. Validity, reliability,
proaches to qualitative inquiry can be found measurability, and generalizability were the
in work on feminist methods (Reinharz dimensions that received the greatest atten-
1992; Harding 1991; Fonow and Cook 1991; tion in judging evaluation research propos-
Gluck and Patai 1991), criticai theory (Fonte als and reports. Indeed, evaluators con-
2001; Lather 1986; Comstock 1982), and criti- cerned about increasing a study's usefulness
cai ethnography (Thomas 1993; Simon and often called for ever more methodologically
Dippo 1986). rigorous evaluations to increase the validity
of findings, thereby supposedly compelling
decision makers to take findings seriously.
Evaluation Standards
By the late 1970s, however, program staff
and Principies
and funders were becoming openly skepti-
The evaluation profession has adopted cal about spending scarce funds on evalua-
standards that call for evaluations to be use- tions they couldn't understand and/or
ful, practical, ethical, and accurate (Joint found irrelevant. Evaluators were being
Committee 1994). In 1995, the American asked to be "accountable" just as program
staff were supposed to be accountable. The Unlike the traditionally aloof stance of basic
questions emerged with uncomfortable di- researchers, evaluators are challenged to
rectness: Who will evaluate the evaluators? take responsibility for use. Implementation
How will evaluation be evaluated? It was in of a utility-focused, feasibility-conscious,
this context that professional evaluators be- propriety-oriented, and accuracy-based
gan discussing standards. evaluation requires situational responsive-
The most comprehensive effort at devei- ness, methodological flexibility, multiple
oping standards was hammered out over evaluator roles, political sophistication, and
five years by a 17-member committee ap- substantial doses of creativity (Patton
pointed by 12 professional organizations 1997a).
with input from hundreds of practicing While the standards and principies offer a
evaluation professionals. Just prior to publi- generic set of criteria for judging the quality
cation, Dan Stufflebeam (1980), chair of the of evaluations, many different models and
committee, summarized the results as fol- viewpoints coexist under this broad um-
lows: brella (Stufflebeam, Madeus, and Kellaghan
2000; Greene 2000; Patton 1997a; Worthen,
The standards that will be published essen- Sanders, and Fitzpatrick 1996). Indeed, one
tially call for evaluations that have four fea- can find in evaluation examples of evalua-
tures. These are utility, feasibily, propriety and tors applying any of the four sets of criteria
accuracy. And I think it is interesting that the already reviewed. The traditional scientific
Joint Committee decided on that particular or- research criteria are the basis for evaluation
der. Their rationale is that an evaluation research as represented by Rossi, Freeman,
should not be done at ali if there is no prospect and Lipsey (1999) and Huey-Tsyh Chen and
for its being useful to some audience. Second, Peter H. Rossi (1987). Constructivist criteria
it should not be done if it is not feasible to con- applied to evaluation provide the founda-
duct it in political terms, or practicality terms, tion for Four th Generation Evaluation (Guba
or cost effectiveness terms. Third, they do not and Lincoln 1989) and sensitivity to multiple
think it should be done if we cannot demon- stakeholder perspectives (Greene 1998a,
s trate that it will be conducted fairly and ethi- 1998b, 2000). The artistic and evocative crite-
cally. Finally, if we can demonstrate that an ria inform "connoisseurship evaluation"
evaluation will have utility, will be feasible (Eisner 1985, 1991). Criticai change criteria
and will be proper in its conduct, then they undergird empowerment evaluation (Fet-
said we could turn to the difficult matters of terman 2000a), diversity-inclusive evalua-
the technical adequacy of the evaluation. tion (Mertens 1998), and aspects of delib-
(p- 90) erative democratic evaluation that involve
values-based advocating for democracy
In 1994, revised standards were pub- (House and Howe 2000). Spanning this di-
lished following an extensive review span- versity and variety of practice is a general
ning several years (Joint Committee 1994). understanding that those who use evalua-
While some changes were made in the 30 in- tions apply both "truth tests" (Are the find-
dividual standards, the overarching frame- ings accurate and valid?) and "utility tests"
work of four primary criteria remained un- (Are the findings relevant and useful?)
changed: utility, feasibility, propriety, and (Weiss and Bucuvalas 1980). This involves
accuracy Taking the standards seriously has attending to and balancing legitimate con-
meant looking at the world quite differently. cerns about both technical quality and util-
ity of findings (Greene 1990). Stufflebeam hand, we do not take narratives at face value,
(2001) has prepared the Evaluation Values as complete and accurate representations of
and Criteria Checklist to help evaluators reality. We believe that stories are usually con-
and their clients consider an appropriate structed around a core of facts or life events,
range of generic values and criteria as they yet allow a wide periphery for freedom of in-
identify those that will undergird particular dividuality and creativity in selection, addi-
evaluations. tion to, emphasis on, and interpretation of
these "remembered facts."
Life stories are subjective, as is one's self or
Clouds and Cotton: Mixing identity. They contain "narrative truth" which
and Changing Perspectives may be closely linked, loosely similar, or far
removed from "historical truth." Conse-
The five frameworks just reviewed show quently, our stand is that life stories, when
the range of criteria that can be brought to properly used, may provide researchers with
bear in judging a qualitative study. They can a key to discovering identity and understand-
also be viewed as "angles of vision" or "al- ing itboth in its "real" or "historical" core,
ternative lenses" for expanding the possibil- and as narrative construction. (p. 8)
ities available, not only for critiquing in-
quiry but also for undertaking it (Peshkin The remainder of this chapter will elabo-
2001). While each set of criteria manifests a rate some of the most prominent of these
certain coherence, many researchers mix competing criteria that affect judgments
and match approaches. The work of Tom about the quality and credibility of qualita-
Barone (2000), for example, combines aes- tive inquiry and analysis. Which criteria you
thetic, political (criticai change), and choose to emphasize in your work will de-
constructivist elements. As an evaluator, I pend on the purpose of your inquiry, the val-
have worked with and mixed criteria from ues and perspectives of the audiences for
ali five frameworks to match particular de- your work, and your own philosophical
signs to the needs and interests of specific and methodological orientation. Operating
stakeholders and clients (Patton 1997a). But within any particular framework and using
any particular evaluation study has tended any specific set of criteria will invite criti-
to be dominated by one set of criteria with a cism from those who judge your work from
second set as possibly secondary. a different framework and with different cri-
Mixing and combining criteria mean teria. (For examples of the vehemence of
dealing with tensions between them. After such criticisms between those using tradi-
reviewing the tensions between traditional tional social science criteria and those using
social science criteria and postmodern artistic narrative criteria, see Bochner 2001
constructivist criteria, narrative researchers or English 2000.) Understanding that criti-
Lieblich, Tuval-Mashiach, and Zilber (1998) cisms (or praise) flow from criteria can help
assert "a middle course," but that middle you anticipate how to position your inquiry
course reveals the very tensions they are try- and make explicit what criteria to apply to
ing to supersede as they work with one leg in your own work as well as what criteria to of-
each camp. fer others given the purpose and orientation
of your work.
We do notadvocate total relativism that treats But it's not always easy to tell whether
ali narratives as texts of fiction. On the other someone is operating from a realist, con-
cHassiNo.^KBS i m glnrlsi
' iJi lîri^ÎIÎ-lri-Jiriî-i-aS^4 t" 0f rniVvi.Tfty^MIT Wir !!"!. .M!i,'r.i;v! i,l ^! VA ! l V

iiiiVn;;! iMI"i'|j.in!.=;!y 'j'iiTI!i:!!i-i!:!; ii'"!1.!- j'.'Vi ri-: /ii:!! n-:!' .:;'::i!i !:=:T: i:! .Vl i":! 1 !
.. :i"i;iia:TjrhjVh!."' :V;!H n!!MnJf&ii!ii iTij J C iiiJ iW.Aru U AilTt-y: .
-
TCT I;i.!:'.I; =: N I T I W I N
: ! ! : R R : I? ,; I\ R R |I .vs ^' V "h,'!!!'".' :M'TV'.' r : . i-î:
M
! ' ii:{ii.!.'i|:li1:'jii{i ,'h jrrV! irVMi 'jj!'' "!! >
! : ' ir!' ""! '!
1 1 'i> i'i" '
I :*! ;:' ;; =: i.- > ".'i iVli' "rn V i !.1'
, i'i"'I!IR!II .!!::'J:' I: : M
!
I
! I I"I!' FJI=;|"{ :-:".!I-::-N-:I5 . - O S
5
S
0 {'"'Si 1::: 1! Si'i i i !
> '
! - j'-
<", "V i;
> (} "
i'ii iVi " !." !; I'i!; I !:":;1, .'fO! !.i=l'!S!!' i.V! 1!!', --
j ;-:!.'MjtfnTIr;VirI'!.!rs <r;,] fnPT.T:f !h.j' 'j;V:'
'i !'".> ;:!j -=|| ; I; j V ' \ II! .I! ^. Y-V;
: twiA rt .'/ M'-V!: ;i;:"! !: ri1 ' \ r i;! ;i : ;
" i iiîKV -ii

]iWi i-M-r! ^k a(Lir
/ .!!I:;!:!!.:I!''. |':! : tfilM.iVi Irj.' l n:! v tM!!: I Y L ^ F F E M
iVlK CC.V IVii., i.ki!i.!""rl-!|iViTViV.' M)!
.tjGH&fCh i"Hs:iI' ttffm jty* c^J.1 ij
!'.!':; ;:! i ii ri trr.' - :ri'i -r !! ru" \ !v. "'J; ' i; J:ij.; " o
L I J - B - A ^ R I ^ L I I I T L L HRTNRT F I R A S ;<!.-IH'S.;! VI!! N I ! : ' 13,"
&XZCVS V..' ':V>ii.r<VV.:!':.'.! 0' !'!!!
.!.; !:':.'','I::J J:I - R R R ^ A . M I r,"n :;:'
RG
:ii!':i ii''' :;.'!inViJ !.".'>!! !=."-' .!\X,ft" rJ
LdtHjit iVitf ! I ' ! . : ; ' ! ir.iJ - V J ^ - , ' V rniprn : !;'/:; iYs:
" A ';='.:' : = I - - . J > : ' VS; JVI } KM I: >>& \ ! i !:-: S' V R
:
i;! i ;! .V f.r/.'!' I' !-i T ,; r!.;' rn h r r I.; IM J; J; }\.
L pwh SI-|=I S; .I, i I IVI I " . >I ] R N I- O R M '
! ' ??
. >: I: th* N : N ' ; J: j'!?!/!:!. / I iii.ri LI!:'!:.!i'fi!!!' "!'I;I.;:;;;!;;. H'M! VV-C;'<I]I.'
1
: thr ItWr r I':i li.! i!ii;ii1.:|:v! |T! 'tXfatfbiV iM r. i' j-.i :'.:!.; k->': ' 'ri! i i th" -r H,;Vj { n " i;! - iT.!.;'.'! r s;
:i!iVii- V-:i:"::' iVi .viwnr?SV O F L V I
.!ii!i:v; iViLTii:i'i.v]-s ,"|i i'' i ! - y \p !::rwi- ir
iiTiii' .1 " .*t!!';!!.!: /p r!ri!vy.:i'i";.' l! -fj/il^J
iWiikJ'.ii.irrilri-r r!n>a|i'i i.;;ri'i:><;:ii'"".!'!: ).!.'i". ! 1
ItuAm. i-i TI! !i::I !:VC.!; " jx .li :;
rC! ii iV iA -iiiV i'!" " r-i-!",;' jr; \\ \ \\
IÎJR ':;-}! !:!^::: 1 ; >'I ;-: I.I '.: RN-1 I: ! ; , R!,;; V K V
-itfi !ri.!i"M i!i!:-i" i|tV! !";:]| rn,!; i'iy. !:i.'h:! njrCiiVi! ,!
structionist, artistic, activist, or evaluative !5. Credibility

framework. Indeed, the criteria can shift
quickly. Consider this example. My
six-year-old son, Brandon, was explaining The credibility of qualitative inquiry de-
to me a geography science project he had pends on three distinct but related inquiry
done for school. He had created an ecologi- elements:
cal display out of egg cartons, ribbons, cot-
ton, bottle caps, and styrofoam beads. rigorous methods for doing fieldwork that
"These are three mountains and these are yield high-quality data that are system-
four valleys," he said, pointing to the egg atically analyzed with attention to issues
cup arrangement. "And is that a cloud?" I of credibility;
asked, pointing to the big hunk of cotton. He the credibility of the researcher, which is de-
looked at me, disgusted, as though I'd said pendent on training, experience, track
just about the dumbest thing he'd ever record, status, and presentation of self;
heard. "That's a piece of cotton, Dad." and
0 philosophical beliefin the value of qualitative counter such a suspicion before it takes root.
inquiry, that is, a fundamental apprecia- One strategy involves discussing one's pre-
tion of naturalistic inquiry, qualitative dispositions, making biases explicit, to the
methods, inductive analysis, purposeful extent possible, and engaging m mental
sampling, and holistic thinking. cleansing processes (e.g., epoche in phe-
nomenological analysis; see Chapter 8). Or
one may simply acknowledge one's orienta-
\3. Rigor: Strategies tion as a feminist researcher or criticai theo-
for Enhancing the rist and move on from there.
Quality of Analysis However one approaches the issue, being
able to report that you engaged in a system-
Chapters 6 and 7 focused on rigorous tech- atic search for alternative themes, divergent
niques for increasing the quality of data col- patterns, and rival explanations enhances
lected during fieldwork (observing, inter- credibility. This can be done both induc-
viewing, and document gathering), while tively and logically. Inductively it involves
Chapter 8 reviewed systematic analysis looking for other ways of organizing the
strategies. However, at the heart of much data that might lead to different findings.
controversy about qualitative findings are Logically it means thinking about other logi-
doubts about the nature of the analysis. Sta- cal possibilities and then seeing if those pos-
tistical analysis follows formulas and rules, sibilities canbe supported by the data. When
while, at the core, qualitative analysis de- considering rival organizing schemes and
pends on the insights and conceptual capa- competing explanations, your mind-set
bilities of the analyst. Qualitative analysis shouldn't be focused on attempting to dis-
depends from the beginning on astute pat- prove the alternatives; rather, you look for
tern recognition, a process epitomized in data that support alternative explanations. Fail-
health research by the scientist working on ure to find strong supporting evidence for
one problem who suddenly notices a pat- alternative ways of presenting the data or
tern related to a quite different problem contrary explanations helps increase confi-
and thus discovers Viagra. As Pasteur dence in the original, principal explana tion
posited, "Chance favors the prepared you generated. Comparing alternative pat-
mind." Here, then, are some techniques that terns will not typically lead to clear-cut "yes
prepare the mind for insight while also en- there is support" versus "no there is not sup-
hancing the credibility of the resulting anal- port" kinds of conclusions. You're searching
ysis. for the best fit. This requires assessing the
weight of evidence and looking for those
Integrity in Analysis: Generating patterns and conclusions that fit the prepon-
and Assessing Rival Conclusions derance of data. Keep track of and report al-
ternative classification systems, themes, and
One barrier to credible qualitative find- explanations that you considered and
ings stems from the suspicion that the ana- "tested" during data analysis. This demon-
lyst has shaped findings according to pre- strates intellectual integrity and lends con-
dispositions and biases. Whether this may siderable credibility to the final set of find-
have happened unconsciously, inadver- ings offered, especially if explanations are
tently, or intentionally (with malice afore- proffered. As Yin (1999a) has observed, anal-
thought) is not the issue. The issue is how to ysis of rival explanations in case studies con-
stitutes a form of rigor in qualitative analysis been identified, our understanding of those
parallel to the rigor of experimental designs patterns and trends is increasedby consider-
aimed at eliminating rival explanations. ing the instances and cases that do not fit
A formal and forced approach to engag- within the pattern. These may be exceptions
ing rival conclusions draws on the legal sys- that prove the rule. They may also broaden
tem^ reliance on opposing perspectives bat- the "rule," change the "rule," or cast doubt
tling it out in the courtroom. The advocacy- on the "rule" altogether. Analytic induction
adversary model suggested by Wolf (1975) (see Chapter 8) makes analysis of negative
developed in response to concerns that eval- cases a centerpiece of its analytical strategy
uators could be biased in their conclusions. for revising and fine tuning hypotheses and
So, to balance biases, two teams engage in conclusions (Denzin 1989c).
the evaluation. The avocacy team gathers In the Southwest Field Training Project
and presents information that supports the involving wilderness education, virtually
proposition that the program is effective; ali participants reported significant "per-
the adversary team gathers information that sonal growth" as a result of their participa-
supports the conclusion that the program tion in the wilderness experiences; however,
ought to be changed or terminated. A varia- the two people who reported "no change"
tion of this strategy would be to arbitrarily provided particularly useful insights into
create advocacy and adversary teams only how the program operated and affected par-
during the analysis stage so that both teams ticipants. These two had crises going on
work with the same set of data but each team back home that limited their capacity to "get
organizes and interprets those data to sup- into" the wilderness experiences. The pro-
port different and opposite conclusions. An- ject staff treated the wilderness experiences
other variation would be for a lone analyst as fairly self-contained, closed-system expe-
to organize data systematically into pro and riences. The two negative cases opened up
con sets to see what each yielded. The weak- thinking about "baggage carried in from the
ness of the advocacy-adversary approach is outside world," "learning-oriented mind-
that it emphasizes contrasts and opposite sets," and a "readiness" factor that subse-
conclusions to the detriment of synthesis quently affected participant selection and
and integra tion. It forces data sets into com- preparation.
bat with each other. Such oversimplification No specific guidelines can tell you how
of complex and multifaceted findings is a and how long to search for negative cases or
primary reason that advocacy-adversary how to find alternative constructs and hy-
evaluation is rarely used (in addition to be- potheses in qualitative data. Your obligation
ing expensive and time-consuming). Still, it is to make an "assiduous search... until no
highlights the importance of engaging in further negative cases are found" (Lincoln
some systematic analysis of alternative and and Guba 1986:77). You then report the basis
rival conclusions. for the conclusions you reach about the sig-
nificance of negative or deviant cases.
Negative cases also provide instructive
Negative Cases opportunities for new learning in formative
evaluations. For example, in a health educa-
Closely related to testing alternative con- tion program for teenage mothers where the
structs is the search for and analysis of nega- large majority of participants complete the
tive cases. Where patterns and trends have program and show knowledge gains, an im-
portant component of the analysis should rection makes the most sense given the clues
include examina tion of reactions from drop- (data) that are available. Such writing adds
outs, even if the sample is small for the drop- credibility by showmg the analyst's authen-
out group. While the small proportion of tic search for what makes most sense rather
dropouts may not be large enough to make a than marshaling ali the data toward a single
difference in a statistical analysis, qualita- conclusion. Indeed, the whole tone of a re-
tively the dropout feedback may provide port feels different when the researcher is
criticai information about a niche group, willing to openly consider possibilities other
specific subculture, and/or clues to pro- than those finally settled on as most reason-
gram improvement. able. Compare the approach of weighing al-
Readers of a qualitative study will make ternatives to the report where ali the data
their own decisions about the plausibility of lead in a single-minded fashion, in a rising
alternate explanations and the reasons why crescendo, toward an overwhelming pre-
deviant cases do not fit within dominant sentation of a single point of view. Perfect
patterns. But I would note that the section of patterns and omniscient explanations are
the report that involves exploration of alter- likely to be greeted skepticallyand for
native explanations and consideration of good reason: The human world is not per-
why certain cases do not fali into the main fectly ordered and human researchers are
pattern can be among the most interesting not omniscient. Humility can do more than
sections of a report to read. When well writ- certamty to enhance credibility. Dealing
ten, this section of a report reads something openly with the complexities and dilemmas
like a detective study in which the analyst posedby negative cases is both intellectually
(detective) looks for clues that lead m differ- honest and politically strategic.
ent directions and tries to sort out which di-
Triangulation
B y combining multiple observers, theories, methods, and data sources,

[researchers] can hope to overcome the intrinsic bias that comes from
single-methods, single-observer, and single-theory studies.
Norman K. Denzin (1989c:307)
Chapter 5 on design discussed the bene- locate yourself at their intersection. The
fits of using multiple data collection tech- notion of triangulatmg also works meta-
niques, a form of triangulation, to study the phorically to call to mind the world's stron-
same setting, issue, or program. You may re- gest geometric shapethe triangle. The
call from that discussion that the term trian- logic of triangulation is based on the premise
gulation is taken from land surveying. that no single method ever adequately
Knowing a single landmark only locates you solves the problem of rival explanations. Be-
somewhere along a line m a direction from cause each method reveals different aspects
the landmark, whereas with two landmarks of empirical reality, multiple methods of
you can take bearings in two directions and data collection and analysis provide more
grist for the research mill. Combinations of sentially the same result. The point is to test
interviewing, observation, and document for such consistency. Different kinds of data
analysis are expected in much fieldwork. may yield somewhat different results be-
Studies that use only one method are more cause different types of inquiry are sensitive
vulnerable to errors linked to that particular to different real-world nuances. Thus, un-
method (e.g., loaded interview questions, derstanding inconsistencies in findings
biased or untrue responses) than studies across different kinds of data can be
that use multiple methods in which different illuminative and important. Finding such
types of data provide cross-data consistency inconsistencies ought not be viewed as
checks. weakening the credibility of results, but
It is in data analysis that the strategy of rather as offering opportunities for deeper
triangulation really pays off, not only in pro- insight into the relationship between in-
viding diverse ways of looking at the same quiry approach and the phenomenon under
phenomenon but in adding to credibility by study. Tll comment briefly on each of these
strengthening confidence in whatever con- types of triangulation.
clusions are drawn. Four kinds of triangula-
tion can contribute to verification and vali-
dation of qualitative analysis:
METHODS TRIANGULATION:
RECONCILING QUALITATIVE AND
1. Methods triangulation: Checking out the
QUANTITATIVE DATA
consistency of findings generated by
different data collection methods Methods triangulation often involves
comparing and integrating data collected
2. Triangulation ofsources: Checking out the through some kind of qualitative methods
consistency of different data sources with data collected through some kind of
within the same method quantitative methods. Such efforts flow
from a pragmatic approach to mixed meth-
3. Analyst triangulation: Using multiple an- ods analysis that assumes potential co?npati-
alysts to review findings bility and seeks to discover the degree and
nature of such compatibility (Tashakkori
4. Theory/perspective triangulation: Using and Teddlie 1998:12). This is seldom
multiple perspectives or theories to in- straightforward because certain kinds of
terpret the data questions lend themselves to qualitative
methods (e.g., developing hypotheses or
By triangulating with multiple data sources, theory in the early stages of an inquiry, un-
observers, methods, and/or theories, re- derstanding particular cases in depth and
searchers can make substantial strides in detail, getting at meanings in context, cap-
overcoming the skepticism that greets sin- turing changes in a dynamic environment),
gular methods, lone analysts, and sin- while other kinds of questions lend them-
gle-perspective interpreta tions. selves to quantitative approaches (e.g., gen-
However, a common misconception eralizing from a sample to a population,
about triangulation involves thinking that testing hypotheses, making systematic com-
the purpose is to demonstra te that different parisons on standardized criteria). Thus, it is
data sources or inquiry approaches yield es- common that quantitative methods and
qualitative methods are used n a comple- responsibilities for different kinds of data.
mentary fashion to answer different ques- The Trend study involved an analysis of
tions that do not easily come together to pro- three social experiments designed to test the
vide a single, well-mtegrated picture of the concept of using direct cash housing allow-
situation. ance payments to help low-income families
Given the varying strengths and weak- obtain decent housing on the open market.
nesses of qualitative versus quantitative ap- The analysis of qualitative data from a par-
proaches, the researcher using different ticipant observation study produced results
methods to investigate the same phenome- that were at variance with those generated
non should not expect that the findings gen- by analysis of quantitative data. The credi-
erated by those different methods will auto- bility of the qualitative data became a central
matically come together to produce some issue in the analysis.
nicely integrated whole. Indeed, the evi-
dence is that one ought to expect initial con- The difficulty lay in conflicting explanations
flicts in findings from qualitative and quan- or accounts, each based largely upon a differ-
titative data and expect those findings to be ent kind of data. The problems we faced in-
received with varying degrees of credibility. volved not only the nature of observational
It is important, then, to consider carefully versus statistical inferences, but two sets of
what each kind of analysis yields and give preferences and biases within the entire re-
different interpretations the chance to arise search t e a m . . . .
and be considered on their merits before fa- Though qualitative/quantitative tension is
voring one result over the other based on not the only problem which may arise in re-
methodological biases. search, I suggest that it is a likely one. Few re-
Shapiro (1973) has described in de tail her searchers are equally comfortable with both
struggle to resolve basic differences between types of data, and the procedures for using the
qualitative data and quantitative data in her two together are not well developed. The ten-
study of Follow Through classrooms; she dency is to relegate one type of analysis or the
eventually concluded that some of the con- other to a secondary role, according to the na-
flicts between the two kinds of data were a ture of the research and the predilections of the
result of measuring different things, al- investigators.... Commonly, however, obser-
though the ways in which different things va tional data are used for "generating hypoth-
were measured were not immediately ap- eses," or "describing process." Quantitative
parent until she worked to sort out the con- data are used to "analyze outcomes," or "ver-
flicting findings. She began with greater ify hypotheses." I feel that this division of la-
trust in the data derived from quantitative bor is rigid and limting. (Trend 1978:352)
methods and ended by believing that the
most useful information came from the The 1980 meeting of the Society of Ap-
qualitative data. plied Anthropology in Denver included a
An article by M. G. Trend (1978) of ABT symposium on the problems encountered
Associates has become required reading for by anthropologists participating in teams in
anyone becoming involved in a team project which both quantitative and qualitative
that will involve collecting and analyzing data were being collected. The problems
both qualitative and quantitative data and they shared were stark evidence that quali-
where different members of the team have tative methods were typically perceived as
Mixing Methods
Qualitative Inquiry Quantitative Analysis
( Last year you

s had 2 home
I runs ali season.
This year you
have 5 in one
month. Whafs
<u the difference?
(O
u
X.
2
X!
c
f}
c
2
IS
3
O
"!
nJ
-E
exploratory and secondary when used in Qualitative and quantitative data can be
conjunction with quantitative/experimen- fruitfully combined to elucidate comple-
tal approaches. When qualitative data sup- mentary aspects of the same phenomenon.
ported quantitative findings, that was icing For example, a community health indica tor
on the cake. When qualitative data con- (e.g., teenage pregnancy rate) can provide a
flicted with quantitative data, the qualita- general and generalizable picture of an is-
tive data have often been dismissed or ig- sue, while case studies of a few pregnant
nored. A strategy of methods triangulation, teenagers can put faces on the numbers and
then, doesn't magically put everyone on the illuminate the stories behind the quantita-
same page. While valuing and endorsing tri- tive data. This becomes even more powerful
angulation, Trend (1978) suggested that "we when the indicator is broken into categories
give different viewpoints the chance to arise, (e.g., those under age 15, those 16 and older)
and postpone the immediate rejection of in- with case studies illustrating the implica-
formation or hypotheses that seem out of tions of and rationale for such categoriza-
joint with the majority viewpoint. Observa- tion.
tionally derived explanations are particu- In essence, triangulation of qualitative
larly vulnerable to dismissal without a fair and quantitative data constitutes a form of
trial" (pp. 352-53). comparative analysis. The question is, What
A Story of Triangulation: Testing Conclusions

E X H I B I T 9.2 With More Fieldwork
Economists Lawrence Katz and Jeffrey ing to say they wanted access to better jobs
Liebman of Harvard and Jeffrey R. Kling of and schools, and what we came to understand
Princeton were trying to interpret data from a was their consuming fear of random crime;
federal housing experiment that involved ran- the need the mothers felt to spend every min-
domly assigning people to a program that ute of their day making sure their children
would help them get out of the siums. The were safe" (Uchitele 2001:4).
evaluation focused on the usual outcomes of By adding qualitative, fied-based inter-
improved school and job performance. How- view data to their study, Kling, Liebman, and
ever, to get beyond the purely statistical data, Katz (2001) came to a new and different un-
they decided to conduct face-to-face inter- derstanding of the program's impacts and
views with residents in an inner-city poor participants' motivations based on interview-
communtyafterstudying the results ofa pre- ing the people directly affected, listening to
liminarysurvey with the people who were par- their perspectives, and including those per-
ticipating in this program. spectives in their analysis.
Professor Liebman commented to a New
York Times reprter: "1 thought they were go-
does each analysis contribute to our under- B comparing observations with inter-
standing? reas of convergence increase views;
confidence in findings. reas of divergence D comparing what people say in public
open windows to better understanding the
with what they say in private;
multifaceted, complex nature of a phenome-
non. Deciding whether results have con- o checking for the consistency of what
verged remains a delicate exercise subject to people say about the same thing over
both disciplined and creative interpretation. time;
Focusing on the degree of convergence rather B comparing the perspectives of people
than forcing a dichotomous choicethat the from different points of view, for exam-
different kinds of data do or not converge ple, in an evaluation, triangulating staff
yields a more balanced overall result. views, client views, funder views, and
views expressed by people outside the
program; and
TRIANGULATION OF
B checking interviews against program
QUALITATIVE DATA SOURCES
documents and other written evidence
that can corrob orate what interview re-
The second type of triangulation involves
spondents report.
triangulating data sources. This means com-
paring and cross-checking the consistency
of information derived at different times Quite different kinds of data can be
and by different means within qualitative brought together in a case study to illumi-
methods. It means nate various aspects of a phenomenon.
Smith and Kleine (1986) triangulated histor- based evaluation (assessing the stated out-
ical analyses, life history interviews, and comes of the program) and a second that
ethnographic participant observations to il- undertook a "goal-free" evaluation in which
luminate the roles of powerful actors in eval- the evaluators assess clients' needs and pro-
uating an innovative educational project. gram outcomes without focusing on stated
As we found with methods triangulation, goals (see Chapter 4). Comparing the results
triangulation of data sources within qualita- of the goals-based team with those of the
tive methods may not lead to a single, totally goal-free team provides a form of analytical
consistent picture. The point is to study and triangulation for determining program ef-
understand when and why these differences fectiveness.
appear. The fact that observational data pro-
duce different results than interview data REVIEW BY INQUIRY
does not mean that either or both kinds of PARTICIPANTS
data are "invalid," although that may be the
case. More likely, it means that different Having those who were studied review
kinds of data have captured different things the findings offers another approach to ana-
and so the analyst attempts to understand lytical triangulation. Researchers and evalu-
the reasons for the differences. Either consis- ators can learn a great deal about the accu-
tency in overall patterns of data from differ- racy, completeness, faimess, and perceived
ent sources or reasonable explanations for validity of their data analysis by having the
differences in data from divergent sources people described in that analysis react to
can contribute significantly to the overall what is described and concluded. To the ex-
credibility of findings. tent that participants in the study are unable
to relate to and confirm the description and
analysis in a qualitative report, questions are
TRIANGULATION WITH raised about the credibility of the findings.
MULTIPLE ANALYSTS Alkin, Daillak, and White (1979), in study-
ing how evaluations were used, presented
A third kind of triangulation is investiga-
each case study to the people in the setting
tor or analyst triangulation, that is, using
described and asked them for both verbal
multiple as opposed to singular observers or
and written reactions. They then included
analysts. Triangulating observers or using
those written reactions in the report.
several interviewers helps reduce the poten-
tial bias that comes from a single person do-
ing ali the data collection and provides Obtaining the reactions of respondents to
means of more directly assessing the consis- your working drafts is time-consuming, but
tency of the data obtained. Triangulating ob- respondents may (1) verify that you have re-
servers provides a check on bias in data col- flected their perspectives; (2) inform you of
lection. A related strategy is triangulating sections that, if published, could be problem-
analyststhat is, having two or more per- atic for either personal or political reasons;
sons independently analyze the same quali- and (3) help you to develop new ideas and m-
tative data and compare their findings. terpretations. (Glesne 1999:152)
In evaluation, an interesting form of team
triangulation has been used. Michael How important can participant feedback
Scriven (1972b) used two separate teams, be, not only to confirm findings but to make
one that conducted a traditional goals- sure the right questions are being asked?
Massachusetts Institute of Technology re- AUDIENCE REVIEW AS

searcher Eric Von Hipple reported that 77% CREDIBILITY TRIANG ULATION
of the innovations in equipment used to
Reflexive triangulation (Exhibit 2.2 in
make semiconductor and printed circuit
Chapter 2) adds the audience's reactions to
boards and 67% of the breakthroughs re-
the triangulation mix. Evaluation consti-
ported in four major types of scientific in-
tutes a particular challenge in establishing
struments came from customers (Gladwell
credibility because the ultimate test of the
1997:47). In phenomenological terms, the
credibility of an evaluation report is the re-
real-world lived experience of customers is
sponse of primary intended users and read-
driving technological innovation among
er s of that report. Their reactions revolve
those able to hear it.
around/ace validity. On the face of it, is the re-
Collaborative and participatory inquiry
por tbelievable? Are the data reasonable? Do
builds in participants' review of findings as
the results connect to how people under-
a matter of course. However, investigative
stand the world? In seriously soliciting us-
inquiry (Douglas 1976) aims at exposing
ers' reactions, the evaluator 's perspective is
what goes on beyond the public eye and is
joined to the perspective of the people who
often antagonistic to those in power, so their
must use the findings. House (1977) sug-
responses would not typically be used to re-
gests that the more "naturalistic" the evalua-
vise conclusions but might be used to at least
tion, the more it relies on its audiences to
offer them an opportunity to provide con-
reach their own conclusions, draw their own
text and an alternative interpretation. Some
generalizations, and make their own inter-
researchers worry that sharing findings
pretations:
with participants for their reactions will un-
dermine the independence of their analysis.
Unless an evaluation provides an explanation
Others view it as an important form of trian-
for a particular audience, and enhances the
gulation. In an Internet listserv discussion of
understanding of that audience by the content
this issue, one researcher reported this expe-
and form of the argument it presents, it is not
rience:
an adequate evaluation for that audience,
even though the facts on which it is based are
I gave both transcripts and a late draft of find- verifiable by other procedures. One indicator
ings to participants in my study. I wondered of the explanatory power of evaluation data is
what they would object to. I had not promised the degree to which the audience is per-
to alter my conclusions based on their feed- suaded. Hence, an evaluation may be "true" in
back, but I had assured them that my aim was the conventional sense but not persuasive to a
being sure not to do them harm. My findings particular audience for whom it does not serve
included some significant criticisms of their as an explanation. In thefullest sense, then, an
efforts that I feared/expected they might ob- evaluation is dependent both on the person who
ject to. Instead, their review brought forth some makes the evaluative statement and on the person
new information about initia tives that had not who receives it. (p. 42, emphasis added)
previously been mentioned. And their pri-
mary objection was to my not giving the credit Understanding the interaction and mutu-
for their successes to a wider group in the com- ality between the evaluator and the people
munity. What I learned was not to make as- who use the evaluation, as well as relation-
sumptions about participants' thinking. ships with participants in the program, is
criticai to understanding the human side of is nicely illustratedby a story about the great
evaluation. This is part of what gives evalua- artist Pablo Picasso.
tionand the evaluatorsituational and Marketing of fakes of his paintings
interpersonal "authenticity" (Lincoln and plagued Picasso. His friends became in-
Guba 1986). Appendix 9.1 at the end of this volved in helping check out the authenticity
chapter provides an experiential account of supposed genuine originais. One friend in
from an evaluator dealing with issues of particular became active in this regard and
credibility while building relationships with brought several paintings to Picasso, ali of
program participants and evaluation users; which he identified as fake. A poor artist
her reflections provide a personal, in-depth who hoped to profit from having obtained a
description of what authenticity is like from Picasso before the great artisfs works had
the perspective of one participant observer. become so valuable sent his painting for in-
spection via the friend. Again Picasso pro-
nounced it a forgery.
EXPERT AUDIT REVIEW
"But I saw you paint this one with my
A final review alternative involves using very own eyes," protested the friend.
experts to assess the quality of analysis or, "I can paint false Picassos as well as any-
where the stakes for externai credibility are one," retorted Picasso.
especially high, performing a metaevalu-
ation or process audit. An externai audit by a THEORY TRIANGULATION
disinterested expert can render judgment
about the quality of data collection and anal- A fourth kind of triangulation involves
ysis. "That part of the audit that examines using different theoretical perspectives to
the process results in a dependablity judg- look at the same data. Chapter 3 presented a
ment, while that part concerned with the number of general theoretical frameworks
product (data and reconstructions) results in derived from divergent intellectual and dis-
a confirmabily judgment" (Lincoln and Guba ciplinary traditions. More concretely, multi-
1986:77, emphasis added). Such an audit ple theoretical perspectives can be brought
would need to be conducted according to to bear on specialized substantive issues.
appropriate criteria. For example, it would For example, one might examine interviews
not be fair to audit an aesthetic and evoca- with therapy clients from different psycho-
tive qualitative presentation by traditional logical perspectives: psychotherapy, Ges-
social science standards, or vice versa. But talt, Adlerian, and behavioral psychology.
within a particular framework, expert re- Observations of a group, community, or or-
views can increase credibility for those who ganization can be examined from a Marxian
are unsure how to distinguish high-quality or Weberian perspective, a conflict or func-
work. That, of course, is the role of the doe- tionalist point of view. The point of theory
toral committee for graduate students and triangulation is to understand how differing
peer reviewers for scholarly journals. Prob- assumptions and premises affect findings
lems arise when peer reviewers apply tradi- and interpreta tions.
tional scientific criteria to constructivist sub- A concrete version of theory triangula-
missions, and vice versa. In such cases, the tion for evaluation would involve examin-
review or audit itself lacks credibility. The rng the data from the perspectives of various
challenge of getting the right expert, one stakeholder positions. It is common for di-
who can apply an appropriately criticai eye, vergent stakeholders to disagree about pro-
gram purposes, goals, and ineans of attain- servation (because it is rarely possible to
ing goals. These differences represent observe ali situations even within a sin-
different "theories of action" (Patton 1997a) gle setting)
that can cast the same findings in different Limitations from the time periods dur-
perspective-based lights. ing which observations took place, that
is, constraints of temporal sampling
s Limitations based on selectivity in the
THOUGHTFUL, SYSTEMATIC
people who were sampled either for ob-
TRIANGULATION
servations or interviews, or selectivity in
Ali of these different types of triangula- document sampling
tionmethods triangulation, triangulation
of data sources, analyst triangulation, and
In reporting how purposeful sampling
theory or perspective triangulationoffer
strategies affect findings, the analyst returns
strategies for reducing systematic bias and
to the reasons for having made initial design
distortion during data analysis. In each case,
decisions. Purposeful sampling involves
the strategy involves checking findings
studying information-rich cases in depth
against other sources and perspectives. Tri-
and detail to understand and illuminate im-
angulation, in whatever form, increases
portant cases rather than generalizing from
credibility and quality by countering the
a sample to a population (see Chapter 5). For
concern (or accusation) that a study's find-
instance, sampling and studying highly suc-
ings are simply an artifact of a single
cessful and unsuccessful cases in an inter-
method, a single source, or a single investi-
vention yield quite different results than
gator's blinders.
studying a typical case or a mix of cases. Peo-
Design Checks: Keeping ple unfamiliar with purposeful samples
may think of small samples as "biased," a
Methods and Data in Context
perception that undermines credibility in
One possible source of distortion in quali- their minds. In communicating findings,
tative findings concerns how design deci- then, it becomes important to emphasize
sions affect results. For example, purposeful that the issue is not one of dealing with a dis-
sampling strategies provide a limited num- torted or biased sample, but rather one of
ber of cases for examination. When inter- clearly delineating the purpose and limita-
preting findings, then, it becomes important tions of the sample studiedand therefore
to reconsider how design constraints may being careful about extrapolating (much less
have affected the data available for analysis. generalizing) the findings to other situa-
This means considering the rival method- tions, other time periods, and other people.
ological hypothesis that the findings are due Reporting both methods and results in their
to methodological idiosyncrasies. proper contexts will avoid many controver-
By their nature, qualitative findings are sies that result from yielding to the tempta-
highly context and case dependent. Three tion to overgeneralize from purposeful sam-
kinds of sampling limitations typically arise ples. Keeping findings in context is a
in qualitative research designs: cardinal principie of qualitative analysis.
Mulla Nasrudin was once called upon to
Limitations in the situations (criticai make this point to his monarch. Although he
events or cases) that are sampled for ob- was supposed to be a wise and holy man,
Nasrudin was accused of being almost illit- by designing an evaluation to answer the
erate. One day the ruler of his country de- focused questions of specific primary in-
cided to put this to the test. tended users (Patton 1997a). However, as we
"Write something for me, Nasrudin," looked at examples of lessons learned being
said he. included as conclusions in a variety of clus-
"I would willingly do so, but I have taken ter evaluation reports, Ricardo Millett, for-
an oath never to write so much as a single let- mer director of evaluation at the W. K.
ter again," replied Nasrudin. Kellogg Foundation, and 1 began discussing
"Well, write something in the way in the deteriorating meaningfulness of the
which you used to write before you decided phrases "lessons learned" and "best prac-
not to write, so that I can see what it was tices." As these phrases became widely
like." used, they began to be applied to any kind of
"I cannot do that, because every time you insight, evidentially based or not. We began
write something, your writing changes thinking about what would constitute a
slightly through practice. If 1 wrote now, it "high-quality lessons learned" and decided
would be something written for now." that one's confidence in the transferability or
"Then bring me an example of his writ- extrapolated relevance of a supposed lesson
ing, anyone who has one," ordered the ruler. learned would increase to the extent that it
Someone brought a terrible scrawl that was supported by multiple sources and
the Mulla had once written to him. types of learnings. Exhibit 9.3 presents a list
"Is this your writing?" asked the mon- of kinds of evidence that could be accumu-
arch. lated to support a proposed lesson learned,
"No," said Nasrudin. "Not only does making it more worthy of application and
writing change with time, but reasons for adaptation to new settings if it has inde-
writing change. You are now showing a pendent triangulated support from a variety
piece of writing done by me to demonstra te of perspectives. Questions for generating
to someone how he should not write" (Shah lessons learned are also listed. Thus, for ex-
1973:92). ample, the lesson that designing an evalua-
tion to answer the focused questions of spe-
High-Quality Lessons Learned cific primary intended users enhances
evaluation use is supported by research on
The notions of identifying and articulat- use, theories about diffusion of innovation
ing "lessons learned" and "best practices" and change/ practitioner wisdom, cross-case
have become popular purposes of cross-case analyses of use, the profession's articulation
analyses in multisite organizational studies of standards, and expert testimony. For ex-
and cluster evaluations that aim to build ample, House's reflections about general-
knowledge comparatively. Rather than be- izability in The Logic ofEvaluative Argument
ing stated in the form of traditional scientific (1977) constitute an example of expert testi-
empirical generalizations, lessons learned mony in support of the lesson learned about
and best practices more often take the form evaluation use:
of principies of practice that must be adapted
to particular settings in which the principie In evaluation, the social and psychological
is to be applied. For example, a lesson contexts become particularly relevant and the
learned from research on evaluation use is knowledge less certain. Under those condi-
that evaluation use will likely be enhanced tions argumenta tion aimed at gaining the ad-
E X H I B I T 9.3 High-Quality Lessons Learned
High-quality lessons learned: Knowledge that can be applied to future action and derived
from screening according to specific criteria:
Evaluation findings-patterns across programs
Basic and applied research
Practice wisdom and experience of practitioners
Experiences reported by program participants/clients/intended beneficiaries
Expert opinion
Cross-disciplinary connections and patterns
Assessment of the importance of the lesson learned
Strength of the connection to outcomes attainment
The idea is that the greaterthe number of supporting sources for a "lesson learned," the more
rigorous the supporting evidence, and the greater the triangulation of supporting sources, the
more confidence one has in the significance and meaningfulness of a lesson learned. Lessons
learned with only one type of supporting evidence would be considered a "lessons learned hypoth-
esis." Nested within and cross-referenced to lessons learned should be the actual cases from which
practice wisdom and evaluation findings have been drawn. A criticai principie here is to maintain
the contextual frame for lessons learned, that is, to keep lessons learned grounded in their context.
For ongoing learning, the trick is to follow future supposed applications of lessons learned to test
their wisdom and relevance over time in action in new settings.
Questions for qeneratinq hiqh-qualitv lessons learned

1. What is meant by a "lesson"?
2. What is meant by "learned"?
3. By whom was the lesson learned?
4. What is the evidence supporting each lesson?
5. What is the evidence the lesson was learned?
6. What are the contextual boundaries around the lesson (i.e., under what conditions
does it apply)?
7. Is the lesson specific, substantive, and meaningful enough to guide practice in some
concrete way?
8. Who else is likely to care about this lesson?
9. What evidence will they want to see?
10. How does this lesson connect with other lessons?
herence and increasing the understanding of sal audience of ali rational men with the neces-
particular audiences is more appropriate. Per- sity of his conclusions.
suasion claims vaiidity only for particular au- Persuasion is directly related to action.
diences and the intensity with which Even though evaluation information is less
particular audiences accept the evaluation certain than scientific information addressed
findings is a measure of this effectiveness. The to a universal audience, persuasion is effective
evaluator does not aim at convincing a univer- in promoting action because it focuses on a
particular audience and musters information gram is a recovering alcoholic. This can
with which this audience is concerned. (p. 6) either enhance or reduce credibility depend-
ing on how it has enhanced or detracted
High-quality lessons learned, then, repre- from data gathering and analysis, Either
sent principies extrapolated from multiple way, the analyst needs to deal with it in re-
sources and independently triangulated to porting findings. In a similar vein, it is only
increase transferability as cumulative honest to report that the evaluator of a fam-
knowledge working hypotheses that can be ily counseling program was going through a
adapted and applied to new situations, a difficult divorce at the time of fieldwork.
form of pragmatic utilitarian generaliza- No definitive list of questions must be ad-
bility, if you will. The pragmatic bias in this dressed to establish invs tigator credibility.
approach reflects the wisdom (dare one say The principie is to report any personal and
lesson learned) of Samuel Johnson: "As gold professional information that may have af-
which he cannot spend will make no man fected data collection, analysis, and inter-
rich, so knowledge which he cannot apply pretationeither negatively or positively
will make no man wise." in the minds of users of the findings. For
example, health status should be reported if
it affected one's stamina in the field. (Were
you sick part of the time? The fieldwork for
m The Credibility
evaluation of an African health project was
of the Researcher conducted over three weeks during which
The prvio us sections have reviewed strate- time the evaluator had severe diarrhea. Did
gies for enhancing the quality and credibil- that affect the highly negative tone of the re-
ity of qualitative analysis, searching for ri- port? The evaluator said it didn't, but I'd
val explanations, explaining negative cases, want to have the issue out in the open to
triangulation, and keeping data in context. make my own judgment.) Background char-
Technical rigor in analysis is a major factor acteristics of the researcher (e.g., gender,
in the credibility of qualitative findings. age, race, ethnicity) may be relevant to re-
This section now takes up the issue of how port in that such characteristics can affect
the credibility of the inquirer affects the way how the researcher was received in the set-
findings are received. ting under study and what sensitivities the
Because the researcher is the instrument inquirer brings to the issues under study.
in qualitative inquiry, a qualitative report In preparing to interview farm families in
should include some information about the Minnesota, I began building up my toler-
researcher. What experience, training, and ance for strong coffee a month before the
perspective does the researcher bring to the fieldwork. Being ordinarily a non-coffee
field? Who funded the study and under drinker, I knew my body would be jolted by
what arrangements with the researcher? 10 to 12 cups of coffee a day doing interviews
How did the researcher gain access to the in farm kitchens. In the Caribbean, when in-
study site? What prior knowledge did the re- terviewing farmers, I had to increase my tol-
searcher bring to the research topic and erance for rum because some interviews
study site? What personal connections does took place in rum shops. These are matters
the researcher have to the people, program, of personal preparationboth mental and
or topic studied? For example, suppose the physicalthat affect perceptions about the
observer of an Alcoholics Anonymous pro- quality of the study. Preparation and train-
ing for fieldwork, discussed at the beginning The credibility of your findings and interpre-
of Chapter 6, should be reported as part of ta tions depends upon your careful attention to
the study's methodology. establishing trustworthiness Time is a ma-
jor factor in the acquisition of trustworthy
data. Time at your research site, time spent in-
terviewing, and time building sound relation-
Considering Investigator ships with respondents ali contribute to
Effects: Varieties of Reactivity trustworthy data. When a large amount of
time is spent with your research participants,
Another factor to consider and report they less readily feign behavior or feel the
concerns how the presence of an observer or need to do so; moreover, they are more likely
evaluator may have affected what was ob- to be frank and comprehensive about what
served. There are four primary ways in they tell you. (Glesne 1999:151)
which the presence of an outside observer,
or the fact that an evaluation is taking place,
On the other hand, prolonged engage-
can distort the findings of a study:
ment may actually increase reactivity as the
researcher becomes more a part of the set-
1. reactions of those in the setting (e.g.,
ting and begins to affect what goes on
program participants and staff) to the
through prolonged engagement. Thus,
presence of the qualitative field-
whatever the length of inquiry or method of
worker;
data collection, researchers have an obliga-
tion to examine how their presence affects
2. changes in the fieldworker (the measur- what goes on and what is observed.
ing instrument) during the course of the
data collection or analysis, that is, what
It is axiomatic that observers must record
has traditionally been called instrumen-
what they perceive to be their own reactive ef-
tation effects;
fects. They may treat this reactivity as bad and
attempt to avoid it (which is impossible), or
3. the predispositions, selective percep- they may accept the fact that they will have a
tions, and/or biases of the inquirer; and reactive effect and attempt to use it to advan-
t a g e . . . . The reactive effect will be measured
4. researcher incompetence (including by daily field notes, perhaps by interviews in
lack of sufficient training or prepara- which the problem is pointedly inquired
tion). about, and also in daily observations. (Denzin
1978b :200)
Problems of reactivity are well docu-
mented in the anthropological literature, Anxieties that surround an evaluation
which is one of the prime reasons why quali- can exacerbate reactivity. The presence of an
tative methodologists advocate long-term evaluator can affect how a program opera tes
observations that permit an initial period as well as its outcomes. The evaluator's
during which observers and the people in presence may, for example, create a halo ef-
the setting being observed get a chance to fect so that staff performs in an exemplary
get used to each other. This increases trust- fashion and participants are motivated to
worthiness and that supports credibility "show off." On the other hand, the presence
both within and outside the study setting. of the evaluator may create so much tension
and anxiety that performances are below point where it was impossible to do good ob-
par. Some forms of program evaluation, es- servations. Lillian laughed and suggested to
pecially "empowerment evaluation" and me that what I was experiencing was the
"intervention-oriented evaluation" (Patton way those classrooms actually were. She
1997a: Chapter 5) turn this traditional threat went on to note that this was common
to validity into an asset by designing data among visitors to schools; they were always
collection to enhance achievement of the de- concerned that the teacher, knowing visitors
sired program outcomes. For example, at the were coming, whipped the kids into shape
simplest levei, the observation that "what for those visitors. She suggested that under
gets measured gets done" suggests the the best of circumstances a teacher might get
power of data collection to affect outcomes kids to move out of habitual patterns into
attainment. A leadership program, for ex- some model mode of behavior for as much
ample, that includes in-depth interviewing as 10 or 15 minutes, but that, habituai pat-
and participant journal writing as ongoing terns being what they were, kids would
forms of evaluation data collection may find rapidly revert to normal behaviors and
that participating in the interviewing and whatever artificiality might have been intro-
writing reflectively have effects on partici- duced by the presence of the visitor would
pants and program outcomes. Likewise, a likely become apparent.
community-based AIDS awareness inter- Evaluators and researchers should strive
vention can be enhanced by having commu- to neither overestimate nor underestimate
nity participants actively engaged in identi- their effects but to take seriously their re-
fying and doing case studies of criticai sponsibility to describe and study what
community incidents. In short, a variety of those effects are.
reactive responses are possible, some that A second concern about evaluator effects
support program processes, some that inter- arises from the possibility that the evaluator
fere, and many that have implications for in- changes during the course of the evaluation.
terpreting findings. Thus, the evaluator has In Chapter 7 on interviewing I offered sev-
a responsibility to think about the problem, eral examples including how, in a study of
make a decision about how to handle it in child sexual abuse, those involved were
the field, attempt to monitor evaluator/ob- deeply affected by what they heard. One of
server effects, and reflect on how reactivities the ways this sometimes happens in anthro-
may have affected findings. pological research is when participant ob-
Evaluator effects are often considerably servers "go na tive" and become absorbed
overrated, particularly by evaluators. There into the local culture. The epitome of this in a
is more than a slight touch of self-impor- shorter-term observation is the story of the
tance in some concerns about reactivity. observers who became converted to Chris-
Lillian Weber, director of the Workshop Cen- tianity while observing a Billy Graham cru-
ter for Open Education, City College School sade (Lang and Lang 1960). Evaluators
of Education, New York, once set me sometimes become personally involved
straight on this issue, and I pass her wisdom with program participants or staff and there-
on to my colleagues. In doing observations fore lose their sensitivity to the full range of
of open classrooms, I was concemed that my events occurring in the setting.
presence, particularly the way kids flocked Johnson (1975) and Glazer (1972) have re-
around me as soon as I entered the class- flected on how they and others have been
room, was distorting the situation to the changed by doing field research. The con-
sensus of advice on how to deal with the difference stances and end up with different
probiem of changes in the observers as a re- conclusions.
sult of involvement in research is similar to Consider the interviewing stance of em-
advice about how to deal with the reactive pathic neutrality introduced in Chapter 2 and
effects createdby the presence of observers. elaborated in Chapter 7. An empathically
neutral inquirer will be perceived as caring
It is central to the method of participant obser- about and interested in the people being
vation that changes will occur in the observer; studied, but neutral about the content of
the important point, of course, is to record what they reveal. House (1977) balances the
these changes. Field notes, introspection, and caring, interested stance against independ-
conversations with informants and colleagues ence and impartiality for evaluators, a
provide the major means of measuring this distance that also applies to those working ac-
m e n s i o n , . . . for to be insensitive to shifts in cording to the standards of traditional sci-
one's own attitudes opens the way for placing ence.
naive interpretations on the complex set of
events under analysis. (Denzin 1978b :200) The evaluator mustbe seen as caring, as inter-
ested, as responsive to the relevant arguments.
The third concern about inquirer effects He must be impartial rather than simply ob-
has to do with the extent to which the predis- jective. The impartiality of the evaluator must
positions or biases of the evaluator may af- be seen as that of an actor in events, one who is
fect data analysis and interpretations. This responsive to the appropriate arguments but
issue carries mixed messages because, on in whom the contending forces are balanced
the one hand, rigorous data collection and rather than non-existent. The evaluator must
analytical procedures, like triangulation, are be seen as not having previously decided in fa-
aimed at substantiating the validity of the vor of one position or the other. (House
data and minimizing inquirer biases; on 1977:45-46)
the other hand, the interpretative and
constructivist perspectives remind us that But neutrality and impartiality are not
data from and about humans inevitably rep- easy stances to achieve. Denzin (1989b) cites
resent some degree of perspective rather a number of scholars who have concluded,
than absolute truth. Getting close enough to as he does, that every researcher brings pre-
the situation observed to experience it first- conceptions and interpretations to the prob-
hand means that researchers can learn from iem being studied, regardless of methods
their experiences, thereby generating per- used.
sonal insights, but that closeness makes their
objectivity suspect. "For social scientists to Ali researchers take sides, or are partisans for
refuse to treat their own behavior as data one point of view or another. Value-free inter-
from which one can learn is really tragic" pretive research is impossible. This is the case
(Scriven 1972a:99). In effect, ali of the proce- because every researcher brings preconcep-
dures for validating and verifying analysis tions and interpretations to the probiem being
that have been presented in this chapter are studied. The term hermeneutical rcle or situa-
aimed at reducing distortions introduced by tion refers to this basic fact of research. Ali
inquirer predisposition. Still, people who scholars are caught in the circle of interpreta-
use different criteria in determining eviden- tion. They can never be free of the herme-
tial credibility will come at this issue from neutical situation. This means that scholars
must state beforehand their prior interpreta- which the evaluator's findings can be
tions of the phenomenon being investigated. trusted; that is, trustworthiness can be un-
Unless these meanings and values are clari- derstood as one dimension of perceived
fied, their effects on subsequent interpreta- methodological rigor (Lincoln and Guba
tions remam clouded and often misunder- 1986; Glesne 1999). For better or worse, the
stood. (Denzin 1989b:23) trustworthiness of the data is tied directly to
the trustworthiness of the person who col-
lects and analyzes the dataand his or her
Earlier (Exhibit 9.1)1 presented five sets of
demonstrated competence. Competence is
criteria for judging the quality of qualitative
demonstrated by using the verification and
inquiry. Those varying frameworks offer
validation procedures necessary to establish
different perspectives on how inquirers
the quality of analysis and thereby building
should deal with concerns about bias. Neu-
a "track record" of quality work.
trality and impartiality are expected when
qualitative work is being judged by tradi-
tional scientific criteria or by evaluation Intellectual Rigor
standards, thus the source of House's admo-
nition quoted above. In contrast, construc- The thread that runs through this discus-
tivist analysts are expected to deal with sion of credibility is the importance of intel-
these issues through conscious and commit- lectual rigor, professional integrity, and
ted reflexivityentering the hermeneutical methodological competence. There are no
circle of interpretation and therein reflecting simple formulas or clear-cut rules about
on and analyzing how their perspective inhow to do a credible, high-quality analysis.
teracts with the perspectives they encounter. The task is to do one's best to make sense of
Artistic inquirers often deal with issues of things. A qualitative analyst returns to the
how they personally relate to their work by data over and over again to see if the con-
invoking aesthetic criteria: Judge the work structs, categories, explanations, and inter-
on its artistic merits. When criticai change preta tions make sense, if they really reflect
criteria are applied in judging reactivity, the the nature of the phenomena. Creativity, in-
issue becoines whether, how, and to what ex- tellectual rigor, perseverance, insight
tent the inquiry furthered the cause or en- these are the intangibles that go beyond the
hanced the well-being of those involved and routine application of scientific procedures.
studied; neutrality is eschewed in favor of As Nobel prize-winning physicist Percy
explicitly using the inquiry process to facili- Bridgman put it: "There is no scientific
tate change, or at least illuminate the condi- method as such, but the vital feature of a sci-
tions needed for change. entisfs procedure has been merely to do his
The politics of evaluation mean that indi- utmost with his mind, no holds barred"
vidual evaluators must make their own (quoted in Mills 1961:58).
peace with how they are going to describe
what they do. The meaning and connota-
SL The Paradigms
tions of words such as objectivity, subjectivity,
Debate and Credibility
neutrality, and impartiality will have to be
worked out with particular stakeholders in We come now to the third leg of the credibil-
specific evaluation settings. Essentially, ity triangle, the first two having been (1) rig-
these are ali concems about the extent to orous methods for doing fieldwork that yield
QUIDEUNF*; FCIF NM,\VR IN MM^^MHMM.

O H M S or ?;;i;;U -";I.TUPR !: H E 5 6 A R C I I
* Auhî^jji-vii-iinii '.H'--iii.d|i:" iiiilXi 1 i j; '-.;!' iii I ii cnsfti o r ^ i H n

* 'ii "!! k ! ;.h::)].!i!:! \y- n-ni- -- =,=!.=! i sn i - 1:;;- n i *: ":=i I i:: m
* .:,;=! :,=:!!! i\v? ; ii nvi;\ "C|JIji: MMW' fn11HIi|iiI1 !y U II! Mi : I " V 1 TI.I
Illllil6ll!!lilil|!lllilllllill!
.'!ilnhc WO o: i* ntttfliy lAll no! r-:= ' ii^LiblSI 1f !;<3i:I'!; =ii i.v! li!
||||||||||||||||||||||
' : : " ! " I I : . I ! S.! : KI LP .II ; H ' - I M >} I R R=I:=- H A N D I J - : J -I 1 S;; .I:'! U !;; :; W . B ? I 11 >,\'
th"! kyi-n ji l.i. ii !:i jj; i =:n" dn!;.-for';'1' w r !. H"'!!''! I'!:i! n !!'*
+ PA!'!'=!! !UI G I I L I J B :-|;1 : : T ;'! I : D-1|.!; nri i.'ri.,L <
: :I'.IIFI,<.
s-RA-hi!V r B ' H : ! : > = ' ;r.fi.;ipnrn!
tik':. .Vi! "iU" ! d i:'!: i.'i J] iJ a II !f .! ! lhe iry.
* CL i. ;! V ii i.i! ;:=!:! YW i;: a! M:! .i iii '<
! .A b 1 d -H'::: i i. i i? H
!< i ii = i r= : U. \: :::. w. i 1 1 \\
* !;..;: Mv
: iii.iinh^jM^.Hriii i*lr std ffrr hrH pisptt Y :n !'!:. ii ;!.
* fa "M K I :=! r s 1 p , =::=:! I ! J: r\ n -S . : I :;.A) I & N =:I R ; : A R T O J W ^ S1 T SRu t i A - V R ,;-I I; I I J TI
aii-J !.!;.'.:==.il';-. i;ii.i;rrii'n'.r;ri> iir-d tirtC'.".!i'- ;:!.I; !iJi!V-!.!:' rlji.-J:!!!f= i' i"i!ui''I Li
n h n n iMi^fltl^^^^^^^^ n i i f h i H^y
t Hf H^ny Hy
i NIhiI Ou iinJillilIlllllllllnli^
* | !:';:; pf i II u I.n-; s-s ik rs' t :; I .=; D \ fcu ii i! i . iv y = : ; I B V ! . :I |.M ! I I I ! " 1.1 !. :I''I 1 1
:;N:=:;. oii-i-iiiisHiiiii',, k SLr= : >:! =:"MI:^ i^^svie^ ikii!!:! i->ii .ji: w:I f,,i!,, Mi;,.i.V
miKmmwiwMiWm^mmmmKam
high-quality data that are systematically an- interpretivism. How the debate is framed
alyzed with attention to issues of credibility depends on the perspectives that people
and (2) the credibility of the researcher; which bring to it and the language available to
is dependent on training, experience, track them to talk about it, The debate is rooted in
record, status, and presentation of self. We philosophical differences about the nature
take up now the issue of philosophical beliefin of reality and epistemological differences
the value of qualitative inquiry, that is, a fun- about what constitutes knowledge and how
damental appreciation of naturalistic in- it is created. This has come to be called "the
quiry, qualitative methods, inductive analy- paradigms debate," a paradigm in this case
sis, purposeful sampling, and holistic being a particular worldview where phiios-
thinking. ophy and methods intersect to determine
The use of qualitative methods can be what kinds of evidence one finds acceptable.
quite controvrsia! The controversy stems The literature on the paradigms debate is ex-
from the long-standing debate in science tensive (cf. Denzin and Lincoln 2000b;
over how best to study and understand the Tashakkori and Teddlie 1998; Patton 1997a:
world. As discussed in earlier chapters, the Chapter 12; Shadish 1995a, 1995b, 1995c;
debate sometimes takes the form of qualita- Guba 1991; Fetterman 1988a, 1988b; Lincoln
tive versus quantitative methods, or science and Guba 1985, 1986, 2000; Cronbach 1975;
versus phenomenology, or positivism ver- Guba and Lincoln 1981; Reichardt and Cook
sus constructivism, or realism versus 1979; Rist 1977; Campbell 1974, 1999a,
1999b). The point here is to alert those new to odologists (hereafter known as "quais")
the debate that it has been and can be in- found themselves together on a train travel-
tense, divisive, emotional, and rancorous. ing to the same professional meeting. The
Both scientists and nonscientists often quais, ali of whom had tickets, observed that
hold strong opinions about what consti- the quants had only one ticket for their
tutes credible evidence. These opinions are whole group.
paradigm derived and paradigm dependent "How can you ali travei on one ticket?"
because a paradigm constitutes a worldview asked a qual.
built on implicit assumptions, accepted defi- "Wehave our methods," replied a quant.
nitions, comfortable habits, values defended Later, when the conductor came to punch
as truths, and beliefs projected as reality. As tickets, ali the quants slipped quickly behind
such, paradigms are deeply embedded in the door of the toilet. When the conductor
the socialization of adherents and practitio- knocked on the door, the head quant slipped
ners telling them what is important, legiti- their one ticket under the door, thoroughly
mate, and reasonable. Paradigms are also fooling the conductor.
normative, telling the practitioner what to On their return from the conference, the
do without the necessity of long existential two groups again found themselves on the
or epistemological consideration. But it is same train. The qualitative researchers, hav-
this aspect of paradigms that constitutes ing learned from the quants, had schemed to
both their strength and their weakness share a single ticket. They were chagrined,
their strength in that it makes action possi- therefore, to learn that, this time, the statisti-
ble, their weakness in that the very reason cians had boarded with no tickets.
for action is hidden in the unquestioned as- "We know how you traveled together
sumptions of the paradigm. with one ticket," revealed a qual, "but how
Given the often controversial nature of can youpossibly get away with no tickets?"
qualitative findings and the necessity, on oc- "We have new methods," replied a quant.
casion, to be able to explain and even defend Later, when the conductor approached,
the value and appropriateness of qualitative ali the quais crowded into the toilet. The
approaches, the sections that follow will head statistician followed them and
briefly discuss the most common concerns. knocked authoritatively on the toilet door.
Tll then tell you how to make the case that The quais slipped their one and only ticket
the debate is over, or as my teenage daughter under the door. The head quant took the
says, "That's so 10 minutes ago." ticket and joined the other quants in a differ-
First, however, to set the context and ac- ent toilet. The quais were subsequently dis-
knowledge that the debate continues for covered without tickets, publicly humili-
many, let me share a story to illustrate the an- ated, and tossed off the train atits next stop.
tagonisms that sometimes undergird (and
undermine) the debate. A former student Beyond the Numbers Game
sent me the following story, which she had
received as an e-mail chain letter, a matter of Philosopher of science Thomas H. Kuhn
interest only because it suggests widespread (1970), having studied extensively the value
distribution. systems of scientists, observed that "the
Once upon a time, not so very long ago, a most deeply held values concern predic-
group of statisticians (hereafter known as tions" and "quantitative predictions are
"quants") and a party of qualitative meth- preferable to qualitative ones" (pp. 184-85).
Measuring Responses
PLAYING THE HUNBERS GAME:

TURHING THOUGHTS INTO SCALES
The methodological status hierarchy in sci- valid, and meaningless (for examples, see
ence ranks "hard data" above "soft data" Hausman 2000; Huff and Gis 1993). Indeed,
where "hardness" refers to the precision of Gephart (1988) has designated "ethnosta-
statistics. Qualitative data, then, carry the tistics" as a form of ethnographic study of
stigma of "being soft." This carries over into groups that routinely produce statistics, fo-
the public arena, especially in the media and cusing on the technical and operational as-
among policymakers, creating what has sumptions involved in the production of sta-
been called "the tyranny of numbers" tistics and deconstructing statistics as a
(Eberstadt, Eberstadt, and Moynihan 1995). rhetorical device in research and the public
How can one deal with a lingering bias arena.
against qualitative methods? The point, however, is not to be anti-
The starting point is understanding and numbers. The point is to be pro-meaningful-
being able to communicate the particular ness. Thus, by knowing the strengths and
strengths of qualitative methods (Chapters 1 weaknesses of both quantitative and quali-
and 2) and the kinds of evaluation and other tative data, you can help those with whom
applications for which qualitative data are you dialogue focus on really important
especially appropriate (Chapter 4). It is also questions rather than, as sometimes hap-
helpful to understand the special seductive- pens, focusing primarily onhow to generate
ness of numbers in modern society. Num- numbers. The really important questions
bers convey a sense of precision and accu- are, What's worth knowing? What data will
racy even if the measurements that yielded be most illuminative? Most useful? How can
the numbers are relatively unreliable, in- the design be appropriately matched to the
inquiry purpose? In evaluation, what design imply opinion rather than fact, intuition
is most appropriate for the type of evalua- rather than logic, impression rather than
tion needed (formative, developmental, confirmation. Chapter 2 briefly discussed
summative), the stage of program develop- concerns about objectivity versus subjectiv-
ment, and the priority information needs of ity, but I return to the issue here to address
primary stakeholders? In policy formula- how these concerns affect the credibility and
tion, one must understand where, how, and utility of qualitative analysis.
when qualitative data can inform and influ- Social scientists are exhorted to eschew
ence policy processes, a matter examined in subjectivity and make sure that their work is
depth by experienced policy analyst Ray "objective." The conventional means for
Rist (2000). controlling subjectivity and maintaining ob-
Moreover, as noted in discussing the jectivity are the methods of quantitative so-
value of methods triangulation, the issue cial science: distance from the setting and
need not be quantitative versus qualitative people being studied, formal operation-
methods, but rather how to combine the alism and quantitative measurement, ma-
strengths of each in a multimethods ap- nipulation of isolated variables, and experi-
proach to research and evaluation. Qualita- mental designs. Yet, the ways in which mea-
tive methods are not weaker or softer than sures are constructed in psychological tests,
quantitative approaches. Qualitative meth- questionnaires, cost-benefit indica tors, and
ods are different. Furthermore, "we no longer routine management information systems
need to regard qualitative research as provi- are no less open to the intrusion of biases
sional, because qualitative studies have al- than making observations in the field or ask-
ready assembled a usable, cumulative body ing questions in interviews. Numbers do not
of knowledge" (Silverman 1997:1). Given protect against bias; they merely disguise it.
that context, let's examine some ways of Ali statistical data are based on someone's
reframing old quantitative-qualitative de- definition of what to measure and how to
bate issues. measure it. An "objective" statistic such as
the consumer price index is really made up
Beyond Objectivity and of very subjective decisions about what con-
sumer items to include in the index. Peri-
Subjectivity: New Concepts,
odically, govemment economists change the
New Language
basis and definition of such ndices.
Scriven (1972a) has insisted that quantita-
Science places great value on objectivity.
tive methods are no more synonymous with
Often the primary reason decision makers
objectivity than qualitative methods are syn-
commission an evaluation is to get objective
onymous with subjectivity:
data from an independent and objective
source externai to the program being evalu-
ated. The charge that qualitative methods Errors like this are too simple to be explicit.
are inevitably "subjective" casts an asper- They are inferred confusions in the ideological
sion connoting the very antithesis of scien- foundations of research, its interpretations, its
tific inquiry. Objectivity is traditionally con- application It is increasingly clear that the
sidered the sine qua non of the scientific influence of ideology on methodology and of
method. To be subjective means to be biased, the latter on the training and behavior of re-
unreliable, and irrational. Subjective data searchers and on the identification and dis-
bursement of support is staggeringly pow- lournalism in general and investigative jour-

erful. Ideology is to research what Marx sug- nalism in particular are moving away from the
gested the economic factor was to politics and criterion of objectivity to an emergent crite-
what Freud took sex to be for psychology. rion usually labeled " f a i r n e s s . " . . . Objectivity
(P- 94) assumes a single reality to which the story or
evaluation must be isomorphic; it is in this
sense a one-perspective criterion. It assumes
Scriven's lengthy discussion of objectiv-
that an agent can deal with an objective (or an-
ity and subjectivity in educational research
other person) in a nonreactive and noninter-
deserves careful reading by students and
active. It is an absolute criterion.
others concerned by this distinction. He
lournalists are coming to feel that objectiv-
skillfully detaches the notions of objectivity
ity in that sense is unattainable....
and subjectivity from their traditionally nar-
Enter "fairness" as a substitute criterion. In
row associations with quantitative and qual-
contrast to objectivity, fairness has these fea-
itative methodology, respectively. He pres-
tures:
ents a clear explanation of how objectivity
has been confused with consensual valida- It assumes multiple realities or truths
tion of something by multiple observers. hence a test of fairness is whether or not
Yet, a little research will yield many in- "both" sides of the case are presented, and
stances of "scientific blunders" (Youngson there may even be multiple sides.
1998) where the majority of scientists (or
other people) were factually wrong while It is adversarial rather than one-perspective
one dissenting observer described things as in nature. Rather than trying to hew the line
they actually were (Kuhn 1970). with the truth, as the objective reprter
Qualitative rigor has to do with the qual- does, the fair reprter seeks to present each
ity of the observations madeby an evaluator. side of the case in the manner of an advo-
Scriven emphasizes the importance of being cateas, for example, attorneys do in mak-
factual about observations rather than being ing a case in court. The presumption is that
distant from the phenomenon being stud- the public, like a jury, is more likely to reach
ied. Distance does not guarantee objectiv- an equitable decision after having heard
ity; it merely guarantees distance. Never- each side presented with as much vigor and
theless, in the end, Scriven (1998) still finds commitment as possible.
the ideal of objectivity worth striving for as a
counter to bias, and he continues to find the It is assumed that the subjecfs reaction to
language of objectivity serviceable. the reprter and interactions between them
In contrast, Lincoln and Guba (1986), as heavily determines what the reprter per-
noted earlier, have suggested replacing the ceives. Hence one test of fairness is the
traditional mandate to be objective with an length to which the reprter will go to test
emphasis on trustivorthiness and authenticity his own biases and rule them out.
by being balanced, fair, and conscientious in
taking account of multiple perspectives, It is a relative criterion that is measured by
multiple interests, and multiple realities. balance rather than by isomorphism to en-
They have suggested that researchers and during truth.
evaluators can learn something about these
attributes from the stance of invstigative Clearly, evaluators have a great deal to learn
journalists. from this development. (Guba 1981:76-77)
Earlier in this chapter, I discussed the futile search for truth, the topic we consider
credibility of the inquirer and noted that in the next section. As this is written we are
trustworthiness of the inquirer is one dimen- still searching for language and terminology
sion of rigor. The issue, then, is not really to transcend the old and outdated divisions
about objectivity in the abstract, but about of objective versus subjective. No consensus
researcher credibility and trustworthiness, about new terminology has emerged, and
about fairness and balance. How, then, does given the five different sets of criteria for
one deal with concerns about objectivity? judging qualitative inquiry Iidentified at the
It is helpful to know that scholarly philos- beginning of this chapter, it seems unlikely
ophers of science now typically doubt the that a consensus is on the horizon. This can
possibility of any one or any method being be liberating because it opens up the possi-
totally "objective." But subjectivity, even if bility of getting beyond the meaningless ab-
acknowledged as inevitable (Peshkin 1988), stractions of objectivity and subjectivity and
carries negative connotations at such a deep moving instead to carefully selecting descrip-
levei and for so many people that the very tive methodological language that best de-
term can be an impediment to mutual un- scribes your own inquiry processes and pro-
derstanding. For this and other reasons, as a cedures. That is, don't label those processes
way of elaborating with any insight the na- as "objective," "subjective," "trustworthy,"
ture of the research process, the notion of "neutral," "authentic," or "artistic." De-
subjectivity may have become as useless as scribe them and what you bring to them and
the notion of objectivity. how you've reflected on them, and then let
the reader be persuaded, or not, by the intel-
The death of the notion that objective truth is lectual and methodological rigor, meaning-
attainable in projects of social inquiry has been fulness, value, and utility of the result. In the
generally recognized and widely accepted by meantime, be very careful how you use par-
scholars who spend time thinking about such ticular terms in specific contexts, a point
matters. . . . I will take this recognition as a made nicely by the following cautionary
starting point in calling attention to a second tale.
corpse in our midst, an entity to which many During a tour of America, former British
refer as if it were still alive. Instead of explor- Prime Minister Winston Churchill attended
ing the meaning of subjectivity in qualitative a buffet luncheon at which chicken was
educational research, I want to advance the served. As he returned to the buffet for a sec-
notion that following the failure of the objecti- ond helping he asked, "May I have some
vists to maintain the viability of their episte- more breast?"
mology, the concept of subjectivity has been His hostess, looking embarrassed, ex-
likewise drained of its usefulness and there- plained that "in this country we ask for
fore no longer has any meaning. Subjectivity, I white meat or dark meat."
feel obliged to report, is also dead. (Barone Churchill, taking the white meat he was
2000:161) offered, apologized and returned to his ta-
ble.
Barone (2000:169-70) goes on to argue in The next morning the hostess received a
favor of the "criterion of criticai persuasive- beautiful orchid from Churchill with the fol-
ness" based on "neopragmatisim," essen- lowing card: "I would be most obliged if you
tially elevating a search for utility above the would wear this on your white meat."
Completely Objective
Perspective
Reflections on Truth and Utility as Criteria of Quality
L ady, I do not make up things. That is lies. Lies are not true. But the truth
could be made up if you know how. And that's the truth.
Lily Tomlin
One paradigm-related belief that affects sounding, arrogant, and philosophical dis-
how people react to qualitative data in- course on the question "What is truth?" in
volves how they think about the idea of the expectation that the public officials pres-
"Truth/' "Do you, as a qualitative re- ent would be alienated and dismiss my testi-
searcher, swear to tell the truth, the whole mony. So I did not reply, "That depends on
truth and no thing but the truth?" I was once what truth means." I said simply: "Certainly
asked this very question by a hostile school I promise to respond honestly." Notice the
researcher at a public meeting who sought to shift from truth to honesty.
embarrass me in front of a school board in a Lancelot Andrews, a 17th-century priest,
debate about standardized tests. I knew observed that Pontius Pilate is recorded to
from previous discussion that he had read, have asked Jesus at his trial, "What is
in a previous edition of this book, this very Truth?" Pilate asked his question, Andrews
section expressing doubt about the utility of observed, "and then some other matter took
truth as a criterion of quality and I suspected him in the head, and so he rose and went his
that he hoped to lure me into an academic- way before he had his answer." While the
question "What is truth?" may be intrinsi- The entire October 2000 issue of the tech-
cally rhetorical and unanswerable, beliefs nology and business magazine Forbes ASAP
about the nature of truth affect how one was devoted to writings on "what is true?"
views the findings of research and evalua- with contributions from 50 people drawn
tions. So the question remains: "Do you, as a from history, science, media, religion, busi-
qualitative researcher, swear to tell the truth, ness, technology, and popular culture. Sur-
the whole truth and nothing but the truth?" prise. They didn't agree.
The researcher applying traditional social Earlier I cited the research of Weiss and
science criteria might respond, "I can show Bucuvalas (1980) that decision makers apply
you truth insofar as it is revealed by the both truth tests and utility tests to evalua-
data." tion. Truth, in this case, however, means rea-
The constructivist might answer: "I can sonably accurate and believable data rather
show you multiple truths." than data that are true in some absolute
The artistically inclined might suggest sense. Sawy policymakers know better than
that "fiction gets at truth better than nonfic- most the context- and perspective-laden na-
tion" and that "beauty is truth." ture of competing truths. Qualitative in-
The criticai theorist could explain that quiry can present accurate data on various
"truth depends on one's consciousness" or perspectives, including the evaluator's per-
the activist might say, "I offer you praxis. spective,' without the burden of determining
Here is where I take my stand. This is true for that only one perspective must be true.
__ _ //
me. Smith (1978), pondering these questions, has
The pragmatic evaluator might reply, "I noted that in order to act in the world we of-
can show you what is useful. What is useful ten accept either approximations to truth or
is true." even untruths:
Finding Truth can be a heavy burden. I
once had a student who was virtually para- For example, when orie drives from city to city,
lyzed in writing an evaluation report be- one acts as if the earth is flat and does not try to
cause he wasn't sure if the patterns he calculate the earth's curva ture in plarining the
thought he had uncovered were really true. I trip, even though acting as if the earth is flat
suggested that he not try to convince himself means acting on an untruth. Therefore, in our
or others that his findings were true in any study of evaluation methodology, two criteria
absolute sense but, rather, that he had done replace exact truth as paramount: practical
the best job he could in describing the pat- utility and levei of certainty. The levei of cer-
terns that appeared to him to be present in tainty required to make an adequate judgment
the data and thathe present those patterns as under the law differs dependmg on whether
his perspective based on his analysis and in- one is considering an administrative hearing,
terpretation of the data he had collected. an inquest, or a criminal case. Although it
Even if he believed that what he eventually seems obvious that much greater certainty
produced was Truth, sophisticated people about the nature of things is required when
reading the report would know that what he legislators set national and educational policy
presented was no more than his perspective, than when a district superintendent decides
and they would judge that perspective by whether to continue a local program, the rhet-
their own commonsense understandings oric in evaluation implies that the same high
and use the information according to how it levei of certainty is required of both cases. If
contributed to their own needs. we were to first determine the levei of cer-
TPUT- V?FSrJ5 ^ i ^ V i N l V )
fliMnuEiii ivi:!'ii i.r OIC-N =:I:I:J.-KS:1 W l-ft.rta.qM.VrnMl v .i iVi:i!.:':|!!':,!!!!:!-I i^-y r:.''

mY.^v;i:: w n-iV V !! :;!!u* r^t vn- WVIIIK.'.':- TI-'"!'I:. .II',
Y> rv.'-f - .. 11 I ! I .!.I !. !.:; I:,:; M n r atai T 1 " i:

, , , 1 j i 1 i ' i I'i I . I ! i: 1 . I.i K I I ! ! T I ;' .::-! : n p i-n-
E lYHJrt .!i ^ I ! . - n :
I V J V . .VL i!i-:!I.
: !.'I!--IVII, ":.11WMX1!tfVXiSVlVII'.?Mij'JI
ivi.':i! IV. !c u-iVei-t !!!!.'! .':; a
- 1
!'=). 7 r. i '.!I 111.111' i. iVi i. Vi '"sv: ' |VI!;I ,;i:; .. M' | l-^V = !i
!i!n.ii.11.!.'i! ftjiVuir >vivc'\ p:i,-p," cpjih:iij: Oir
i vi 11 i. i i.'' i' ' i.-:' I-I.I ii-Vi:! I I:-.-:V,- '-:i i:y-: ;;ri
!.!: iiijjVi'i"I"S:ii'iI1 ,:i.!:;!I;'-:::'. P s;s:r. ih;: P W . V i 'I-1 -I i !! i. I.V i ' ! ' | . V ,V.I 111.I IV! ,:! i , " ! ' !V ii i'.'i \i Vi -
I ! " 1 !I'iV-I |L-DL iVi.Ti:' MD \, \\-A\> ' \>V IH>!!!!R- ! i: I VL" '.-i I.'.!.\:., :: c I! I txli:'.'. I !'!!!.':!:!;! V ri:'/
j.: 11!i!Ri Ili'i!I! :t '!! 1=!:. POJ!'.ii:::s-=-; i".'.'i i" m
: , : r i ^.TiV, !Vi:i-,! =;i 'V! i;!. i! iV- !ifii1!.;!
: ii .: S.r J UMip,\'OM WM ,1"ttl !hf !.!i!!rr:rnI =.Y!:.u!1.=!.'. d.!!ihi:i! ti jflj"
: . n . !' 111=1 11 y? r !=:i ii': s.i-i i :!i vi cvy .I V ; ! ) , .'vvil'
rj 'li1: l.i.'|.-!!1!:!.,.,i! i.I-.vl.-i.VvilJ: Ifl 1"iM
rii" !!,:;!!: ::!!ii!:."::.!:!/ :! :!' .'i !: vi":! r i =
! : :.i<j-J .=i':- :: i .!! :I: s; . !Vl =i" i:!,:;';!:! !'!!! !"
rvii"'VIII-I-vir. , 11'itdi, :.I!=:' SCUPLI .H.' 1: M - W \v 11! : 11 l ! ! irs l.!i.i : :i i-!i !::, ! T! !;! I !.:i!,
!! !ii li i.'i'iii" ' ii.!! i! ij 11':!!!; I ; o.' j. sfc-r n r -:=-:=" ii ri' i' -i 11 i ,<!|. .v/iVi ! ! i.:!!|.i i i I i 'I i!' i ! 11 j '.'Vi
IL. 1 !!:VL:'I!.'I - .RVI :.=," I V I . !IFC 4 9 I I !i-l1 i - i I'j1 11 l' V '"!'. I i I ! 1 i I! "i! i ! i I i<' i'! i ii i i i' ii i!1'!1
KMrt!.,'!' i';.Vil.v, i1 !.*iji.'i.'",!1-!1 i-i=i .';i:- Mi* ' i'. i" 1 - nn 11rr.i wj'::i! I:!i n1 i!'!:!:!!!:; 21!^
rii'rlui.'rl Li.! i.'!.'iUi t :if:=I!
tainty desired in a specific case, we could then and contextual approach to truth in empha-
more easily choose appropriate methods. Nat- sizing the "pragmatic validation" of find-
uralistic descriptions give us greater certainty ings in which the results of qualitative analy-
in our understanding of the nature of an edu- sis are judged by their relevance to and use
cational process than randomized, controlled by those to whom findings are presented.
experiments do, but less certainty in our This criterion of utility can be applied not
knowledge of the strength of a particular ef- only to evaluation but also to qualitative
fect [0]ur first concern should be the prac- analyses of ali kinds, including textual anal-
tical utility of our knowledge, not its ultimate ysis. Barone (2000), having rejected objectiv-
truthfulness. (p. 17) ity and subjectivity as meaningless criteria
in the postmodern age, makes the case for
In studying evaluation use (Patton pragmatic utility as follows:
1997a), I found that decision makers did not
expect evaluation reports to produce
"Truth" in any fundamental sense. Rather, If ali discourse is culturally contextual, how
they viewed evaluation findings as addi- do we decide which deserves our attention
tional information that they could and did and respect? The pragmatists offer the crite-
combine with other information (political, rion of usefulness for this p u r p o s e Anidea,
experiential, other research, colleague opin- like a tool, has no intrinsic value and is "true"
ions, etc.), ali of which fed into a slow, evolu- only in its capacity to perform a desired ser-
tionary process of incrementai decision vice for its handler within a given situation.
making. Kvale (1987) echoed this interactive When the criterion of usefulness is applied to
context-bound, historically situated transac- treason by the king's ministers, wise men
tions between itself and a text, it helps us to charged with advising on matters of great
judge which textual experiences are to be vai- import. Nasrudin was charged with going
ued. . . . The gates are opened for textual en- from village to village inciting the people by
counters, in any inquiry genre or tradition, saying, "The king's wise men do not speak
that serve to fulfill an important human pur- truth. They do not even know what truth is.
pose. (pp. 169-70) They are confused." Nasrudin was brought
before the king and the court. "How do you
plead, guilty or not guilty?"
Focusing on the connection between
"I am both guilty and not guilty," replied
truth tests and utility tests shifts attention
Nasrudin.
back to credibility and quality, not as abso-
"What, then, is your defense?"
lute generalizable judgments but as contex-
Nasrudin turned and pointed to the nine
tually dependent on the needs and interests
wise men who were assembled in the court.
of those receiving our analysis. This obliges
"Have each sage write an answer to the fol-
researchers and evaluators to consider care-
fully how they present their work to others, lowing question: 'What is water?' "
with attention to the purpose to be fulfilled. The king commanded the sages to do as
That presentation should include reflections they were asked. The answers were handed
on how your perspective affected the ques- to the king, who read to the court what each
tions you pursued in fieldwork, careful doc- sage had written.
umentation of ali procedures used so that The first wrote: "Water is to remove
others can review your methods for bias, thirst."
and being open in describing the limita tions The second: "It is the essence of life."
of the perspective presented. Appendix 9.1 The third: "Rain."
contains an in-depth description of how one The fourth: "A clear, liquid substance."
qualitative inquirer, engaged in program The fifth: "A compound of hydrogen and
"documentation," dealt with these issues in oxygen."
a long-term participant observer relation- The sixth: "Water was given to us by God
ship. This excerpt, titled "A Documenter's to use in cleansing and purifying ourselves
Perspective," is based on the documenteis before prayer."
research journal and field notes. It moves the The seventh: "It is many different
discussion from abstract philosophizing to thingsrivers, wells, ice, lakes, so it de-
day-to-day, in-the-trenches fieldwork en- pends."
counters aimed at sorting out what is true The eighth: "A marvelous mystery that
(small t) and useful. defies definition."
As one additional source of reflection on The ninth: "The poor man's wine."
these issues, perhaps the following Sufi Nasrudin turned to the court and the
story will provide some guidance about the king, "I am guilty of saying that the wise
difference between truth and perspective. men are confused. I am not, however, guilty
Sagely, in this encounter, Nasrudin gathers of treason because, as you see, the wise men
data to support his proposition about the na- are confused. How can they know if I have
ture of truth. Here's the story. committed treason if they cannot even de-
Mulla Nasrudin was on trial for his life. cide what water is? If the sages cannot agree
He was accused of no less a crime than on the truth about water, something they
consume every day, how can one expect The king ordered that Nasrudin be set
that they can know the truth about other free.
things?"
From Generalizations to Extrapolations and Transferability
T he trouble with generalizations is that they don't apply to particulars.
Yvonna S. Lincoln and Egon G. Guba (1985:110)
case studies share the problem of being

The pragmatic criterion of utility leads to
highly localized. Findings from a study,
the question of what one can do with quali-
experimental or naturalistic in design, can
tative findings. Certainly, the results illumi-
be generalized according to five princi-
nate a particular si tua tion or small number
pies:
of cases. But what of utility beyond the lim-
ited case or cases studied? Can qualitative
1. The Principie of Prximal Similarity. We
findings be generalized?
generalize most confidently to appli-
Chapter 5 discussed the logic and value of
cations where treatments, settings,
purposeful sampling with small, but care-
populations, outeomes, and times are
fully selected, information-rich cases. Cer-
most similar to those in the original re-
tain kinds of small samples, for example, a
search. . . .
criticai case, are selected and studied pre-
cisely because they have broader relevance. 2. The Principie of Heterogeneity of Irrele-
Other sampling strategies, for example, ex- vancies. We generalize most confi-
treme cases (exemplars of excellence or fail- dently when a research finding contin-
ure), are selected for their potential to yield ues to hold over variations in persons,
insights about principies that might be ap- settings, treatments, outcome mea-
plied elsewhere. However, purposeful sam- sures, and times that are presumed to
pling is not widely understood. Thus, quali- be conceptually irrelevant. The strat-
tative inquirers may encounter a predis- egy here is identifying irrelevancies,
position toward large, random samples and and where possible including a di-
disbelief in (or ignorance about) the value of verse array of them in the research so
small, purposeful samples. It is important in as to demonstrate generalization over
responding to such concerns that one fully them....
understands the relative strengths and
weaknesses of different sampling strategies. 3. The Principie of Discriminant Validity.
Nor are qualitative and quantitative sam- We generalize most confidently when
ples incompatible. Chapter 5 discusses sev- we can show that it is the target con-
eral mutually reinforcing combinations. struct, and not something else, that is
necessary to produce a research find-
Shadish (1995a) argued that the core prin-
ing. .. .
cipies of generalization apply to both experi-
ments and ethnographies (or qualitative 4. The Principie of Empirical Interpolation
methods generally). Both experiments and and Extrapolation. We generalize most
confidently when we can specify the In suggesting that generalizations have

range of persons, settings, treatinents, not stood up well in the sciences, Cronbach
outcomes, and times over which the (1975) offered an alternative strategy that
finding holds more strongly, less constitutes excellent advice for the qualita-
strongly, or not ali. The strategy here is tive analyst:
empirical exploration of the existing
range of instances to discover how that Instead of making generalization the ruling
range might generate variability in the consideration in our research, I suggest that
finding for instances not studied we reverse our priorities. An observer collect-
5. The Principie of Explanation. We general- ing data in a particular situation is in a posi-
ize most confidently when we can spection to appraise a practice or proposition in
ify completely and exactly (a) which that setting, observing effects in context. In
parts of one variable (b) are related to trying to describe and account for what hap-
which parts of another variable (c) pened, he will give attention to whatever vari-
through which mediating processes (d) ables were controlled, but he will give equally
with which salient interactions, for then careful attention to uncontrolled conditions,
we can transfer only those essential to personal characteristics, and to events that
components to the new application to occurred during treatment and measurement.
which we wish to generalize. The strat- As he goes from situation to situation, his first
egy here is breaking down the finding task is to describe and interpret the effect anew
into component parts and processes so in each locale, perhaps taking into account fac-
as to identify the essential ones. tors unique to that locale or series of events
(Shadish 1995a: 424-26) When we give proper weight to local condi-
tions, any generalization is a working hypoth-
Still, deeper philosophical and episte- esis, not a conclusion. (pp. 124-25)
mological issues are embedded in concerns

about generalizing. What's desirable or Robert Stake (1978,1995,2000), master of
hoped for in science (generalizations across case methods, concurs with Cronbach that
time and space) runs into considerations the first priority is to do justice to the specific
about what's possible. Lee J. Cronbach case, to do a good job of "particularization"
(1975), one of the major figures in psy- before looking for patterns across cases. He
chometrics and research methodology, has quotes William Blake on the subject: "To
given considerable attention to the issue of generalize is tobe an idiot. To particularize is
generalizations. He has concluded that so- the lone distinction of merit. General
cial phenomena are too variable and context knowledges are those that idiots possess."
bound to permit very significant empirical Stake (1978) continued:
generalizations. Cronbach also compared
generalizations in natural sciences with Generalization may not be ali that despicable,
what was likely to be possible in behavioral but particularization does deserve praise. To
and social sciences. His conclusion is that know particulars fleetingly, of course, is to
"generalizations decay. At one time a con- know next to nothing. What becomes useful
clusion describes the existing situation well, understanding is a full and thorough knowl-
at a later time it accounts for rather little vari- edge of the particular, recognizing it also in
ance, and ultimately is valid only as history" new and foreign contexts. That knowledge is a
(p. 122). form of generalization too, not scientific in-
duction but naturalistic generalization, ar- Guba (1978) considered three alternative
rived at by recognizing the similarities of positions that might be taken in regard to the
objects and issues in and out of context and by generalizability of naturalistic inquiry find-
sensing the natural covariations of happenings:
ings. To generalize this way is to be both intu-
itive and empirical, and not idiotic. (p. 6) 1. Generalizability is a chimera; it is im-
possible to generalize in a scientific
sense at a l i . . . .
Stake extends "naturalistic generaliza-
tions" to include the kind of learning that 2. Generalizability continues to be impor-
readers take from their encounters with spe- tant, and efforts should be made to meet
cific case studies. The "vicarious experi- normal scientific criteria that pertain to
ence" that comes from reading a rich case it
account can contribute to the social con- 3. Generalizability is a fragile concept
struction of knowledge that, in a cumulative whose meaning is ambiguous and
sense, builds general, if not necessarily whose power is variable. (pp. 68-70)
generalizable, knowledge.
Having reviewed these three positions,
Readers assimilate certain descriptions and Guba proposed a resolution that recognizes
assertions into memory. When a researcher's the diminished value and changed meaning
narrative provides opportunity for vicarious of generalizations and echoes Cronbach's
experience, readers extend their memories of emphasis, cited above, on treating conclu-
happenings. Naturalistic, ethnographic case sions as hypotheses for future applicability
materiais, to some extent, parallel actual expe- and testing rather than as definitive.
rience, feeding into the most fundamental
processes of awareness and understanding... The evaluator should do what he can to estab-
[to permit] naturalistic generalizations. The lish the generalizability of his findings Of-
reader comes to know some things told, as if ten naturalistic inquiry can establish at least
he or she had experienced it. Enduring mean- the "limiting cases" relevant to a given situa-
ings come from encounter, and are modified tion. But in the spirit of naturalistic inquiry he
and reinforced by repeated encounter. should regard each possible generalization
In life itself, this occurs seldom to the indi- only as a working hypothesis, to be tested
vidual alone but in the presence of others. In a again in the next encounter and again in the
social process, together they bend, spin, con- encounter after that. For the naturalistic in-
solida te, and enrich their understandings. We quiry evaluator, premature closure is a cardi-
come to know what has happened partly in nal sin, and tolerance of ambiguity a virtue.
terms of what others reveal as their experi- (Guba 1978:70)
ence. The case researcher emerges from one
social experience, the observation, to choreo- Guba and Lincoln (1981) emphasized ap-
graph another, the report. Knowledge is so- preciation of and attention to context as a
cially constructed, so we constructivists natural limit to naturalistic generalizations.
believe, and, in their experiential and contex- They ask, "What can a generalization be ex-
tual accounts, case study researchers assist cept an assertion that is context free? [Yet] it
readers in the construction of knowledge. is virtually impossible to imagine any human be-
(Stake 2000:442) havior that is not heavily mediated by the context
in which it occurs" (p. 62). They proposed be particularly useful when based on infor-
substituting the concepts "transferability" mation-rich samples and designs, that is,
and "fittingness" for generalization when studies that produce relevant information
dealing with qualitative findings: carefully targeted to specific concerns about
both the present and the future. Users of
The degree of transferability is a direct function evaluation, for example, will usually expect
of the similarity between the two contexts, evaluators to thoughtfully extrapolate from
what we shall call "fittingness." Fittingness is their findings in the sense of pointing out
defined as degree of congruence between lessons learned and potential applications to
sending and receiving contexts. If context A future efforts. Sampling strategies in quali-
and context B are "suffciently" congruent, tative evaluations can be planned with the
then working hypotheses from the sending stakeholders' desire for extrapolation in
origina ting context may be applicable in the mind.
receiving context. (Lincoln and Guba
1985:124)
S The Credibility Issue
in Retrospect: Increased
Cronbach and Associates (1980) have of-
fered a middle ground in the methodologi- Legitimacy for Qualitative
cal paradigms debate over generalizability, Methods
They found little value in experimental de- Beyond the Qualitative-
signs that are so focused on carefully con- Quantitative Debate
trolling cause and effect (internai validity)
that the findings are largely irrelevant be-
This chapter has reviewed ways of en-
yond that highly controlled experimental
hancing the quality and credibility of quali-
situation (externai validity). On the other
tative analysis by dealing with three distinct
hand, they were equally concerned about
but related inquiry concerns:
entirely idiosyncratic case studies that yield
little of use beyond the case study setting.
o rigorous methods for doing fieldwork that
They were also skeptical that highly specific
yield high-quality data that are system-
empirical findings would be meaningful un-
atically analyzed with attention to issues
der new conditions. They suggested instead
of credibility;
that designs balance depth and breadth, re-
alism and control so as to permit reasonable the credibility ofthe researcher, which is de-
"extrapolation" (pp. 231-35). pendent on training, experience, track
Unlike the usual meaning of the term gen- record, status, and presentation of self;
eralization, an extrapolation clearly connotes and
that one has gone beyond the narrow con- philosophical beliefin the value of qualitative
fines of the data to think about other applica- inquiry, that is, a fundamental apprecia-
tions of the findings. Extrapolations are tion of naturalistic inquiry, qualitative
modest speculations on the likely applica- methods, inductive analysis, purposefui
bility of findings to other situations under sampling, and holistic thinking.
similar, but not identical, conditions. Extrap-
olations are logical, thoughtful, case de- The debate between qualitative and
rived, and problem oriented rather than sta- quantitative methodologists has often been
tistical and probabilistic. Extrapolations can strident. In recent years, the debate has soft-
ened. A consensus has gradually emerged evaluations focused largely on quantitative

that the important challenge is to appro- measurement of clear, specific goals and ob-
priately match methods to purposes, ques- jectives. With the widespread social and ed-
tions, and issues and not to universally ad- ucational experimentation of the 1960s and
vocate any single methodological approach early 1970s, evaluation designs were aimed
for ali inquiry situations. Indeed, eminent at comparing the effectiveness of different
methodologist Thomas Cook, one of eval- programs and treatments through rigorous
uation^ luminaries, pronounced in his controls and experiments. This was the pe-
keynote address to the 1995 International riod when the quantitative/experimental
Evaluation Conference in Vancouver that paradigm dominated. By the middle 1970s,
"qualitative researchers have won the quali- the paradigms debate had become a major
tative-quantitative debate/' focus of evaluation discussions and writ-
Won in what sense? ings. By the late 1970s, the alternative quali-
Won acceptance. tative/ naturalistic paradigm had been fully
The validity of experimental methods articulated (Guba 1978; Patton 1978; Stake
and quantitative measurement, appropri- 1975, 1978). During this period, concern
ately used, was never in doubt. Now, quali- about finding ways to increase use became
tative methods have ascended to a levei of predominant in evaluation, and evaluators
parallel respectability. I have found in- began discussing standards. A period of
creased inferest in and acceptance of qualita- pragmatism and dialogue followed, during
tive methods in particular and multiple which calls for and experiences with multi-
methods in general. Especially in evalua- ple methods and a synthesis of paradigms
tion, a consensus has emerged that research- became more common. The advice of
ers and evaluators need to know and use a Cronbach and Associates (1980), in their im-
variety of methods to be responsive to the portant book on reform of program evalua-
nuances of particular empirical questions tion, was widely taken to heart: "The evalua-
and. the idiosyncrasies of specific stake- tor will be wise not to declare allegiance to
holder needs. either a quantitative-scientific-summative
The credibility and respectability of qual- methodology or a qualitative-naturalistic-
itative methods vary across disciplines, uni- descriptive methodology" (p. 7).
versity departments, professions, time peri- Signs of dtente and pragmatism now
ods, and countries. In the field I know best, abound. Methodological tolerance, flexibil-
program evaluation, the increased legiti- ity, eclecticism, and concern for appropriate-
macy of qualitative methods is a function of ness rather than orthodoxy now character-
more examples of useful, high-quality eval- ize the practice, literature, and discussions
uations employing qualitative methods and of evaluation. Several developments seem to
an increased commitment to providing use- me to explain the withering of the method-
ful and understandable information based ological paradigms debate.
on stakeholders' concerns. Other factors that
contribute to increased credibility include 1. The articulation of professional stan-
more and higher-quality training in qualita- dards has emphasized methodological ap-
tive methods and the publication of a sub- propriateness rather than paradigm ortho-
stantial qualitative literature. doxy (Joint Committee 1994). Within the
The history of the paradigms debate par- standards as context, the focus on conduct-
allels the history of evaluation. The earliest ing evaluations that are useful, practical,
ethical, and accurate, and accumulation of Moreover, the upshot of ali the develop-
practical evaluation experience during the mental work in qualitative methods is that,
past two decades, has reduced paradigms as documented in Chapter 3, today there is
polarization. as much variation among qualitative re-
searchers as there is between qualitatively
2. The strengths and weaknesses of both and quantitatively oriented scholars.
quantitative/experimental methods and
qualitative/naturalistic methods are now
5. Support for methodological eclecti-
better understood. In the original debate,
cism from major figures and institutions in
quantitative methodologists tended to at-
evaluation increased methodological toler-
tack some of the worst examples of qualita-
ance. When eminent measurement and
tive evaluations while the qualitative evalu-
methods scholars such as Donald Campbell
ators tended to hold up for critique the
and Lee ). Cronbach began publicly recog-
worst examples of quantitative/experi-
nizing the contributions that qualitative
mental approaches. With the accumulation
methods could make, the acceptability of
of experience and confidence, exemplars of
qualitative/naturalistic approaches was
both qualitative and quantitative ap-
greatly enhanced. Another important en-
proaches have emerged with corresponding
dorsement of multiple methods came from
analyses of the strengths and weaknesses of
the Program Evaluation and Methodology
each. This has permitted more balance and a
Division of the U.S. General Accounting Of-
better understanding of the situations for
fice (GAO), which arguably did the most
which various methods are most appropri-
important and influential evaluation work
ate as well as grounded experience in how
at the national levei. Under the leadership of
to combine methods.
Assistant Comptroller General and former
3. A broader conceptualization of evalu- American Evaluation Associationpresident
ation, and of evaluator training, has di- (1995) Eleanor Chelimsky, the GAO pub-
rected attention to the rela tion of methods to lished a series of methods manuais includ-
other aspects of evaluation, such as use, and ing Case Study Evaluations (1987), Prospective
has therefore reduced the intensity of the Methods (1989), and The Evaluation Synthesis
methods debate as a topic unto itself. (1992). The GAO manual Desigtiing Evalua-
Methods decisions are now framed in a tions (1991) puts the paradigms debate to
broader context of use that, I believe, has re- rest as it describes what constitutes a
duced the intensity of the paradigms de- "strong evaluation":
bate, a debate that often went on in absolute,
context-free terms. Strength is not judged by adherence to a par-
ticular paradigm. It is determined by use and
4. Advances in methodological sophisti- technical adequacy, whatever the method,
cation and diversity within both paradigms within the context of purpose, time and re-
have strengthened diverse applications to sources.
evaluation problems. The prolifera tion of Strong evaluations employ methods of
books and journals in evaluation, including analysis that are appropriate to the question,
but not limited to methods contributions, support the answer with evidence, document
has converted the field into a rich mosaic the assumptions, procedures, and modes of
that cannot be reduced to quantitative ver- analysis, and rule out the competing evidence.
sus qualitative in primary orientation. Strong studies pose questions clearly, address
them appropriately, and draw inferences com- 7. There is increased advocacy of and ex-
mensurate with the power of the design and perience in combining qualitative and
the availability, validity, and reliability of the quantitative approaches. The Reichardt
data. Strength should not be equated with and Rallis (1994) volume The Qualitative-
complexity. Nor should strength be equated Quantitative Debate: New Perspectives just
with the degree of statistical manipulation of cited also included these themes: "blended
data. Neither infatuation with complexity nor approaches," "integrating the qualitative
statistical incantation makes an evaluation and quantitative," "possibilities for integra-
stronger. tion," "qualitative plus quantitative," and
The strength of an evaluation is not de- "working together." (For elaboration of
fined by a particular method. Longitudinal, these reasons for the withering of the para-
experimental, quasi-experimental, before- digms debate, see Patton 1997a: 290-99.)
and-after, and case study evaluations can be
either strong or w e a k . . . . That is, the strength
of an evaluation has to be judged within the
Matching Claims
context of the question, the time and cost con- and Criteria
straints, the design, the technical adequacy of The withering of the methodological par-
the data collection and analysis, and the pre- adigms debate holds out the hope that stud-
sentation of the findings. A strong study is ies of ali kinds can be judged on their merits
technically adequate andusefulinshort, it according to the claims they make and the
is high in quality. (pp. 15-16) evidence marshaled in support of those
claims. The thing that distinguished the five
6. Evaluation professional societies have sets of criteria introduced at the beginning of
supported exchanges of views and high- this chapter is that they support different
quality professional practice in an environ- kinds of claims. Traditional scientific claims,
ment of tolerance and eclecticism. The eval- constructivist claims, artistic claims, criticai
uation professional societies and journals change claims, and evaluation claims will
serve a variety of people from different dis- tend to emphasize different kinds of conclu-
ciplines who operate in different kinds of or- sions with varying implica tions. In judging
ganiza tions at different leveis, in and out of claims and conclusions, validity of the
the public sector, and m and out of universi- claims made is only partly related to the
ties. This diversity, and opportunities to ex- methods used in the process.
change views and perspectives, has contrib-
uted to the emergent pragmatism, eclec- Validity is a property of knowledge, not meth-
ticism, and tolerance in the field. A good ex- ods. No matter whether knowledge comes
ample is the volume of New Directions for from an ethnography or an experiment, we
Program Evaluation titled "The Qualitative- may still ask the same kind of questions about
Quantitative Debate: New Perspectives" the ways in which that knowledge is valid. To
(Reichardt and Rallis 1994). The tone of the use an overly simplistic example, if someone
eight distinguished contributions in that claims to have nailed together two boards, we
volume is captured by such phrases as do not ask if their hammer is valid, but rather
"peaceful coexistence," "each tradition can whether the two boards are now nailed to-
learn from the other," "compromise solu- gether, and whether the claimant was, in fact,
tion," "important shared characteristics," responsible for that result. In fact, this particu-
and "a call for a new partnership." lar claim may be valid whether the nail was set
BF^LITY IS AMEtiflUUSi
i::i|'!!'i!,i'!!ri,-i ri'* n !i.':n;r>'i m,' rV'..t!" : r I.V ri.;; .* or!!!::!!':! W '!!!:-S ! . V i . - y Mr
-rv/ JEi i:'.-! .:; ri; i'i< =." iy iVil" " i V V1 ! j-' I; (M.i II'I i' I.! IV II ! ! I;!.; " I - N " I 1 y\ >.}!
li '' ii iTiV i:! iy I" i.:v r=' > i.V !:'i.! i.',-i" i -1 -T r- ! I\ V I M RI! . ' J X ' !T:i '' L :L .!>!! ;;!!: H H : I I :!- 1
I ' ! i! 'rij.iiii :.'" !.I i..ii-I i:I!'I.I V I ! .'I <!':' = N R

.Vi :!!!.!!n:!.!iI';! ^ /!UiM; Sn'ihii"r j<fliu.'i'W- 11
!.. . ... !'is

! A- iJri ! ; i'!': n.'':" i " i i V != ' r=i"
aUl r !:',-.'
!
IliilliillIilllilliliiiliiiliiilISlII
i .tttrf I i.> >:t ! :V 11 |'.! K . VI.' i r!.- :.-i! 1 ! ' ,ID !.=:i<v.' iî =ViVii'iiivf-
.rhA' !' "jt! ! .;rhri |i':| n :'.':!':=;;! rrri/i !y i.;: an: .iii!.!!.iAV-i' 'ii ij',I:"V'Vi!;!"i"i ii*i
1
: n; n/n.j Du . . j=V;'i il r* iriJ i= iia-Ii hY& i ! ',y i i: îcn^ i ! n'" ;\ YI.;:

!':ij :.':! r iYri ilV! * i.L'. nV.i i.u=.:n "" ; f : !i i'.' .i :'i i.R-<? iri' A i.Vi , I.i i.i I !. ! ! J - .Vi
: !
ijli;!.!!^ :;!::hi".,i: ::V;I".:. nivr i Vi/pi \y g-r!.;;' .; i: i:'!.i >. i!'i '' ' h Vi" V:':-i in." r! ; !:-; vivi
:!!"!:! hVixi ,!:!:!.: !n: iViv i-ms-n:' rfw-j ri Vh Vi = ;, I : ; I.i:;ir> J F 1! : ':' ;'!
.:.':!: .iiXii:^ !!'?:; J.:i .!.':.I ! ri:" N >:, ;,Vi iy : R - '*.i!,i':;:i: i'(' \> tyii .'>.-WTV !
j:a|r.Vi I !i> jAr/y-ril V: jh: .v; Ti srfai h i! 'i'i VI Vi i'.i i il a i' r-ii ::!.!! iVi.1 i-vr; ,!" \ i -,;;
ctiVitf.r i!i'i!i';!!'iii- <M: " Mtfi ':'.!;=' :i:'i v. i i o h" " i i: ;: ':.|: C>'.\ OiV!'!!'(! slVi!
L I L ^ "sj\ :;ii: >.'ii'- i".-:^:;T;:"n:T!:<ViM n ! '
: ! I:'' : '
! ! : V* - m . . : I..'
! : A\\- - P -
lijrtTOT =M
: : h i! - '!:'i''::i i' i? * ' ctf
! i ! , , ! i ^ i :i i.ii.i V i! V V*I i ' i i " i Vi ;TV; {.": =-. rv!;; '
!
s=== 1 = ; !=:; . ifeij: i i i ii'O ilV.i il'i i.V '.i", !^ !.!/' i:i:Vi'!i-;ri|iri\:yiri-.ii! ! V:i.A
i: !.I.!.V!iV|!' h ;' :''"
, : ! I; '::;-; V ^ : ' . <" . I ^ ' ^ T L L T I K N B I H I I
! !
i'V-Li-Vi !.; i; !'l rr ri !.5. :.! n.-n !:- jm !.;.!' ir s i' iYi " Ji i i i. i-i Vi ij' i! !! !i'r:j!. /!T!: =! vil;' ' "-r
!.;: i !:=; \\ n.^nV ; tf, i: j.vf -:;
! i >: iVi i - ptf H/i h" :":1.;! ITI? LYOC i'^ i1
VO.Ky lVsh fla fsA Vi !,:<! ni ik irn-vi JjKWifci^y ^' !i i'Vvi'i'! ! 'i- :.i' 1 ilTiV.
in place by a hammer, an airgun, or the butt of dence carefully and be open to the possi-
a screwdriver, A hammer does not guarantee bility that what has been learned most
successful nailing, successful nailing does not from a particular inquiry is how to do it
require a hammer, and the validity of the claim better next time. Canadian-bom bacteriol-
is in principie separate from which tool was ogist Oswald Avery, discoverer of DNA as
used. The same is true of methods in the social the basic genetic material of the cell,
behavioral sciences. (Shadish 1995a:421) worked for years in a small laboratory at
the hospital of the Rockefeller Institute in
This brings us back to a pragmatic focus New York City. Many of his initial hypoth-
on the utility of findings as a point of entry eses and research conclusions turned out,
for determirdng what's at stake in the claims upon further investiga tion, to be wrong.
made in a study and therefore what criteria His colleagues marveled that he never
to use in assessing those claims. As I noted in turned argumentative when findings
opening this chapter, judgments about cred- countered his predctions and he never be-
ibility and quality depend on criteria. And came discouraged. He was committed to
though this chapter has been devoted to learning and was often heard telling his
ways of enhancing quality and credibility, students: "Whenever you fali, pick up
ali such efforts ultimately depend on the something."
willingness of the inquirer to weigh the evi-
APPENDIX 9.1
5 S S
Case Study: A Documenter's Perspective
Introduction. This appendix provides a reflective case study, by Beth Alberty, of

the struggle experienced by one internai, forma tive program evaluator of an in-
novative school art program as she tried to figure out how to provide useful in-
formation to program staff from the voluminous qualitative data she collected.
Beth begins by describing what she means by "documentation" and then shares
her experiences as a no vice in analyzing the data, a process of moving from a
mass of documentary material to a unified, holistic document.
Documentation
Documentation, as the word is commonly used, may refer to "slice of life"

recordings in various media or to the marshaling of evidence in support of a
position or point of view. We are familiar with "documentary" films; we re-
quire lawyers or journalists to "document" their cases. Both meanings con-
tribute to my view of what documentation is, but they are far from describing
it fully. Documentation, to my mind, is the interpretive reconstitution of a fo-
cai event, setting, project, or other phenomenon, based on observation and on
descriptive records setin the context of guiding purposes and commitments.
I have always been a staff member of the situations I have documented,
rather than a consultant or an employee of an evaluation organization. At
first this was by accident, but now it is by conviction: My experience urges
that the most meaningful evaluation of a program's goals and commitments
is one that is planned and carried out by the staff and that such an evaluation
contributes to the program as well as to externai needs for information. As a
staff member, I participate in staff meetings and contribute to decisions. My
relationships with other staff members are close and reciprocai. Sometimes I
provide services or per form functions that directly fulfill the purposes of the
programfor example, working with children or adults, answering visitors'
questions, writing proposals and reports. Most of my time, however, is spent
planning, collecting, reporting, and analyzing documentation.
First Perceptions
With this context in mind, let me turn to the beginning plunge. Observing
is the heart of documenting and it was into observing that I plunged, coming
up delighted at the apparent ease and swiftness with which I could fish in-
5 9 0 tI. ANALYSIS, INTERPRETATION, AND REPORTING
sight and ideas from the ceaseless ocean of activity around me. Indeed, the fact
that observing (and record-keeping) does generate questions, insight, and mat-
ters for discussion is one of many reasons why records for any documentation
should be gathered by those who actually work in the setting.
My observing took many forms, each offering a different way of releasing
questions and ideasinteractive and noninteractive observations were tran-
scribed or discussed with other staff members, and thereby rethought; chil-
dren^ writing was typed out, the attention to every detail involving me in what
the child was saying; notes of meetings and other events were rewritten for the
record; and so on. Handling such detail with attention, I found, enabled me to
see into the incident or piece of work m a way I hadn't on first look. Connec tions
with other things I knew, with other observations I made, or questions I was
puzzling over seemed to prolifera te during these processes; new perceptions
and new questions began to form.
I have heard others describe similarly their delighted discovery of the pro-
vocativeness of record-keeping processes. The teacher who begins to collect
children's art, without perhaps even having a particular reason for the collect-
ing, will, just by gathering the work together, begin to notice things about them
that he or she had not seen beforehow one child's work influences another's,
how really different (or similar) are the trees they make, and so on. The
in-school advisor or resource teacher who reviews ali his or her contacts with
teachersas they are recorded or in a special meeting with his or her col-
leaguesmay begin, for example, to see patterns of similar interest m the re-
quests he or she is getting and thus become aware of new possibilities for rela-
tionships within the school.
My own delight in this apparently easy access to a first levei of insight made
me eager to collect more and more, and I also found the sheer bulk of what I
could collect satisfying. As I collected more records, however, my enthusiasm
gradually changed to alarm and frustration. There were so many things that
could be observed and recorded, so many perspectives, such a complicated his-
tory! My feelings of wanting more changed to a feeling of needmg to get every-
thing. It wasn't enough for me to know how the program worked nowI felt I
needed to know how it got started and how the present workings had evolved.
It wasn't enough to know how the central part of the program workedI felt I
had to know about ali its spinoff activities and from ali points of view. I was
quickly drawn into a fear of losing something significant, something I might
need later on. Likewise, in my early observations of class sessions, I sought to
write down everything I saw. I have had this experience of wanting to get ev-
erything in every setting in which I have documented, and I think it is not
unique.
I was fortunate enough to be able to indulge these feelings and to learn from
where they led me. It did become clear to me after a while that my early ambi-
tions for documenting everything far exceeded my time and, mdeed, the needs
of the program. Nevertheless, there was a sense to them. Collecting so much
Enhancing Quality andCredibilitylJ.615
was a way of getting to know a new setting, of orienting myself. And, not know-
ing the setting, I couldn't know what would turn out to be important in "recon-
stituting" it; also, the purpose of "reconstituting" it was sufficiently broad to in-
clude any number of possibilities from which I had not yet selected. In fact, I
found that the first insights, the first connections that came from gathering the
records were a significant part of the process of determining what would be im-
portant and what were the possibilities most suited to the purposes of the docu-
mentation. The process of gathering everything at first turned out to be impor-
tant and, I think, needs to be allowed for at the beginning of any documenting
effort. Even though much of the material so gathered may remain apparently
unused, as it was in my documenting, in fact it has served its purpose just in be-
ing collected. A similar process may be required even when the documenter is
already familiar with the setting, since the new role entails a new perspective.
The first connections, the first pattems emerging from the accumulating re-
cords were thus a valuable aspect of the documenting process. There came a
moment, however, when the data I had collected seemed more massive than
was justified by any thought I'd had as a result of the collecting. I was ill at ease
because the first pattems were still fairly unformed and were not automatically
turning into a documentation in the full sense I gave earlier, even though I rec-
ognized them as part of the documentary data. Particularly, they did not func-
tion as "evaluation." Some further development was needed, but what? "What
do I do with them now?" is a cry I have heard regularly since then from teachers
and others who have been collecting records for a while.
I began with the relatively simple procedure of rereading everything I had
gathered. Then I returned to rethink what my purposes were, and sought out
my original resources on documentation. Rereading qualitative references,
talking with the staff of the school and with my staff colleagues, I began to
imagine a shape 1 could give to my records that would make a coherent repre-
sentation of the program to an outside audience.
At the same time I began to rethink how I could make what I had collected
more useful to the staff. Conceiving an audience was very important at this
stage. I will be retuming to this moment of transition from initial collecting to
rethinking later, to analyze the entry into interpretation that it entails. Descrip-
tively, however, what occurred was that I began to see my observations and re-
cords as a body with its own configura tions, interrelationships, and possibili-
ties, rather than simply as excerpts of the larger program that related only to the
program. Obviously, the observations and records continued to have meaning
through their primary relationship to the setting in which they were made, but
they also began to have meaning through their secondary relationships to each
other.
These secondary relationships also emerge from observation as a process of
reflecting. Here, however, the focus of observation is the setting as it appears in
and through the observations and records that have accumulated, with ali their
representation of multiple perspectives and longitudinal dimensions. These
observations in and through records"thickened observations"are of

course confirmed and added toby continuing direct observation of the setting.
Beginning to see the records as a body and the setting through thickened ob-
servation is a process of integrating data. The process occurs gradually and re-
quires a broad base of observation about many aspects of the program over
some period of time. It then requires concentrated and systematic efforts to find
connections within the data and weave them into patterns, to notice changes in
what is reported, and find the relationship of changes to what remains con-
stant. This process is supported by juxtaposing the observations and records in
various ways, as well as by continuai return to reobserve the original phenome-
non. There is, in my opinion, no way to speed up the process of documenting.
Reflectiveness takes time.
In retrospect I can identify my own approach to an integration of the data as
the time when I began to give my opinions on long-range decisions and inter-
pretations of daily events with the ease of any other staff member. Up to the mo-
ment of transition, I shared specific observations from the records and talked
them over as a way of gathering yet more perspectives on what was happening.
I was aware, however, that my opinions or interpretations were still personal.
They did not yet represent the material 1 was collecting.
Thus, it may be that integration of the documentary material becomes ap-
parent when the documenter begins to evince a broad perspective about what
is being documented, a perspective that makes what has been gathered avail-
able to others withoutprecluding their ownperceptions. This perspective is not
a fixed-point view of a finished picture, both the view and the picture con-
structed somehow by the documenter in private and then unveiled with a
flourish. It is also not a personal opinion; nor does it arise from placing a prede-
termined interpretive structure or standard on the observations. The perspec-
tive results from the documenter's own currentbest integration of the many as-
pects of the phenomenon, of the teachers' or staff's aims, ideas, and current
struggles, and of their historical development as these have been conveyed in
the actions that have been observed and the records that have been collected.
As documenter, my perspective of a program or a classroom is like my per-
spective of a landscape. The longer I am in it, the sharper defined become its
features, its hills and valleys, forests and fields, and the folds of distance; the
more colorful and yet deeply shaded and nuanced in tone it appears; the more
my memory of how it looks in other weather, under other skies, and in other
seasons, and my knowledge of its living parts, its minute detail, and its history
deepen my viewing and valuing of it at any moment. This landscape has con-
stancy in its basic configura tions, but is also always changing as circumstances
move it and as my perceptions gather. The perspective the documenter offers to
others must evoke the constancy, coherence, and integrity of the landscape, and
its possibilities for changing its appearance. Without such a perspective, an or-
ganization or integration that is both personal and inf ormed by ali that has been
gathered by myself and by others in the settingothers could not share what I
have seencould not locate familiar landmarks and reflect on them as they ex-
Enhancing Quality and CredibilitylJ.617
hibit new relationships to one another and to less familiar aspects. Ali that ma-
terial, ali those observations and records, would be a lifeless and undoubtedly
dusty pile.
The process of forming a perspective in which the data gathered are inte-
grated into an organic configuration is obviously a process of interpretation. I
had begun documenting, however, without an articulated framework for inter-
pretation or a format for representa tion of the body of records, like the theoreti-
cal framework researchers bring to their data. Of course, there was a frame-
work. Coneeptions of artistic process, of learning and development, were
inherent in the program, but these were not explicit in its goals as a program to
provide certain kinds of service. The plan of the documentation had called for
certain results, but there was no specified format for presentation of results.
Therefore, my entry into interpretation became a struggle with myself over
what I was supposed to be doing. It was a long internai debate about my re-
sponsibilities and commitments.
When I began documenting this particular school's art program, for exam-
ple, I had priorities based on my experience and personal commitments. It
seemed to me self-evidently important to provide art activities for children and
to try and connect these to other areas of their learning. I knew that art was not
something that could be "learned" or even experienced on a once-a-week basis,
so I thought it was important to help teachers find various ways of integrating
art and other activities into their classrooms. I had already made a personal esti-
mate that what I was documenting was worthwhile and honest. I had found
points of congruence between my priorities and the program. I could see how
the various struetures of the program specified ways of approaching the goals
that seemed possible and that also enabled the elabora tion of the goals.
This initial commitment was diffuse; I felt a kind of general enthusiasm and
interest for the efforts I observed and a desire to explore and be helpful to the
teachers. In retrospect, however, the commitment was suffciently energizing
to sustain me through the early phases of collecting observations and records,
when I was not sure what these would lead to. Rather than restricting me, the
commitment freed me to look openly at every thing (as reflected in the early en-
thusiasm for collecting everything). Obviously, it is possible to begin docu-
menting from many other positions of relative interest and investment, but I
suspect that even if there is no particular involvement in program content on
the part of the documenter, there mustbe at least some idea of being helpful to
its staff. (Remember, this was a formative evaluation.) Otherwise, for example,
the process of gathering data may be circumscribed.
At the point of beginning to "do something" with the observations and re-
cords, I was forced to specify the original commitment, to rethink my purposes
and goals. Rereading the observations and records as a preliminary step in re-
working to address different audiences, I found myself at first reading with an
idea of "balancing" success and failure, an idea that constricted and trivialized
the work I had observed and recorded. Thankfully, it was immediately evident
from the data itself that such balance was not possible. If, during ten days of ob-
5 9 4 tI . ANALYSIS, INTERPRETATION, AND REPORTING
serva tion, a child's experience was intense one day and characterized by rowdy
socializing the other nine, a simple weigh-off would not establish the success or
failure of the child's experience. The idea was ludicrous. Similarly, the staff
might be thorough in its planning and follow-through on one day and disorga-
nized on another day, but organiza tion and planning were clearly not the total-
ity of the experience for children.
Such trade-offs implied an externai, stereotyped audience awaiting some
kind of quantitative proof, which I was supposed to provide in a disinterested
way, like an externai, summative evaluator. The "balance d view" phase was
also like my early record-gathering of everything. What I was documenting
was still in fragments for me, and my approach was to the particulars, to every
detail.
A second approach to interpreting, also brief, took a slightly broader view of
the data, a view that acknowledged my original estimate of program value and
attempted to specify it. Perceiving through the data the landscape-like configu-
rations of program strengths, I made assessments that included statements of
past mistakes or inadequacies like minor "flaws" in the landscape (a few odd
billboards and a garbage dump in one of Poussin's dreams of classical Italy, for
example) rather than debits on a balance sheet. Here again, the implication was
of an externai audience, expecting some absolute of accomplishment. The
"flaws" could be "minor" only by reference to an implied major flawthat of
failing to carry out the program goals altogether.
The formulation of strength subsuming weakness could not withstand the
vitality of the records I was reading. The reality the data portrayed became
clearer as the inadequacy of my first formulations of how to interpret the docu-
mentary material was revealed. Similarly, the implications of externai audience
expectations were not justified by the actuality of my relationship to the pro-
gram and staff. My state d goal as documenter had been originally to set up re-
cord-keeping proce dures that would preserve and make available to staff and
to other interested persons aspects of the beginnings and workings of the pro-
gram, and to collect and analyze some of the material as an assessment of what
further possibilities for development actually existed. My goals had not been to
evaluate in the sense of an externai judgment of success or failure.
Thinking over what other approaches to interpretation were possible, I re-
called that I had gathered documentary materiais quite straightforwardly as a
participant, whose engagement was initially through recognition of shared
convictions and points of congruence with the program. Perhaps, I decided, I
could share my viewpoint of the observations just as straightforwardly, as a
participant with a particular point of view. In examining this possibility, I came
to a view of interpreting observational data as a process of "rendering," much
as a performer renders a piece of classical music. The interpretation follows a
text closelyas a scientist might say, it sticks closely to the facts. But it also re-
flects the performer, specifically the performer's particular manner of engage-
ment in the enterprise shared by text and performer, the enterprise of music.
The same relationship could exist, it seemed to me, between a body of observa-
tions and records gathered participatively and as documenter. The relationship

would allow my personal experience and viewpoint to enhance rather than dis-
tort the data. Indeed, I would becoine their voice.
Through this relationship I could make the observations available to staff
and to other audiences in a way that was flexible and responsive to their needs,
purposes, and standards. In so doing, of course, the framework of inherent con-
ceptions underlying the work of the program would be incorporated. Thus, to
interpret the observational data I had gathered, I had to reaffirm and clarify my
relationship, my attachment to and participation in the program.
My initial engagement, with its strong coloring of prior interests and ideas,
had never meant that I understood or was sympathetic with every goal or prac-
tice of every participant of theprogram ali the time. In any joint enterprise, such
as a school or program, there are diverse and multiple goals and practices. Part
of the task of documenting is to describe and make these various understand-
ings, points of view, and practices visible so that participants can reflectively
consider them as the basis for planning. No participant agrees on ali issues and
points of practice. Part of being a participant is exploring differences and how
these illuminate issues or contribute to practice. My participation allowed me
to examine and extend the interests and ideas I came with as well as observing
and recording those other people brought. In this process my engagement was
deepened, enabling me to make assessments closer to the data than my first
readings brought. These assessments are evaluation in its original sense of
"drawing-value-from," an interactive process of valuing, of giving weight and
meaning.
In the context of renewed engagement and deepened participation, assess-
ments of mistakes or inadequacies are construed as discrepancies between a
particular practice and the intent behind it, between immediate and long-range
purposes. The discrepancy is not a flaw in an otherwise perfect surface,
butlike the discrepancy in a child's understanding that stimulates new learn-
ingis the occasion for growth. It is a sign of life and possibility. The burden of
the discrepancy can lie either with the practice or with the intent, and that is the
point for further examination. Assessment can also occur through the observa-
tion of and search for underlying themes of continuity between present and
past intent and practice, and the point of change or transformation in continu-
ity. Whereas discrepancy will usually be a more immediate trigger to evalua-
tion, occasions for the considera tion of continuity may tend to be longer-
rangeplanning for the comingyear, contemplating changes in staff and func-
tion, or commemorating an anniversary.
I have located the documenter as participant, internai to the program or set-
ting, gathering and shaping data in ways that make them available to partici-
pants and potentially to an externai audience. Returning to the image of a land-
scape, let me cominent on the different forms availability assumes for these
different audiences.
Participant access to the landscape through the documenter's perspective
cannot be achieved through ponderous written descriptions and reports on
620tI.ANALYSIS, INTERPRETATION, AND REPORTING
what has been observed but must be concentrated in interaction. Sometimes

this may require the development of special or regular structuresa series of
short-term meetings on a particular issue or problem; an occasional event that
sums up and looks ahead; a regular meeting for another kind of planning. But
many times the need is addressed in very slight forms, such as a comment in
passing about something a child or adult user is doing, or about the appearance
of a display, or the recounting of another staff member's observation. I do not
mean that injecting documentation into the self-assessment process is a jug-
gling act or some feat of manipula tion; merely that the documenter must be
aware that his or her role is to keep things open and that, while the observations
and records are a resource for doing this, a sense of the whole they create is also
essential. The landscape is, of course, changed by the new observations offered
by fellow viewers.
The externai audience places different requirements on the documenter who
seeks to represent to it the documentary perspective. By externai audience I re-
fer to funding agencies, supervisors, school boards, institutional hierarchies,
and researchers. Proposals, accounts, and reports to these audiences are gener-
ally required. They can be burdensome because they may not be organically re-
lated to the process of internai self-reflection and because the externai audience
has its own standards, purposes, and questions; it is unfamiliar with the setting
and with the documenter, and it needs the time offered by written accounts to
return and review the material. The externai audience will need more history
and formal description of the broad aspects than the internai audience, with
commentary that indicates the significance of recent developments. This need
can be met in the overall organization, arrangement, and introduction of docu-
ments, which also convey the detail and vividness of daily activity.
To limit the report to conventional format and expectations would probably
misrepresent the quality of thought, of relating, of self-assessment that goes
into developing the work. If there is intent to use the occasion of a report for re-
flectionfor example, by including staff in the development of the reportthe
reporting process can become meaningful internally while fulfilling the legiti-
mate externai demands for accounting. Naturally, such a comment engages the
externai audience in its own evaluative reflections by evoking the phenomenon
rather than reducing it.
In closing, I return to what I see as the necessary engaged participation of the
documenter in the setting being documented, not only for data-gathering but
for interpretation. Whatever authenticity and power my perspective as
documenter has had has come, I believe, from my commitment to the develop-
ment of the setting I was documenting and from the opportunities in it for me to
pursue my own understanding, to assess and reassess my role, and to come to
terms with issues as they arose.
We come to new settings with prior knowledge, experience, and ways of un-
derstanding, and our new perceptions and understandings build on these. We
do not simply look at things as if we had never seen anything like them before.
When we look at a cluster of light and dark greens with interstices of blue and
some of deeper browns and purples, what we identify is a tree against the sky.
Similarly, in a classroom we do not think twice when we see, for example, a
child scratching his head, yet the same phenomenon might be more strictly de-
scribed as a particular combination of forms and inovements. Our daily func-
tioning depends on this kind of apparently obvious and mundane interpreta-
tion of the world. These interpretations are not simply personal opinion
though they certainly may be uniquenor are they made up. They are instead
organizations of our perceptions as "tree" or "child scratching" and they corre-
spond at many points with the phenomena so described.
It is these organizations of perception that convey to someone else what we
have seen and that make objects available for discussion and reflection. Such
organizations need not exclude our awareness that the tree is also a cluster of
colors or that the child scratching his head is also a small human form raisingits
hand in a particular way. Indeed, we know that there could be many other ways
to describe the same phenomena, including some that would be completely nu-
mericalbut not necessarily more accurate, more truthful, or more useful! Af-
ter ali, we organize our perceptions in the context of immediate purposes and
relationships. The organizations must correspond to the context as well as to
the phenomenon.
Facts do not organize themselves into concepts and theories just by being
looked at; indeed, except within the framework of concepts and theories, there
are no scientific facts but only chos. There is an inescapable a priori element in
ali scientific work. Questions must be asked before answers can be given. The
questions are ali expressions of our interest in the world; they are at bottom val-
uations. Valuations are thus necessarily involved already at the stage when we
observe facts and carry on theoretical analysis and not only at the stage when
we draw political inferences from facts and valuations (Myrdal 1969:9).
My experience suggests that the situation in documenting is essentially the
same as what I have been describing with the tree and the child scratching and
what Myrdal describes as the process of scientific research. Documentation is
based on observation, which is always an individual response both to the phe-
nomena observed and to the broad purposes of observation. In documentation
observation occurs both at the primary levei of seeing and recording phenom-
ena and at secondary leveis of reobserving the phenomena through a volume of
records and directly, at later moinents. Since documentation has as its purpose
to offer these observations for reflections and evaluation in such a way as to
keep alive and open the potential of the setting, it is essential that observations
at both primary and secondary leveis be interpreted by those who have made
them. The usefulness of the observations to others depends on the docu-
menteis rendering them as finely as he or she is able, with as many points of
correspondence to both the phenomena and the context of interpretation as
possible. Such a rendering will be an interpretation that preserves the phenom-
ena and so does not exclude but rather invites other perspective.
Of course, there is a role for the experienced observer from outside who can
see phenomena freshly; who can suggest ways of obtaining new kinds of infor-
ma tion about it, or, perhaps more important, point to the significance of already
existing procedures or data; who can advise on technical problems that have
arisen within a documentation; and who can even guide efforts to interpret and
integrate documentary information. I am stressing, however, that the outside
observer in these instances provides support, not judgment or the criteria for
judgment.
The documenter's obligation to interpret his or her observations and those
reflected in the records being collected becomes increasingly urgent, and the in-
terpretations become increasingly significant, as ali the observers in the setting
become more knowledgeable about it and thus more capable of bringing range
and depth to the interpretation. Speaking of the weight of her observations of
the Manus over a period of some 40 years to great change, Margaret Mead clari-
fies the responsibility of the participant-observer to contribute to both people
studied and to a wider audience the rich individual interpretation of his or her
own obs eiva tions:
Uniqueness, now, in a study like this (of people who have come under the continu-
ing influence of contemporary world culture), lies in the relationships between the
fieldworker and the material. I still have the responsibility and incentives that
come from the fact that because of my long acquaintance with this village I can per-
ceive and record aspects of this people's life that no one else can. But even so, this
knowledge has a new edge. This material will be valuable only if I myself can orga-
nize it. In traditional fieldwork, another anthropologist familiar with the area can
take over one's notes and make them meaningful. But here it is my individual con-
sciousness that provides the ground on which the lives of these people are figures.
(Mead 1977:282-83)
In documenting it seems to me the contribution is ali the greater, and ali the
more demanded, because what is studied is one's own setting and commit-
ment.
SOURCE: Used with permission of Beth Alberty.
Answers to riddles
presented on pages 538-9.
Riddle Number One: Who Am I? Observer
Riddle Number Two: Who Am I? Interviewer
Riddle Number Three: Who Am I? Participant in field settings
Riddle Number Four: Who Am I? Interpreter

"Refeiêrvces
Abbey, Edward. 1968. Desert Solitaire: A Season in the Wilderness. New York: Ballantine.
Ackoff, Russell. 1999a. Ackoffs Best: His Classic Wriings on Management. New York:
John Wiley.
. 1999b. Re-Creating the Corporation: A Design of Organiza tions for the 2 lst Cen-
tury. Oxford, XJK: Oxford University Press.
. 1987. The Art of Problem Solving Accompanied by Ackoffs Fables. New York: John
Wiley.
Ackoff, Russell and Fred Emery. 1982. On Purposeful Systems. Salinas, CA: Intersys-
tems.
Academy for Educational Development (AED). 1989. Handbookfor Excellence in Focus
Group Research. Washington, DC: Academy for Educational Development.
AEA Task Force on Guiding Principies for Evaluators. 1995. "Guiding Principies for
Evaluators." New Directions for Program Evaluation 66 (summer): 19-34, Guiding
Principies for Evahiators, edited by William R. Shadish, D. L. Newman, M. A.
Scheirer, and C. Wye. San Francisco: Jossey-Bass.
Agar, Michael. 2000. "Border Lessons: Linguistic 'Rich Points' and Evaluative Under-
standing." Nezv Directions for Evaluation 86 (summer): 93-109. San Francisco:
Jossey-Bass.
. 1999. "Complexity Theory: An Explora tion and Overview." Field Methods 11
(2, November): 99-120.
. 1986. Speaking of Ethnography. Qualitative Research Methods Series, Vol. 2.
Beverly Hills, CA: Sage.
Agar, Michael H. and H. S. Reisinger. 1999. "Numbers and Pattems: Heroinlndicators
and What They Represent." Human Organization 58 (4, winter): 365-74.
Alasuutari, Pertti. 1995. Researching Culture. Thousand Oaks, CA: Sage.
Alkin, M. 1997. "Stakeholder Concepts in Program Evaluation." In Evaluation for Edu-
cational Productivity, edited by A. Reynolds and H. Walberg. Greenwich, CT: JAI.
RI
RI 6 [3. QUALITATIVE RESEARCH AND EVALUATION
Alkin, Marvin C. 1972. "Wider Context Goals and Goals-Based Evaluators." In Evalua-
tion Comment: The Journal of Educational Evaluation. Center for the Study of Evalua-
tion, UCLA, 3 (4, December): 10-11.
Alkin, Marvin C., Mary Andrews, G. L. Lewis, H. Manhertz, L. Sandinann, and J. West.
1989. Externai Evaluation ofthe Caribbean Agricultural Extension Project. Bridgetown,
Barbados: U.S. Agency for International Development.
Alkin, Marvin C., Richard Daillak, and Peter White. 1979. Using Evaluations: Does Evalu-
ation Make a Difference? Beverly Hills, CA: Sage.
Alkin, Marvin C. and Michael Q. Patton. 1987. "Working Both Sides of the Street." New
Directions for Program Evaluation 36 (winter): 19-32, The Client Perspective on Evalua-
tion, edited by leri Nowakowski. San Francisco: lossey-Bass.
Allen, Charlotte. 1997. "Spies Like Us: When Sociologists Deceive Their Subjects." Ln-
gua Franca (November): 31-38.
Allison, Mary Ann. 2000. "Enriching Your Practice With Complex Systems Thinking."
OD Practitioner 32 (3): 11-22.
Anderson, Barry. 1980. The Complete Thinker. Englewood Cliffs, NJ: Prentice Hall.
Anderson, Richard B. 1977. "The Effectiveness of Follow Through: What Have We
Learned?" Presented at the annual meeting of the American Educational Research
Association, New York City, April 5.
Anderson, Virgnia and Lauren Johnson. 1997. Systems Thinking Basics: From Concepts to
Causai Loops. Williston, VT: Pegasus Communications.
Arcana, Judith. 1983. Every Mother's Son: The Role of Mothers in the Making ofMen. Lon-
don: The Women's Press.
. 1981. Our Mothers' Daughters. London: The Woinen's Press.
Arditti, Rita. 1999. Searchingfor Life: The Grandmothers ofthe Plaza de May o and the Disap-
peared Children of Argentina. Berkeley: University of Califrnia Press.
Arendt, Hannah. 1968. Between Past and Future: Eight Exercises in Political Thought. New
York: Viking.
Argyris, Chris. 1982. Reasoning, Learning and Action: Individual and Organizational. San
Francisco: lossey-Bass.
Argyris, Chris, Robert Putnam, and Diana M. Smith. 1985. Action Science. San Francisco:
lossey-Bass.
Argyris, Chris and Donald Schon. 1978. Organizational Learning: A Theory of Action Per-
spective. Reading, MA: Addison-Wesley.
Armstrong, David. 1992, Managing by Storying Around. New York: Doubleday.
Asimov, Isaac. 1983. "Creativity Will Dominate Our Time After the Concepts of Work
and Fun Have Been Blurred by Technology." Personnel Admmistrator 28 (2): 42-46.
Atkinson, Paul. 1992. Understanding Ethnographic Texts. Qualitative Research Methods
Series, Vol. 25. Newbury Park, CA: Sage.
Atkinson, Robert. 1998. The Life Story Interview. Qualitative Research Methods Series,
Vol. 44. Thousand Oaks, CA: Sage.
Aubel, ludi. 1993. Participatory Program Evaluation: A Manual for Involving Stakeholders in
the Evaluation Process. Dakar, Senegal: Catholic Relief Services, under a U.S. AID
grant.
Aubrey, Robert and Paul Cohen. 1995. Working Wisdom: Learning Organizations. San
Azumi, Koya and lerald Hage. 1972. "Towards a Synthesis: A Systems Perspective."
Pp. 511-22 in Organizational Systems, edited by Koya Azumi and lerald Hage. Lexing-
ton, MA: D. C. Heath.
Baert, Patrick. 1998. Social Theory in the Tioentieth Century. New York: New York Univer-
sity Press.
Baldwin, lames. 1990. Notes ofa Native Son. Boston: Beacon.
References J. R3
Bali, Michael S. and Gregory W. H. Smith. 1992. Analyzing Visual Data. Qualitative Re-
search Methods Series, Vol. 24. Newbury Park, CA: Sage.
Bandler, Richard and JohnGrinder. 1975a. Patterns ofthe Hypnotic Techniques of Milton H.
Erikson, M.D. Vol. 1. Cupertino, CA: Meta Publica tions.
. 1975b. The Structure ofMagic. Vols. 1 and 2. Paio Alto, CA: Science and Behavior
Books.
Barker, Roger G. 1968. Ecological Psychology. Stanford, CA: Stanford University Press.
Barker, Roger G. and P. Schoggen. 1973. Qualities of Commtinihj Life: Methods of Mea-
suring Environment and Behavior Applied to an American and an English Town. San Fran-
cisco: Jossey-Bass.
Barker, Roger G. and H. F. Wright. 1955. Midwest and Its Children. New York: Harper &
Row.
Barker, Roger G., H. F. Wright, M. F. Schoggen, andL. S. Barker. 1978. Habitats, Environ-
ments, and Human Behavior. San Francisco: Jossey-Bass.
Barone, Tom. 2000. Aesthetics, Politics, and Educational Inquiry: Essays and Examples. New
York: Peter Lang.
Barton, David, Mary Hamilton, and Roz Ivanic. 1999. Sitnated Literacies: Reading and
Writing in Context. New York: Routledge.
Bartunek, Jean M. and Meryl Reis Louis. 1996. Insider/Outsider Team Research. Qualita-
tive Research Methods Series, Vol. 40. Thousand Oaks, CA: Sage.
Bateson, Gregory. 1978. "The Pattern Which Connects." CoEvolution Quarterly (sum-
mer): 5-15. Originally a speech for the Lindisfarne Association, October 17,1977, at
the Cathedral of St. John the Divine, Manhattan, NY.
Bateson, Mary Catherine. 2000. Full Circles, Overlapping Lives: Culture and Generation in
Transition. New York: Random House.
Bawden, R. J. and R. G. Packhain. 1998. "Systemic Praxis in the Education of the Agri-
cultural Systems Practitioner." Systems Research And Behavioral 15 (5, September-Oc-
tober): 403-12.
Becker, Howard S. 1985. Outsiders: Studies in the Sociology ofDeviance. New York: Free
Press.
. 1970. Sociological Work: Method and Suhstance. Chicago: Aldine.
. 1967. "Whose Side Are We On?" Social Problems 14:239-248.
. 1953. "Becoming a Marijuana User." American Journal of Sociology 59:235-42.
Becker, Howard and Blanche Geer. 1970. "Participant Observation and Interviewing: A
Comparison." In Qualitative Methodology, edited by W. J. Filstead. Chicago:
Markham.
Beebe, James. 2001. Rapid Assessment Process. Walnut Creek, CA: AltaMira.
Belenky, M. F., B. M. Clinchy, N. R. Goldberger, and J. M. Tarule. 1986. Women s Way of
Knowing: The Development ofSelf Voice, and Mind. New York: Basic Books.
Benko, S. and A. Sarvimaki. 2000. "Evaluation of Patient-Focused Health Care From a
Systems Perspective." Systems Research and Behavioral Science 17 (6, November-
December): 513-25.
Benmayor, Rina. 1991. "Testimony, Action Research, and Empowerment: Puerto Rican
Women and Popular Education." Pp. 159-74 in Women's Words: The Feminist Practice
of Oral History, edited by Sherna Berger Gluck and Daphne Patai. New York:
Routledge.
Benson, Alexis P., D. Michelle Hinn, and Claire Lloyd, eds. 2001. Visions of Quality: How
Evaluators Define, Understand, and Represent Program Quality. Advances in Program
Evaluation Vol. 7. Amsterdam, the Netherlands: Elsevier Science.
Bentz, Valerie Malhotra and Jeremy J. Shapiro. 1998. Mindful Inquiry in Social Research.
Thousand Oaks, CA: Sage.
Berens, Linda V. and Dario Nardi. 1999. The 16 Personality Types: Description for SelfDis-
covery. New York: Telos.
Berger, Peter and T. Luckmann. 1967. The Social Construction of Reality: A Treatise in the
Sociology of Knowledge. Garden City, NY: Anchor.
Berland, Jody. 1997. "Nationalism and the Modernist Legacy: Dialogues With Innis."
Culture and Policy 8 (3): 9-39.
Bernard, H. Russell. 2000. Social Research Methods: Qualitative and Quantitative Ap-
proaches. Thousand Oaks, CA: Sage.
, ed. 1998. Handbook of Methods in Cultural Anthropology. Walnut Creek, CA:
AltaMira.
. 1995. Research Methods in Anthropology: Qualitative and Quantitative Approaches.
Walnut Creek, CA: AltaMira.
Bernard, H. Russell and Gery W. Ryan. 1998. "Textual Analysis: Qualitative and Quanti-
tative Methods." Pp. 595-646 in Handbook of Methods in Cultural Anthropology, edited
by H. Russell Bernard. Walnut Creek, CA: AltaMira.
Bernthal, N. 1990. Motherhood Lost and Found: The Experience of Becoming an Adoptive
Mother to a Foreign Bom Child. Unpublished doctoral dissertation, Graduate College,
The Union Institute, Cincinnati, OH.
Bhaskar, R. A. 1975. A Realist Theory of Science. Leeds, UK: Leeds Books.
Bierce, Ambrose. [1906j 1999. The DeviVs Dictionary. New York: Oxford University Press.
Binnendijk, Annette L. 1986. AID's Experience With Contraceptive Social Marketing: A Syn-
thesis of Project Evaluation Findings. A.I.D. Evaluation Special Study No. 40. Washing-
ton, DC: U.S. Agency for International Development.
Blumer, Herbert. 1978. "Methodological Principies of Empirical Science." In Sociological
Methods: A Sourcebook, edited by Norman K. Denzin. New York: McGraw-Hill.
. 1969. Symbolic Interactionism. Englewood Cliffs, NJ: Prentice Hall.
. 1954. " W h a t l s Wrong With Social Theory?"' American Sociological Review 19:3-10.
Boas, Franz. 1943. "Recent Anthropology." Science 98:311-14,334-37.
Bochner, Arthur P. 2001. "Narrative's Virtues." Qualitative Inquiry 7 (2): 131-57.
Bochner, Arthur P. and Carolyn Ellis. 2001. Ethnographically Speaking. Oxford, UK:
Rowman & Littlefield.
Bochner, Arthur P , Carolyn Ellis, and L. Tillman-Healy. 1997. "Relationships as
Stories." Pp. 307-24 in Handbook of Personal Relationships: Theory, Research, and In-
terventions. 2d ed., edited by S. Dick. New York: John Wiley.
Bogdan, R. C. and S. K. Biklen. 1992. Qualitative Research for Education. Boston: Allyn &
Bacon.
Boring, E. G. 1942. Sensation and Perception in the History ofPsychology. Bloomington, IN:
Appleton Century.
Borman, KathrynM. and Judith P. Goetz. 1986. "Ethnographic and Qualitative Research
Design and Why It Doesn't Work." American Behavioral Scientist 30 (1, September-
October): 42-57.
Boruch, R. and D. Rindskopf. 1984. "Data Analysis." Pp. 121-58 in Evaluation Research
Methods. 2d ed., edited by L. Rutman. Beverly Hills, CA: Sage.
Boston Women's Teachers' Group. 1986. The Effect ofTeaching on Teachers. Nor th Dakota
Study Group on Evaluation monograph series. Grand Forks: Center for Teaching
and Learning, University of North Dakota.
Boulding, Kenneth E. 1985. Human Betterment. Beverly Hills, CA: Sage.
Boxill, Nancy A. 1990. Homeless Children: The Watchers and the Waiters. New York: Ha-
worth.
Boyatzis, Richard E. 1998. Transforming Qualitative Information: Thematic Analysis and
Code Development. Thousand Oaks, CA: Sage.
Brady, Ivan. 2000. "Anthropological Poetics." Pp. 949-79 in Handbook of Qualitative Re-
search. 2d ed., edited by Norman K. Denzin and Yvonna S. Lincoln. Thousand Oaks,
CA: Sage.
. 1998. " A Gift of the Journey." Qualitative Inquiry 4 (4, Deceinber): 463.
References J. R5
Brajuha, Mario and L. Hallowell. 1986. "Legal Intrusion and the Politics of Fieldwork:
The Impact of the Brajuha Case." Urban life 1 (4): 454-78.
Brandon, Paul R., Marlene A. Lindberg, and Zhigand Wang. 1993. "Involving Program
Beneficiaries in the Early Stages of Evaluation: Issues of Consequential Validity and
Influence." Educational Evaluation and Policy Analysis 15 (4): 420-28.
Braud, William and Rosemarie Anderson. 1998. Transpersonal Research Methods for the
Social Sciences: Honoring Human Experience. Thousand Oaks, CA: Sage.
Bremer, ]., E. Cole, W. Irelan, and P. Rourk. 1985. A Review ofAID's Experience in Priva te
Sector Development. A.I.D. Program Evaluation Report No. 14. Washington, DC: U.S.
Agency for International Development.
Brewer, J. and A. Him ter. 1989. Multimethod Research: A Synthesis of Styles. Newbury
Park, CA: Sage.
Brinkley, David. 1968. "Public Broadcasting Laboratory." Interview on Public Broad-
casting Service, December 2.
Brislin, Richard W., K. Cushner, C. Cherrie, and Mahealani Yong. 1986. Intercultural In-
teractions: A Practical Guide. Beverly Hills, CA: Sage.
Brizuela, B. M. J . P. Stewart, R. G. Carrillo, and J. G. Berger. 2000. Acts of Inquiry in Quali-
tative Research. Reprint Series No. 34. Cambridge, MA: Harvard Educational Review.
Brock, James, Richard Schwaller, and R. L. Smith. 1985. "The Social and Local Govern-
ment Impacts of the Abandonment of the Milwaukee Railroad in Montana." Evalua-
tion Review 9 (2): 127-43.
Brookfield, Stephen. 1994. "Tales From the Dark Side: A Phenomenography of Adult
Criticai Reflection." International Journal oftifelong Education 13 (3): 203-16.
Brown, John Seely, Alan Collins, and Paul Duguid. 1989. "Situated Cognition and the
Culture of Learning." Educational Researcher 18 (1, January-February): 32-42.
Brown, Judith R. 1996. The I in Science: Training to Utilize Subjectivity in Research. Oslo,
Norway: Scandinavian University Press.
Browne, Angela. 1987. When Battered Women Kill. New York: Free Press.
Bruce, Christine and Rod Gerber. 1997. Phenomenographic Research: An Annotated Bibliog-
raphy. Occasional Paper 95.2. Centre for Applied Environmental and Social Educa-
tion Research. Brisbane, Australia: QUT Publications.
Bruner, Edward M. 1996. "My Life in an Ashram." Qualitative Inquiry 2 (3, September):
300-19.
Bruyn, Severyn. 1966. The Human Perspective in Sociology: The Methodology of Participant
Observation. Englewood Cliffs, NJ: Prentice Hall.
. 1963. "The Methodology of Participant Observation." Human Organization
21:224-35.
Buber, Martin. 1923.1 and Thou. New York: Macmillan Library.
Buckholt, Mareia. 2001. Women's Voices ofResilience: Female Adult Abuse Survivors Define
the Phenomenon. Unpublished doctoral dissertation, Graduate College, The Union
Institute, Cincinnati, OH.
Bullogh, Robert V., Jr. and Stefinee Pinnegar. 2001. "Guidelines for Quality in Autobio-
graphical Forms of Self-Study Research." Educational Researcher 30 (3): 13-21.
Bunch, Eli Haugen. 2001. "Quality of Life of People With Advanced HIV/AIDS in Nor-
way." Groimded Theory Review 2:30-42.
Bunge, Mario. 1959. Causality. Cambridge, MA: Harvard University Press.
Burdell, Patricia and Beth Blue Swadener. 1999. "Criticai Personal Narrative and
Autoethnography in Education: Reflections on a Genre." Educational Researcher 28
(6): 21-26.
Burns, Tom and G. M. Stalker. 1972. "Models of Mechanistic and Organic Structure."
Pp. 240-55 in Organizational Systems, edited by Koya Azumi and lerald Hage. Lex-
ington, MA: D. C. Heath.
Bussis, Arme, Edward A. Chittenden, and Marianne Amarei. 1973. Methodology in Edu-
cational Evaluation and Research. Princeton, NJ: Educational Testing Service.
Buxton, Amity. 1982. Children's Journais: Further Dimensions o/AssessingLanguage Devel-
opment. North Dakota Study Group on Evaluation monograph series. Grand Forks:
Center for Teaching and Learning, University of North Dakota.
Cambei, Ali Bulent. 1992. Applied Chos Theory: A Paradigm for Complexity. New York:
Academic Press.
Campbell, Donald T. 1999a. "Legacies of Logical Positivism and Beyond." Pp. 131-44 in
Social Experimentation, by Donald T. Campbell and M. Jean Russo. Thousand Oaks,
CA: Sage.
. 1999b. "On the Rhetorcal Use of Experiments." Pp. 149-58 in Social Experimen-
tation, by Donald T. Campbell and M. Jean Russo. Thousand Oaks, CA: Sage.
. 1974. "Qualitative Knowing in Action Research." Presented at the annual meet-
ing of the American Psychological Association, New Orleans, LA.
Campbell, Donald T. and M. Jean Russo. 1999. Social Experimentation. Thousand Oaks,
CA: Sage.
Campbell, Jeanne L. 1983. Factors and Conditions Influencing Useftness of Planning, Eval-
uation and Reporting in Schools. Unpublished doctoral dissertation, University of
Minnesota.
Carchedi, G. 1983. "Class Analysis and the Study of Social Forms." Pp. 347-67 in Beyond
Method, edited by Gare th Morgan. Beverly Hills, CA: Sage.
Carini, Patricia F. 1979. The Art of Seeing and the Visibility of the Person. North Dakota
Study Group on Evaluation monograph series. Grand Forks: Center for Teaching
and Learning, University of North Dakota.
. 1975. Observation and Description: An Alternative Methodfor the Investigation of
Human Phenomena. North Dakota Study Group on Evaluation monograph series.
Grand Forks: Center for Teaching and Learning, University of North Dakota.
Carlin, George. 1997. Brain Droppings. New York: Hyperion.
Casse, Pierre and Surinder Deol. 1985. Managing Intercultural Negotiations: Guidelines for
Trainers and Negotiators. Washington, DC: SIETAR International.
Castaneda, Carlos. 1973. Journey to Ixtlan. New York: Pocket Books.
Cedillos, Jose Hilrio. 1998. "Mayan Fragments and Bricolage: Roots of Layered Con-
sciousness." Unpublished manuscript.
. Forthcoming. The Bricolage Arts: The Postmodernist Search for Shamanic Jazz.
Cernea, Michael, ed. 1991. Putting People First: Sociological Variables in Rural Develop-
ment. 2d ed. New York: Oxford University Press.
Cernea, Michael M. and Scott E. Guggeriheim. 1985. "Is Anthropology Superfluous in
Farming Systems Research?" World Bank Reprint Series No. 367. Washington, DC:
World Bank.
Cervantes Saavedra, Miguel de. 1964. Don Quixote. New York: Signet Classics.
Chagnon, Napoleon. 1992. Yanomamo: The Last Days ofEden. New York: Harcourt Brace.
Chamberlayne, P., J. Bornat, and T. Wengraf. 2000. The Turn to Biographical Methods in So-
cial Science. London: Routledge.
Chambers, Erve. 2000. "Applied Ethnography." Pp. 851-69 in Handbook of Qualitative Re-
CA: Sage.
Charmaz, Kathy. 2000. "Grounded Theory: Objectivist and Constructivist Methods."
Pp. 509-35 in Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin
and Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
Charon, Rita, M. G. Greene, and R. D. Adelman. 1998. "Qualitative Research in Medi-
cine and Health Care: Questions and Controversy, a Response. Journal of General In-
ternai Medicine 13 (January): 67-68.
References J. R7
Chatterjee, A. 2001. "Language and Space: Some Interactions." Trends in Cognitive Sci-
ences 5 (2): 55-61.
Checkland, Peter. 1999. Systems Thinking, Systems Practice: A 30-Year Retrospective. New
York: lohn Wiley.
Chen, Huey-Tsyh and Peter H, Rossi. 1987. "The Theory-Driven Approach to Validity"
Evaluation and Program Planning 10:95-103.
Chew, Siew Tuan. 1989. Agroforestry Projects for Small Parmers. A.I.D. Evaluation Special
Study No. 59. Washington, DC: U.S. Agency for International Development.
Cheyne, V. 1988. Growing Up in a Patherless Home: The Female Experience. Ann Arbor, MI:
University Microfilms International. Unpublished doctoral thesis, Graduate Col-
lege, The Union Institute, Cincinnati, OH.
Chibnik, M. 2000. "The Evolution of Market Niches in Oaxacan Woodcarving." Ethnol-
ogy 39 (3, summer): 225-42.
Christians, Clifford G. 2000. "Ethics and Politics in Qualitative Research." Pp. 133-55 in
Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin and Yvonna S.
Lincoln. Thousand Oaks, CA: Sage.
Church, Kathryn. 1995. Forbidden Narratives: Criticai Autobiography as Social Science. To-
ronto, Ontario, Canada: University of Toronto Press.
Cialdini, Robert B. 2001. "The Science ofPersuasion." Scientific American 284 (2): 76-81.
Clark, J. 1988. The Experience ofthe Psychologically Androgynous Male. Unpublished doc-
toral dissertation, Graduate College, The Union Institute, Cincinnati, OH.
Clarke, I. and W. Mackaness. 2001. "Management 'Intuition': An Interpreta tive Account
of Structure and Content of Decision Schemas Using Cognitive Maps." Journal of
Management Studies 38 (2): 147-72.
Cleveland, Harlan. 1989. The Knowledge Executive: Leadership in an Information Society.
New York: E. P. Dutton.
Coffey, Amanda and Paul Atkinson. 1996. Making Sense of Qualitative Data: Complemen-
tary Research Strategies. Thousand Oaks, CA: Sage.
Cole, Andra L. and J. Gary Knowles. 2000. Doing Reflexive Life History Research. Walnut
Creek, CA: AltaMira.
Coles, Robert. 1990. The Spiritual Life of Children. Boston: Houghton Mifflin.
. 1989. The Call of Stories: Teaching and the Moral Imagination. Boston: Houghton
Mifflin.
Collins, Jim. 2001. "Levei 5 Leadership: The Triumph of Humility and Fierce Resolve."
Harvard Business Review 79 (1, January): 67-76,175.
Comstock, D. E. 1982. "A Method for Criticai Research." Pp. 370-90 in Knowledge and
Values in Social and Educational Research, edited by E. Bredo and W. Feinberg. Phila-
delphia: Temple University Press.
Connolly, Deborah R. 2000. Homeless Mothers: Face to Face With Women and Poverty. Min-
neapolis: University of Minnesota Press.
Connor, Ross. 1985. "International and Domestic Evaluation: Comparisons and In-
sights." Pp. 19-28 in Culture and Evaluation, edited by Michael Quinn Patton. San
Conrad, Joseph. 1960. Heart ofDarkness. New York: Dell.
Conroy, Dennis L. 1987. A Phenomenological Study ofPolice OJficers as Victims. Unpub-
lished doctoral thesis, Graduate College, The Union Institute, Cincinnati, OH.
Constas, M. A. 1998. "Deciphering Postmodern Educational Research." Educational Re-
searcher 27 (9): 36-42.
Cook, Thomas D. 1995. "Evaluation Lessons Learned." Plenary address at the Interna-
tional Evaluation Conference "Evaluation '95," November 4, Vancouver, BC.
Cook, Thomas D., Laura C. Leviton, and William R. Shadish, Ir. 1985. "Program Evalua-
tion." Pp. 699-777 in Handbook of Social Psychology, Theory and Method. Vol. 1. 3d ed.,
edited by G. Lindzey and E. Aronson. New York: Random House.
Cook, Thomas D. and Charles S. Reichardt, eds. 1979. Qualitative and Quantitative
Methods in Evaluation Research. Beverly Hills, CA: Sage.
Cooke, N. J. 1994. "Varieties of Knowledge Elicitation Techniques." International Journal
of Human-Computer Studies 41 (6): 801-49.
Cooper, Harris. 1998. Synthesizing Research. Thousand Oaks, CA: Sage.
Coulon, Alain. 1995. Ethnomethodology. Qualitative Research Methods Series, Vol. 36.
Cousins, J. Bradley and Lorna M. Earl, eds. 1995. Participatory Evaluation in Education:
Studies in Evaluation Use and Organizational Learning. London: Falmer.
. 1992. "The Case for Participatory Evaluation." Educational Evaluation and Policy
Analysis 14:397-418.
Covey, Stephen R. 1990. The 7 Habits of Highly Effective People: Powerful Lessons in Personal
Change. New York: Fireside.
Cox, Gary B. 1982. "Program Evaluation." Pp. 338-51 in Handbook on Mental Health Ad-
ministration, edited by Michael S. Austin and William E. Hersley. San Francisco:
lossey-Bass.
Craig, P. 1978. The Heart ofa Teacher: A Heuristic Study of the lnner World of Teaching. Ann
Arbor, MI: University Microfilms International.
Creswell, lohn W. 1998. Qualitative Inquiry and Research Design: Choosing Among Five Tra-
ditions. Thousand Oaks, CA: Sage.
Cronbach, Lee ]. 1988. "Playing With Chos." Educational Researcher 17 (6, August-Sep-
tember): 46-49.
. 1982. Designing Evahiations of Educational and Social Programs. San Francisco:
lossey-Bass.
. 1975. "Beyond the Two Disciplines of Scientific Psychology." American Psychol-
ogist 30:116-27.
Cronbach, Lee I. and Associates. 1980. Toward Reform of Program Evaluation. San Fran-
cisco: lossey-Bass.
Crosby, Philip B. 1979. Quality Is Free: The Art of Making Quality Certain. New York:
McGraw-Hill.
Crotty, Michael. 1998. The Foundations of Social Research: Meaningand Perspective in the Re-
search Process. London: Sage.
Curry, Constance. 1995. Silver Rights. Chapei Hill, NC: Algonquin.
Czarniawska, Barbara. 1998. A Narrative Approach to Organization Studies. Qualitative
Research Methods Series, Vol. 43. Thousand Oaks, CA: Sage.
Cziko, Gary A. 1989. "Unpredictability and Indeterminism in Human Behavior: Argu-
ments and Implications for Educational Research." Educational Researcher 18 (3,
April): 17-25.
Dalgaard, K. A., M. Brazzel, R. T. Liles, D. Sanderson, and E. Taylor-Powell. 1988. Issues
Programming in Extension. Washington, DC: Extension Service, U.S. Department of
Agriculture.
Dart, lessica. 2000. Personal e-mail communication. For more information online about
the "most significant changes" monitoring system, go to http:// www. egroups.com/
group / MostSignificantChanges.
Dart, J. J., G. Drysdale, D. Cole, and M. Saddington. 2000. "The Most Significant Change
Approach for Monitoring an Australian Extension Project." PLA Notes 38:47-53. Lon-
don: International Institute for Environment and Development.
Davies, Rick J. 1996. "An Evolutionary Approach to Facilitating Organisational
Learning: An Experimentby the Christian Commission for Development in Bangla-
desh." Swansea, UK: Centre for Development Studies.
Davis, Kingsley. 1947. "Final Note on a Case of Extreme Social Isolation." American Jour-
nal of Sociology 52 (March): 432-37.
References J. R9
. 1940. "Extreme Social Isolation of a Child." American Journal of Sociology 45 (Jan-

uary): 554-65.
De Bono, Edward. 1999. Six Thinking Hats. New York: Little, Brown.
DeCramer, Gary. 1997.Minnesota's District/Area Transportation PartnershipProcess. Vol. 1,
Cross-Case Analysis. Vol. 2, Case Studies and Other Perspectives. Minneapolis: Center
for Transportation Studies, University of Minnesota.
De Munck, V. 2000. "Introduction: Units for Describing and Analyzing Culture and So-
ciety." Ethnology 39 (4): 279-92.
Denning, Stephen. 2001. The Springboard: How Storytelling Ignites Action in Knowl-
edge-Era Organizations. Portsmouth, NH: Butterworth-Heinemann.
Denny, Terry. 1978. "Storytelling and Educational Understanding." Occasional Paper
No. 12, Evaluation Center. Kalamazoo: Western Michigan University.
Denzin, Norman K. 2001. Interpretive Interactionism. 2d ed. Thousand Oaks, CA: Sage.
. 2000a. "Aesthetics and the Practices of Qualitative Inquiry." Qualitative Inquiry
6 (2): 256-65.
. 2000b. "Rock Creek History" Symbolic Interaction 23 (1): 71-81.
. 1997a. "Coffee With Anselm." Qualitative Family Research 11 ( 1 , 2 November):
16-18.
. 1997b. Interpretive Ethnography: Ethnographic Practices for the 21 st Century. Thou-
sand Oaks, CA: Sage.
. 1991. Images of Postmodern Society: Social Theory and Contemporary Cinema. Lon-
don: Sage.
. 1989a. Interpretive Biography. Qualitative Research Methods Series, Vol. 17.
Newbury Park, CA: Sage.
. 1989b. Interpretive Interactionism. Newbury Park, CA: Sage.
. 1989c. The Research Act: A Theoretical Introduction to Sociological Methods. 3d ed.
Englewood Cliffs, NJ: Prentice Hall.
. 1978a. "The Logic of Naturalistic Inquiry." In Sociological Methods: A Sourcebook,
edited by Norman K. Denzin. New York: McGraw-Hill.
. 1978b. The Research Act: A Theoretical Introduction to Sociological Methods. 2d ed.
New York: McGraw-Hill.
Denzin, Norman K. and Yvonna S. Lincoln, eds. 2000a. Handbook of Qualitative Research.
2d ed. Thousand Oaks, CA: Sage.
. 2000b. "Introduction: The Discipline and Practice of Qualitative Research."
Pp. 1-28 in Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin and
Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
Deutsch, Claudia H. 1998. "The Guru of Doing It Right Still Sees Much Work to Do."
New York Times, November 15, p. B5.
Deutscher, Irwin. 1970. "Words and Deeds: Social Science and Social Policy." Pp. 27-51
in Qualitative Methodology, edited by W. J. Filstead. Chicago: Markham.
Dewey, John. 1956. The Child and the Ciirriculuj?i. Chicago: University of Chicago Press.
Dobbert, Marion L. 1982. Ethnographic Research: Theory and Application for Modern Schools
and Societies. New York: Praeger.
Domaingue, Robert. 1989. "Community Development Through Ethnographic Futures
Research." Journal of Extension (summer): 22-23.
Douglas, Jack D. 1976. Investigative Social Research: Individual and Team Field Research.
Douglass, Bruce and Clark Moustakas. 1985. "Heuristic Inquiry: The Internai Search to
Know." Journal of Humanistic Psychology 25 (3, summer): 39-55.
Douglass, W. A. 2000. "In Search of Juan de Onate: Confessions of a Crypto-
essentialist." Journal of Anthropological Research 56 (2): 137-62.
Drass, Kriss and Charles Ragin. 1992. QCA: Qualitative Comparative Analysis. A DOD
software program distributed by the Publications Office, Institute for Public Policy,
Northwestern University. Evanston, IL: Northwestern University.

Duckworth, Eleanor. 1978. The African Frimary Science Program: An Evaluation and Ex-
tended Thoughts. North Dakota Study Group on Evaluation monograph series. Grand
Forks: Center for Teaching and Learning, University of North Dakota.
Dunn, Stephen. 2000. "Empathy." The New Yorker, April 10, p. 62.
Durkin, Tom. 1997. "Using Computers in Strategic Qualitative Research." Pp. 92-105 in
Context and Method in Qualitative Research, edited by Gale Miller and Robert
Dingwall. Thousand Oaks, CA: Sage.
Durrenberger,E.P. and S.Erem. 1999. "The WeakSuffer What They Must: A Natural Ex-
periment in Thought and Structure." American Anthropologist 101 (4): 783-93.
Eberstadt, Nicholas, Nicolas Eberstadt, and Daniel Patrick Moynihan. 1995. The Tyr-
anny ofNimbers: Mismeasiirement aiidMisride,Yfash.mgton/I)C: American Enterprise
Institute Press.
Edmunds, Stahrl W. 1978. Alternative U.S. Futures: A Policy Analysis of Individual Choices
in a Political Economy. Santa Monica, CA: Goodyear.
Edwards, Ward, Mareia Guttentag, and Kurt Snapper. 1975. " A Decision-Theoretic Ap-
proach to Evaluation Research." In Handbookof Evaluation Research. Vol. 1, edited by
E. L. Struening and M. Guttentag. Beverly Hills, CA: Sage.
Eichelberger, R. Tony. 1989. Disciplined Inquiry: Understanding and Doing Educational Re-
search. White Plains, NY: Longman.
Eichenbaum, Luise and Susie Orbach. 1983. Understanding Women: A Feminist Psychoan-
alytic Approach. New York: Basic Books.
Eisner, Elliot W. 1997. "The New Frontier m Qualitative Research Methodology." Quali-
tative Inquiry 3 (3, September): 259-73.
. 1996. "Should a Novel Count as a Disser ta ti on in Education?" Research in the
Teaching of English 30 (4): 403-27.
. 1991. The Enlightened Eye: Qualitative Inquiry and the Enhancement of Educational
Practice. New York: Macmillan.
. 1988. "The Primacy of Experience and the Politics of Method." Educational Re-
searchers (June/July): 15-20.
. 1985. The Art of Educational Evaluation: A Personal View. London: Falmer.
Elliott, John. 1976. Developing Hypotheses About Classrooms From Teachers Practical Con-
struets. North Dakota Study Group on Evaluation monograph series. Grand Forks:
Center for Teaching and Learning, University of North Dakota.
Ellis, Carolyn. 1986. FisherFolk: Two Communities on Chesapeake Bay. Lexington: Univer-
sity Press of Kentucky.
Ellis, Carolyn and Arthur P. Bochner, 2000. " Autoethnography, Personal Narrative, Re-
flexivity: Researcher as Subject." Pp. 733-68 in Handbook of Qualitative Research.
2d ed., edited by Norman K. Denzin and Yvonna S. Lincoln. Thousand Oaks, CA:
Sage.
. 1996. Composing Ethnography: Alternative For ms of Qualitative Writing. Walnut
Creek, CA: AltaMira.
Elmore, Richard F. 1976. "Follow Through Planned Variation." In Social Program Imple-
mentation, edited by Walter Williams and Richard F. Elmore. New York: Academic
Press.
English, Fenwick W. 2000. " A Criticai Appraisal of Sara Lawrence-Lightfoofs Portrai-
ture as a Method of Educational Research." Educational Researcher 29 (7): 21-26.
Ensler, Eve. 2001. The Vagina Monologues: The V-Day Edition. New York: Villard.
Eoyang, Glenda H. 1997. Coping With Chos: Seven Simple Tools. Cheyenne, WY: Lagumo.
Erickson, Fred. 1973. "What Makes School Ethnography 'Ethnographic'?" Anthropology
and Education Quarterly 4 (2): 10-19.
Erickson, Ken and Donald Stull. 1998. Doing Team Ethnography: Warnings and Advice.
Qualitative Research Methods Series, Vol. 42. Thousand Oaks, CA: Sage.
References !fj. Rll
Ericsson, K. Anders and Herbert Alexander Simon. 1993. Protocol Analysis: Verbal Re-
ports as Data. Cambridge: MIT Press.
Fadiman, Clifton, ed. 1985. The Little, Brown Book of Anecdotes. Boston: Little, Brown.
Farming Systems Support Project (FSSP). 1987. Bibliography ofReadings in Farming Sys-
tems. Gainesville: University of Florida Institute of Food and Agricultural Sciences.
. 1986. Diagnosis in Farming Systems Research and Extension. Vol. 1. Gainesville:
University of Florida Institute of Food and Agricultural Sciences.
Fehrenbacher, Harry L v Thomas R. Owens, and Joseph F. Haehnn. 1976. The Use of Stu-
dent Case Study Methodology in Program Evaluation. Research Evaluation Develop-
ment Paper Series No. 10. Portland, OR: Northwest Regional Educational Labora-
tory.
Feiman, Sharon. 1977. "Evaluatmg Teacher Centers." School Review 8:395-411.
Feldman, Martha S. 1995. Strategies for Interpreting Qualitative Data. Qualitative Re-
search Methods Series, Vol. 33. Thousand Oaks, CA: Sage.
Ferguson, Cecile. 1989. The Use and Impact of Evaluation by Decision Makers: Four Austra-
lian Case Studies. Unpublished doctoral thesis, Macquarie University, Australia.
Festinger, Leon. 1956. When Prophecy Fails: A Social and Psychological Study. New York:
HarperCollins College Division.
Fetterman, David M. 2000a. Foundations of Empowerment Evaluation: Step by Step. Thou-
. 2000b. "Summary of the STEP Evaluation and Dialogue." American Journal of
Evaluation 21 (2, spring-summer): 239-259.
. 1989. Ethnography: Step by Step. Newbury Park, CA: Sage.
. 1988a. "Qualitative Approaches to Evaluating Education." Educational Re-
searcher 17 (8, November): 17-23.
. 1988b. Qualitative Approaches to Evaluation in Education: The Silent Scientific Revo-
lution. New York: Praeger.
, ed. 1984. Ethnography in Educational Evaluation. Beverly Hills, CA: Sage.
Fetterman, David M., A. I- Kaftarian, and A. Wandersman, eds. 1996. Empowerment Eval-
uation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks,
CA: Sage.
Fielding, Nigel G. 2000. "The Shared Fate of Two Innovations in Qualitative Methodol-
ogy: The Relationship of Qualitative Software and Secondary Analysis of Archived
Qualitative Data." Qualitative Social Research [Online] 1 (3, December). Available
from http:/ /caqdas.soc.surrey.ac.uk/news.
. 1995. "Choosing the Right Qualitative Software Package." Data Archive Bulletin
[Online] 58. Available from http://caqdas.soc.surrey.ac.uk/choose.htm.
Fielding, Nigel G. and lane L. Fielding. 1986. Linking Data. Qualitative Research
Methods Series, Vol. 4. Beverly Hills, CA: Sage.
Fielding, Nigel G. and Raymond M. Lee. 1998. Computer Analysis and Qualitative Re-
search. Thousand Oaks, CA: Sage.
Filstead, William ]., ed. 1970. Qualitative Methodology. Chicago: Markham.
Fitz-Gibbon, Carol Taylor and Lynn Lyons Morris. 1987. How to Design a Program Evalua-
tion. Newbury Park, CA: Sage.
Fitzpatrick, lacqueline, lan Secrist, and Debra ]. Wright. 1998. Secrets for a Successful Dis-
ser tation. Thousand Oaks, CA: Sage.
Fitzsimmons, Ellen L. 1989. "Alternative Extension Scenarios." Journal of Extension 28
(3, fali): 13-15.
Fonow, Mary Margaret andludith A. Cook. 1991. Beyond Methodology: Feminist Scholar-
ship as Lived Research. Bloomington: Indiana University Press.
Fontana, Andra and lames H. Frey. 2000. "The Interview: From Structured Questions
to Negotiated Text." Pp. 645-72 in Handbook of Qualitative Research. 2d ed., edited by
Norman K. Denzin and Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
Fonte, John. 2001. "Why There Is a Culture War: Gramsci and Tocqueville in America."
Policy Review 104 (January): 14-23.
Foucault,Michel. 1988. "The Aesthetics of Existence." In Politics, Philosophy, Culture: In-
terviews and Other Writings 1977-1984, edited by L. D. Kritzman. New York:
Routledge.
. 1972. The Archaeology of Knowledge and the Discourse on Language. New York: Pan-
theon.
Frake, Charles. 1962. "The Ethnographic Study of Cognitive Systems." In Anthropology
and Human Behavior, edited by T. Gladwin and W. H. Sturtlevant. Washington, DC:
Anthropology Society of Washington.
Frank, A. 2000. "Illness and Autobiographical Work." Qualitative Sociology 23:135-56.
. 1995. The Wounded Storyteller: Body, IUness, and Ethics. Chicago: University of
Chicago Press
Freire, Paulo. 1973. Education for Criticai Consciousness. New York: Seabury.
Fricke, JohnG. and Raj Gill. 1989. "Participative Evaluations." Canadian Journal of Evalu-
ation 4 (1, April/May): 11-26.
. 1970. Pedagogy of the Oppressed. New York: Seabury.
Frow, John and Meaghan Morris. 2000. "Cultural Studies." Pp. 315-46 in Handbook of
Qualitative Research. 2d ed., edited by Norman K. Denzin and Yvonna S. Lincoln.
Fuller, Steve. 2000. Thomas Kuhn: A Philosophical History ofOur Times. Chicago: Univer-
sity of Chicago Press.
Gahan, Clia and Mike Hannibal 1998. Doing Qualitative Research Using QSR.NUD.IST.
Gallucci, M. and M. Perugini 2000. " A n Experimental Test of a Game-Theoretical Model
of Reciprocity." Journal ofBehavioral Decision Making 13 (4): 367-89.
Galt, D. L. and S. B. Mathema. 1987. "Farmer Participation in Farming Systems Re-
search." In Farmng Systems Support Project Newsletter, Vol. 5, No. 7. Gainesville: Uni-
versity of Florida Institute of Food and Agricultural Sciences.
Gamson, Joshua. 2000. "Sexualities, Queer Theory, and Qualitative Research." Pp. 347-
65 in Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin and
Garcia, Samuel E. 1984. Alexander the Great: A Strategy Revieio. Report No. 84-0960,
Maxwell Air Force Base. Montgomery, Alabama: Air Command and Staff College.
Garfinkel, Harold. 1967. Studies in Ethnomethodology. Englewood Cliffs, NJ: Prentice
Hall.
Geertz, Chfford. 2001. "Life Among the Anthros." The New York Reviciv of Books 48 (2,
February 8): 18-22.
. 1973. "Deep Play: Notes on the Balinese Cockfight" Pp. 412-53 in The Interpreta-
tion of Cultures. New York: Basic Books.
Gentile, J. Ronald. 1994. "Inaction Research: A Superior and Cheaper Alternative for
Educational Researchers." Educational Researcher 23 (2): 30-32.
Gephart, Robert P., Jr. 1988. Ethnostatistics: Qualitative Foundations for Quantitative Re-
search. Qualitative Research Methods Series, Vol. 12. Newbury Park, CA: Sage.
Gergen, Mary M. and Kenneth J. Gergen. 2000. "Qualitative Inquiry: Tensions and
Transformation." Pp. 1025-46 in Handbook of Qualitative Research. 2d ed., edited by
Norman K. Denzin and Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
Gharajedaghi, lamshid. 1985. Toward a Systems Theory of Organization. Salinas, CA:
Intersystems.
Gharajedaghi, Jamshid and Russell L. Ackoff. 1985. "Toward Systemic Education of
Systems Scientists." Systems Research 2 (1): 21-27.
Gilgun, Jane. 1999. "Fingernails Painted Red: A Feminist Semiotic Analysis of a 'Hot'
Text." Qualitative Inquiry 5 (2): 181-207.
References J. R13
. 1996. "Huinan Development and Adversity in Ecological Perspective, Part 2,

Three Patterns." Families in Society 77:459-576.
. 1995. "We Shared Something Special: The Moral Discourse of Incest Perpetra-
tors." Journal ofMarriage and the Family 57:265-81.
. 1994. "Avengers, Conquerors, Playmates, and Lovers: A Continuum of Roles
Played by Perpetrators of Child Sexual Abuse." Families in Society 75:467-80.
. 1991. "Resilience and the Intergenerational Transmission of Child Sexual
Abuse." Pp. 93-105 in Family Sexual Abuse: Frontline Research and Evaluation, edited by
Michael Quinn Patton. Newbury Park, CA: Sage.
Gilgun, Jane and Laura McLeod. 1999. "Gendering Violence." Studies in Symbolic In ter-
action 22:167-93.
Gilligan, Carol. 1982. In a Different Voice: Psychological Theory and Women's Development.
Cambridge, MA: Harvard University Press.
Giorgi, A. 1971. "Phenomenology and Experimental Psychology." In Duques tie Studies
in Phenomenological Psychology, edited by A. Giorgi, W. Fisclmer, and R. Von
Eckartsberg. Pittsburgh, PA: Duquesne University Press.
Gladwell, Malcolm. 2000. "Annals of Medicine." The New Yorker, March 13, pp. 55-56.
. 1997. "Just Ask for It: The Real Key to Technological Innovation." The New
Yorker, April 7, pp. 45-49.
Gladwin, Christina H. 1989. Ethnographic Decision Tree Modeling. Qualitative Research
Methods Series, Vol. 19. Newbury Park, CA: Sage.
Glaser, Barney G. 2001. "Doing Grounded Theory." Grounded Theory Review 2:1-18.
. 2000. "The Future of Grounded Theory." Grounded Theory Review 1:1-8.
, ed. 1993. Examples of Grounded Theory: A Reader. Mill Valley, CA: Sociology
Press.
. 1978. Theoretical Sensitivity: Advances in the Methodology of Grounded Theory. Mill
Valley, CA: Sociology Press.
Glaser, Barney G. and Anselm L. Strauss. 1967, Discovery of Grounded Theory: Strategies
for Qualitative Research. Chicago: Aldine.
Glass, Ronald David. 2001. "On Paulo Freire's Philosophy of Praxis and the Founda-
tions of Liberation Education." Educational Researcher 30 (2): 15-25.
Glazer, Myron. 1972. The Research Adventure: Promise and Problems of Fieldwork. New
York: Random House.
Gleick, James. 1987. Chos: Making a New Science. New York: Penguin.
Glennon, Lynda M. 1983. "Synthesism: A Case of Feminist Methodology." Pp. 260-71 in
Beyond Method, edited by Gare th Morgan. Beverly Hills, CA: Sage.
Glesne, Corrine. 1999. Becoming Qualitative Researchers: An Introduction. 2d ed. New
York: Longman.
. 1997. "That Rare Feeling: Representing Research Through Poetic Transcrip-
tion." Qualitative Inquiry 3 (2, June): 202-21.
Gluck, Sherna Berger and Daphne Patai, eds. 1991. Women's Words: The Feminist Practice
of Oral History. New York: Routledge.
Godet, Michel. 1987. Scenarios and Strategic Management. London: Butterworths.
Goffman, Erving. 1961. Asylums: Essays on the Social Situation of Mental Patients and Other
Inmates. Garden City, NY: Anchor.
Golden-Biddle, Karen and Karen D. Locke. 1997. Composng Qualitative Research. Thou-
Golembiewski, Bob. 2000. "Three Perspectives on Appreciative Inquiry." OD Practitio-
ner 32 (1): 53-58.
Goodall, H. L., Jr. 2000. Writing the New Ethnography. Walnut Creek, CA: AltaMira.
Goodenough, W. 1971. Culture, Language, and Society. Reading, MA: Addison-Wesley.
Goodson, Ivor and Martha Foote. 2001. "Testing Times: A School Case Study." Education
Policy Analysis Archives 9 (2, January 15): 1-10.
Gore, Jennifer M. andKennethM. Zeichner. 1995. Connecting Action Research to Genuine

Teacher Development. Pp. 203-14 in Criticai Discourses on Teacher Development, edited
by John Smyth. London: Cassell.
Graham, Robert J. 1993. "Decoding Teaching: The Rhetoric and Politics of Narrative
Form." Journal of Natural Inquiry 8 (1, fali): 30-37.
Graue, M. Elizabeth and Daniel J. Walsh. 1998. Studying Children in Context: Theories,
Methods, and Ethics of Studying Children. Thousand Oaks, CA: Sage.
Grbich, Carol. 1998. Qualitative Research in Health: An Introdnction. Thousand Oaks, CA:
Sage.
Greenbaum, Thomas L. 1997. The Handbook for Focus Group Research. 2d ed. Thousand
Oaks, CA: Sage.
Greene, Jennifer C. 2000. "Understanding Social Programs through Evaluation." Pp.
981-99 in Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin and
. 1998a. "Balancing Phiiosophy and Practicality in Qualitative Evaluation." Pp.
35-49 in Proceedings ofthe Stake Symposium on Educational Evaluation, edited by Rita
Davis. Champaign/Urbana: University of Illinois.
. 1998b. "Qualitative Interpretive Interviewing." Pp. 135-54 in Educational Re-
search for Educational Productivity. Advances in Educational Productivity, Vol. 7, ed-
ited by A. J. Reynolds and H. J. Walberg. Greenwich, CT: JAI.
. 1990. "Technical Quality Versus User Responsiveness in Evaluation Practice."
Evaluation and Program Planning 13 (3): 267-74.
Greig, Anne and layne Taylor. 1998. Doing Research With Children. Thousand Oaks, CA:
Sage.
Grinder, John, J. DeLozier, and R. Bandler. 1977. Patterns ofthe Hypnotic Techniques of Mil-
ton Erickson, M.D. Vol. 2. Cupertino, CA: Meta Publica tions.
Guba, Egon G,, ed. 1991. The Paradigm Dialog. Newbury Park, CA: Sage.
. 1981. "Investigative Reporting." Pp. 67-86 in Metaphors for Evaluation, edited by
Nick L. Smith. Beverly Hills, CA: Sage.
. 1978. Toward a Methodology of Naturalistic Inquiry in Educational Evaluation. CSE
Monograph Series in Evaluation No. 8. Los Angeles: Center for the Study of Evalua-
tion, University of Califrnia, Los Angeles.
Guba, Egon G. and Yvonna S. Lincoln. 1990. "Can There Be a Human Science?" Person-
Centered Revieio 5 (2): 130-54.
. 1989. Fourth Generation Evaluation. Newbury Park, CA: Sage.
. 1988. "Do Inquiry Paradigms Imply Inquiry Methodologies?" Pp. 89-115 in
Qualitative Approaches to Evaluation in Education: The Silent Scientific Revolution, ed-
ited by D. Fetterman. New York: Praeger.
. 1981. Effective Evaluation: Improving the Usefulness of Evaluation Results Through
Responsive and Naturalistic Approaches. San Francisco: Jossey-Bass.
Gubrium, Jaber F. and James Holstein. 2000. "Analyzing Interpretive Practice." Pp.
487-508 in Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin and
Gubrium, Jaber F. and Andra Sankar. 1993. Qualitative Methods in Aging. Newbury
Park, CA: Sage.
Guerrero, Sylvia H., ed. 1999a. Gender-Sensitive & Feminist Methodologies: A Handbook for
Health and Social Researchers. Quezon City: University of the Philippines Center for
Women's Studies.
, ed. 1999b. Selected Readings on Health and Feminist Research: A Sourcebook.
Quezon City: University of the Philippines Center for Women's Studies.
Guerrero-Manalo, Stella. 1999. Child Sensitive Interviewing: Pointers in Interviewing
Child Victims of Abuse." Pp. 195-203 in Gender-Sensitive & Feminist Methodologies: A
Handbook for Health and Social Researchers, edited by S. H. Guerrero. Quezon City:
References J. R15
University of the Philippines Center for Women's Studies.

Hacking, Ian. 2000. The Social Construction of What Cambridge, MA: Harvard Univer-
sity Press.
Ha 11, Nina, ed. 1993. Exploring Chos: A Guide to the New Science of Disorder. New York:
Norton.
Hallowell, L. 1985. "The Outcome of the Brajuha Case: Legal Implications for Sociolo-
gists." Footnotes, American Sociological Association 13 (1): 13.
Hamel, Jacques with S. Dufour and D. Fortin. 1993. Case Study Methods. Qualitative Re-
search Methods Series, Vol. 32. Newbury Park, CA: Sage.
Hamon, Raeann R. 1996. "Bahamian Life as Depicted by Wives' Tales and Other Old
Sayings." Pp. 57-88 in The Methods and Methodologies of Qualitative Family Research,
edited by Marvin B. Sussman and Jane F. Gilgun. New York: Haworth.
Handwerker, W. Penn. 2001. Quick Ethnography. Walnut Creek, CA: AltaMira.
Harding, Sandra. 1991. Whose Science? Whose Knowledge? Thinking From Women's Lives.
Ithaca, NY: Cornell University Press.
Harkreader, Steve A. and Gary T. Henry. 2000. "Using Performance Measurement Sys-
tems for Assessing the Merit and Worth of Reforms." American Journal of Evaluation
21 (2, spring-summer): 151-70.
Harper, Douglas. 2000. "Reimagining Visual Methods. Pp. 717-32 in Handbook of Quali-
tative Research. 2d ed., edited by Norman K. Denzin and Yvonna S. Lincoln. Thou-
Harris, P. R. and R. T. Moran. 1979. Managing Cultural Differences. Houston, TX: Gulf.
Hart, L.K. 1999. "Culture, Civilization, and Demarcation at the Northwest Borders of
Greece." American Ethnologist 26 (1, February): 196-220.
Harvey, C. and I. Denton. 1999. "To Come of Age: The Antecedents of Organizational
Learning." Journal of Management Studies 36 (7, December): 897-918.
Harwood, Richard R. 1979. Small Farm Development: Understanding and Improving
Farming Systems in the Humid Tropics, Boulder, CO: Westview.
Hausman, Carl. 2000. Lies We Live By. New York: Routledge.
Hawka, S. 1986. The Experience ofFeeling Unconditionally Loved. Ann Arbor, MI: Univer-
sity of Microfilms International. Doctoral thesis, Graduate College, The Union Insti-
tute, Cincinnati, OH.
Hayano, D. M. 1979. "Autoethnography: Paradigms, Problems, and Prospects." Human
Organization 38:113-20.
Hayes, T. A. 2000. "Stigmatizing Indebtedness: Implications for Labeling Theory. Sym-
bolic Interaction 23 (1): 29-46.
Headland, T., Kenneth Pike, and M. Harris, eds. 1990. "Emics and Etics: The Insider/
Outsider Debate." Frontiers of Anthropology 7.
Heap, James L. 1995. "Constructionism in the Rhetoric and Practice of Fourth-Genera-
tion Evaluation." Evaluation and Program Planning 18 (1): 51-61.
Hbert, Yvonne M. 1986. "Naturalistic Evaluation in Practice: A Case Study." Nezu Direc-
tionsfor Program Evaluation 30 (March): 3-22, Naturalistic Evaluation, edited by David
D. Williams. San Francisco: lossey-Bass.
Heilein, Robert A. 1973. The Notebooks of Lazarus Long. New York: G.P.Putnam's
Sons.
Helmer, Olaf. 1983. Looking Forward: A Guide to Futures Research. Beverly Hills, CA: Sage.
Hendricks, Michael. 1982. "Oral Policy Briefings." Pp. 249-58 in Communication Strat-
egies in Evaluation, edited by N. L. Smith. Beverly Hills, CA: Sage.
Heron, John. 1996. Cooperative Inquiry: Research Into the Human Condition. Thousand
Oaks, CA: Sage.
Hertz, Rosanna, ed. 1997. Reflexivity and Voice. Thousand Oaks, CA: Sage.
Heydebrand, Wolf V. 1983. "Organization and Praxis." Pp. 306-20 in Beyond Method,
edited by Gareth Morgan. Beverly Hills, CA: Sage.
Higginbotham, J. B. and K. K. Cox. 1979. Focus Group Interviews. Chicago: American

Marketing Association.
Hill, Michael R. 1993. Archival Strategies and Techniques: Analytical Field Research. Quali-
tative Research Methods Series, Vol. 31. Newbury Park, CA: Sage.
Hirsh, Sandra K. and Jean M. Kummerow. 1987. Introduction to Type in Organizational
Settings. Paio Alto, CA: Consulting Psychologists Press.
Hodder, Ian. 2000. "The Interpretation of Documents and Material Culture." Pp. 703-15
in Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin and Yvonna
S. Lincoln. Thousand Oaks, CA: Sage.
Hoffman, Lynn. 1981 .Foundations of Family Therapy: A Conceptual Frameworkfor Systems
Theory. New York: Basic Books.
Holbrook, Terry L. 1996. "Document Analysis: The Contrast Between Official Case Re-
cords and the Journal Woman on Welfare." Pp. 41-56 in The Methods and Methodologies
of Qualitative Family Research, edited by Marvin B. Sussman and Jane F. Gilgun. New
York: Haworth.
Holland, J. H. 1998. Emergence: From Chos to Order. Reading, MA: Helix.
. 1995. Hidden Order: How Adaptation Builds Complexity. Reading, MA: Perseus.
Holley, Heather and Jlio Arboleda-Florez. 1988. "Utilization Isn't Everything." Cana-
dian Journal of Program Evaluation 3 (2, October/November): 93-102.
Hollinger, David. A. 2000. "Paradigms Lost." The New York Times Book Revieiu, May 28,
p. 23.
Holmes, Robyn M. 1998. Fieldwork With Children. Thousand Oaks, CA: Sage.
Holstein, James A. and Jaber F. Gubrium. 1995. The Active Interview. Qualitative Re-
search Methods Series, Vol. 37. Thousand Oaks, CA: Sage.
Holte, John, ed. 1993. Chos: The New Science. Nobel Conference 26. Saint Peter, MN:
Gustavus Adolphus College.
Holtzman, JohnS. 1986. "Rapid Reconnaissance Guidelines for Agricultural Marketing
and Food Systems Research in Developing Countries." Working Paper No. 30, De-
partment of Agricultural Economics, Michigan State University, Lansing.
Hopson, Rodney, ed. 2000. How and Why Language Matters in Evaluation. New Directions
for Evaluation 86 (summer). San Francisco: Jossey-Bass.
House, Ernest. 1991. "Confessions of a Responsive Goal-Free Evaluation." Evaluation
Practice 12 (1, February): 109-13.
. 1978. "Assumptions Underlying Evaluation Models." Educational Researcher
7:4-12.
. 1977. The Logic of Evaluative Argument. CSE Monograph Series in Evaluation
No. 7. Los Angeles: Center for the Study of Evaluation, University of Califrnia, Los
Angeles.
House, E. R. and K. R. Howe. 2000. "Deliberative Democratic Evaluation." New Direc-
tions for Evaluation 85 (spring): 3-12, Evaluation as a Democratic Process: Promoting In-
clusion, Dialogue, and Deliberation, edited by Katherine E. Ryan and Lizanne
DeStefano. San Francisco: Jossey-Bass.
Huff, Darrell and Irving Gis. 1993. Hoio to Lie With Statistics. New York: Norton.
Hull, Bill. 1978. Teachers' Seminars on Children's Thinking. North Dakota Study Group on
Evaluation monograph series. Grand Forks: Center for Teaching and Learning, Uni-
versity of North Dakota.
Human Services Research Institute. 1984. Assessing and Enhancing the Quality ofHmnan
Services. Boston: Human Service Research Institute.
Humphrey, Derek. 1991. Final Exit. Eugene, OR: Hemlock Society.
Humphreys, Laud. 1970. Tearoom Trae: Impersonal Sex in Public Places. New York: Al-
dine de Gruyter.
Hunt, Scott A. and Robert D. Benford. 1997. "Dramaturgy and Methodology." Pp. 106-
18 in Context and Method in Qualitative Research, edited by Gale Miller and Robert
References J. R17

Hurty, Kathleen. 1976. "Reportby the Women's Caucus." Proceedings: Educational Evalu-
ation and Public Policy, A Conference. San Francisco: Far West Laboratory for Educa-
tional Research and Development.
Husserl, Edmund. 1967. "The Thesis of the Natural Standpoint and Its Suspension. "
Pp. 68-79 in Phenomenology, editedby J. J. Kockelmans. GardenCity, NY: Doubleday.
. 1913. Ideas. London: George AUen and Unwin. Republished 1962, New York:
Collier.
Ihde, D. 1977. Experimental Phenomenology. New York: Putnam.
Ivanic, Roz. 1998. Writing and Identity: The Discoursal Construction ofldentity in Acaemic
Writing. Studies in Written Language and Literacy, Vol. 5. Amsterdam: lohn
Benjamins.
Jacob, Evelyn. 1988. "Clarifying Qualitative Research: A Focus on Traditions." Educa-
tional Research 17 (1, January-February): 16-24.
. 1987. "Qualitative Research Traditions: A Review." Review of Educational Re-
search 57 (1): 1-50.
James, William. [1902] 1999. The Varieties of Religious Experience. New York: Random
House.
lanesick, Valerie J. 2000. "The Choreography of Qualitative Research Design: Minuets
Improvisations, and Crystalization. Pp. 379-99 in Handbookof Qualitative Research. 2d
ed., editedby NormanK. Denzin and YvonnaS. Lincoln. Thousand Oaks, CA: Sage.
. 1998. "Stretching" Exercises for Qualitative Researchers. Thousand Oaks, CA:
Sage.
Janowitz, Morris. 1979. "Where Is the CuttingEdge of Sociology?" Sociological Quarterly
20: 591-93.
Jarvis, Sara. 2000. Getting the Log Out of Our Own Eyes: An Exploration of Individual and
Team Learning in a Public Human Services Agency. Unpublished doctoral dissertation,
Graduate College, The Union Institute, Cincinnati, OH.
Jervis, Kathe. 1999. Between Home and School: Cultural Interchange in an Elementary Class-
room. Teacher's College, Columbia University. New York: National Center for Re-
structuring Education, Schools and Teaching.
lohnson, Allen and Ross Sackett, 1998. "Direct Systematic Observation of Behavior."
Pp. 301-31 iriHandbook of Methods in Cultural Anthropology, editedby H. Russell Ber-
nard. Walnut Creek, CA: AltaMira.
lohnson, Jeffrey C. 1990. Selecting Ethnographic Informants. Qualitative Research
Johnson, John M. 1975. Doing Field Research. Beverly Hills, CA: Sage.
Johnston, Bruce F., Allen Hoben, D. W. Dijkerman, and W. K. Jaeger. 1987. An Assessment
ofA.I.D. Activities to Promote Agricultural and Rural Development in Sub-Saharan frica.
A.I.D. Evaluation Special Study No. 54. Washington, DC: U.S. Agency for Interna-
tional Development.
Johnstone, B. 2000. "The Individual Voice in Language." Annual Review of Anthropology
29:405-24.
Joint Committee on Standards for Educational Evaluation. 1994. The Standards for Pro-
gram Evaluation. Thousand Oaks, CA: Sage.
Jones, JamesH. 1993. Bad Blood: The Tuskegee Syphilis Experiment. New York: Free Press.
Jones, Michael Owen. 1996. Studying Organizational Sxjmbolism. Qualitative Research
Methods Series, Vol. 39. Thousand Oaks, CA: Sage.
Jorgensen, Danny L. 1989. Participant Observation: A Methodology for Human Studies.
lunker, Buford H. 1960. Field Work: An Introduction to the Social Sciences. Chicago: Uni-
versity of Chicago Press.
Juran, Joseph M. 1951. Quality Control Handbook. New York: McGraw-Hill.
"Kalamazoo Schools." 1974. American School Board Journal (April): 32-40.

Kanter, Rosabeth Moss. 1983. The Change Masters: bnovationfor Productivity in the Ameri-
can Corporation. New York: Simon & Schuster.
Kaplowitz, M. D. 2000. "Statistical Analysis of Sensitive Topics in Group and Individual
Interviews." Quality & Quantity 34 (4, November): 419-31.
Kartunnen, Lauri. 1973. "Remarks on Presuppositions." Presented at the Texas confer-
ence Performances, Conversational Implicature and Presuppositions, March.
Katz, Louis. 1987. The Experience of Personal Change. Unpublished doctoral dissertation,
Graduate College, The Union Institute, Cincinnati, OH.
Katzer, Jeffrey, Kenneth H. Cook, and Wayne W. Crouch. 1978. Evaluating Information: A
Guidefor Users of Social Science Research. Reading, MA: Addison-Wesley.
Kegan, Jerome. 1982. The Evolving Self: Problem and Process in Human Development. Cam-
bridge, MA: Harvard University Press.
Kelley, Tom and Jonathan Littman. 2001. The Art oflnnovation: Lessons in Creativity From
Ideo, America's Leading Design Firm. Garden City, NY: Doubleday.
Kemmis, Stephen and Robin McTaggart. 2000. "Participatory Action Research." Pp. 567-
606 in Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin and
Kenny, M. L. 1999. "No Visible Means of Support: Child Labor in Urban Northeast
Brazil." Human Organization 58 (4, winter): 375-86.
Kibel, Barry M. 1999. Success Stories as Hard Data: An Introduction to Results Mapping.
New York: Kluwer Academic/Plenum.
Kim, Daniel H. 1999. Introduction to Systems Thinking. Williston, VT: Pegasus Communi-
cations.
. 1994. Systems Archetypes II: Using Systems Archetypes to Take Effective Action.
Williston, VT: Pegasus Communications.
. 1993. Systems Archetypes I. Williston, VT: Pegasus Communications.
Kimmel, Allan J. 1988. Ethnics and Values in Applied Social Research. Newbury Park, CA:
Sage.
Kincheloe, Joe L. and Peter McLaren. 2000. "Rethinking Criticai Theory and Qualita-
tive Research." Pp. 279-313 in Handbook of Qualitative Research. 2d ed., edited by Nor-
man K. Denzin and Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
King, Jean A. 1995. "Involving Practitioners in Evaluation Studies: How Viable Is Col-
laborative Evaluation in Schools." Pp. 86-102 m Participatory Evaluation in Education:
Studies in Evaluation Use and Organizational Learning, edited by J. Bradley Cousins
and Lorna Earl. London: Falmer.
King, Jean A. and M. Peg Lonnquist. 1994a. "The Future of Collaborative Action Re-
search: Promises, Problems and Prospects." Unpublished paper, College of Educa-
tion, University of Minnesota, Minneapolis, based on a presentation at the annual
meeting of the American Educational Research Association, Atlanta, GA, 1993.
. 1994b. " A Review of Writing on Action Research: 1944-Present." Unpublished
paper, Center for Applied Research and Educational Improvement, University of
Minnesota, Minneapolis.
King J e a n A., Lyrrn L. Morris, and Carol T. Fitz-Gibbon. 1987. How to Assess Program Im-
plementation, Newbury Park, CA: Sage.
King, Jean A. and Ellen Pechman. 1982. Improving Evaluatiofi Use in Local Schools. Wash-
ington, DC: National Institute of Education.
Kirk, Jerome and M. L. Miller. 1986. Reliability and Validity in Qualitative Research.
Kleining, Gerhard and Harald Witt. 2000. "The Qualitative Heuristic Approach: A
Methodology for Discovery in Psychology and the Social Sciences. Rediscovering
the Method of Introspection as an Example." Frum: Qualitative Social Research [On-
linel 1 (1, January). Available from http://qualitative-research.net/fqs.
References J. R19
Kling, Jeffrey R., Jeffrey B. Liebman, and Lawrence F. Katz. 2001. "Bullets Don't Got No
Name: Consequences of Fear in the Ghetto." Paper presented at the conference
Mixed Methods sponsored by the MacArthur Network on Successful Pathways
Through Middle Childhood, January 25, Santa Monica, CA.
Kloman, Erasmus H., ed. 1979. Cases in Accountability: The Work of GAO. Boulder, CO:
Westview.
Kneller, G. F. 1984. Movements of Thought in Modern Education. New York: John Wiley.
Kopala, Mary and Lisa A. Suzuki. 1999. Using Qualitative Methods in Psychology. Thou-
Kramer, Peter D. 1993. Listening to Prozac. New York: Pengum.
Krenz, Claudia and Gilbert Sax. 1986. "What Quantitative Research Is and Why It Does-
n't Work." American Behavioral Scientist 30(1, September-October): 58-69.
Krishnamurti, J. 1964. Think on These Things. New York: Harper & Row.
Krueger, Otto and Janet M. Thuesen. 1988. Type Talk. New York: Delacorte.
Krueger, Richard A. 1997a. Analyzing and Reporting Focus Group Residts. The Focus
Group Kit, Vol. 6. Thousand Oaks, CA: Sage.
. 1997b. Developing Questions for Focus Groups. The Focus Group Kit, Vol. 3. Thou-
. 1997c. Moderating Focus Groups. The Focus Group Kit, Vol. 5. Thousand Oaks,
CA: Sage.
. 1994. Focus Group Interviews: A Practical Guide for Applied Research. 2d ed. Thou-
Krueger, Richard A. and Mary Anne Casey. 2000. Focus Group Intervieios: A Practical
Guide for Applied Research. 3d ed. Thousand Oaks, CA: Sage.
Krueger, Richard A. and lean A. King. 1997. Involving Community Members in Focus
Groups. The Focus Group Kit, Vol. 4. Thousand Oaks, CA: Sage.
Kuhn, Thomas. 1970. The Structure of Scientific Revolutions. Chicago: University of Chi-
cago Press.
Kuhns, Eileen and S. V. Martorana, eds. 1982. Qualitative Methods for Institutional Re-
search. San Francisco: Jossey-Bass.
Kulish, Nicholas. 2001. "Ancient Split of Assyrians and Chaldeans Leads to Mod-
ern-Day Battle Over Census." Wall Street Journal, March 12, p. 1.
Kushner, Saville. 2000. Personalizing Evaluation. London: Sage.
Kvale, Steinar. 1996. InterViews: An Introduction to Qualitative Research Interviewing.
. 1987. "Validity in the Qualitative Research Interview." Methods: A Journal for Hu-
man Science 1 (2, winter): 37-72.
Ladson-Billings, Gloria. 2000. "Racialized Discourses and Ethnic Epistemologies."
Pp. 257-77 in Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin
and Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
Lahey, Lisa, E. Souvaine, R. Kegan, R. Goodman, and S. Felix. n.d. (about 1988). "A
Guide to the Subject-Object Interview: Its Administration and Interpretation." Cam-
bridge, MA: Subject-Object Research Groups, Harvard Graduate School of Educa-
tion. Mimeo.
Lalonde, Bernadette I. D. 1982. "Quality Assurance." Pp. 352-75 in Handbook on Mental
Health Administration, edited by Michael J. Austin and William E. Hershey. San Fran-
cisco: lossey-Bass.
Lang, K. and G. E. Lang. 1960. "Decisions for Christ: Billy Graham in New York City." In
Identity and Anxiety, edited by M. Stein, A. J. Vidich, and D. M. White. New York: Free
Press.
Lather, P. 1986. "Research as Praxis." Harvard Educational Review 56 (3): 257-77.
Lawrence-Lightfoot, Sara. 2000. Respect: An Exploration. Cambridge, MA: Perseus.
. 1997. "Illumination: Framing the Terrain." Pp. 41-59 in The Art and Science ofPor-
traiture, by S. Lawrence-Lightfoot and J. H. Davis. San Francisco: Jossey-Bass.
Lawrence-Lightfoot, Sara and Jessica Hoffman Davis. 1997. The Art and Science ofPor-
traiture. San Francisco: Jossey-Bass.
LeCompte, MargaretD. and Jean Schensul. 1999. Designingand Conducting Ethnographic
Research. Ethnographer's Toolkit, Vol. 1. Walnut Creek, CA: AltaMira.
Lee, Penny. 1996. The WhorfTheory Complex: A Criticai Reconstruction. Amsterdam Studies
in Theory and History ofLinguistic Science, Series 3, Studies in History of Language. Vol.
81. Philadelphia: John Benjamins.
Lee, Thomas W. 1998. Using Qualitative Methods in Organizational Research. Thousand
Oaks, CA: Sage.
Leeuw F., R. Rist, and R. Sonnichsen, eds. 1993. Comparative Perspectives on Evaluation
and Organizational Learning. New Brunswick, NJ: Transaction.
Leonard, Elmore. 2001. "Anecdotes." Week in Review. New York Times, March 11, p. 7.
Levin, B. 1993. "Collaborative Research in and With Organizations." Qualitative Studies
in Education 6 (4): 331-40.
Levin-Rozalis, Miri. 2000. "Abduction: A Logical Criterion for Programme and Project
Evaluation." Evaluation 6 (4): 415-32.
Levi-Strauss, Claude. 1966. The Savage Mind. 2d ed. Chicago: University of Chicago
Press.
Levitt, Norman. 1998. "Why Professors Believe Weird Things." Skeptic 6 (3): 28-35.
Levy, P. F. 2001. "The Nutlsland Effect: When Good Teams Go Wrong." Harvard Business
Review 79 (3): 51-59,163.
Lewis, P. J. 2001. "The Story of I and the Death of a Subject." Qualitative Inquiry 7 (1, Feb-
ruary): 109-28.
Lieblich, Amia, Rivka Tuval-Mashiach, and Tamar Zilber. 1998. Narrative Research: Read-
ing, Analysis, and Interpretation. Thousand Oaks, CA: Sage.
Liebow, Elliot. 1967. Tally's Comer. Boston: Little, Brown.
Lincoln, Yvonna S. 1990. "Toward a Categorical Imperative for Qualitative Research."
Pp. 277-95 in Qualitative Inquiry in Education: The Continuing Debate, edited by Elliot
Eisner and Alan Peshkin. New York: Teachers Coliege Press.
. 1985. Organizational Theory and Inquiry: The Paradigm Revolution. Beverly Hills,
CA: Sage.
Lincoln, Yvonna S. and Egon G. Guba. 2000. "Paradigmatic Controversies, Contradic-
tions, and Emerging Confluences." Pp. 163-88 in Handbookof Qualitative Research.
Sage.
. 1986. "But Is It Rigorous? Trustworthiness and Authenticity in Naturalistic
Evaluation." New Directions for Program Evaluation 30 (summer): 73-84, Naturalistic
Evaluation, edited by David D. Williams. San Francisco: Jossey-Bass.
. 1985. Naturalistic Inquiry. Beverly Hills, CA: Sage.
Lofland, John. 1971. Analyzing Social Settings. Belmont, CA: Wadsworth.
Lofland, John and L. H. Lofland. 1984. Analyzing Social Settings. Belmont, CA: Wads-
worth.
Lonner, Walter J. and John W. Berry. 1986. Field Methods in Cross-Cultural Research.
Louis, M. R. 1983. "Organizations as Culture Bearing Milieux." In Organizational Sym-
bolism, edited by L. R. Pondy, G. Morgan, P. J. Frost, Samuel B. Bacharach, & T. C.
Dandridge. Greenwich, CT: JAI.
Love, Amold 1.1991. Internai Evaluation: Building Organizations From Within. Newbury
Park, CA: Sage.
Mabry, L., ed. 1997. Evaluation and thePostmodem Dilemma. Advances inProgram Evalu-
ation, Vol. 3. Greenwich, CT: JAI.
References J. R21
MacBeth, Douglas. 2001. On Reflexivity in Qualitative Research." Qualitative Inquiri/

7(1): 35-68.
MacDonald, B. 1987. "Evaluation and Control of Education." In Issues and Methods in
Evaluation, edited by R. Murphy and H. Torrance. London: Paul Chapman.
MacQueen, Kathleen M. and Bobby Milstein. 1999. "A Systems Approach to Qualitative
Data Management and Analysis." Field Methods 11 (1): 27-39.
Madriz, Esther. 2000. "Focus Groups in Feminist Research." Pp. 835-50 in Handbook of
Qualitative Research. 2d ed., edited by Norman K. Denzin and Yvonna S. Lincoln.
Maguire, Patrcia. 1996. "Considering More Feminist Participatory Research: What's
Congruency Got to Do With It?" Qualitative Inquiry 2 (1, March): 106-18.
Mairs, Nancy. 1997. Voice Lessons: On Becoming a (Woman) Writer. Boston: Beacon.
Manning, Peter K. 1987. Semiotics and Fieldwork. Qualitative Research Methods Series,
Vol. 7. Newbury Park, CA: Sage.
Marino, Rocco A. 1985. How Adolescent Sons Perceive and Describe the Impact of the Fa-
ther-Son Relationship on Their Own Sense of Self-Identity. Doctoral dissertation, Gradu-
ai e College, The Union Institute, Cincinnati, OH.
Mark, M. M., G. T. Henry, and G. Julnes. 2000. Evaluation: An Integra ted Framework for
Understanding, Gniding, and Improving Public and Nonprofit Policies and Programs. San
Francisco: Jossey-Bass.
Marshall, Catherine and Gretchen Rossman. 1989. Designing Qualitative Research. New-
bury Park, CA: Sage.
Marx, Leo. 1999. "The Struggle Over Thoreau." The New York Review of Books 46 (11):
60-64.
Maslow, Abraham H. 1966. The Psychology of Science. New York: Harper & Row.
. 1956. "Toward a Humanistic Psychology." Etc. 13:10-22.
Mathews, Ruth, J. K. Matthews, and Kathleen Speltz. 1989. Female Sexual Ojfenders. Or-
well, VT: Safer Society Press.
Matthews, Jane K., Jodie Raymaker, and Kathleen Speltz. 1991. "Effects of Reunification
on Sexually Abusive Families." Pp, 147-61 in Family Sexual Abuse: Frontline Research
and Evaluation, edited by Michael Quinn Patton. Newbury Park, CA: Sage.
Maxwell, Joseph A., Philip G. Bashook, and Leslie J. Sandlow. 1987. "Combining
Ethnographic and Experimental Methods in Educational Evaluation: A Case Study."
Pp. 568-90 in Evaluation Studies Reviezv Annnal, No. 12, edited by William R. Shadish,
Jr. and Charles S. Reichardt. Newbury Park, CA: Sage.
McClure, Gail. 1989. Organizational Culture as Manifest in Criticai Incidents: A Case Study
ofthe Faculty ofAgriculture, University ofthe West Indies. Unpublished doctoral disser-
tation, University of Minnesota, Minneapolis.
McCracken, Grant. 1988. The Long Interview. Qualitative Research Methods Series, Vol. 13.
McGuigan, Jim. 1998. Cultural Methodologies. Thousand Oaks, CA: Sage.
McLaughlin,Milbrey. 1976. "Implementation as Mutual Adaptation." In Social Program
Implementation, edited by Walter Williams and Richard F. Elmore. New York: Aca-
demic Press.
McNamara, Carter. 1996. Evaluation of a Group-Managed, Multi-Technique Management
Development Program That Includes Action Learning. Unpublished doctoral disserta-
tion, Graduate College, The Union Institute, Cincinnati, OH.
Mead, George H. 1934. Mind, Self and Society. Chicago: University of Chicago Press.
Mead, Margaret. 1977. Letters From the Field, 1925-1975. New York: Harper & Row.
Meeker, Joseph W. 1980. The Comedy of Survival: In Search ofan Environmental Ethic. Los
Angeles: Guild of Tutors Press. Reprinted 1997, University of Arizona Press.
Merleau-Ponty, Maurice. 1962. The Phenomenology of Perception. London: Routledge &
Kegan Paul.
Merriam, John E. and Joel Makower. 1988. Trend Watching: How the Media Create Trends
and How to Be the First to Uncover Them. New York: Tilden Press, American Manage-
ment Association (AMACOM).
Merriam, Sharon. 1997. Qualitative Research and Case Study Applications in Education. San
Francisco: Jossey-Bass.
Mertens, Donna M. 1999. "Inclusive Evaluation: Implications of Transforma tive Theory
for Evaluation. American Journal of Evaluation 20 (1, winter): 1-14.
. 1998. Research Methods in Education and Psychology: Integrating Diversity With
Quantitative and Qualitative Approaches. Thousand Oaks, CA: Sage.
Merton, R., M. Riske, and P. L. Kendall. 1956. The Focused Intervieiu. New York: Free
Press.
Messick, S. 1989. "Validity." Pp. 13-103 in Educational Measurement. 3d ed., edited by
R. L. Linn. New York: American Council on Education/Macmillan.
Meyers, William R. 1981. The Evaluation Enterprise. San Francisco: Jossey-Bass.
Miles, Matthew B. and A. M. Huberman. 1994. Qualitative Data Analysis: An Expanded
Sourcebook. 2d ed. Newbury Park, CA: Sage.
. 1984. Qualitative Data Analysis: A Sourcebook ofNezv Methods. Beverly Hills, CA:
Sage.
Milgram, Stanley. 1974. Obedience to Authority. New York: Harper & Row.
Milius, Susan. 1998. "When Worlds Collide." Science 154 (6): 92-93.
Miller, Gale. 1997. "Contextualizing Texts: Studymg Organizational Texts." Pp. 77-91 in
Context and Method in Qualitative Research, edited by Gale Miller and Robert
Miller, Sally and Patrcia Winstead-Fry. 1982. Family Systems Theory andNursing Practice.
East Norwalk, CT: Appleton & Lange.
Miller, William L. and Benjamin F. Crabtree. 2000. "Clinicai Research." Pp. 607-32 in
Mills, C. Wright. 1961. The Sociological Imagination. New York: Oxford University Press.
Minnich, Elizabeth. Forthcoming. Transforming Knowledge. 2d ed. Philadelphia: Temple
University Press.
. 1999. "What's Wrong With Civic Life? Remembering Political Wellsprings of
U.S. Democratic Action." The Good Society 9 (2): 7-14.
. 1990. Transforming Knowledge. Philadelphia: Temple University Press.
Mitchell, Richard. 1979. Less Than Words Can Say: The Underground Grammarian. Boston:
Little, Brown.
Mitchell, Richard G., Jr. 1993. Secrecy and Fieldwork. Qualitative Research Methods Se-
ries, Vol. 29. Newbury Park, CA: Sage.
Montgomery, Jason and Willard Fewer. 1988. Family Systems and Beyond. New York: Hu-
man Science Press.
Moos, Rudolf. 1975. Evaluating Correctional and Community Settings. New York: Wiley
Interscience.
Morgan, David L. 1997a. The Focus Group Guidebook. The Focus Group Kit, Vol. 1. Thou-
. 1997b. Planning Focus Groups. The Focus Group Kit, Vol 2. Thousand Oaks, CA:
Sage.
. 1988. Focus Groups as Qualitative Research. Qualitative Research Methods Se-
ries, Vol. 16. Newbury Park, CA: Sage.
Morgan, Gareth. 1989. Creative Organizational Theory: A Resourcebook. Newbury Park,
CA: Sage.
. 1986. Images of Organization. Beverly Hills, CA: Sage.
, ed. 1983. Beyond Methods: Strategies for Social Research. Beverly Hills, CA: Sage.
Morris, Edmund. 2000. Dutch: A Memoir ofRonald Reagan. New York: Random House.
References 13. R23
Morris, M. W. 2000. "The Lessons We (Don't) Learn: Counterfactual Thinking and Orga-
nizational Accountability After a Close Call." Administrativa Science Quarterly 45 (4):
737-65.
Morrison, David. 1999. "The Role of Observation." Skeptical Briefs 9 (1): 8.
Morse, Janice M., ed. 1997. Completing a Qualitative Project. Thousand Oaks, CA: Sage.
. 1991. Qualitative Nursing Research. Newbury Park, CA: Sage.
Morse, Janice M. and Peggy Anne Field. 1995. Qualitative Research Methods for Health Pro-
fessionals. Thousand Oaks, CA: Sage.
Morse, Janice M., Janice Penrod, and Judith Hupcey. 2000. "Qualitative Outcome Anal-
ysis: Evaluating Nursing Interventions for Complex Clinicai Phenomena." Journal of
Nursing Scholarship 32 (2): 125-30.
Moustakas, Clark. 1997. Relationship Play Therapy. Northvale, NJ: Jason Aronson.
. 1995. Being-In, Being-For, Being-With. Northvale, NJ: Jason Aronson.
. 1994. Phenomenological Research Methods. Thousand Oaks, CA: Sage.
. 1990a. "Heuristic Research: Design and Methodology." Person-Centered Review
5 (2): 170-90.
. 1990b. Heuristic Research: Design, Methodology, and Applications. Newbury Park,
CA: Sage.
. 1988. Phenomenology, Science and Psychotherapy. Sydney, Nova Scotia, Canada:
Family Life Institute, University College of Cape Breton.
. 1981. Rhythms, Rituais and Relationships. Detroit, MI: Center for Huinanistic
Studies.
. 1975. The Touch of Loneliness. Englewood Cliffs, NJ: Prentice Hall.
. 1972. Loneliness and Love. Englewood Cliffs, NJ: Prentice Hall.
. 1961. Loneliness. Englewood Cliffs, NJ: Prentice Hall.
Mueller, Marsha R. 1996. Immediate Outeomes ofLower-Income Participants in Minnesota's
Universal Access Early Childhood Fairly Education. St. Paul, MN: Department of
Children, Families, and Learning.
Mueller, Marsha R. and Jody Fitzpatrick. 1998. "Dialogue With Marsha Mueller." Amer-
ican Journal of Evaluation 19 (1): 97-98.
Murali, M. Lakshmanan, ed. 1995. Chos in Nonlinear Oscillators: Controllingand Synchro-
nization. World Scientific Series on Nonlinear Science, Series A: Monographs and
Treatises. New York: World Scientific.
Murray, Michael and Kerry Chamberlain. 1999. Qualitative Health Psychology: Theories
and Methods. Thousand Oaks, CA: Sage.
Mwaluko G. S. and T. B. Ryan. 2000. "The Systemic Nature of Action Learning
Programmes." Systems Research and Behavioral Science 17 (4, July-August): 393-401.
Myers, Isabel Briggs with Peter Meyers. 1995. Gifts Differing. Paio Alto, CA: Consulting
Psychologists Press.
Myrdal, Gunnar. 1969. Objectivity in Social Research. New York: Random House/Pan-
theon.
Nadei, Lynn and Daniel Stein, eds. 1995. The 1993 Lectures in Complex Systems. Santa Fe
Institute Studies in the Sciences of Complexity. Lectures, Vol. 6. Boulder, CO:
Perseus.
Nagel, Emest. 1961. The Structure of Science. New York: Harcourt, Brace and World.
Naisbitt, John. 1982. Megatrends: Ten Neiu Directions Transforming Our Lives. New York:
Warner Books.
Naisbitt, John and Patricia Aburdene. 1990: Megatrends 2000: Ten New Directions for the
1990s. New York: William Morrow.
Nash, Roderick. 1986. Wilderness and the American Mind. New Haven, CT: Yale Univer-
sity Press.
Neimeyer, Greg ]., ed. 1993. Constructivist Assessment: A Casebook. Newbury Park, CA:
Sage.
Newman, Diana and Robert Brown. 1996. Applied Ethicsfor Program Evaluation. Thou-
Noblit, George W. and R. Dwight Hare. 1988. Meta-Ethnography: Synthesizing Qualitative
Studies. Newbury Park, CA: Sage.
Nussbaum, Martha. 2001. "Disabled Lives: Who Cares?" The New York Review o/Books 48
(1, January 11): 34-37.
Oakley, A. 1981. "Interviewing Women: A Contradiction in Terms." Pp. 30-61 in Doing
Eeminist Research, edited by H. Roberts. London: Routledge & Kegan Paul.
Ogbor, I- O. 2000. "Mythicizing and Reification in Entrepreneurial Discourse: Ideol-
ogy-Critique of Entrepreneurial Studies." Journal of Management Studies 37 (5, luly):
605-35.
Olesen, Virginia L. 2000. "Feminisms and Qualitative Research At and Into the Millen-
nium." Pp. 215-56 in Handbook of Qualitative Research. 2d ed., edited by Norman K.
Denzin and Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
Olson, Ruth Anne. 1974. "A Value Perspective on Evaluation." Marcy Open School,
Minneapolis Public Schools. Mimeo.
Ormerod, Paul. 2001. Butterfly Economics: A New General Theory of Social and Economic
Behavior. New York: Basic Books.
Owens, Thomas, Joseph F. Haehnn, and Harry L. Fehrenbacher. 1976. The Use of Multiple
Strategies in the Evaluation ofan Experience-Based Career Education Program. Research
Evaluation Development Paper Series No. 9. Portland, OR: Northwest Regional Ed-
ucational Laboratory.
Packer, Martin and Richard Addison. 1989. Enter ing the Circle: Hermeneutic Investigation
in Psychology. Albany: State University of New York Press.
Padgett, DeborahK. 1998. Qualitative Methods in Social Work Research: Challenges and Re-
wards. Thousand Oaks, CA: Sage.
Page, Reba N. 2000. "The Turn Inward in Qualitative Research." Pp. 3-16 in Acts of In-
quiry in Qualitative Research, edited by B. M. Brizuela, I- P. Stewart, R. G. Carrillo, and
J. G. Berger. Reprint Series No. 34. Cambridge, MA: Harvard Educational Review.
Palmer, Laura. 1988. Shrapjtel in the Heart. New York: Vintage.
Palmer, R. E. 1969. Hermeneutics. Evanston, IL: Northwestern University Press.
Palumbo, Dennis J., ed. 1987. The Politcs of Program Evaluation. Newbury Park, CA:
Sage.
Panati, Charles. 1987. Extraordinary Origins of Everyday Things. New York: Harper &
Row.
Parameswaran, Radhika. 2001. "Feminist Media Ethnography in ndia: Exploring
Power, Gender, and Culture in the Field." Qualitative Inquiry 7 (1, February): 69-103.
Park, Clair Claiborne with Oliver Sacks. 2001. Exiting Nirvana: A Daughter's Life With
Autism. Boston: Little, Brown.
Parlett, Malcolm and David Hamilton. 1976. "Evaluation as Illumination: A New Ap-
proach to the Study of Innovatory Programs." In Evaluation Studies Review Annual,
Vol. 1, edited by G. V. Glass. Beverly Hills, CA: Sage.
Partnow, Elaine. 1978. The Quotable Woman, 1800-0n. Garden City, NY: Anchor.
Patton, Michael Quirin. 2000. "Language Matters." Nezv Drections for Evaluation 86
(summer): 5-16, Hoiv and Why Language Matters in Evaluation, edited by Rodney
Hopson. San Francisco: lossey-Bass.
. 1999a. Grand Canyon Celebration: A Father-Son Journey ofDiscovery. Amherst, NY:
Prometheus.
. 1999b. "On Enhancing the Quality and Credibility of Qualitative Analysis."
Health Services Research 34 (5, Part 2, December): 1189-208.
. 1999c. "Organizational Development and Evaluation." Special issue of Cana-
dian Journal of Program Evaluation, pp. 93-113.
References J. R25
. 1999d. "Some Framing Questions About Racism and Evaluation." American

Journal of Evaluation 20 (3, fali): 437-51.
. 1998. "Discovering Process Use." Evaluation 4 (2): 225-33.
. 1997a. Utilization-Focused Evaluation: The New Century Text. 3d ed. Thousand
Oaks, CA: Sage.
. 1997b. "View Toward Distinguishing Empowerment Evaluation and Placing It
in a Larger Context." Evaluation Practice 18 (2): 147-63.
. 1996a. Inside the Doctoral Dissertation. 2-hr. videotape. Cincinnati, OH: The Un-
ion Institute. Online at www.tui.edu.
. 1996b. "A World Larger Than Formative and Summative." Evaluation Practice 17
(2): 131-44.
. 1994. "Developmental Evaluation." Evaluation Practice 15 (3): 311-20.
, ed. 1991. Family Sexual Abuse: Frontline Research and Evaluation. Newbury Park,
CA: Sage.
. 1990. "Humanistic Psychology and Qualitative Research: Shared Principies and
Processes." Person-Centered Review 5 (2): 191-202.
. 1988a. "Extension's Future: Beyond Technology Transfer." Knowledge 1 (4,June):
476-91.
. 1988b. "Integrating Evaluations Into a Program for Increased Utility and
Cost-Effectiveness." New Directions for Program Evaluation 39 (fali), Evaluation Utili-
zation, edited by John A. McLaughlin, Larry J. Weber, Robert W. Covert, and Robert
B. Ingle. San Francisco: Jossey-Bass.
. 1988c. "Paradigms and Pragmatism." Pp. 116-37 in Qualitative Approaches to
Evaluation in Education: The Silent Scientific Revolution, edited by David M. Fetter-
man. New York: Praeger.
. 1988d. "Query: The Future and Evaluation." Evaluation Practice 9 (4): 90-93.
. 1987a. Creative Evaluation. 2d ed. Newbury Park, CA: Sage.
. 1987b. "The Extension Organization of the Future." Journal of Extension 15
(spring): 22-24.
, ed. 1985. Culture and Evaluation. New Directions for Program Evaluation 25
(March). San Francisco: lossey-Bass.
. 1981. Practical Evaluation. Beverly Hills, CA: Sage.
. 1978. Utilization-Focused Evaluation. Beverly Hills, CA: Sage.
. 1975. Altemative Evaluation Research Paradigms. North Dakota Study Group on
Evaluation monograph series. Grand Forks: Center for Teaching and Learning, Uni-
versity of North Dakota.
Patton, Michael Quinn with Brandon Q. T. Patton. 2001. "What's in a Name? Heroic No-
menclatura in the Grand Canyon." Plateau Journal 4 (2, winter): 16-29.
Patton, Michael Quinn and Stacey Stockdill. 1987. "Summative Evaluation of the Tech-
nology for Literacy Center." St. Paul, MN: Saint Paul Foundation.
Paul, Jim. 1994. What I Learned Losing a Million Dollars. Chicago: Infrared.
Pawson, R. and N. Tilley. 1997. Realistic Evaluation. London: Sage.
Payne, Stanley L. 1951. The Art of Asking Questions. Princeton, NJ: Princeton University
Press.
Pedler, M., ed. 1991. Action Learning in Practice. Aldershot Hauts, UK: Gower.
Peito, Pertti J. and Gretel H. Peito. 1978. Anthropological Research: The Structure of Inquiry.
Cambridge, UK: Cambridge University Press.
Perkyl, Anssi. 1997. "Reliability and Validity in Research Based on Transcripts."
Pp. 201-20 in Qualitative Research: Theory, Method and Practice, edited by David
Silverman. London: Sage.
Percy, Walker. 1990. The Message in the Bottle. New York: Noonday.
Perls, Fritz. 1973. The Gestalt Approach and Eye Witness to Therapy. Paio Alto, CA: Science
and Behavior Books.
Perrone, Vito, ed. 1985. Portraits ofHigh Schools. Carnegie Foundation for the Advance-
ment of Teaching. Lawrenceville, NJ: Princeton University Press.
. 1977. The Abuses of Standardized Testing. Bloomington, IN: Phi Delta Kappa Edu-
cational Foundation.
Perrone, Vito and Michael Quinn Patton with Barbara French. 1976. Does Accountability
Count Without Teacher Support? Minneapolis: Minnesota Center for Social Research,
University of Minnesota.
Peshkin, Alan. 2001. "Angles of the Vision: Enhancing Perception in Qualitative Re-
search." Qualitative Inquiry 7 (2): 238-53.
. 2000a. "The Nature of Interpretation in Qualitative Research." Educational Re-
searcher 17 (7, October): 17-22.
. 2000b. Permissible Advantage? The Moral Consequences of Elite Schooling.
Mahwah, NJ: Lawrence Erlbaum,
. 1997. Places of Memory: Whiteman's Schools and Native American Communities.
Sociocultural, Political, and Historical Studies in Education. Mahwah, NJ: Lawrence
Erlbaum.
. 1988. "In Search of SubjectivityOne's Own." Educational Researcher 29 (9, De-
cember): 5-9.
. 1986. God's Choice: The Total World of a Fundamentalist Christian School. Chicago:
University of Chicago Press.
. 1985. "Virtuous Subjectivity: In the Participant-Observer's I's." Pp. 267-68 in
Exploring Clinicai Methods for Social Research, edited by David N. Berg and Kenwyn K.
Smith. Beverly Hills, CA: Sage.
Peters, Thomas J. 1987. Thriving on Chos: Handbookfor a Management Revolution. New
York: Knopf.
Peters, Thomas J. and Robert H. Waterman, Ir. 1982. In Search of Excellence: Lessons From
America's Best-Run Companies. New York: Harper & Row.
Pettigrew, Andrew M. 1983. "On Studying Organizational Cultures." Pp. 87-104 in
Qualitative Methodology, edited by lohn Van Maanan. Beverly Hills, CA: Sage.
Philliber, Susan. 1989. Workshop on Evaluating Adolescent Pregnancy Prevention Pro-
grams, Children's Defense Fund Conference, Washington, DC, March 10.
Pietro, Daniel Santo. 1983. Evaluation Sourcebook For Private and Voluntary Organizations.
New York: American Council of Voluntary Agencies for Foreign Service.
Pike, Kenneth. 1954. Language in Relation to a Unified Theory of the Structure of Human
Behavior. Vol. 1. University of Califrnia: Summer Institute of Linguistics. Repub-
lished in 1967, The Hague, the Netherlands: Mouton.
Pillow, Wanda S. 2000. "Deciphering Attempts to Decipher Postmodern Educational
Research." Educational Researcher 29 (5, June-July): 21-24.
Pirsig, Robert M. 1991. Lila: An Inquiry Into Morais. New York: Bantam.
. 1984. Zen and theArt of MotorcycieMaintenance: An Inquiry Into Values, New York:
Bantam.
Polanyi, Michael. 1967. The Tacit Dimension. Reprinted 1983. Magnolia, MA: Peter
Smith.
. 1962. Personal Knowledge. Chicago: University of Chicago Press.
Porter, Michael E. and Mark R. Kramer. 1999. "Philanthropy's New Agenda: Creating
Value." Harvard Business Review 78 (6, November-Deceinber): 121-30.
Potter, J. 1996. Representing Reality: Discourse, Rhetoric and Social Construction. London:
Sage.
Powdermaker, Hortense. 1966. Stranger and Friend. New York: Norton.
Preskill, Hallie and R. T. Torres. 1999. Evaluative Inquiry for Learning in Organizations.
Preskill, Stephen and Robin Smith Jacobvitz. 2000. Stories of Teaching: A Foundation for
Educational Renewal. Englewood Cliffs, NJ: Prentice Hall.
References J. R27
Preskill, Stephen L. and Hallie Preskill. 1997. "Meeting the Postmodern Challenge:
Pragmatism and Evaluative Inquiry for Organizational Learning." Pp. 155-69 in
Evaluation and the Postmodern Dilemma. Advances in Program Evaluation, Vol. 3, ed-
ited by L. Mabry. Greenwich, CT: JAI.
Pressley, Michael and Peter Afflerbach. 1995. Verbal Protocols of Reading: The Nature of
Constructively Responsive Reading. Mahwah, NJ: Lawrence Erlbaum.
Private Agencies Collaborating Together (PACT). 1986. Participatory Evaluation. New
York: Private Agencies Collaborating Together.
Program Evaluation Division (PED). 2001. Early Childhood Education Programs: Program
Evaluation Report. Report No. 01-01. St. Paul, MN: Office of the Legislative Auditor.
Punch, Maurice. 1997. Dirty Business: Exploring Corporate Misconduct. London: Sage.
. 1989. "ResearchingPolice Deviance: A Personal Encounter With theLimitations
and Liabilities of Fieldwork." British Journal of Sociology 40 (2): 177-204.
. 1986. The Politics and Ethics of Fieldwork. Qualitative Research Methods Series,
Vol. 3. London: Sage.
. 1985. Conduct Unbecoming: Police Deviance and Control. London: Tavistock.
Putnam, H. 1990. Realism With a Human Face. Cambridge, MA: Harvard University
Press.
. 1987. The Many Faces of Realism. LaSalle, IL: Open Court.
Radavich, David. 2001. "On Poetry and Pain." A View From the Loft 24 (6, January): 3-6,
17.
Ragin, Charles C. 2000. Fuzzy-Set Social Science. Chicago: University of Chicago Press.
. 1987. The Comparative Method: Moving Beyond Qualitative and Quantitative Strat-
egies. Berkeley: University of Califrnia Press.
Ragin, Charles C. and Howard S. Becker, eds. 1992. What Is a Case? Exploring the Founda-
tions of Social Inquiry. Cambridge, UK: Cambridge University Press.
Raia, Anthony and Newton Margulies. 1985. "Organizational Development: Issues,
Trends, and Prospects." Pp. 246-72 in Human Systems Development, edited by R. Tan-
nenbaum, N. Margulies, and F. Massarik. San Francisco: Jossey-Bass.
Ramachandran, V. S. and Sandra Blakeslee. 1998. Phantoms in the Brain: Probing theMys-
teries of the Human Mind. New York: William Morrow.
Reed, John H. 2000. "Paying for Interviews." Posting on EvalTalk Internet listserv of the
American Evaluation Association, September 1. Posted from Arlington, VA:
TecMRKT Works.
Reichardt, Charles S. and Thomas D. Cook. 1979. "Beyond Qualitative Versus Quantita-
tive Methods." Pp. 7-32 in Qualitative and Quantitative Methods in Evaluation Research,
edited by Thomas D. Cook and Charles S. Reichardt. Beverly Hills, CA: Sage.
Reichardt, Charles S. and Sharon F. Rallis, eds. 1994. The Qualitative-Quantitative Debate:
New Perspectives. New Directions for Program Evaluation 61 (spring). San Francisco:
Jossey-Bass.
Reinharz, Shulamit. 1992. Feminist Methods in Social Research. New York: Oxford Univer-
sity Press.
Rettig, Kathryn, Vicky Chiu-Wan Tam, and Beth Maddock Magistad. 1996. "Using Pat-
tern Matching and Modified Analytic Induction in Examining Justice Principies in
Child Support Guidelines." Pp. 193-222 in The Methods and Methodologies of Qualita-
tive Family Research, edited by Marvin B. Sussman and Jane F. Gilgun. New York:
Haworth.
Rhee, Y. 2000. "Complex Systems Approach to the Study of Politics." Systems Research
and Behavioral Science 17 (6, November-December): 487-91.
Rheingold, Howard. 2000. They Have a Word for It: A Lightheated Lexicon ofUntranslatable
Words and Phrases. 2d ed. Louisville, KY: Sarabande.
. 1988. They Have a Word for It: A Lightheated Lexicon of Untranslatable Words and
Phrases. Los Angeles: Tarcher.
Ribbens, Jane and Rosalind Edwards. 1998. Feminist Dilemmas in Qualitative Research;
Public Knowledge and Private Lives, London: Sage.
Richardson, Laurel. 2000a. "Evaluating Ethnography." Qualitative Inquiry 6 (2, June):
253-55.
. 2000b. "Writing: A Method of Inquiry." Pp. 923-48 in Handbook of Qualitative Re-
CA: Sage.
Richardson, Miles. 1998. "Poetics in the Field and on the Page." Qualitative Inquiry 4 (4,
December): 451-62.
Riessman, Catherine Kohler. 1993. Narrative Analysis. Newbury Park, CA: Sage.
Rist, Ray C. 2000. "Influencing a Policy Process With Qualitative Research." Pp. 1000-17
in Handbook of Qualitative Research. 2d ed., edited by Norman K. Denzin and Yvonna
S. Lincoln. Thousand Oaks, CA: Sage.
. 1977. "On the Relations Between Educational Research Paradigms: From Dis-
dain to Detente." Anthropology and Education Quarterly 8:42-49.
Robinson, C. A.,Jr. 1949. Alexander the Great, the Meeting ofEast and West in World Govern-
ment and Brotherhood. New York: Dutton.
Rog, Deborah. 1985. A Methodological Analysis of Evaluability Assessment. Unpublished
doctoral dissertation, Vanderbilt University, Nashville, TN.
Rogers, B. L. and M. B. Wallerstein. 1985. PL 480 Title I: Impact Evaluation Results and Rec-
ommendations. A.I.D. Program Evaluation Report No. 13. Washington, DC: U.S.
Agency for International Development.
Rogers, Carl. 1977. On Personal Power. New York: Delacorte.
. 1969. "Toward a Science of the Person." In Readings in Humanistic Psychology, ed-
ited by A. Sutich and M. Vich. New York: Free Press.
. 1961. On Becoming a Person. Boston: Houghton Mifflin.
Rogers, Everett. 1962. Dijfusion of Innovations. New York: Free Press.
Rogers, Patricia ]., Timothy A. Hacsi, Anthony Petrosino, and Tracy A. Huebner, eds.
2000. Program Theory in Evaluation: Challenges and Opportunities. New Directions for
Evaluation 87 (fali). San Francisco: Jossey-Bass.
Ronai, Carol Rambo. 1999. "The Next Night Sous Rature: Wrestling With Derrida's Mi-
mesis." Qualitative Inquiry 5 (1, March): 114-29.
Rorty, Richard. 1994. "Method, Social Science, and Social Hope." In The Postmodern
Tum: New Perspectives on Social Theory, edited by Stephen Seidman. Cambridge, UK:
Cambridge University Press.
Rose, Dan. 1990. Living the Ethnographic Life. Qualitative Research Methods Series, Vol. 23.
Roseanne. 2001. " W h a t T v e Learned." Esquire (March): 194.
Rosenblatt, Paul C. 1985. The Family in Business. San Francisco: lossey-Bass.
Rosenthal, Rob. 1994. Homeless in Paradise: A Map of the Terrain. Philadelphia: Temple
University Press.
Rossi, Peter H., Howard E. Freeman, and Mark W. Lipsey. 1999. Evaluation: A Systematic
Approach. 6th ed. Thousand Oaks, CA: Sage.
Rossi, Peter H. and W. Williams, eds. 1972. Evaluating Social Programs: Theory, Practice,
ajd Politics. New York: Seminar Press.
Rossman, Gretchen B. and Sharon F. Rallis. 1998. Learning in the Field: An Introduction to
Qualitative Research. Thousand Oaks, CA: Sage.
Rubin, Herbert J. and Irene S. Rubin. 1995. Qualitative Interviewing: The Art of Hearing
Data. Thousand Oaks, CA: Sage.
Rudestam, Kjell E. and RaeR. Newton. 1992. Survivmg Your Dissertation. Newbury Park,
CA: Sage.
Ruhleder, Karen. 2000. "The Virtual Ethnographer: Fieldwork in Distributed Electronic
Environments." Field Methods 12 (1, February): 3-17.
References J. R29
Ryan, Gery W. and H. Russell Bemard. 2000. "Data Management and Analysis
Methods." Pp. 769-802 in Handbook of Qualitative Research. 2d ed., edited by Norman
K. Denzin and Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
Sacks, Oliver. 1985. The Man Who Mis took His Wifefor a Hat. New York: Summit.
. 1973. Awakenings. New York: Harper & Row.
Safire, William and Leonard Safire. 1991. Leadership. New York: Fireside.
Salmen, Lawrence F. 1987. Listen to the People: Participant-Observer Evaluation of Develop-
ment Projects. New York: Oxford University Press for the World Bank.
Sanday, Peggy Reeves. 1983. "The Ethnographic Paradigm." Pp. 19-36 in Qualitative
Methodology, edited by John Van Maanen. Beverly Hills, CA: Sage.
Sanders, William. 1976. The Sociologist as Detective. 2d ed. New York: Praeger.
Sandmann, Lorilee R. 1989. Educational Program Development Approaches Associated With
Eastern Caribbean Extension Programs. Unpublished doctoral dissertation, University
of Wisconsin-Madison.
Sands, Deborah M. 1986. "Farming Systems Research: Clarification of Terms and Con-
cepts." Pp. 87-104 in Experimental Agriculture, Farming Systems Series. Vol. 22. Cam-
bridge, UK: Cambridge University Press.
Sands, G. 2000. A Principal at Work: A Story of Leadership for Building Sustainable Capacity
ofa School. Unpublished Ed.D thesis, Centre for Leadership, Management and Policy,
Faculty of Education, Queensland University of Technology, Brisbane, Australia.
Schein, Edgar H. 1985. Organizational Culture and Leadership. San Francisco: Jossey-Bass.
Schensul, Jean and MargaretD. LeCompte, eds. 1999. Ethnographer's Toolkit 7 vols. Wal-
nut Creek, CA: AltaMira.
Schlechty, P. and G. Noblit. 1982. "Some Uses of Sociological Theory in Educational
Evaluation." In Policy Research, edited by Ron Corwin. Greenwich, CT: JAI.
Schmidt, Mary R. 1993. "Alternative Kinds of Knowledge and Why They Are Ignored."
Public Administration Review 53 (6): 526-31.
Schoggen, P. 1978. "Ecological Psychology and Mental Retardation." Pp. 33-62 in Ob-
serving Behavior. Vol. 1, Theory and Applications in Mental Retardation, edited by G.
Sackett. Baltimore: University Park Press.
Schon, D. A. 1987. Educating the Reflective Practitioner: Toward a New Design for Teaching
and Learning in the Professions. San Francisco: Jossey-Bass.
. 1983. The Reflective Practitioner: How Professionals Think in Action. New York: Ba-
sic Books.
Schorr, Lisbeth B. 1988. Within Our Reach: Breaking the Cycle ofDisadvantage. New York:
Doubleday.
Schultz, Emily, ed. 1991. Dialogue at theMargins: Whorf, Bakhtin, and Linguistic Relativity.
Madison: University of Wisconsin Press.
Schultz, Stephen J. 1984. Family Systems Therapy: An Integration. Northvale, NJ: Jason
Aronson.
Schutz, Alfred. 1977. "Concepts and Theory Forma tion in the Social Sciences." In Under-
standing and Social Inquiry, edited by F. R. Pallmayr and T. A. McCarthy. Notre Dame,
IN: University of Notre Dame Press.
. 1970. Ou Phenomenology and Social Relations. Chicago: University of Chicago
Press.
. 1967. The Phenomenology ofthe Social World. Evanston, IL: Northwestern Univer-
sity Press.
Schwandt, Thomas A. 2001. Dictionary of Qualitative Inquiry. 2d rev. ed. Thousand Oaks,
CA: Sage.
. 2000. "Three Epistemological Stances for Qualitative Inquiry: Interpretivism,
Hermeneutics, and Social Constructivism. Pp. 189-214 in Handbook of Qualitative Re-
CA: Sage.
. 1997a. Qualitative Inquiry: A Dictionary o/Terms. Thousand Oaks, CA: Sage.

. 1997b. "Whose Interests Are Being Served? Program Evaluation as a Concep-
tual Practice of Power." Pp. 89-104 in Evaluation and the Postmodern Dilemma. Ad-
vances in Program Evaluation, Vol. 3, edited by L. Mabry. Greenwich, CT: JAI.
. 1989. "Recapturing Moral Discourse in Evaluation." Educational Researcher 18
(8): 11-16, 34.
Schwartzman, Helen B. 1993. Ethnography in Organizations. Qualitative Research
Scott, Myrtle and Susan J. Eklund. 1979. "Ecological Methods in the Study of Adminis-
tra tive Behavior." Presented at the 1979 American Educational Research meetings,
San Francisco.
Scriven, Michael. 1998. "The Meaning of Bias." In Stake Symposium on Educational Evalu-
ation. Urbana: CIRCE, University of Illinois.
. 1993. Hard-Won Lessons in Program Evaluation. New Directionsfor Program Evalua-
tion 58. San Francisco: Jossey-Bass.
. 1976. "Maximizing the Power of Causai Investiga tion: The Modus Operandi
Method." Pp. 120-39 in Evaluation Studies Annual Review 1, edited by G. V. Glass.
. 1972a. "Objectivity and Subjectivity in Educational Research." In Philosophical
Redirection of Educational Research: The Seventy-First Yearbook ofthe National Society for
the Study of Education, edited by L. G. Thomas. Chicago: University of Chicago Press.
. 1972b. "Prose and Cons About Goal-Free Evaluation." Evaluation Comment
3:1-7.
Scudder, T. 1999. "The Emerging Global Crisis and Development Anthropology: Can
We Have an Impact?" The 1999 Malinowski Award Lecture. Human Organization 58
(4, winter): 351-64.
Searle, Barbara, ed. 1985. Evaluation in World Bank Education Projects: Lessons From Three
Case Studies. Report No. EDT 5. Washington, DC: World Bank.
Senge, Peter M. 1990. The Fifth Disciple: TheArt and Practice ofthe Learning Organization.
New York: Doubleday.
Shadish, William R. 1995a. "The Logic of Generaliza tion: Five Principies Common to
Experiments and Ethnographies." American Journal of Community Psychology 23 (3):
419-28.
. 1995b. "Philosophy of Science and the Quantitative-Qualitative Debates: Thir-
teen Common Errors." Evaluation and Program Planning 18 (1): 63-75.
. 1995c. The Quantitative-Qualitative Debates: "DeKuhnifying the Conceptual
Context." Evaluation and Program Planning 18 (1): 47-49.
Shah, Idries. 1973. The Subtleties ofthe Inimitable Mulla Nasrudin. New York: Dutton.
. 1972, The Exploits ofthe Incomparable Mullah Nasrudin. New York: Dutton.
Shaner, W. W., P. F. Philipp, and W. R. Schmehl. 1982a. Farming Systems Research and De-
velopment: Guidelines for Developing Countries. Boulder, CO: Westview.
. 1982b. Readings in Farming Systems Research and Development. Boulder, CO:
Westview.
Shank, Gary D. 2002. Qualitative Research: A Personal Skills Approach. Englewood Cliffs,
NJ: Prentice Hall.
Shapiro, Edna. 1973. "Educational Evaluation: Rethinking the Criteria of Competence."
School Revieiv 81 (August): 523-49.
Shaw, Gordon, Robert Brown, and Philip Bromiley. 1998. "Strategic Stories: How 3M Is
Rewriting Business Planning." Harvard Business Review 76 (3, May-June): 41-50.
Shepard, L. 1993. "Evaluating Test Validity." Review of Research in Education 19:405-50.
Shils, Edward A. 1959. "Social Inquiry and the Autonomy of the Individual." In The Hu-
man Meaning ofthe Social Sciences, edited by D. Lerner. Cleveland, OH: Meridian.
References J. R31
Silverman, David. 2000. "Analyzing Talk and Text." Pp. 821-34 in Handbook of Qualitative
Research. 2d ed., edited by Norman K. Denzin and Yvonna S. Lincoln. Thousand
Oaks, CA: Sage.
, ed. 1997. Qualitative Research: Theory, Method and Practice. London: Sage.
Silverzweig, Stan and Robert R Alien. 1976. "Changing the Corporate Culture." Sloan
Management Reviezv 17 (3): 33-49.
Simic, Charles. 2000. "Tragicomic Soup." The New York Review of Books 47 (9): 8-11.
Simmons, Richard. 1985. Farming Systems Research: A Review. World Bank Technical Pa-
per No. 43. Washington, DC: World Bank.
Simon, R. I. and D. Dippo. 1986. "On Criticai Ethnographic Work." Anthropology and Ed-
ucation Quarterly 17 (4): 195-202.
Sims, Calvin. 2001. "Stone Age Ways Surviving, Barely: Indonesian Village Is Caught
Between Worlds Very Far Apart." New York Times, March 11, p. 6.
Smith, Dorothy. 1979. "A Sociology for Women." In The Pristn of Sex, edited by J. A.
Sherman and E. T. Beck. Madison: University of Wisconsin Press.
Smith, John K. 1991. "Goodness Criteria: Alterna tive Research Paradigms and the Prob-
lem of Criteria." Pp. 167-87m The Paradigm Dialogue, edited by Egon Guba. Newbury
Park, CA: Sage.
Smith, John Maynard. 2000. "The Cheshire Cat's DNA: Review of the Century of the
Gene." New York Reviezv of Books 47 (20, December 21): 43-46.
Smith, Louis M. and Paul F. Kleine. 1986. "Qualitative Research and Evaluation: Trian-
gulation and Multimethods Reconsidered." New Directions for Program Evaluation 30
(summer): 55-72, Naturalistic Evaluation, edite d by David D. Williams. San Francisco:
Jossey-Bass.
Smith, Midge F. 1989. Evaluability Assessment: A Practical Approach. Norwell, MA:
Kluwer-Nijhoff.
Smith, Nick, ed. 1981. Metaphors for Evaluation: Sources ofNezu Methods. Beverly Hills,
CA: Sage.
. 1980. "Evaluation Utilization: Some Needed Distinctions." Evaluation Network
Newsletter 16:24-25.
. 1978. "Truth, Complementarity, Utility, and Certainty." CEDR Quarterly
11:16-17.
Snow, D. A. 1980. "The Disengagement Process: A Neglected Problem in Participant
Observation Research." Qualitative Sociology 3 (2): 100-22.
Sociometrics. 1989. Evaluating Programs Aimed at Preventing Teenage Pregnanes. Paio
Alto, CA: Sociometrics.
Sonnemann, Ulrich. 1954. Existence and Therapy:An Introduction to Phenomenological Psy-
chology & Existential Analysis. New York: Gr une & Stratton.
Sonnichsen, Richard C. 2000. High Impact Internai Evaluation. Thousand Oaks, CA: Sage.
. 1993. "Can Governments Lean?" In Compara tive Perspectives on Evaluation and
Organizational Learning, edited by F. Leeuw, R. Rist, and R. Sonnichsen. New Bruns-
wick, NJ: Transaction.
Sorensen, Peter F., Therese F. Yaeger, and Dave Nicoll. 2000. "Appreciative Inquiry: Fad
or Important Focus for O D ? " OD Practitioner 32 (1): 3-5.
Spindler, George and Lorie Hammond. 2000. "The Use of Anthropological Methods in
Educational Research." Pp. 17-25 in Acts of Inquiry in Qualitative Research, edited by
B. M. Brizuela, J. P. Stewart, R. G. Carrillo, and J. G. Berger. Reprint Series No. 34.
Cambridge, MA: Harvard Educational Review.
Stake, Robert E. 2000. "Case Studies." Pp. 435-54 in Handbook of Qualitative Research.
Sage.
. 1998. "Hoax?" In Stake Symposium on Educational Evaluation. Urbana: CIRCE,
University of Illinois.
. 1995. The Art o/Case Study Research. Thousand Oaks, CA: Sage.
. 1978. "The Case Study Method in a Social Inquiry." Educational Researcher 7:5-8.
. 1975. Evaluating the Arts in Education: A Responsive Approach. Columbus, OH:
Charles E. Merrill.
Stake, Robert E., L. Bresler, and L. Mabry. 1991. Custom and Cherishing: The Arts in Ele-
mentary Schools. Urbana: Council for Research in Music Education, University of Illi-
nois.
Stanfield, J. H. 1999. "Slipping Through the Front Door: Relevant Social Sciences in the
People of Color Century." American Journal of Evaluation 20 (3, fali): 415-35.
Steiriberg, D. 1.1983. Irrigation and AIDs Experience: A Consideration Based on Evaluation.
A.I.D. Program Evaluation Report No. 8. Washington, DC: U.S. Agency for Interna-
tional Development.
Stenhouse, Lawrence. 1977. Case Study as a Basis for Research in a Theoretical Contemporary
History of Education. East Anglia, UK: Centre for Applied Research in Education, Uni-
versity of East Anglia.
Stewart, Alex. 1998. The Ethnographer's Method. Qualitative Research Methods Series,
Vol. 46. Thousand Oaks, CA: Sage.
Stewart, Edward C. 1985. American Cultural Patterns: A Cross-Cidtural Perspective.
Yarmouth, ME: Intercultural Press.
Stockdill, S. H., R. M. Duhon-Sells, R. A. Olson, and M. Q. Patton. 1992. "Voices in the
Design and Evaluation of a Multicultural Education Program: A Developmental Ap-
proach." New Directions for Program Evaluation 53 (spring): 17-34, Minority Issues in
Program Evaluation, edited by Anna-Marie Madison. San Francisco: lossey-Bass.
Stoecker, R. 1999. "Are Academics Irrelevant? Roles for Scholars in Participatory Re-
search." American Behavioral Scientist 542 (5): 840-54.
Storm, lim and Michael Vitt. 2000. Master of Creative Philanthropy: The Story of Russ
Ewald. Minneapolis, MN: Philanthropoid.
St. Pierre, Elizabeth Adams. 2000. "The Call for Intelligibility in Postmodern Educa-
tional Research." Educational Researcher 29 (5): 25-28.
Strauss, Anselm and Juliet Corbin. 1998. Basicsof Qualitative Research: Techniques and Pro-
cedures for Developing Grounded Theory. 2d ed. Thousand Oaks, CA: Sage.
, eds. 1997. Grounded Theory in Practice. Thousand Oaks, CA: Sage.
. 1990. Bfls/cs of Qualitative Research: Grounded Theory Procedures and Techniques.
Strike, Kenneth. 1972. "Explaining and Understanding: The Impact of Science on Our
Concept of Man." In Philosophical Redirection of Educational Research: The Seventy-Pirst
Yearbook of the National Society for the Study of Education, edited by L. G. Thomas. Chi-
cago: University of Chicago Press.
Stringer, Ernest T. 1996. Action Research: A Handbook for Practitoners. Thousand Oaks,
CA: Sage.
Stufflebeam, Daniel L. 2001. "Evaluation Values and Criteria Checklist." Posted online
at the Western Michigan University's Evaluation Center's Evaluation Checklists
Web site: www.wmich.edu/evalctr/checklists.
. 1980. "An Interview With Daniel L. Stufflebeam." Educational Evaluation and
Policy Analysis 2 (4): 90-92.
Stufflebeam, Daniel L., George F. Madeus, and Thomas Kellaghan, eds. 2000. Evaluation
Models: Viezvpoints on Educational and Human Services Evaluation. 2d ed. Boston:
Kluwer.
Suchman, Edward. 1967. Evaluation Research: Principies and Practice in Public Service and
Social Action Programs. New York: Russell Sage.
Sussman, Marvin B. and lane F. Gilgun, eds. 1996. The Methods and Methodologies of Qual-
itative Family Research. New York: Haworth.
References J. R33
Symon, Gillian and Catherine Cassell. 1998. Qualitative Methods and Analysis in Organi-
za tional Research: A Practical Guide. Thousand Oaks, CA: Sage.
Tallmadge, John. 1997. Meeting the TreeofLife. Salt Lake City: University of Utah Press.
Tashakkori, Abbas and Charles Teddlie. 1998. Mixed Methodology: Combining Qualitative
and Quantitative Approaches. Thousand Oaks, CA: Sage.
Taylor, Steven J. and Robert Bogdan. 1984. Introduction to Qualitative Research Methods:
The Search for Meaning. 2d ed. New York: John Wiley.
Tedlock, Barbara. 2000. "Ethnography and Ethnographic Representation." Pp. 455-86 in
Tesch, R. 1990. Qualitative Research: Analysis Types and Softivare Tools. New York: Falmer.
Textor, Robert. 1980. A Handbook on Ethnographic Futures Research. Stanford, CA: Stan-
ford University Cultural and Educational Future Research Project
Thomas, Jim. 1993. Doing Criticai Ethnography. Qualitative Research Methods Series,
Vol. 26. Newbury Park, CA: Sage.
Thomas, W. I. and D. Thomas. 1928. The Child in America. New York: Knopf.
Thompson, Linda. 1992. "Feminist Methodology for Family Studies." Journal of Mar-
riage and the Family 54 (1): 3-18.
Tierney, Patrick. 2000a. Darkness in El Dorado: Hozv Scientists and Journalists Devastated the
Amazon. New York: Norton.
2000b. "The Fierce Anthropologist" The Neiu Yorker, October 9, pp. 50-61.
Tierney, William. 2000. "Undaunted Courage: Life History and the Postmodern Chal-
lenge." Pp. 537-65 in Handbook of Qualitative Research. 2d ed., edited by Norman K.
Denzin and Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
Tikunoff, William with B. Ward. 1980. Interactive Research and Development on Teaching.
San Francisco: Far West Laboratory for Educational Research and Development.
Tilney, John S., Jr. and James Riordan. 1988. Agricultural Policy Analysis and Planning: A
Summary ofTwo Recent Analyses of A.I.D.-Supported Projects Worldwide. A.I.D. Evalua-
tion Special Study No. 55. Washington, DC: U.S. Agency for International Develop-
ment.
Torres, Rosalie, Hallie Preskill, and Mary Piontek. 1996. Evaluation Strategies for Commu-
nicating and Reporting: Enhancing Learning in Organizations. Thousand Oaks, CA: Sage.
Travisano, Richard. 1998. "On Becoming Italian American: An Autobiography of an
Ethnic Identity." Qualitative Inquiry 4 (4, December): 540-63.
Tremmel, Robert. 1993. "Zen and the Art of Reflective Practice in Teacher Education."
Harvard Educational Review 63 (4): 434-58.
Trend, M. G. 1978. "On the Reconciliation of Qualitative and Quantitative Analyses: A
Case Study." Human Organization 37:345-54.
Trochim, William M. K., ed. 1989. "Concept Mapping for Evaluation and Planning."
Special issue of Evaluation and Program Planning 12 (1).
Trow, Martin. 1970. Comment on "Participant Observation and Interviewing: A Com-
parison." In Qualitative Methodology, edited by W. I- Filstead. Chicago: Markham.
Tucker, Eugene. 1977. "The Follow Through Planned Variation Experiment: What Is the
Pay-Off?" Presented at the annual meeting of the American Educational Research
Association, New York City, April 5.
Turksever, A. and G. Atalik. 2001. "Possibilities and Limita tions for the Measurement of
the Quality of Life in Urban Aras." Social Indicators Research 53 (2, February): 163-87.
Turner, Aaron. 2000. "Embodied Ethnography: Doing Culture." Social Anthropology 8
(1): 51-60.
Turner, JonathanH. 1998. The StructureofSociological Theory. Belmont, CA: Wadsworth.
Turner, Roy, ed. 1974. Ethnomethodology: Selected Readings. Baltimore: Penguin.
Turpin, Robin. 1989. "What Is and Is Not Politics inEvaluation?" Evaluation Practice 10,1
(February): 54-57.
Uchitelle, Louis. 2001. "By Listening, Three Economists Show Slums Hurt the Poor."
New York Times, February 18, p. B4.
U.S. General Accounting Office (GAO). 1998. Emerging Drug Problems. Washington, DC:
General Accounting Office.
. 1992. The Evaluation Synthesis. Washington, DC: General Accounting Office.
. 1991. Designing Evaluations. Washington, DC: General Accounting Office.
. 1989. Prospective Methods: The Prospective Evaluation Synthesis. Washington, DC:
General Accounting Office.
. 1987. Case Study Evaluations. Transfer Paper 9. Washington, DC: General Ac-
counting Office.
United Way of America. 1996. Measuring Program Outcomes: A Practical Approach. Alex-
andria, VA: Effective Practices and Measuring Impact for United Way of America.
Uphoff, Norman. 1991. " A Field Guide for Participatory Self-Evaluation." Special issue,
Evaluation of Social Development Projects. Community Development Journal 26 (4):
271-85.
van denHoonaard, Will C. 1997. Working With Synthesizing Concepts: Analytical Field Re-
search. Qualitative Research Methods Series, Vol. 41. Thousand Oaks, CA: Sage.
Van Maanen, John, ed. 1998. Qualitative Studies in Organizations. Thousand Oaks, CA:
Sage.
. 1988. Tales ofthe Field: On Writing Ethnography. Chicago: University of Chicago
Press.
Van Manen, Max. 1990. Researching Lived Experience: Human Science for an Action Sensi-
tive Pedagogy. New York: State University of New York.
Vesneski, W. and Kemp, S. 2000. "Families as Resources: Exploring Washington's Fam-
ily Group Conferencing Project." Pp. 312-23 in Family Group Conferencing: New Direc-
tions in Community-Centere Child and Family Practice, edited by G. Burford and J.
Hudson. New York: Aldine de Gruyter.
Vidich, Arthur J. and Standford M. Lyman. 2000. "Qualitative Methods: Their History
in Sociology and Anthropology" Pp. 37-84 in Handbook of Qualitative Research. 2d ed.,
edited by Norman K. Denzin and Yvonna S. Lincoln. Thousand Oaks, CA: Sage.
Von Bertalanffy, Ludwig. 1976. General System Theory: Foundations, Development, Appli-
cations. New York: George Braziller.
Von Oech, Roger. 1998. A Whack on the Side ofthe Head: Hozv You Can Be More Creative.
New York: Warner.
Wadsworth, Yoland. 1993a. How Can Professionals Help Groups Do Their Own Participa-
tory Action Research? Melbourne, Australia: Action Research Issues Association.
. 1993b. What Is Participatory Action Research? Melbourne, Australia: Action Re-
search Issues Association.
. 1984. Do It Yourself Social Research. Melbourne, Australia: Victorian Council
of Social Service and Melbourne Family Care Organization in association with Allen
and Unwin.
Wagoner, David. 1999. "Lost." Traveling Light: Collected and Nezv Poems. Champaign:
University of Illinois Press.
Waldrop, M. M. 1992. Complexity: The Emerging Science at the Edge ofOrder and Chos.
New York: Simon & Schuster.
Walker, Joyce. 1996. "Letters in the Attic: Private Reflections of Women, Wxves and
Mothers." Pp. 9-40 in The Methods and Methodologies of Qualitative Family Research,
edited by Marvin B. Sussman and Jane F. Gilgun. New York: Haworth.
Wallace, Ruth A. and Alison Wolf. 1980. Contemporary Sociological Theory. Englewood
Cliffs, NJ: Prentice Hall.
Wallerstein, Iminanuel. 1980. The Modem World System. San Diego, CA: Academic Press.
Walston, J. T. and R. W. Lissitz. 2000. "Computer-Mediated Focus Groups." Evaluation
Review (5, October): 457-83.
References J. R35
Walters, Jonathan. 1992. "The Cult of Total Quality." Governing: The Magazine of States
and Localities, May, pp. 38-42.
Warren, Marion K. 1984. AID and Education: A Sector Report on Lessons Learned. A.I.D.
Program Evaluation Report No. 12. Washington, DC: U.S. Agency for International
Development.
Waskul, D., M. Douglass, and C. Edgley. 2000. "Cybersex: Outercourse and the
Enselfment of the Body." Symbolic Interaction 23 (4): 375-97.
Wasserman, Gary and Alice Davenport. 1983. Power to the People: Rural Electrification
Sector Summary Report. A.I.D. Program Evaluation Report No. 11. Washington, DC:
U.S. Agency for International Development.
Wasson, C. 2000. "Ethnography in the Field of Design." Human Organization 59 (4, win-
ter): 377-88.
Watkins, Jane Magruder and David Cooperrider. 2000. "Appreciative Inquiry: A
Transformative Paradigm." OD Practitioner 32 (1): 6-12.
Watkins, K. E. and V. J. Marsick. 1993. Sculpting the Learning Organization. San Francisco:
Jossey-Bass.
Watson, Graham and Jean-Guy Goulet. 1998. "What Can Ethnomethodology Say About
Power?" Qualitative Inquiry 4 (1, March): 96-113.
Wax, Rosalie H. 1971. Doing Fieldwork: Warnings and Advice. Chicago: University of Chi-
cago Press.
Webb, Eugene J., Donald T. Campbell, Richard Schwartz, and Lee Sechrest. 1966. Unob-
trusive Measures: Nonreactve Research in the Social Sciences. Chicago: Rand McNally.
Webb, Eugene J. and Karl E. Weick. 1983. "Unobtrusive Measures in Organizational
Theory: A Rejninder." Pp. 209-24 in Qualitative Methodology, edited by John Van
Maanen. Beverly Hills, CA: Sage.
Weidman, Emmaline. 1985. Dancing With a Demon: A Heuristic Investigation ofjealousy.
Unpublished doctoral dissertation, Graduate College, The Union Institute, Cincin-
nati, OH.
Weiss, Carol. 1972. Evaluation Research: Methods of Assessing Program Effectiveness.
Englewood Cliffs, NJ: Prentice Hall.
Weiss, Carol H. and Michael Bucuvalas. 1980. "Truth Test and Utility Test: Decision
Makers' Frame of Reference for Social Science Research." American Sociological Re-
view (April): 302-13.
Weiss, Heather B. 2001. "Strategic Communications: From the Director's Desk." The
Evaluation Exchange 7 (1): 1.
Weiss, Heather B. and Jennifer C. Greene. 1992. "An Empowerment Partnership for
Family Support and Education Programs and Evaluations." Family Science Review 5
(1,2, February / May): 145-63.
Wheatley, Margaret. 1992. Leadership in the New Science. San Francisco: Berrett- Koehler.
White, Michael and David Epston. 1990. Narrative Means to Therapeutic Ends. New York:
Norton.
Whitehead, Alfred N. 1958. Modes ofThought. New York: Capricorn.
Whiting, Robert. 1990. You Gotta Have Wa. New York: Vintage.
Wholey, Joseph S. 1994. "Assessing the Feasibility and Likely Usefulness of Evalua-
tion." Pp. 15-39 in Handbook ofPractical Program Evaluation, edited by J. Wholey, H.
Hatry, and K. Newcomer. San Francisco: Jossey-Bass.
. 1979. Evaluation: Promise and Performance. Washington, DC: Urban Institute.
Whyte, William Foote, ed. 1989. Action Research for the Twenty-First Century: Participa-
tion, Reflection, and Practice. Special issue of American Behavioral Scientist 32 (5,
May/June).
. 1984. LearningFrom the Field: AGuide From Experience. Beverly Hills, CA: Sage.
. 1943. Street Comer Society. Chicago: University of Chicago Press.
Wildavsky, A. 1985. "The Self-Evaluating Organization." Pp. 246-65 in Program Evalua-

tion: Patterns and Directions, edited by E. Chelimsky. Washington, DC: American So-
ciety for Public Administration.
Wilkinson, Alec. 1999. "Notes Left Behind." The New Yorker, February 15, pp. 44-49.
Williams, Brackette F. 1991. Stains onMy Name, War in My Veins: Guyana and the Politics of
Cultural Struggle. Durham, NC: Duke University Press.
Williams, Walter. 1976. "Implementation Analysis and Assessment." In Social Program
Implementation, edited by Walter Williams and Richard F. Elmore. New York: Aca-
demic Press.
Wilson, E. 0 . 1 9 9 8 . "Back to the Enlightenment: We Must Know, We Will Know." Free In-
quiry 18 (4): 21-22.
Wilson, Paul. 1999. "The First Laugh." Transia tion of a speech by President Vclav
Havei upon receiving the Open Society Prize awarded by the Central European Uni-
versity in Budapest in 1999. The New York Review ofBooks 46 (20): 59.
Wilson, Stacy. 2000. "Construct Validity and Reliability of a Performance Assessment
Rubric to Measure Student Understanding and Problem Solving m College Physics:
Implications for Public Accountability in Higher Education." Doctoral dissertation,
University of San Francisco. Dissertation Abstracts International AAT 9970526.
Wirth, Louis. 1949. "Preface." In Ideology and Utopia, by K. Mannheim. New York: Har-
court Brace lovanovich.
Wisp, L. 1986. "The Distinction Between Sympathy and Empathy: To Call For th a Con-
cept, a Word Is Needed." Journal ofPersonality and Social Psychology 50:314-21.
Wolcott, Harry F. 1992. "Posturing in Qualitative Inquiry." Pp. 3-52 in The Handbook of
Qualitative Research in Education, edited by M. D. LeCompte, W. L. Milroy, and J.
Preissle. New York: Academic Press.
. 1990. Writing Up Qualitative Research. Qualitative Research Methods Series, Vol.
20. Newbury Park, CA: Sage.
. 1980. "How to Look Like an Anthropologist Without Really Being One." Prac-
ticing Anthropology 3 (2): 56-59.
Wolf, Robert L. 1975. "Trial by lury: A New Evaluation Method." Phi Delta Kappan,
November.
Wolf, Robert L. and Barbara L. Tymitz. 1978. "Whatever Happened to the Giant Wom-
bat: An Investigation of the Impact of the Ice Age Mammals and Emergence of Man
Exhibit." Washington, DC: National Museum of Natural History, Smithsonian Insti-
tute.
Worthen, Blaine R., James R. Sanders, & Jody L. Fitzpatrick. 1996. Program Evaluation:
Alternative Approaches and Practical Giddelines. Reading, MA: Addison-Wesley.
Wright, H. F. 1967. Recording and Anahjzing Child Behavior. New York: Harper & Row.
Yin, Robert K. 1999a. "Rival Explanations as an Alternative to 'Reforms as Experi-
ments.' " In Validity and Social Experimentation: Donald CampbelVs Legacy, edited by
Leonard Bickman. Thousand Oaks, CA: Sage.
. 1999b. "Strategies for Enhancing the Quality of Case Studies." Presentation at
Health Sciences Research conference, Qualitative Methods in Health Sciences Re-
search. Bethesda, MD: Cosmos.
. 1994. Case Study Research: Design and Methods. Applied Social Research
Methods, Vol. 5. Thousand Oaks, CA: Sage.
. 1989. Case Study Research: Design and Methods. Rev. ed. Newbury Park, CA: Sage.
Youngson, Robert. 1998. Scientific Blunders: A Brief History ofHow Wrong Scientsts Can
Sometimes Be. New York: Caroll & Graf.
Zaner, R. M. 1970. The Way of Phenomenology: Criticism as a Philosophical Discipline. New
York: Pegasus.
jAu+no^ ZJv\c\ejz
Abbey, E., 290 Arendt, H v 188

Aburdene, R, 201 Argyris, C., 145,163,179
Academy for Educational Development (AED), 388 Armstrong, D., 196
Ackoff, R., 120,122 Asmov, I., 512
Ackoff, R. L 120,121 Atalik, G 150
Addison, R 115,497,498 Atkinson, P., 83,495
Adelman, R. D., 96 Atkinson, R v 404
AEA Task Force, 205,543,549, 586 Aubel,J., 183
Afflerbach, R, 385 Aubrey, R., 179
Agar, M., 81,123,124,166,167,363 Avery, O-, 588
Agar, M. H 81 Azumi, K., 119
Alasuutari, R, 405
Alkin, M., 169,170,187
Alkin, M. G, 174,195,452, 560 Baert, P., 95
Allen, C 271, 272 Baldwin,}., 88,459
Allen, R. F., 81 Bali, M. S v 308
Allison, M-A, 123 Bandler, R-, 237,245,355,369
Amarei, M., 421 Barker, L. S 118
Anderson, B., 513,514 Barker, R. G., 118
Anderson, E., 314 Barone, T., 116,118, 548, 549,551, 576,579
Anderson, R., 405 Barton, D., 114
Anderson, R. B., 162 Barhrnek, J. M., 267
Anderson, V., 120 Bashook, P. G., 250
Andrews, M., 195,577 Bateson, G., 198,481
Arboieda-Florez, J 174,192 Bateson, M. C., 86
Arcam, J-, 402 Bawden, R. J., 180
Arditti, R., 402,412 Becker, H. S., 21,125,264, 438,447, 457,495
M II
Beebe, J., 194 Brown, A., 96

Belenky, M. F 5, 6,7,433 Brown, J. R., 59, 64,65
Bellah, R., 459 Brown, J. S., 16
Benford, R. D v 432,499 Brown, R., 196,407
Benko, S., 121 Browne, A., 8,193,232,297,403-404,406,434, 438
Benmayor, R., 549 Bruce, C 104
Bennett, W., 459 Bruner, E. M., 86
Benson, A. P., 148 Bruyn, S., 48,61,328,329
Bentz, V. M., 134 Buber, M., 64
Berens, L. V., 507 Buckholt, M 238
Berger, J. G., 76,452 Bucuvalas, M., 550, 578
Berger, P 99,102,379 Bullogh, R. V.,Jr 571
Berland, J., 391 Bunch, E. H., 492
Bernard, H. R., 95, 230,446, 447,449,453, 455,465, 493 Bunge, M., 363
Bernthal, N 108 Burdell, P 552
Berra, Y., 191 Bums, T., 119
Berry, J. W., 394 Bussis, A., 421
Bhaskar, R. A., 95 Buxton, A., 193
Bierce, A., 110
Bilken, S. K 494
Binnendijk, A. L., 501 Cade, J. F. ]., 330
Blake, W., 582 Cambei, A. B., 123
Blakeslee, S., 11 Campbell, D. T., 92, 93,192,239,247, 270,292,471,
Bloom, A., 459 571,586
Blumer, H 112,125, 278,456 Campbell, J. L 174,474,534
Boas, F., 455 Carchedi, G., 131
Bochner, A. R, 85, 86,88, 89,115,118,478,548,551 Carini, P. F 193,274,328
Bogdan, R., 69,95,111,454 Carlm, G v 177
Bogdan, R. G, 494 Carlyle, T v 441
Boring, E. G., 56 Camilo, R. G., 76,452
Borman, K. M v 50 Casey, M. A., 386,388
Bomat, J., 89,115,478 Casse, P 391
Boruch, R., 71 Cassell, C., 405
Boston Women's Teachers' Group, 183 Castaneda, C., 309
Boulding, K. E., 146 Cedillos, J. H 401,402
Bowerman, B., 204 Cernea, M., 175
Boxill, N. A., 20 Cernea, M. M., 121
Boyatzis, R. E., 452,463, 465 Cervantes Saavedra, M. de, 379,380
Brady, I., 124,432 Chagnon, N., 273,326
Brajuha, M., 416 Chamberlain, K., 405
Brandon, P. R., 548 Chamberlayne, P, 89,115,478
Braud, W., 405 Chambers, E., 81,84
Brazzel, M-, 194 Charmaz, K., 128
Bremer, ]., 501 Charon, R., 96
Bresler, L., 452 Chatterjee, A., 504
Brewer, J., 248 Checkland, P., 120
Bridgman, P., 205,570 Cheimsky, E., 586
Brinkley, D., 96 Chen, H-T, 550
Brislin, R. W 391 Cherrie, C., 391
Brizuela, B. M 76, 452 Chew, S. T., 501
Brock, J., 59 Cheyne, V., 108
Bromiley, P., 196 Chibnik, M., 216
Brookfield, S., 483 Chittenden, E. A., 421
Author Index lj. 13
Christans, C. G., 407 Daillak, R., 174, 452,560

Church, K., 88 Dalgaard, K. A., 194
Churchill, W., 576 Dart,J., 196
Cialdini, R. B., 495 Dart, J. J., 196,198
Clark, J 108 Davenport, A., 501
Clarke, I., 471 Davies, R. J., 196
Cleveland, H., 146 Da Vinci, L., 56
Clinchy, B. M 5, 6,7,433 Davis, J. H., 55,404, 432
Coffey, A., 495 Davis, K 45
De Bono, E.,514
Cohen, B. B 399
DeCramer, G., 199
Cohen, P., 179
DeLozier, J., 237
Cole, A. L., 404
Deming, W. E., 146
Cole, D., 196,198
De Munck, Vv 493
Cole, E., 501
Denning, S., 195
Coles, R., 116,314,402,403 Denny, T., 195
Collms, A., 16 Denton, J., 180
Collins, J., 232, 271
Denzin, N. K., 21,48,79,80,88,100,101,104,116,124,
Comstock, D. E., 549 125,133, 247, 265,278,401,404, 438,450, 451,470,
Comte, A., 92 478,486,487, 493,503, 506,543,546, 548,554, 555,
Confucius, 361 567, 569,570, 79,571
Connolly, D. R., 271 Deol, S 391
Connor, R., 394 Deuscher, C. H., 61,146
Conrad, J., 318,319 Dewey, ]., 60, 71
Conroy, D. L., 456,457 Dijkerman, D. W., 501
Constas, M. A., 101 Dilthey, Wv 114
Cook T. D., 571 Dippo, D., 549
Cook, J. A., 549 Dobbert, M. L., 82
Cook, K. H., 260 Domaingue, R., 200
Cook, T. D., 68, 239, 585 Douglas, J. D., 270, 310, 312,561
Cooke, N.J.,350 Douglass, B., 107,108,183,483
Cooper, H., 500 Douglass, M., 112
Cooperrider, D., 181,182 Douglass, W. A., 81,486
Drass, K., 492
Corbin, J., 125,127,128, 239,453,454,465,487,488,
Dreikurs, R., 25
489,490,491,492
Drysdale, G 196,198
Coulon, A., 111
Duckworth, E., 183
Cousins, J. B., 184,185,187, 269
Dufour, S., 298
Covey, S. R., 7,232, 434
Duguid, P., 16
Cox, G. B., 148
Duhon-Sells, R. M., 98
Cox, K. K 385 Durin, S., 52
Crabtree, B. F., 400 Durkin, T., 442
Craig, R, 108,487 Durrenberger, E. P., 42
Creswell, J. W., 79,104,132
Cronbach, L. J., 12,123, 571, 582, 583,586
Cronbach, L. J. and Associates, 584,585 Earl, L. M., 184,185,187, 269
Crosby, P. B., 146 Eberstadt, N v 573
Crotty, M., 79,86,97,99,114,115,131 Edgley,C., 112
Crouch, W. W., 260 Edmunds, S. W., 200
Curry, G, 437 Edwards, R 129
Cushner, K., 391 Edwards, W., 165
Czamiawska, B., 118 Eichelberger, R. T., 107,115
Cziko, G. A., 124 Eichenbaum, L., 130
Einstein, A., 12 Frow,J 391

Eisner, E. W., 87,172,173, 409,550 Fuller, S 99
Eklund, S.}., 309
Eliot, T. S., 165
Elliott, J., 183 Gahan, C., 446
Ellis, C 85,86, 88,89,118,272,548 Galileo, 237
Elmore, R. F., 162 Gallucci, M., 312
Emerson, R. W., 260 Galt, D. L., 122
Emery, E, 122 Gamson, J., 130
English, F. W., 404, 551 Garcia, S. E., 38
Ensler, E 548 Gardiner, B v 314
Eoyang, G. H., 123 Garfinkel, H 110,111
Epston, D., 116 Geer, B., 21,264
Erem, S v 42 Geertz, C v 273,327,340,438
Erickson, F., 202 Gis, I., 573
Erickson, K., 83 Gentile, J. R., 222
Erickson, M. H., 237 Gephart, R. P., Jr., 573
Ericsson, K. A., 385 Gerber, R 104
Gergen, K. J 482
Gergen, M. M., 482
Fadiman, C., 50 Gharajedaghi, Jv 119,120,121
Farming Systems Support Project (FSSP), 121,122 Gilgun, ]., 95,193,403,494
Fehrenbacher, H. L 452,524 Gilgun, J. F 405, 406,452
Feiman, S., 165,166 Gill, R., 184
Felix, S 395 Gilligan, C., 65
Ferguson, B., 326 Giorgi, A v 105
Ferguson, C., 174 Gladwell, M 43,561
Festinger, L., 271 Gladwin, C. H., 83
Fetterman, D. M., 79, 82,183,187,220, 269,303,550, Glaser, B. G., 57, 67,125,127,215,324,454,488, 489,
571 491, 492,545
Fewer, W., 121 Glass, G. V., 247
Field, P. A., 405 Glass, R. D., 549
Fielding, J. L-, 247 Glazer, M., 568
Fielding, N. G., 247,443,446,492, 493 Gleick, J., 123,124,126
Fieldman, M. S., 499 Glennon, L. M., 129
Filstead, W.}., 53 Glesne, C., 87,432,452,502,546,548,560,567,570
Fitz-Gibbon, C. T., 162, 245 Gluck, S. B., 549
Fitzpatrick, ]., 497, 503,550 Godet, M., 200
Fitzsimmons, E. L., 200 Goetz, J. P., 50
Fonow, M. M., 549 Goffman, E., 438
Fontana, A., 340,342,388 Goldberger, N. R., 5, 6, 7,433
Fonte, J 131, 549 Golden-Biddle, K., 432
Foote, M., 147 Golembiewski, B., 182
Fortin, D., 298 Goodali, H. L., Jr., 83, 85,548
Foucault, M., 546,579 Goodenough, W., 81
Frake, C., 458 Goodman, R., 395
Frank, A., 118 Goodson, I., 147
Freeman, H. E., 550 Gore, J. M 179
Friedan, B., 459 Goulet, J-G, 102,111
Freire, P., 549 Graham, R.}., 115
Frey, J. H., 340,342,388 Gramsci, A., 131
Fricke, J. G., 184 Graue, M. E., 402
Author Index lj. 15
Grbch, G, 405 Heinlein, R. A., 636

Greenbaum, T. L., 390 Helmer, O., 200
Greene, J. G, 65, 68, 98,183, 550, 551 Hendricks, M., 511
Greene, M. G 96 Henry, G. T 95,220
Greig, A., 402 Heraclitus, 54
Grinder, J 237, 245,355,369 Heron, J., 183
Grout, M., 60 Hertz, R., 65, 495
Guba, E. G., 14,39, 44, 50, 67, 71,79,96, 98,171,190, Heydebrand, W. V., 131
225,246, 254,252,323,465, 466,546,550,554, 562, Higginbotham, J. B., 385
570, 571,575, 581,583, 584,585 Hill, M. R., 293
Gubrium, J 111, 482 Hinn, D. M., 148
Gubrium, J. E, 404,405 Hirsh, S. K., 507
Guerrero, S. H., 129,269,402 Hoben, A., 501
Guerrero-Manalo, S., 402 Hodder, I., 295
Guggenheim, S. E., 121 Hoffman, L., 121
Guttentag, M., 165 Holbrook, T. L., 395
Holland, J. H., 124
Holley, H 174,192
Hacking, I., 102 Hollinger, D. A., 99
Hacsi, T. A., 163 Holmes, R. M., 402
Haehnn, J. F., 452, 524 Holstein, J 111, 482
Hage, J., 119 Holstein, J. A., 404
Halcolm, 1-2,3,37,38, 75,137,143-144, 207, 209-211, Holte, J., 123
257,259,299,317, 330,339-340,402, 418,429, 431, Holizman, J. S., 122
467-468, 500, 506, 515,541-542, A1-A2 Hopson, R v 112
Hall, N., 123 Horowitz, R., 314
Hallowell, L 416 House, E 87,170,171,172,561,564,569, 570
Hamel, ]., 298,452 House, E. R., 185, 186, 187, 550
Hamilton, D., 172 Howe, K. R., 186,187,550
Hamilton, M., 114 Huberman, A. M., 94,133, 433,465,471, 546
Hammond, L., 81 Huebner, T. A., 163
Hamon, R. R., 395 Huff, D., 573
Handwerker, W. R, 194 Hugo, V., 62
Hannibal, M., 446 Hull, B., 183
Hansen, E., 322 Human Services Research Institute, 148
Harding, S., 549 Humphrey, D., 146
Hare, R. D., 500 Humphreys, L., 272
Harkreader, S. A., 220 Hunt, S. A-, 432,499
Harper, D., 104,308, 482 Hunter, A., 248
Harris, M 268 Hupcey, ]., 151
Harris, P. R., 391
Hurty, K., 505
Hart, L. K., 81
Husserl, E., 104,105,248,482,483,485
Harvey, C., 180
Harwood, R. R., 122
Hausman, C., 573
Ihde, D., 485
Hawka, S 108
Irelan, W., 501
Hayano, D. M-, 85
Ivanic, R., 88,114
Hayes, T. A., 112
Headland, T v 268
Heap, J. L 102
Jacob, E., 76,118,119,132
Hbert, Y. M., 452
Jacobvitz, R. S., 118
Heidegger, M., 248,482
Jaeger, W. K., 501
James, W., 50 Kramer, M. R., 151

Janesick, V.J., 432,477 Kramer, P. D., 330
Janowitz, M v 505 Krenz, C., 50
Jarvis, S., 179 Krishnamurti, J., 299,301
Jervis, K., 81 Krueger, O., 507
Johnson, A., 229 Krueger, R. A., 386,387,388,390
Johnson, J. C., 83 Kuhn, T., 61,71, 99,100,572,575
Johnson, J. M., 312,568 Kuhns, E., 76
Johnson, L., 120 Kulish, N., 461
Johnson, S., 566 Kummerow, J. M., 507
Johnston, B. R, 501 Kushner, S v 65,118,176,186,187
Johnstone, B., 89 Kvale, S 114,374,407,579
Joint Committee on Standards for Educational Evalu-
ation, 407,543,549,550,585
Jones, J.H., 271 Ladson-Bilhngs, G., 130
Jones, M. O., 113 Lahey, L 395
Jorgensen, D. L., 312 Laonde, B. I. D., 148
Julnes, G., 95 Lang, G. E., 267,568
Junker, B. H 270 Lang, K., 267, 568
Juran, J. M 146 Lather, P., 549
Lawrence-Lightfoot, S., 55,63, 297,404,432,439
LeCompte, M. D., 83
Kafka, F 310 Lee, B 335,338
Kaftarian, A. J 79,183,187, 269 Lee, P 55
Kanter, R. M., 237 Lee, R. L 200
Kaplowitz, M. D., 387, 389 Lee, R. M 443, 446,492,493
Kartunrien, L., 369 Lee, T. W., 405
Katz, L., 108,485 Leeuw Fv 184
Katz, L. R, 559 Leo, R., 271
Katzer, ]., 260 Leonard, E., 450
Kegan, J., 396 Levin, B., 184
Kegan, R., 395 Levin-Rozalis, M., 470
Kellaghan, T., 550 Levi-Strauss, C., 401
Keller, J 314 Leviton, L. C 239
Kelley, X, 514 Levitt, N., 101
Kemmis, S., 269, 398,399 Levy, P. F., 451
Keinp, S., 293 Lewis, G. L., 195
Kendall, P. L., 385 Lewis, P. ]., 65
Kenny, M. L., 81 Lieblich, A., 478,551
Kibel, B. M., 151,196, 465 Liebow, E., 314, 437
Kim, D. H., 120 Liles, R. T., 194
Kimmel, A. J-, 407 Lincoln, Y. S., 14,44, 71,79,80, 96, 97,98,100,104,119,
Kincheloe, J. L 131 133,171,246,252, 254, 401,543,546,550,554,562,
King, J. A., 162,174,183,184,388,390 570,571, 575,583,584
Kipling, R., 276, 278 Lindberg, M. A., 548
Kirk, J., 93 Lipsey, M. W., 550
Kleine, P. F 560 Lissitz, R. W., 389
Kleining, G 109,110 Littman, J., 514
Kling, J. R., 559 Lloyd, C., 148
Kloman, E. H 452 Locke, K. D., 432
Kneller, G. F 114 Lofland, ]., 21, 28,48,125, 262,302,306,320,381,460,
Knowles, J. G., 404 480, 502,503
Kopala, M 405 Lofland, L. H., 125
Author Index lj. 17
Lonner, W. J., 394 Miles, M. B., 94,133,433, 465,471, 546

Lonnquist, M. P, 183 Miigram, S., 270
Louis, M. R., 81, 267 Milius, S., 279
Love, A. J., 179 Mill, J. S., 441
Luckmann, T 99,102 Miller, G., 498,499
Lyman, S. M 81, 84, 95,268,457, 493 Miller, M. L., 93
Miller, S., 121
Miller, W. L., 400
Mabry, L 65, 452 Millett, R., 564
MacBeth, D., 64 Mills, C. W., 205,570
MacDonald, B., 186,187 Milstein, B v 440,444,445,446
Mackaness, W., 471 Minnich, E., 65,129,130,188, 459
MacQueen, K. M 440,444, 445,446 Mitchell, R., 196
Madeus, G. F., 550 Mitchell, R. G.,Jr 269
Madriz, E., 388,389 Montgomery, J., 121
Madsen, R., 459 Moos, R-, 283
Magistad, B. M., 395 Moran, R. T., 391
Maguire, P., 129 Morgan, D. L 388,390
Mairs, N., 65 Morgan, G., 81,119,135
Makower, ]., 201 Morris, E., 89,380
Manhertz, H., 195 Morris, L. L., 162, 245
Manning, P. K 113,493 Morris, M., 391
Margulies, N., 81 Morris, M. W., 496
Marino, R. A., 108 Morrison, D., 277,278
Mark, M, M., 95 Morse, J. M 151,405,502
Marshall, C., 226, 306 Moustakas, C., 8, 9, 53,104,105,107,108,109,110,183,
Marsick, V. ]., 179 405,434,482,483,484,486
Martorana, S. V., 76 Moynihan, D. P., 573
Marx, L., 284 Mueller, M. R., 497
Maslow, A. H 108 MuraU, M. L-, 123
Mathema, S. B 122 Murray, M-, 405
Mathews, R., 193 Musashi, M., 38
Matthews, J. K 193 Mwaluko, G. S., 180
Maxwell, J. A., 250 Myers, I. B., 507
McClintock, B., 60 Myrdal, G., 597
McCIure, G., 394,396,439
McCracken, G., 374,404
McGuigan, }., 405 Nadei, L., 123
McLaren, P., 131 Nagel, E., 363
McLaughlin, M 161,162 Naisbitt, J-, 201
McLeod, L 403 Nardi, D., 507
McNamara, C., 179 Nash, R., 289
McTaggart, R 269,398,399 Neimeyer, G. ]., 96, 547
Mead, G. H., 112 Newman, D., 407
Mead, M., 585 Newton, R. R., 503
Meeker, J. W., 119 Nicoll, D., 181
Merleau-Ponty, M., 105 Noblit, G 480
Merriam, J, E.,201 Noblit, G. W., 500
Merriam, S., 452 Nussbaum, M., 79
Mertens, D. M., 130,187,550
Merton, R., 385
Messick, S., 548 Oakley, A-, 340
Meyers, W. R., 53 Ogbor, J. O., 102
Olesen, V. L., 269 Polanyi, M., 108,111, 487

Olson, R. A., 98 Porter, M. E., 151
Orbach, S., 130 Potter,J., 547
Ormerod, P., 123 Powdermaker, H., 268
Owens, T v 452 Preskill, H 179,181, 220
Owens, T. R., 452,524 Preskill, S 118
Preskill, S. L., 181
Pressley, M 385
Packer, M., 115, 497,498 Preston, M., 230
Packham, R. G 180 Private Agencies Collaborating Together (PACT), 183
Padgett, D. K., 405 Program Evaluation Division (PED), 497
Page, R. N., 76 Punch, M 271, 407,415, 416
Palmer, L 292,293, 395 Putnam, H., 95
Palmer, R. E 114 Putnam, R , 145, 459
Palumbo, D. J 241
Pana ti, C., 204
Parameswaran, R., 53, 54,275,291,299,319,390 Radavich, D., 100
Park, C. C 245 Ragin, C., 492
Parlett, M., 172 Ragin, C. C., 93,447,492,493,545
Partnow, E., 302 Raia, A., 81
Patai, D., 549 Ralli5, S. F., 390,402, 495, 587
Patton, M. Q., 9,10,11,15,17, 51, 64, 68, 78, 89, 93, 98, Ramachandran, V. S., 11
112,121,124,130,138,142,147,161,162,170,173, Raymaker, ]., 193
174,176,179,180,189,190,193,194,195, 201,212, Redfield, R 457
220,241, 252, 282, 290,302,308,313,327,362,394, Reed, J. H., 413, 414
398,401, 406, 434, 437, 455, 459, 464, 472,503,508, Reichardt, C. S 571, 587
511, 514,534,550, 551, 563,564,568,571,579,585, Reinharz, S., 129,183,496,549
587 Reisinger, H. S., 81
Pau], J., 232 Rettig, K 395
Pawson, R., 95 Rhee, Y., 123
Payne, S. L., 353 Rheingold, H v 169,392, 393
Pechman, E., 174 Ribbens,}., 129
Pedler, M 179 Richardson, L., 86, 87,88, 432, 543, 548
Peito, G. H., 265, 268,321, 455 Richardson, M., 87
Peito, P. J 265, 268,321, 455 Riessman, C. K., 115
Penrod, J., 151 Rindskopf, D., 71
Perkyl, A., 93 Riordan, ]., 501
Percy, W., 113 Riske, M 385
Perls, F., 59 Rist, R., 184
Perrone, V., 17,165,193,434, 452 Rist, R. C., 571,574
Perugini, M-, 312 Robinson, C. A., Jr., 38
Peshkin, A., 274, 438,546,551, 576 Rog, D., 164
Peters, T. J., 7,194,231,237, 245, 297,434 Rogers, B. L., 501
Petrosino, A., 163 Rogers, C., 108
Pettigrew, A. M-, 81 Rogers, E., 239
Philipp, P F., 121,122,195 Rogers, P. ]., 163
Philliber, S., 160 Ronai, C. R., 124
Pietro, D. S., 183 Rorty, R 101
Pike, K., 267,268 Rose, D., 83
Pillow, W. S., 101 Roseanne, 348
Pinnegar, S v 571 Rosenblatt, P. C 121
Piontek, M., 179 Rosenthal, R., 462
Pirsig, R. M , 147 Rossi, P. H 170, 550
Author Index lj. 3
Rossman, G., 226,306 Searle, B., 452

Rossman, G. B., 390,402,495 Sechrest, L., 192, 239,270,292
Rourk, P., 501 Secrist, J., 503
Rubin, H. J., 341,392,407,411,415 Senge, P. M 120,179
Rubin, I. S v 341,392,407,411,415 Shadish, W. R 69,71,92,96,571,581,582,588
Rudestam, K. E., 503 Shadish, W. R., Jr., 239
Ruhleder, K., 83 Shah, I., 363,481,564, 636
Russo, M. J., 92 Shakespeare, W., 145,288,289,329
Ryan, G. W., 95,446,455,493 Shaner, W. W 121,122,195
Ryan, T. B., 180 Shank, G. D., 537, 539
Shapiro, E., 49, 61,191,192, 557
Shapiro, J. J., 134
Sackett, R., 229 Shaw, G., 196
Sacks, O., 46,182,245 Shepard, L., 548
Saddington, M., 196,198 Shils, E. A., 269,270
Safire, L., 96 Silverman, D., 113,116,542, 574
Safire, W 96 Silverzweig, S., 81
Salmen, L. F., 394,452 Simic, C., 93
Sand, G., 462 Simmel, G., 110
Sanday, P. R 81 Simmons, R., 122
Sanders, W., 470, 550 Simon, H. A., 385
Sanderson, D., 194 Simon, R. I., 549
Sandlow, L. J., 250 Sims, C., 455
Sandmann, L., 195 Smith, D., 129
Sandmann, L. R., 394 Smith, D. M., 145
Sands, D. M., 245 Smith, G. W. H., 308
Sands, G 122 Smith, J. K., 546
Sankar, A., 405 Smith J . M v 215,459
Santayana, G., 506 Smith, L. M., 560
Sarvimaki, A., 121 Smith, M. E, 164
Sax, G 50 Smith, N 470,504,578
Schein, E. H., 81 Smith, R. L., 59
Schensul, J., 83 Smutylo, T., 153,154
Schlechty, P., 480 Snapper, K., 165
Schleiermacher, E, 114 Snow, D. A., 323
Schmehl, W. R., 121,122,195 Sociometrics, 166
Schoggen, M. F., 118 Sonnemann, U v 104,482
Schoggen, P., 118 Sonnichsen, R., 184
Schon, D., 179 Sonnichsen, R. C., 181
Schon, D. A., 179 Sorensen, P. F., 181
Schorr, L. B 154,158, 231,501 Souvaine, E., 395
Schuitz, E., 55,288 Spacey, K., 61
Schultz, S. J., 121 Speltz, K., 193
Schutz, A., 104,105 Spindler, G., 81
Schwaller, R., 59 St. Pierre, E. A., 579
Schwandt, T. A., 51, 52, 64, 65, 76, 79,92,94, 95,101, Stake, R. E., 171,296,297, 447,449,452,478,480,500,
104,114,132,135,278,482,483,497 506,511,582,583,585
Schwartz, R., 192,239,270, 292 Stalker, G. M., 119
Schwartzman, H. B v 82 Stanfield, J. H 130
Scott, M 309 Stein, D., 123
Scriven, M., 50,56,169,170,467,471,500,560,569, Steinberg, D. I., 501
574, 575 Stenhouse, L., 449
Scudder, T 81 Stewart, A., 83
Stewart, E. C , 391 Travisano, R., 87

Stewart, J. P 76,452 Tremmel, R., 179
Stockdill, S 15,434 Trend, M. G 557,558
Stockdill, S. H., 98 Trochim, W. M. K 471
Stoecker, R 183 Trow, M., 255
Storm, ]., 502 Tse-tung, M., 202
Strassmann, B., 43 Tucker, E., 162
Strauss, A., 125,127,128, 239,453,454,465,487,488, Turksever, A., 150
489,490, 491, 492 Turner, A., 86
Strauss, A. L 57, 67,125,215, 324,454 Turner, J. H 100,101
Strike, K 52 Turner, R 110
Stringer, E. T., 179 Turpin, R., 241
Stufflebeam, D. L., 550, 551 Tuval-Mashiach, R., 478, 551
Stull, D., 83 Tuwaletstwa, P., 279
Suchman, E., 163 Tymitz, B. L., 292, 460,462
Sudermann, H., 341
Sullivan, W. M 459
Sussman, M. B., 405,452 U.S. General Accounting Office (GAO), 93,200,217,
Suzuki, L. A., 405 452,586
Swadener, B. B., 552 Uchtelle, L 559
Swidler, A., 459 United Way of America, 151
Symon, G., 405 Uphoff, N., 184
Tallmadge, J., 27 van den Hoonaard, W. C., 278

Tam, V. C-W, 395 Van Maanen, J., 118,405, 432
Tarule, J. M 5, 6, 7, 433 Van Manen, M., 104,106, 482
Tashakkori, A., 76, 247,248,307,556, 571 Vesneski, W-, 293
Taylor,}., 402 Vidich, A.}., 81,84, 95,268,457,493
Taylor, S. J., 69, 95,111,454 Vitt, M v 502
Taylor-Powell, E., 194 Von Bertalanffy, L., 120
Teddlie, C., 76,247, 248,307,556,571 Von Hipple, E 561
Tedlock, B., 116,391 Von Oech, R 514
Tesch, R., 133 Von Wiese, L., 457
Textor, R., 200
Thomas, D., 96
Thomas, J., 131,134,549 Wadsworth, Y., 183,184
Thomas, W. I., 96 Wagoner, D., 279
Thompson, L., 129,269 Waldrop, M. M., 123
Thoreau, H. D., 504 Walker, ]., 393,395
Thuesen, J. M., 507 Wallace, R. A., 111
Tiemey, P 273,327 Walierstein, I., 120
Tiemey, W., 116 Walierstein, M. B., 501
Tikunoff, W., 400 Walsh, D. J 402
Tilley, N., 95 Walston, J. T., 389
Tlman-Healy, L., 118 Walters, ]., 146
Tilney, J. S.,Jr., 501 Wandersman, A., 79,183,187, 269
Tinbergen, N-, 330 Wang, Z., 548
Tipton, S. M., 459 Wamer, W. L., 314
Toklas, A. B., 322 Warren, M. K., 501
Tomlin, Lily, 577 Waskul, D., 112
Torres, R 179 Wasserman, G., 501
Torres, R. T., 181,220 Wasson, C., 203
Author Index S 111
Waterman, R. H., Jr., 7,231,237, 245,297, 434 Williams, B. F., 44,46,47, 48,54,58
Watkins, J. M 181,182 Williams, W., 161,170
Watkins, K. E., 179 Wilson, E. O., 100
Watson, G., 102,111 Wilson, P., 588
Wax, R. H., 269, 312,314, 329 Wilson, S., 385
Weber, L 568 Winstead-Fry, P., 121
Weber, M 52 Wirth, L 53
Webb, E. J., 192,239, 270,292 Wisp, L., 52
Weick, K. E 292 Witt, H., 109,110
Weidman, E., 108 Wolcott, H. F., 84,133, 506
Weiner, E., 96 Wolf, A., 111
Weiss, C., 170 Wolf, R. L., 292,460,462,554
Weiss, C. H 550,578 Wooden, J., 232
Weiss, H. B., 98,183, 503 Worthen, B. R., 550
Wengraf, T 89,115,478 Wright, D. J., 503
West, C., 459 Wright, H. F., 118
West, J., 195 Wu-Men, 502
Wheatley, M., 123
White, M 116
White, P., 174, 452, 560 Yaeger, T. F 181
Whitehead, A. N., 105 Yin, R. K., 93,298,452,553
Whiting, R., 391 Yong, M 391
Wholey, J. S., 164 Youngson, R., 575
Whorf, B 288,289
Why te, W. R, 81,125,179, 221,269, 273,284, 298,314,
437 Zaner, R. M., 105
Wildavsky, A., 184 Zeichner, K. M., 179
Wilkinson, A., 293 Zilber, T., 478,551
m
Subiect Jrvdex
Abduction, 470 content, 248-257,452-471

Accountability, 149,151,190,199 creativity in, 432^33,438,442,467, 512-515,570
Action research, 145,177,195,213,221-222, 224,269, deductive, 453-456. See also Deduction
274,331,346,388, 436,495,542 documenter's perspective, 287-288,589-598
See also Inaction research during fieldwork, 304,323,331, 436-137
sample question, 225 examples, 5-9,58,433-434, 501,507,508-510,
status, 223,398,542 exercise, 481
Action science, 145 focusing, 439,503-504
Advocacy-adversary model, 554 heuristic, 486-4S7
Advocacy and inquiry, 129-131,570 ideal types, 9
Aesthetic merit, 87, 544,548,570 imaginative variation, 486
Alexander the Great, 37-38 inductive, 55-58, 453-456,470. See also Inductive
Analysis, 34, 248-257,431-534 analysis
as poetry, 87,432,548 interview data variations, 342-348,349
as story, 432 logical, 468-473
auditor for, 562 matrix, 468-474,492
beginning, 436^37,440-442 negative cases, 95,493^94,496,554-555
bracketing, 485-486 no formulas for, 432-434, 466,554,570,588
categories, 351 organizing, 437-438,439,440-441
causes, 470,478-481 paradigms example, 9
challenge of, 432-434 pattems, 442,452-471,485-487, 501-502
classification, 351,457-466 phenomenological, 482-487
coding, 442-447,462^66,489,490,492^93,496, reporting, 439,449-450, 495,502-512,555
516-517,545 See also Audiences; Credibility; Inductive
comparative, 9,56,57,164, 228-230,231, 254,293, analysis; Interpretation; Significance; Rigor;
465,478-481, 492^93,489-491,553-555 Rival Interpretations; Typologies; Units of
computer-assisted, 442-447 analysis
13. 113
similarites focus, 110 discipline-based questions, 216

strategies, 55-64,437-438, 439 sample question, 225
syntheses, 486,487 status, 223
themes, 235, 297,305, 323,442, 452-471, 485-487, Behavior, questions about, 348-349, 352
501-502 Better practices, 220,233, 564-566
triangulation, 467 Bias, 49,51,53, 260,292,569
varying purposes, 434-436 acknowledging, 65, 93,328, 553, 569
Analytic Induction, 91, 94-95, 454, 493-494,554 alternative perspectives on, 570
modified, 494 against qualitative, 573
Anticipatory research, 200-201 controlling, 169,545,563, 569
Anthropology, 81, 89,124, 265,267-268,270, 271,311, grounded theory reducing, 128, 489
321,392,454-455,557 interviewing, 343,367
cognitive, 132 literature review and, 226
ethics, 326-327 methodological, 174
material culture, 293, 295 paradigm-based, 71, 72, 400
Applications, 143-205 sampling issues, 230, 495
summary, 204 sampling the best, 233
Applied research, 213, 217-218, 224,434 See also Triangulation
sample questions, 218,225 Biography, 89,116,132, 450-451,478
status, 223 interpretive, 116
Appreciative inquiry, 181-182 See also Life histories
Appropriateness criterion, 33, 68,145, 255, 585, Boundary problem, 120,225
587-588 Box, thinking outside of, 2
Appropriate applications summary, 204 Bracketing, 106,107
Archives, 293 realities, 111
Art criticism model, 173 Breadth versus depth, 227-228, 254
Artistic criteria, 542, 543,544-545, 547-548,570 Breaking the routine, 202
Assumptions, 135,224,328, 329,336-337, 400,579 Bricolage, 400-402
See also Paradigms Burning questions, 80
Audiences, 9-12,13,434-436,449,503
credibility and, 542-553
reflexivity about, 495 Cartoons, 6,22, 43,57, 62, 70, 82, 90,103,117,180,201,
See also Stakeholders 213,219, 249,256,263,332, 359,366,373, 410,433,
triangulation, 561-562 479, 488, 510, 547,558,573
Audit trail for rigor, 93,562 Case studies, 79,297-298,305,438, 439, 447-452, 478
Authenticity, 51,301, 437,544,546, 562, 575 art of, 432
through rigor, 555 case record, 449-450
through voice, 65, 88-89, 494-495 examples of, 7,15,155-157,197-198, 274,451-452,
Author, about the, 635 501-502,508-509, 518-524
Autism, 332 credibilityof, 553-554
Autobiographical data, 571 focusing, 225-230, 439, 447, 450
Autoethnography, 84-91,132 for legislative monitoring, 199,311
criteria for judging, 87, 542-543,544-545, 548 generalizing from, 93,501, 582-584
defined, 85 in evaluation, 55,152,158,162, 448
example of, 138-142 language of, 195,198
lexicology, 85 layers of analysis, 297-298,447-448
patterns across, 158,200, 438, 501
portfolios, 193
Balance, 51, 65,241, 267-268,325,328,331,415, . prospective, 201
503-504,575-576 rigor in, 553-555
documenter's perspective, 593-598 tradition of, 132
truth and utility, 550-551 See also Purposeful sampling; Sampling; Unique
Basic research, 213,215-216,224,225, 264,434 case orientation syntheses
Subject Index LEI. 115
units of analysis, 228-230,231, 254, 297-298,300, examples, 166,501-502

439,447, 448 See also Evaluation
Category construction, 58 Complexities, 59, 60
See also Analysis; Patterns; Themes; Typologies Complexity theory, 123-124,126,133
Causality, 93,94,478-481,492-493,544,545 Computer-assisted analysis, 442-447
abductive, 470 Conclusions, 506
attribution probiem, 153-154 rival, 553-554,563
constructivist, 98 Confidentiality, 273,286, 294,316, 387,400, 407-412,
modus operandi, 471 416,496
See also Analytic induction new directions in, 411-412
theory of change, 163 See also Ethics; Human subjects protection;
Census categories, 351,461 Informe d consent
Chance, 260,553 Confirmatory research, 193-194, 239-240,323,436
Chos theory, 123-124,126,133 analysis, 436,454,467,562
thrivmg on chos, 194 Conflict model, 270
Clear questions, 361-363 Connoisseurship evaluation, 172-173,550
Classification. See Analysis Consensual validation, 99,467
Clinicai cases, 148,400 Consequential validity, 545, 548
Closeness, 27-28,48-49,50, 67-68,112,171,175, 217, Constant comparative method, 56,125, 239,489-491
262,303,575 See also Grounded theory
Coding procedures, 127,462-466 Constructionism, 79, 96-103,132
sample codebook, 516-517 dualist and monist, 102
See also Analysis; Computer-assisted analysis social, 96-103,133
Collaborative approaches, 122,182-185, 269,320-321, versus constructivism, 97
323,327,331, 388,393, 549 Constructivism, 79, 80, 96-103,132,190,332, 542,543,
analysis, 496-497, 560-561 544-545,546-548
feminist methods, 183, 269,388-389 assumpons, 98
heuristic inquiry, 183 grounded theory, 128
confidentiality and, 412, 496 interviewing, 404
interviewing, 346, 400 responsive evaluation, 171
"nothing without us," 335-338 versus constructionism, 97
principies of, 185 Content analysis. See Analysis
See also Participatory approaches Context, 41, 62-63,262,447,582-584
Coinbjning qualitative and quantitative. See Qualita- constructivism and, 98
tive and quantitative, combming defined, 63
Combining qualitative strategies, 134, 248,265, 287, evaluating outcomes and, 158
294, 306-307,396, 449, 551,552,556 generalizations and, 582-584
case example of, 451-452,518-524,589-598 hermeneutic, 114,115
criteria, 550,551,552 historical, 284-285
interview types, 347-348 preserving, 49,447,480,492,582
omnibus field strategy, 265, 306-307 researcher's perspective as, 64,494-495, 566
triangulation, 559-560 sensitivity, 41, 61-63
Coming of age, 9, 89,139-142,434 setting, 280-283,582
Community development, 200, 273,388 testing and, 191-192
action research, 221, 269 triangulation and, 563-564
Comparative analysis, 56, 57, 293, 465,478-481, Controversies, 34, 68-71,101,222,327,404,553,
492-493, 489-491, 553-555 571-588
example, 9 See also Criteria; Paradigms
ideal-actual, 164 Cooperative inquiry, 183
larger samples, 492-493 Core questions, 134
See also Analysis Correspondence theory, 91-96,102,489,543,544
units of analysis, 228-230, 231,254,297-298, 300 See also Crisis of representation
Comparing programs, 56,164-166,228 Covert observations, 269-273, 277
Creative synthesis, 41, 55, 58,108,486, 487, 548 Criticai change criteria, 542-543,544-545, 548-549,552,
Creativity, 544,548 570
autoethnographic, 86 Criticai ethnography, 131,134,543,548-549,552
in analysis, 86, 467,512-515 Criticai incidents, 47,238,297, 439,451
design, 248-256 See also Criticai case example
in fieldwork, 302 Criticai inquiry, 79
in grounded theory, 127,129 See also Criticai change criteria
in interviewing, 394-405 Criticai theory, 79,129,130-131,133, 543, 548-549, 552,
measuring, 192 553
Credibility, 20, 50-51, 93,242,309,321,398-399,497, See also Criticai change criteria
542-588 Criticai thinking, 513-514
attacking, 92 Cross-case analysis, 57,305,438, 492-493, 500
confirmatory cases, 239 See aso Analysis; Comparative analysis
fieldwork entry, 313, 314 Cross-cultural interviewing, 291,311,323,391-394, 455
paradigm acceptance, 553,570-588 Culture, 81-84
of alternative frameworks, 135, 553 material, 293,295
of qualitative methods, 14, 68-71,242, 260 organizational, 81-82
of small samples, 245-246 popular, 83
of the researcher, 64, 245,552, 566-570, 584 relativism, 100
rigor and, 552,553-566
See also Criteria; Triangulation; Validity
three elements of, 552-553 Dangers, 415-416
through balance, 241 DARE evaluation, 162-163
through realism, 95 Decisions:
through triangulation, 93; 555-566 framework, 135
valuing qualitative inquiry, 553, 584
methods, 49,71-72,77-78,176-177,189,209-257
Crisis of representa tion, 79,100
purpose, 213-223
Criteria, 542-552,562
theoretical orientation, 135
artistic, 542,543,544-545,547-548,570
trade-offs, 223, 225-230,275-276, 401
autoethnography, 87, 542-543, 544-545, 548
utilization-focused, 173-175,202,508,510,513
claims and, 587-588
Deconstruction, 84,90,101,190, 500
connoisseurship, 173
example, 102
constructivist, 542-543,544-545, 546-547
Deduction, 56-57, 67,94,248,252, 453-456, 470,553
criticai change, 542-543, 544-545, 548-549,552, 570
See also Analytic induction
evaluation standards, 542-543, 544-545, 549-551
theory development, 125,470
exhibit comparing, 544-545
Delphi technique, 200
for judging frameworks, 135,544-545
Democratic dialogue, 185-187,190
for judging designs, 72; 247-257, 544-545
Democracy and inquiry, 187-190
for judging findings, 13,23, 50-51,544-545, 560-561,
Description, 23,28,47,48,214,278,280-283,303,331,
562
437-440
for observations, 262,265 in evaluations, 172,199,262, 280-281,285
for qualitative inquiry, 28,51,265 practice writing, 281
for truth-oriented inquiry, 93, 542-543, 544-545 thick, 437-440,451,503-505,592
matched to inquiry purpose, 213, 224,265,542-552, versus explanation, 478-479
562,570 versus interpretation, 480,503-504
mixing, 551-552,562 Designs, 34, 47,247-257
research status distinctions, 223 issues summary, 254
sampling, 238,495 none perfect, 223
See also Credibility paradox, 254-255
traditional research, 542-543,544-545, 570 responsive evaluation, 171-172
truth, 578 trade-offs, 223,225-230,275-276,401
Criticai case example, 99, 236-237 two perspectives, 255
Development, interna tional, 153-154,183, 236, 291, Enlightenment, 100

392, 394,395 Entry into the field, 309, 310-317
Developmental evaluation, 180,220 Environmental scanning, 194
Developmental perspective, 54,167-168 Epiphanies, 451
Dialogue, 181,400, 544, 546 Epistemoogy, 134
democratic, 185-187 See also Paradigms
Differences in kind, 165 Epoche, 484-485, 553
Discovery, 28, 67,107, 323,436,453-454, 467,494 Essence, 106,109,482
Discipline-based questions, 216 See also Heuristic inquiry; Phenomenology
Disconfirming cases, 239-240, 436 Ethics, 241, 269-273, 287,311, 316,326-327,405-415,
See also Negative cases; Rival interpretations 560
Dissertations, 11,33-35,44,68,246,256, 279,301, checklist, 408-409
310-311, 328,346, 416,436,437, 503 See also Human subjects protection; Informed
anxiety about, 500 consent
credibility of, 95 Etic perspective, 84, 267-268, 277,331, 454
foc using, 225, 226-228 Ethnography, 79, 80, 81-84,132,262,275, 303,391
grounded theory and, 127, 487-488 applied, 81
protecting subjects, 271, 346 creative analytic, 86
novis as, 87 criticai, 131,134, 543,548-549
student seeking help, 77-78 embodied, 86
theory and, 136,215 evaluation and, 83
Diversity, documentation of, 164-166, 351 language of, 195,198
Diversity of qualitative inquiry, 76-80,131-135 meta-, 500
applications, 145, 203,204 narrative forms of, 116-117, 552
core questions, 134 new, 85, 203
integrating approaches, 134 quick, 194
Doctoral research. See dissertations See also Autoethnography
Documents as data, 47,171, 293-295 Ethnomethodology, 110-112,132,234, 499
analyzing, 498-499 Ethnostatistics, 573
limitations of, 306-307 EvalTalk, 29
unobtrusive, 191 Evaluability assessment, 164
Documenteis perspective, 287-288,589-598 Evaluand, 218
Drama turgical analysis, 499-500 Evaluation, 151-177,218-221
Dura tion of observations, 273-275, 277, 331,567 case studies in, 55,274, 447-449, 518-524,525-534
Dynamic perspective, 40, 54 constructionist, 97-98
documenting development, 167-168 culture of, 189
defined, 10
design example, 249-256
Ecological psychology, 118-119,132,133 developmental, 180, 220
Early childhood program observation, 23-26 explaining purpose, 407,408
Effects. See Outcomes feedback, 67,197-198, 324-326, 331,506-510,511, 512
Elitist research, 190 first evaluation, 209-211
Emergent design, 40,43-45,173,194,302, 318,330, 436 focusing, 225,226-228,232,234,276,388,435,511
paradox of, 254-255 formative, 42,152,160,164, 212,213,218,220, 221,
protecting human subjects, 246-247, 407-409, 411 224, 273,308,434, 435,542, 554-555
sampling in, 240, 246 goals-based, 147,163,170,560
Emic perspective, 84,267-268,277, 303,331, 363,454 harmonizing values, 176-179
Empathic neutrality, 40, 49-51, 53,365-366, 405, 569 holistic approach, 67,179, 228,287
Empathy, 49,50, 51, 52-54 humanizing, 171,175-176
defined, 52 implementation, 161-162,164,165,199,285
Empiricism, 92 inclusive, 187
Empowerment evaluation, 80,183,190, 220,269, 337, informed consent, 407-408
411-412, 549, 568 interpreting, 468-481
models of, 168-175 Feminist theory, 65, 79,129-130,132,133,190,542,549,

multicultural, 98 553
multivocal, 98 methods, 183,269,388-389, 552
outcomes, 150,151-159,164,166,192,197-198,204, Field notes, 266, 286,289,302-306,309, 316,318,331
241,471-477,518-524,525-534 during interviews, 383
participant observation, 262
Fieldwork, 5, 48,259-338
participatory, 175-191,269,388, 396-399,400,
analysis during, 304,323, 331,436-437
496-497,560-561
closing, 322-324
personalizing, 171,175-176,186
demands, 207
priority setting, 225,511
dimensions of, 277
process, 159-160, 439,471-477
process use of, 180,189-190,220,327, 398 entry, 310-317
purposes for, 213-214, 218-221,223,224, 435 ethnographic, 83
quality assurance and, 147-151 introductions, 311,314
questions for, 438 guidelines summary, 331
realist, 95 layered cases, 297-298
reporting, 435, 438,439, 449-450,495, 502-512,555, omnibus field strategy, 265
561-562 relationships, 310-326, 328,567
sensitizing concepts for, 280, 474-477 routinization, 317-321
standards, 542,543, 545, 549-551 stages, 239,246,331
summative, 14,147,149,164,213,214,218-219,224, strategies, 47-54
434,435, 542 variations summary, 277
systems approach to, 121,167-168 Flexibility, 40, 68, 72,109,175,194, 302,315,331,490,
thinking, 188-190 550
utilization-focused, 68,173-175, 212,464,508,510,
fieldwork example, 45
513
interviewing, 343
Evoca tive inquiry, 84, 542,543,544-545, 547-548
methodological, 202, 248-256
Executive summary, 511-512
Experience, questions about, 348-350, 352 sampling, 240
Experimental designs, 248-256 See also Emergent design
See also Qualitative and quantitative Fruit of qualitative methods, 3,4-5,28
Explana tion, 478-481,546 Focus groups, 112,164,236, 343-344, 385-391,399
conflicting, 557, 558 See also Interviews
rival, 553,563 Focusing research, 223, 225-230,275-276,277
See also Conclusions; Interpretation reports, 503-504,511, 512
Exploratory research, 193,239 Follow through evaluation, 49, 61,162,191,557
Extrapolations, 584 Formative evaluation, 42,152,160,164,212,213,218,
220, 224,273, 275, 308,434,435, 542,554-555
sample question, 225
Face validity, 20, 561
status, 223,542
Factual, 28
versus action research, 221
Faimess, 51,575-576
Foundational questions, 80
Faith-based programs, 175
Fourth generation evaluation, 171
Farming systems research, 121-122,195
Frameworks, alternative, 134-135
Feedback, 42,67,197-198, 324-326,331,506-510
interview, 374-375 See also Criteria; Paradigms
timing, 325-326,331,506-507 Freudian inquiry, 129,130
Feelings: Funders, 44,153
of the observer, 310, 313,315-317, 328,331, 548,569 See also Stakeholders
of interviewers, 403-404,405-406 Futuring applications, 200-201
See also Reflexivity Fuzzy methods, 256
Feelings, questions about, 350,352 Fuzzy set theory, 492-493
Generalizations, 93,94-95,96, 215,224, 544,545,556, through stories, 196

581-584 Humanistic values, 177,183,202
action research, 221 Humanizing evaluation, 171,175-176
alternatives to, 584 Human subjects protection, 191,238, 246-247, 254,
analytic induction, 493 270-273,286, 346,393-394,407-409,411,412
core principies of, 581-582 See also Confidentiality; Ethics; Informed consent
from evaluations, 220 Humanity, common, 318-319, 328
in formative evaluations, 220,221 Hypnosis, 237
lessons learned, 220,500-501, 564-566 Hypotheses, 94-95,194, 252,253, 277,324, 479, 544,
logical, 236-237 545,556,557
naturalistic, 583 alternative to, 193, 277,278
skepticism about, 100,546,582-583 analytic induction, 493-494,554
sampling issues, 230, 244-245,495 grounded theory, 125, 324,454
See also Purposeful sampling; Sample size logic model, 163,470
time-Iimited, 217-218 null, 500
gestalt, 59 rival, 553-554, 563
Gigo's Law, 1 testing qualitative, 96
Goal-free evaluation, 169-170, 560 working, 304,436
Goals-based evaluation, 147,163,170,560
Going native, 84,267,568
Government Performance and Results Act, 151 Ideal-actual comparison, 164
Grand Canyon, 63-64, 89, 281,282, 290,308,437 Ideal types, 459
autoethnography from, 138-142 Ideologically-oriented inquiry, 129-131
Group interview, 17-18, 76,346 See also Criticai change criteria; Orientational quali-
Grounded theory, 11,56, 67, 79,124-129,132,133, tative inquiry
215-216,324,454,487-492,545 flluminative evaluation, 171-172
analytical process, 487-492 Impact evaluation. See Outcomes
influence of, 487 Impartiality, 93,316,569, 570
terminology, 490 Implementation evaluation, 150,161-162,164,165,
objectivist, 128,545 199, 285
theoretical sampling, 239,490 Inaction research, 222
theory bits, 491-492 Inclusion, 186,187
Guidelines for fieldwork, 331 Independent judgment, 93,169,190
See also Credibility; Integrity; Rigor
Indigenous concepts, 454-456, 457-458, 507-508
Halcolm, biography of, 635-635. See also Author Index Individualized outcomes, 152,154,156-159,202, 226,
Harmonizing values, 176-179 241, 438,471-477
Heisenberg uncertainty principie, 326 reports of, 518-524, 525-534
Hermeneutics, 79,113-115,133,497-498 results mapping, 196,465
hermeneutic circle, 114,497-498,569,570 Inductive:
Heuristic inquiry, 107-109,132,183, 234 analysis, 41,55-58,453-456,470,553
analysis process, 486-487 and deductive, 67,125,453-456,470
German alternative tradition, 109-110 methods, 94,248
See (liso Phenomenology theory generation, 125, 487-492
History, 284-285, 293-294,307,439 Infiltration approaches, 310
life, 404, 478 Informal conversational interviews, 285-288, 316,
Holistic perspective, 41,58-61, 62,228,248, 252,273, 342-343, 347-348, 349, 380-381, 411
459 Informal interactions, 285-288
in analysis, 67,447,450,480,492, 497-498,502 Informa tion-rich cases, 46, 230,234, 242,245,563,581
in evaluation, 179,287 See also Purposeful sampling
in genetics, 60 Information systems, 149,168,202, 238,274
in systems analysis, 120,502 qualitative, 241
Informed consent, 246-247,254,270-271,311, 407-411 team, 346,384,386,400, 554, 560

See also Confidentiality; Human subjects types of, 342-348
protection support during, 375
Inner perspectives, 48,340-341 table comparing types, 349
Insight, 48,51,52, 302,304,437, 494 wording questions, 353-374
Integrating qualitative approaches, 134 Interview guide approach, 343-344,347-348,349
See also Combining qualitative strategies Introspection, 104,110,111, 264,299
Integrity, 51, 64,553,570 Investiga tive approach, 270, 561
Interdisciplinary, 121,217 I-Thou, 64
questions, 218
theory development, 216-217
Interludes, between chapter, 33-36,335-338,537-539 Judgment, 331
Internet resources, 29,136, 205,445
focus groups, 389
Interpretation, 50,323, 331,438,465, 470,477-482, Kalamazoo study, 17-20
503-504,571 Key informants, 236,321-322,327,331
connoisseurship, 173 Knowledge:
defined, 480 altemative claims, 135
documenter's perspective, 593-598 and democracy, 188
hermeneutic, 114, 497-498 applied, 217
phenomenological, 106 conscious, 483-484
three forms of, 480 constructed, 102
Interpretivism, 79,115,132,133 disciplinary, 215
Interventions, 54, 56,161-162,217,218, 405 for change, 129
Interview guide approach, 343-344,347-348,349 for its own sake, 215
example, 345,419-421 questions about, 350,352
Interviews, 339-427 intentional, 483-484
analyzing, 438-441,525-534 interdisciplinary, 216-218
controlling, 375-378,415 root problem of, 130
creative approaches, 394-40 self-knowledge, 299, 301,495
cross cultural, 291,311, 391-394 situa ted, 400
focus group, 385-391,399 sociology of, 99,102
group, 112,390-391, 400 stories as, 196
impromptu, 45 transforming, 130
length of, 227 Knowledge-generating evaluations, 220
limitations, 306-307,337,347,401
neutrality, 365-367,405, 569
notes during, 383-384 Labeling theory, 112
observations and, 265,316-317 Language, 100-101,102,574-576
one-shot question, 378-379 cross-cultural, 392-393,455
paying for, 412-415 indigenous, 288-290,362-363,454-455,457-458,
phenomenological, 106 507-508
probes, 344,365,372-374 political, 188
protecting human subjects, 246-247; 269-273,346, See also Sensitizing concepts; Terminology
393-394,405-412,415,416 Leading questions, 343,367
purpose of, 340-341 Learning organizations, 177,179-181,184
rapport, 310, 331,365-366 Legislative monitoring, 23,198-200,241,311
recording data, 286,380-384 Lessons learned, 220,232-233,451,500-501
rewards of, 416-417 high quality, 564-566
See also Questions; Quotations Life histories, 404,478
sequencing questions, 352-353 See also Biography
skills, 27,340,341,379,387,402-405 Limitations, research, 242,246,247,306-307,337,341,
specialized, 402-405 563
Literary ecology, 119 See also Mixed methods; Qualitative and qualitative
Literature review, 226,239 Multiple opera tionalism, 239
Listening actively, 341,417 Multivocal evaluation, 98
Listservs, 29,136,205,445 Myers-briggs type indicator, 507-508
Lived experience, 104,544,561
Logic models, 162-164
Logical analysis, 468-473, 553 Narrative analysis, 115-118, 133,196-197,478,551, 552
Logical empjricism, 92,114 hermeneutic, 497-498
Logical positivism, 92, 94 Narratology, 115-118,133
Lost, 279 Natural experiments, 42,111
Naturalistic inquiry, 39-43,54,126, 248-256, 262
creativity, 302, 401-402, 512-515,544,548
Mapping experiences, 27 criteria for judging, 546
Marxist inquiry, 129,131,133 degrees of, 67,253, 265
See also Orientational qualitative inquiry design, 44,47
Matching methods to purpose, 49, 68, 72,145,212, evaluation and, 171,173
242,255, 267, 573-574,585, 587-588 omnibus field strategy, 265
See also Appropriateness criterion; Criteria rapid reconnaissance, 194,274, 392
Matrix analysis, 468-472 Nature, observing, 284,290
varieties of, 471 Needs assessment, 201
Maximum variation rule, 109 needs, defining, 336-338
Meanings, 147,150,158-159,193, 310, 363, 467-468, Negative cases, 95, 493-494,496, 554-555
477-481 See also Disconfirming cases
hermeneutic, 497-498 Negotiation, 310,435
phenomenological, 485-487 Neutrality, 49-51,53, 328, 569, 570
primacy of, 477 No holds barred, 205, 570
See also Conclusions; Interpretation Nonlinear dynamics, 123-124,133
Meta-evaluation, 211, 562 Nonverbal communication, 290-291
Metaphors, 123-124,125-126, 281,290,432 .
analysis using, 504-506
Methods decisions, 12-14,71-72,77-78,176-177,189, Objective reality, 94,96
202,209-257 Objectivity, 48, 49, 50,93-94,96,312, 487489,544-545,
options summary, 254 574-576
priority setting, 225 challenges to, 109, 548
standards and, 549-550 death of, 576
See also Purposeful sampling; Qualitative and illusion of, 257
Quantitative; Sample size; Sampling in grounded theory, 128, 488-489
trade-offs, 223,225-230,276 politics of, 570
utilization-focused, 173-175,550 skepticism about, 101,548
Mindfulness, 40,134 two views of, 96
Mission fulfillment, 293-294 Observation, 21-26,259-332,278
Mixed methods, 5,13, 68,160,180, 220,247-257, covert, 269-273,277
306-307,331,556-560,574, 585 documenter's perspective, 287-288,589-598
exhibit, 252 duration, 273-275,331, 567
See also Multiple methods; Qualitative and examples, 23-26,262
qualitative focus, 275-276,277,331
Models of evaluation, 168-175 limitations, 242,246, 247,306-307,337, 563
Modus operandi analysis, 471 notable nonoccurrences, 166,295-297, 500
Most significant change story, 196,197-198 onlooker, 265-267
Multicultural evaluation, 98 part-time, 314-317
Multiple methods, 68, 72,92,220,247-257,306-307, preparation for, 260-261
331,585 purposes, 171
Exhibit, 252 sensitizing framework, 276-279, 474-477
sources of data, 279-302 evaluable, 164

stages of, 310-326 implementation and, 161-162,164,285
technology, 307-309 individualized, 152,154,156-159, 226,241
training for, 260-261 lack of measures for, 192
value of, 261-264 mapping, 153
variations in, 265-277 prevention, 166-167
with interviewing, 265,287,316-317 reporting, 518-524,525-534
See also Participant observation results mapping, 196, 465
Observer observed, 314, 318,326-330, 331 sensitizing matrix, 474-477
Observing oneself, 299, 301, 302, 569 unanticipated, 169, 263,288
See also Reflexivity
One-shot questions, 378-379
Ontology, 134 Paradigms, 51,80,174,190,252-257,543,570-588
realist, 101 and criteria distinctions, 544-545
relativitist, 97,98 and practical applications, 145,174,252-253
Open-ended questions, 5,20-21,56,342-348,353-358, and purpose distinctions, 222-223
367 and theoretical frameworks, 134-135
interview length, 227 coming of age, 9,89
response examples, 16,18-19 conflict, 270
Openness, 44,53,109,115,170,171,173,175,203,252, credibility, 553,570-588
318,331,402,514 debate, 68-71, 92,101,119,134,221,252-253,570-588
analytical, 555 defined, 69
literature review and, 226 diversity within, 76
methodological, 252-255 ethnographic, 83
Opinion questions, 350,352 grounded theory, 488-492
Opportunity sampling, 45,240,323, 331 inquiry without, 145
Oral briefings, 511,512 Kuhnian, 71, 99
See also Feedback linear vs organic, 119
Organizational development, 123,163-164,167-168, nonlinear dynamics, 123
177-185,220,262, 388,451 of choices, 257
appreciative inquiry, 181-182 phenomenological, 104-107,482-487
action research, 221 qualitative historical, 133
learning organizations, 177,179-181,184 sociology of knowledge, 99
outcome mapping, 153-154 Participant observation, 4, 21,22, 83,106,262,265-267,
mission fulfillment, 293-294 287,416,569
process studies, 159-160 See also Observation; Reactivity
quality management, 144-146,149-150 effects on observer, 569
reflective practice, 177,179-181 unobtrusive, 191,291-293
sensitizing concepts, 280,474-477 variations in, 253,277
stories and, 195-196,451 Participatory approaches, 175-191,269,320-321, 327,
Organizational narratives, 118,195-196 331,549
Organizational paradigms, 119-120,181 analysis, 496-497,560-561
Orientational qualitative inquiry, 129-131,133,543, feminist inquiry, 129, 269, 388-389
548-549,553 interviewing, 346, 388,396-399,400
in evaluation, 172 "nothing without us," 335-338
See also Criticai change criteria principies of, 185
Original sources, 34 See also Coliaborative approaches
See also Documents Part-time observer, 314-317
Outcomes, 150,151-159,204,241,401, 471-477,492 Patterns, finding, 235, 452^71,485-487
blues song lyrics, 153-154 bracketing, 485-486
case examples, 155-156,475-477,518-524, documenter 's perspective, 589-598
525-534 See also Analysis
change story, 196,197-198 People-oriented inquiry, 27-28
Perceptions, 264,324 Prevention evaluation, 166-167

Personal experience, 40, 47-49, 85-86, 88, 264,303, 324, Probes, 344,365,372-374
326,328,329,331,416-417, 485,548 Problem-solving research, 221
credibility and, 566 Process applications, 159-161
and heuristic inquiry, 108,486-487 Process/outcomes matrix, 471-477
insider-outsider, 331,335-338,368,399 Proof, 2
learning, 329 See also Criteria
of fieldwork, 329, 569 Prospective studies, 200-201
Person alizing evaluation, 171,175-176,186 Protecting data, 441-442
Perspective, 41, 63-66,328,331,332,363,417,478, Publication, 224,434-436,437, 450,502-504
494-495 final reports, 511
emic, etic, 267-268,331,335,454 qualitative joumals, 503
example of, 592-598 Purposeful sampling, 40,45-46,230-243,254,331,332,
of participants, 171-174,176,185,546,560-561 563,581
personal, 328-329 See also Sample size; Sampling; Units of analysis
Rashomon, 332 summary exhibit, 243
Phenomenography, 104, 482, 483 Purposes:
Phenomenology, 69, 79,104-107,132 alternative inquiries, 9-12,13, 23,245, 542-552
analytical process, 482-487 analysis, 434-436,506
and constructivism, 128 criteria and, 544-545
and Verstehen, 52, 69 distinctions, 222-223,254, 434-436,542-552
feminist challenge to, 130 research typology of, 213-223, 434,542-552
psychology example, 8-9 summary exhibit, 224,544-545
See also Heuristic inquiry
various traditions, 482-483
Photography, 281,308, 482 Qualia, 11
Physical environments, 280-283 Qualitative and quantitative:
Poetry, 87,548 combining, 5,14,15, 47,49, 68,160,170,193-194,
Point of view, 328 202,220,234,238,248-257,545, 574
See also Perspective contrasting, 12-15,20-21,51, 56-57, 59-61, 68-71,92,
Politics, 131,186,188,190,555,570 95,119,127,150-151,152,162,165,166-167,168,
sampling based on, 241,495 175,194,227,230, 234,244-245,248-256,326,
See also Criticai change criteria 336-338,353-354,467, 553,555, 556-559,572-588
Portfolios, student, 193 in grounded theory, 127
Portraiture, 404, 432 joke, 572
Positivism, 69, 79, 91-96,132 reconciling, 556-559
See also Logical positivism sampling differences, 46,240
Postmodernism, 50, 65, 79,84, 91, 99-101,132,190, See also Paradigms; Purposeful sampling
332,551,579 triangulating, 556-559
defined, 92 Qualitative applications, 143-205
enlightenment to, 100 summary, 204
logical, 92,94 Qualitative data, 47,145,248-256,286
and skepticism, 99 as a strategic theme, 40,248
Postpositivism, 92-93, 98 defined, 4,47
Power, 100,103, 291,495,545,548-549,561 enhancing, 553-566
construction of knowledge and, 101,130,188 essence of, 457
sharing, 183,337 humanizing effect, 175-176,177
to control thinking, 188,336-337 omnibus field strategy, 265
Pragmatism, 69, 71-72,135-137,143-144,145,146,247, status, 572,585
253-255,307,399, 470,566,576,588 Qualitative traditions, 79-80
in reporting, 511-512 Quality, 145-147,148,150
validation, 579,588 assurance, 147-151,238
Praxis, 65, 79,115,134,180,544-545,546,548-549 assessing, 150
control, 148-149 changing, 93

enhancement, 148-149 constructed, 93, 94,101, 546, 547,548
improvement, 146,149-150,152 expression of, 87
meanings, 147,150 feeling dimension of, 548
metaphysics of, 147 hermeneutic, 115
of life, 150-151 phenomenological, 106
Quality of data, 5,341,383-384, 400,401,405, 440-441, language and, 100,101
513,542-552 inultiple constructions of, 96, 98,101, 546,575-576
autobiographical, 571 objective, 94,544-546
enhancing, 553-566 Reality-oriented inquiry, 91-96, 98,101,132, 543,
See also Criteria; Qualitative and Quantitative 544-545
Quantitative and qualitative. See Qualitative and Reciprocity, 311,312,318,324,329,408,412-415,
Quantitative 561-562
Queer theory, 129,130,133 Recording data, 380-384
Questions, interview, 341-427 Reflection, 104,264, 299,384
clarity of, 353,361-363 criticai, 483
control, 375-378,415 Reflexivity, 35,41, 63, 64-66,79,269,299,330,331,543,
cross-cultural, 291,311,391-394 544-545, 546,570
focus group, 385-391 analytic, 434,494-495,499
neutral, 353 autoethnographic, 87
one shot, 378-379 being reflective, 326,329, 331,417
open-ended, 5,20-21, 341-348,353-358,367 diagram, 66
op tions summary, 352 example of, 589-598
prefacing, 370-372 feminist inquiry and, 129
presupposition, 369-370 questions for, 65-66
probes, 344, 365,372-374 Relationships, 310-326,331
role playing, 367-369 analyzing, 478-481
sequencing, 352-353 See also Documenter's perspective; Ethics;
singular, 353,358-360 Observer observed; Roles
support, 375 Reliability, 53,93,192,261,433,465, 466,544, 545
types of, 342-347,348-352 Reporting, 434-436,438, 449-450,495, 502-512
why? questions, 363-365,366 analytic process, 434, 555
wording, 353-374 examples, 518-524,525-534
Questions not answers, 87, 341 focused, 513
Quotations from interviews, 21, 28,47,284, 286,303, opdons, 439
331,503 poetic forms, 87
examples, 456, 525-534 See also Publication
review by interviewees, 560-561 Representa tion:
transcribing, 380-384, 415,440,441 crisis of 79,100
dualist and monist, 102
objectivist, 102,114
Racism, 130 Relativism, cultural 100, 579
Rapid reconnaissance, 194-195,201, 274,392 Researcher as instruinent, 14, 50,51, 64,109, 299,301,
Rapport, 53,318,331,363,365-366, 405 566
Reactivity, 42,43,191,192, 269, 291-292,301,306, credibility of, 566-570
326-330,331,400, 401,405-406,407,567-570 in analysis, 433,513
Realist theory, 91-96,132, 543,546 Respect, 55,176,190,207, 271,312, 363, 394,417, 545,
grounded theory and, 128 549
transcendental, 94 Responsive evaluation, 171-172
Reality: Resources, 13,254,398,401, 496
ambiguous, 588 Results:
basic research, 215, 542,543, 544-545 mapping, 196
bracketing, 111 See also Outcomes
Revolution, 202 Situa tional responsiveness, 68, 72,330,379,400, 550

Riddles, 537-539,598 Skills, 27,34, 260-261,295, 303, 340, 341, 379,387,
Rigor, 93,260, 340,383,436, 480,544,545 402-405,417,489-490, 496, 513
credibility and, 552,584 Social constructionism. See Constructionism
debates about, 222-223, 572-588 Social environments, 283-284
enhancing, 553-566 Social justice framework, 98,187,545, 548-549
for inductive analysis, 127, 442 Sociology of knowledge, 99,102
intellectual, 570 Software, 442-447
of heuristic inquiry, 108 summary of, 444
paradigm dependent, 174 Sondeos, 195
See nlso Criteria Snowball sampling, 194; 237
Rival interpreta tions, 57,478, 480,493,553-554,563 See also Purposeful sampling
Roles, 310, 321,329, 331 Stages of fieldwork, 310-326, 331
Standardized interview, 344,346-347,348,349
example, 422-427
Sample size, 227-228,242-246 Standards, evaluation, 542,543,545,549-551
larger samples, 492-493 See also Criteria
See also Purposeful sampling State-of-the-art considera tions, 192-193
Sampling: Stenomask, 309
focusing, 225-228,254 Story, 10,47, 89,98,116,117,195-198, 293,406,551
limita tions, 563 analysis and, 432,438, 439,450, 478,502
opportunity, 45 evaluations using, 151-152,199
politics of, 495 evaluation example, 197-198
size, 227-228,242-246,254,297-298, 581 human nature of, 198
summary table, 243 humanize with, 558,559
time, 229 life, 404,478
the best, 231-232, 233 owning one's, 411-412
to redundancy, 246 program's, 199
units of analysis, 228-230,231,254, 297-298,300 Stakeholders in evaluation, 97-98,153,164,171-172,
See also Generaliza tions; Purposeful sampling; 187,189, 236,239,242,335-338,346,470, 472,
Units of analysis 562-563,570
structural variation, 109 intended users, 173-175,189,435
theoretical, 125 mapping, 472
Scenario construction, 200-201 reviewing findings, 560-561
Selective perception, 260, 261,264,321,329,331 Strategic framework, 38-39, 40, 66
Self-awareness, 64,86,264,299, 301,495 Strategic themes. See Themes of qualitative inquiry
See nlso Reflexivity Strategic thinking, 37,39, 50,66-67
Self-esteem, 192 Subjectivity, 50,101,544,574-576
Semiotics, 113,133,499,552 constructivist, 128,544,546-547, 569-570
Sensitizing concepts, 278-279, 348,439,456-457,470 death of, 576
examples, 42,278-279,280, 456,474-477 design, 173
Sensitizing framework, 276-280,301-302,474-477 minimizing, 93
Setting, 280-283 politics of, 570
See also Context rampant, 86
Sequencing questions, 352-353 Substantive contribution, 87
Show don't tell, 89 See also Significance
Significance: Sufi stories, 72-73,257,295,363, 417,481,563,580
determining, 57-58,66,151,295-297, 433,438, Summative evaluation, 14,147,149,164,213,214,
467-468, 218-219,224,434, 435,542
interocular, 467 credibility of, 95,542
reporting, 504,511-512,555 sample question, 225
See also Interpretation status, 223
Singular questions, 358-360 Symbolic interaction, 79, 80,112-113,132
Synthesizing studies, 500-502 See also Grounded theory; Theoretical orientations

See also Creative synthesis Theory-based sampling, 238,490
Synthetic thinking, 120 Theory in use, 163-164
Systems: Theory-method linkage, 125
complexity theory and, 123 Theory of action evaluation, 162-164, 563
defined, 120 Theory of change, 163
elephant story and, 122-123 Theory, program, 163-164
evaluating, 121,167-168, 501-502 limita tions, 337
family, 121 Theory to action continuum, 213, 218, 220, 221,224
farming, 121-122 status distmctions, 223
organizational, 119-120 Theses. See Disserta tions
Systems perspective/theory, 119-123,133 Thick description, 331,437-440,451,503-504,592
Think-aloud protocol, 385
Thinking, 188
Tacit knowledge, 108,111 evaluatively, 187-190
Tape recorders, 307-309,380-384, 414-415 exercises, 188
Team ethnography, 83,195, 269,275, 346,384, 400 Thomas's Theorem, 96
Technology of fieldwork, 307-309 Time frame of questions, 351-353
Technology for Literacy Center, 14-17 Tolerance for ambiguity, 44,242,315, 320,328, 330,
Terminology, 76,110, 222, 291, 311-312,570 437,588
analysis, 453,454-455, 574-576 Trade-offs, 223,225-230, 275-276, 401,594
autoethnographic, 85 Training, 355, 399,496, 513,566, 567
case studies, 195,198, 298 Transferability, 584
cross-cultural, 392-393,454-455, 457-458 Transaction models, 171-172
emic, etic, 267,331,363,454 Transcribing, 380-384,415, 440,441
evaluation, 272,311-312 doing your own, 441
fieldwork, 262 ti ps, 382
generalizations, 584 validating, 561
grounded theory, 490 Treatments, 54,161-162,164,336-337
heuristic inquiry, 486-487 compliance, 163
indigenous, 288-290,454-455, 457-458,507-508 culture as, 83
logic model, 162-163 Triangulation, 93, 247-248,249,254, 261,306,321,331,
participatory, 184-185, 269 504,544,546,555-566
phenomenological, 482-486 and reflexivity, 66, 495,543
political language, 188 True to the data, 58
politics of, 570 Trust, 44, 207,310,312,314,316,318,324, 325,331
research purposes, 213, 223,224 distrust, 325
See also Sensitizing concepts Trustworthiness, 51,544,546, 570,575-576
Tests, standardized, 147,150,158,191-192,193, 548 through voice, 65, 494-495,546,548
Text analysis, 114,580-581 Truth, 96,98, 268, 270, 309,541-542,577-581
See also Hermeneutics; Narrative analysis grounded theory and, 128
Themes, finding, 235,297, 323,452-471, 485-487 heuristic inquiry and, 108
See also Analysis; Documenter's perspective of a text, 116
Themes of qualitative inquiry, 39-71, 248 multiple, 575
alternative, 542-552 phenomenological, 106
practical choices, 66-68 postmodernism and, 99-100, 579
pur and mixed, 252 pragmatic test of, 147
summary table, 40-41 table analysis, 492
Theoretical distinctions, 135,542, 544-545 tests, 550,578
Theoretical orientations, 75-137,542, 543 Truth-oriented inquiry, 91-96,190, 543,544-545
Theoretical traditions summary, 132-133 See also Truth
Theory and qualitative inquiry, 125,194,493-494, 544 Tuskegee experiment, 270-271
basic research, 215, 544-546 Types of qualitative data, 4
See also Qualitative and quantitative Valid belief, 93

Typologies, 457-462,468-473, 507-508 Validity, 14,53,93,114,312,337,383,400, 433, 544, 545
Typology of research purposes, 213,224 of alternative frameworks, 135, 542-552
of claims, 587-588
of small samples, 245-246
Understanding, 49,51,262 hermeneutic, 114
breakthrough, 332 instrument, 192
case-based, 478-479 See also Credibility; Triangulation
constructivist, 544,546 Value-free science, 50,93,569
cross-cultural, 291,311,391-394, 493 Value orienta tions:
hermeneutic, 114,497-498 harmonizing, 176-179
observation-based, 171 Valuing, 147,171,173
phenomenological, 486-487 Verifica tion, 67, 323, 324
prevention, 167 Verstehen, 52-53, 69,467,544,546
See also Verstehen Videotape, 203,281, 308,415
self-understanding, 64 Vietnam Memorial, 292-293
Unintended impacts, 152,169,263,288 Virtual ethnography, 83
Union Institute, ii Vision problem, 202-203
Unique case orientation, 41, 55,57,106,148,172,252, Voice, 6, 35,41, 63-66, 88-89,91,109,494-495, 545, 548,
297,438, 447,449, 450,492, 544 571
particularization, 480,544, 582 and justice, 98, 495
United Way, 151 giving, 98,101,495
Units of activity, 285 minimizing, 93
Units of analysis, 46,55, 228-230, 231,254, 297-298, owning, 65-66, 88,494-495
300,447,448,449 to voiceless, 101,187, 389
comparable, 493
See also Sampling
Unobtrusive observations, 42,191-192,246,264, "War" stories, 241
291-293 Watching and wondering, 332
Utility tests, 550,577-580 Whorf hypothesis, 55,288-289
Utilization-focused evaluation, 10, 68,78,173-175, Why? questions, 363-365,366, 438
202,212,464,508,550 case studies, 478-479
exemplar, 212,509-510 Williams, Brackette:
intended use by intended users, 173,189,508, interview, 44-45, 46,47, 54, 58,73,240
561-562 Women's ways of knowing, 5-7,129,433
process use, 180,189-190, 220,327, 398
reporting, 513
Umpires, 543 Yanomami Indians, 273,326-327
yWjoui tk<s Au+Ko^
Michael Quinn Patton lives in Minnesota uation, Practical Evaluation, How to Use Qual-
where, according to the stated poet laure- itative Methods in Evaluation, and Family Sex-
ate, Garrison Keillor, "ali the women are ual Abuse: Frontline Research and Evaluation.
strong, ali the men are good-looking, and He edited Culture and Evaluation for the
ali the children are above average/' It was journalNei Direction in Program Evaluation.
this interesting lack of statistical variation His creative nonfiction book, Grand Canyon
in Minnesota that led him to qualitative in- Celebration: A Father-Son Journey ofDiscov-
quiry despite the strong quantitative orien- ery, was a finalist for 1999 Minnesota Book
tation of his doctoral studies in sociology at of the Year.
the University of Wisconsin. He serves on He is former President of the American
the graduate faculty of The Union Institute, Evaluation Association and the only recipi-
a nontraditional, interdisciplinary, nonresi- ent of both the Alva and Gunnar Myrdal
dential, and individually designed doctoral Award for Outstanding Contributions to
program. He was on the faculty of the Uni- Useful and Practical Evaluation from the
versity of Minnesota for 18 years, including Evaluation Research Socie ty and the Paul F.
5 years as Director of the Minnesota Center Lazarsfeld Award for Lifelong Contribu-
for Social Research, where he was awarded tions to Evaluation Theory from the Ameri-
the Morse-Amoco Award for innovative can Evaluation Association. The Society for
teaching. Readers of this book will not be Applied Sociology awarded him the 2001
surprised to learn that he has also won the Lester F. Ward Award for Outstanding Con-
University of Minnesota storytelling com- tributions to Applied Sociology
petition.
He has authored five other Sage books: Halcolm made his debut in the first edition
Utilization-Focused Evaluation, Creative Eval- of this book (1980) as a qualitative inquiry
Ia Al
A2 S QUALITATIVE RESEARCH AND EVALUATION
muse and Sufi-Zen teaching master who of- mortal character Lazarus Long, the oldest
fered stories that probed the deeper philo- living member of the human race, who trav-
sophical underpinnings of how we come to eis through time and space offering wisdom
know what we knowor think we know. to mere mortais. Part muse and part alter
Halcolm's musin^.., like his name (pro- ego, part literary character and part schol-
nounced slowly), lead us to ponder "how arly inquirer, Halcolm's occasional appear-
come?" Halcolm was inspired by a combina- ances in this research and evaluation text re-
tion of the character Mulla Nasrudin from mind us to ponder what we think is real,
Sufi stories (Shah, 1972, 1973) and science question what we think we know, and in-
fiction writer Robert Heinlein's (1973) im- quire into how come we think we know it.
"On dissertation proposals on which I have served, Patton's is byfar the most cited qualitative
research text."
Ian Baptiste, Penn State University
This booka resource and training tool for countless applied researchers, evaluators, and
gradu^te studentshas been completely revised with hundreds of new examples and stories
illuminating ali aspects of qualitative inquiry. In this edition, Patton has created the most
comprehensive, systematic, and up-to-date review of qualitative methods available.
The Third Edition has retained and expanded upon the exhibits that highlight and summarize
major issues and guidelines, the summative sections, tables, and figures as well as the sage
advice of the Sufi-Zen master, Halcolm. This revision will help readers integrate and make sense
of the great volume of qualitative works published in the past decade.
"I am dazzled by the material that this book describes and clarifies. He has shifted the focus of
the text to qualitative inquiry in general, which includes qualitative evaluation. New examples
from his own work and that of others serve to clarify and deepen understanding of qualitative
research topics and processes. New discussion of many current issues and debates in qualitative
scholarship (autoethnography, ethical issues of informed consent and confidentiality, focus
group/group interviews, computer-assisted analysis, the complexity of creating criteria for
judging the quality of qualitative research, etc.) will bring readers up-to-date with the variety in
perspectives about (and the variety within) qualitative inquiry. Most of the chapters in the book
have been substantially reorganized in ways that augment the reader's understanding."
Corrine Glesne, author o/Becoming Qualitative Researchers
"Clearly, this is a vastly improved, much more comprehensive, cogently systematic, and timely
reviewa tour de force, one might sayof the field of qualitative research, in terms of the theoretical,
conceptual, methodological, and normative dimensions/foundations of qualitative research.
This is one of the strengths ofthe volume. It seeks to bring together theory and
practice/methods without overburdening one or the otherthis is as rare as
it is commendable, not to mention extremely useful, not only for the
professional researcher, but for the 'non-professional' as well."
Lester Edwin J . Ruiz, New York Theological Seminary
PhotoDisc Images 2 0 0 0 PhotoDisc, Inc.

Please visit our website at vvww.sagepub.com
ISBN 0-7619-1971-6
i\SAGE Publications
S International Educational and Professional Publisher
Thousand Oaks London New Deihi 9 780761 919711

Qualitative Research and Evaluation Methods PDF

Uploaded by

Copyright:

Available Formats

Qualitative Research and Evaluation Methods PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Qualitative Research and Evaluation Methods PDF

Uploaded by

Copyright:

Available Formats

rvalua+ion

Michael Quinn Patton

Michael Quinn Patton

Sage Publications, Inc.

Library of Congress Cataloging-in-Publication Data

Patton, Michael Quinn.

Acquiring Editor: C. Deborah Laughton

PjA"RTT 1 . Conceptual Issues in Qualitative Inquiry 1

1. The Nature of Qualitative Inquiry 3

4. Particularly Appropriate Qualitative Applications 143

P A R T 2 . Qualitative Designs and Data Collection 207

P j A R T 3 . Analysis, Interpretation, and Reporting 429

8. Qualitative Analysis and Interpretation (431^)

9. Enhancing the Quality and Credibility of Qualitative Analysis

About the Author Al

P . A R T 1 , Conceptual Issues in Qualitative Inquiry 1

1. The Nature of Qualitative Inquiry 3

E H 1 H Women's Ways of Knowing: An Example of

2. Strategic Themes in Qualitative Inquiry 37

B B Sample Internet E-mail Discussion Groups (listservs) and

P A R T 2 . Qualitative Designs and Data Collection 207

5. Designing Qualitative Studies 209

A Typology of Research Purposes 224

^QQQQ^IjQQI Family Research Example: Research Questions

6. Fieldwork Strategies and Observation Methods 259

^ ^ Q H Q Summary Guidelines for Fieldwork 330

P;A~RT 3 . Analysis, Interpretation, and Reporting 429

8. Qualitative Analysis and Interpretation 431

I M l i l i Internet Resources and E-mail Discussion Groups (listservs)

UJ23QE&I Case Study: Layers of Possible Analysis 448

9. Enhancing the Quality and Credibility of Qualitative Analysis 541

Subject Index 113

About the Author Al

T he story is told that at the conclusion of a rigorous course in philoso-

In doing this revision, I reviewed over a

W hile part of the task of this revi-

Psychometricians try to measure it

When in doubt, observe and ask questions.

Gigo's law of deduction: Garbage in,, garbage out.

Qualitative inquiry cultivates the most useful of ali human capacities:

Innovators are told: "Think outside the box."

From Halcolm's Inquiry Parables

Three Kinds of Qualitative Data

Recognizing Qualitative Data processes that are part of observable hu-

& Qualitative Findings: T h e m e s , Patterns,

Discovery of an early qualitative evaluation report

monalities and differences. They worked irtforming variations in ways of knowing.

WomerTs Ways of Knowing:

Received knowledge: Women conceive of themselves as capable of receiving, even reproducing

Constructedknowledge: Women view ali knowledge as contextual, experience themselves as

SOURCE: Belenky et al. (1986:15).

IjfPlIH''"1' Coming-of-Age Paradigms

SOURCE: Patton (I999a:333, 335).

UAA.'-' . ; . j : .:, .,.,V

qualitative inquiry. While doing interviews Making Methods Decisions

13. M e t h o d s Choices: Contrasting

ot everything that can be counted counts, and not everything that

Some Guiding Questions and Options for

1. What are the purposes of the inquiry?