UC Irvine: I.T. in Government

UC Irvine
I.T. in Government
Title
Survey Research Methodology in Management Information Systems: An Assessment
Permalink
https://escholarship.org/uc/item/6cs4s5f0
Authors
Pinsonneault, Alain
Kraemer, Kenneth L.
Publication Date
1993-12-01
eScholarship.org Powered by the California Digital Library

University of California
SURVEY RESEARCH METHODOLOGY IN MANAGEMENT INFORMATION SYSTEMS: AN
ASSESSMENT
Alain Pinsonneault
École des Hautes Études Commerciales
5255 Decelles Avenue
Montréal, Québec
Canada H3T 1V6
and
Kenneth L. Kraemer
Graduate School of Management
University of California
Irvine, California
U.S.A. 92717
Working Paper #URB-022
The authors would like to thank the participants of the 1991 Queen's-HEC workshop and two anonymous
reviewers and the Editor in chief of JMIS for their helpful comments on an earlier version of this paper.
This study was supported financially by École des Hautes Études Commerciales de Montréal and the
University of California at Irvine.
ABSTRACT
Survey research is believed to be well understood and applied by MIS scholars. It has been
applied for several years, it is well defined, and it has precise procedures which, when followed closely,
yield valid and easily interpretable data. Our assessment of the use of survey research in the MIS field
between 1980 and 1990 indicates that this perception is at odds with reality. Our analysis indicates that
survey methodology is often misapplied and is plagued by five important weaknesses: (1) single method
designs where multiple methods are needed, (2) unsystematic and often inadequate sampling procedures,
(3) low response rates, (4) weak linkages between units of analysis and respondents, and (5) over reliance
on cross-sectional surveys where longitudinal surveys are really needed. Our assessment also shows that
the quality of survey research varies considerably among studies of different purposes: explanatory
studies are of good quality overall, exploratory and descriptive studies are of moderate to poor quality.
This article presents a general framework for classifying and examining survey research and uses
this framework to assess, review and critique the usage of survey research conducted in the past decade
in the MIS field. In an effort to improve the quality of survey research, this article makes specific
recommendations that directly address the major problems highlighted in the review.
AUTHORS' BIOGRAPHIES
Alain Pinsonneault holds a Ph.d. in administration from University of California at Irvine (1990) and a
M.Sc. in Management Information Systems from Ecole des Hautes Etudes Commerciales de Montreal
(1986). His current research interests include the organizational implications of computing, especially
with regard to the centralization/decentralization of decision making authority and middle managers
workforce; the strategic and political uses of computing, the use of information technology to support
group decision making process; and the benefits of computing. He has published articles in Decision
Support Systems, European Journal of Operational Research, and in Management Information Systems
Quarterly, and one book chapter. He has also given numerous conferences and he is an associate editor
of Informatization and the Public Sector journal. His doctoral dissertation won the 1990 International
Center for Information Technology Doctoral Award.
Kenneth L. Kraemer is the Director of the Public Policy Research Organization and Professor of
Management and Information and Computer Science. He holds a Ph.D. from University of Southern
California. Professor Kraemer has conducted research into the management of computing in
organizations for more than 20 years. He is currently studying the diffusion of computing in Asia-Pacific
countries, the dynamics of computing development in organizations, the impacts of computing on
productivity in the work environment, and policies for successful implementation of computer-based
information systems. In addition, Professor Kraemer is coeditor of a series of books entitled Computing,
Organization, Policy, and Society (CORPS) published by Columbia University Press. He has published
numerous books on computing, the most recent of which being Managing Information Systems. He has
served as a consultant to the Department of Housing and Urban Development, the Office of Technology
Assessment and the United Nations, and as a national expert to the Organization for Economic
Cooperation and Development. He was recently Shaw Professor in Information Systems and Computer
Sciences at the National University of Singapore.
SURVEY RESEARCH METHODOLOGY IN MANAGEMENT INFORMATION SYSTEMS: AN
ASSESSMENT
INTRODUCTION
Science may be said to progress on its methods. The production of knowledge depends very
much on the techniques for collecting, analyzing, and interpreting data and on the way they are applied
[89]. The same may be said of management information systems (MIS). The academic study of MIS
relies very much on the methods used to answer research questions and test research hypotheses, and on
the careful application of these methods. Moreover, since the methods are borrowed for the most part
from established disciplines, the issue of appropriate and skilful application becomes key. And this is
especially the case in survey research where the basic methods have been known since the fifties, but
where the application in many fields continues to fall short of the theoretical ideal. This review of 122
survey-based studies in MIS indicates that survey research in MIS suffers from the same problems that
plague survey research generally: (1) single method designs where multiple methods are needed, (2)
unsystematic and often inadequate sampling procedures, (3) low response rates, (4) weak linkages
between units of analysis and respondents, and (5) over reliance on cross-sectional surveys where
longitudinal surveys are really needed. On the one hand, this is reassuring in that one would expect a
new field to have difficulties at first in adopting and applying methods developed in other fields for its
own problems. On the other hand, it is disappointing, especially when one considers the extent to which
0
survey research is used and the proportion of survey-based studies in MIS that fail to measure up. The
key problem revealed by this article is weaknesses in application of survey methodology, not
inappropriate technical knowledge concerning the methodology.
Assessment of survey research methodology might be done from any of three different
perspectives:
(1) Developing insights into appropriate research methodologies: establish appropriate usage of
different methodologies [7, 36; 60, 61, 70, 74, 76, 79, 95 97].
(2) Examining the quality of existing research methodologies: assess the strengths and weaknesses
of different methodologies as they apply in the MIS field [17, 18, 81, 90, 98].
0
. Teng and Galletta [92] found that almost 25% of all research projects in 1991 used the survey methodology.
Farhoomand [47] found similar results. Teng and Galletta also found that over 50% of the 1503 researchers
surveyed were currently involved in survey research projects.
survey research:survey:10-2-92
(3) Identifying where research is needed: determine areas where the application of specific
methodologies would be most insightful [21, 24, 39, 62, 64, 72, 75].
This paper examines the quality of survey research methodology in MIS, and differs from three major
recent assessments in its comprehensiveness. For example, Newsted, Munro and Huff [81] inventory and
assess data collection instruments. Zmud and Boynton [98] assess survey instruments and survey
measures, whereas Straub [90] describes the evolution of data analysis methods in MIS. This article
focuses on the broader elements of survey research in that it analyzes research design, sampling
procedures, and data collection. The first section describes the database and method used to examine
survey research in MIS, whereas the second section presents the findings of our assessment and the third
section discusses the findings. The last section summarizes the results and recommendations that were
maid throughout this article.
DEFINITION OF SURVEY RESEARCH

There is an important distinction between surveys and survey research. A survey is a means of
"gathering information about the characteristics, actions, or opinions of a large group of people, referred
to as a population" [91]. As such, there are many data collection and measurement processes that are
called surveys--marketing surveys, opinion surveys, and political polls to name some of the most
common. This paper is not about such surveys. Rather, it focuses on surveys that are conducted to
advance scientific knowledge, which we refer to as survey research.
Characteristics of Survey Research

Surveys conducted for research purposes have three distinct characteristics. First, the purpose of
survey is to produce quantitative descriptions of some aspects of the study population. Survey analysis
may be primarily concerned either with relationships between variables, or with projecting findings
descriptively to a predefined population [56]. Survey research is a quantitative method, requiring
standardized information from and/or about the subjects being studied. The subjects studied might be
individuals, groups, organizations or communities; they also might be projects, applications, or systems.
Second, the main way of collecting information is by asking people structured and predefined
questions. Their answers, which might refer to themselves or some other unit of analysis, constitute the
data to be analyzed.
Third, information is generally collected about only a fraction of the study population--a sample--
but it is collected in such a way as to be able to generalize the findings to the population--like service or
manufacturing organizations, line or staff work groups, MIS departments, or various users of information
systems such as managers, professional workers, and clerical workers. Usually, the sample is large
enough to allow extensive statistical analyses.
Appropriate Application of Survey Research in MIS

The nature of survey research can be best understood by comparing it to two other dominant
methods in MIS: case studies and laboratory experiments.
Case studies involve examination of a phenomenon in its natural setting. The researcher has no
control over the phenomenon, but can control the scope and time of the examination. The researcher
may or may not have clearly defined independent and dependent variables. Case studies are most
appropriate when the researcher is interested in the relation between context and the phenomenon of
0
interest.
Laboratory experiments involve examination of a phenomenon in a controlled setting. The
researcher manipulates the independent variables and observes their effects on the dependent variables.
The researcher has direct control over the laboratory conditions and manipulation of the independent
variables, but the researcher can only study phenomena in the present. Laboratory experiments are
especially well-suited to research projects involving relatively limited and well-defined concepts and
propositions that involve individuals or small groups.
In contrast to these two methods, survey research involves examination of a phenomenon in a
wide variety of natural settings. The researcher has very clearly defined independent and dependent
variables and a specific model of the expected relationships which is tested against observations of the
phenomenon. Survey research is most appropriate when:
(a) the central questions of interest about the phenomena are "what is happening?", and "how and
why is it happening?" Survey research is especially well-suited for answering questions about
what, how much and how many, and to a greater extent than is commonly understood, questions
about how and why.
0
. Benbasat et al. [18] indicate that case research is particularly appropriate in two situations: (a) where research
and theory are at their early, formative stages, and (b) where the experiences of the actors are important and the
context of action is critical.
(b) Control of the independent and dependent variables is not possible or not desirable.
(c) The phenomena of interest must be studied in its natural setting.
(d) The phenomena of interest occur in current time or the recent past.
On the other hand, surveys are less appropriate than other methods such as case studies and naturalistic
observation when detailed understanding of context and history of given computing phenomena is
desired.
DATA AND METHOD

The database for this article consists of 141 published articles using survey research which were
culled from major MIS journals between 1980 and 1990. Table 1 distributes the 141 articles by
00
journal. ,
Table 1. Survey articles by journal
Journals Number of Survey Articles

Academy of Management Journal 1 (1%)
Communications of the ACM 18 (13%)
Data Base 15 (11%)
Datamation 2 (1%)
Data Management 1 (1%)
Decision Sciences 2 (1%)
Health, Marketing, and Consumer Behavior 1 (1%)
Information Age 1 (1%)
Information and Management 34 (24%)
Information Processing Management 1 (1%)
Journal of Management Information Systems 12 (9%)
Management Information Systems Quarterly 44 (31%)
Management Science 6 (4%)
Microprocessing and Microprogramming 1 (1%)
Product Inventory Management 1 (1%)
Public Adminstration Review 1 (1%)
Total 141 (100%)
0
. Baroudi and Orlikowski [17], Davis [37], Doll and Torkzadeh [43], Galletta and Lederer [54], Joshzi [66],
Raymond [83], and Torkzadeh [93] are instrument validation studies and are not included in this analysis because the
focus of the present article is on usage and application of survey research as a method to generate knowledge on
particular phenomena.
0
. Another stream is made of surveys about the MIS field itself, like Brancheau and Wethere [24] description of IS
issues and Frand and McLean [51] survey of Business Schools computer use. The other studies of the MIS field are
Alavi [3], Amoroso, Thompson, and Cheney [5], Ball and Harris [11], Barki, Rivard, and Talbot [13], Cheney and
Lyons [28] Couger [29], Dickson and Nechis [40], Frand and Britt [50], Frand, McLean, and Britt [52], Guimaraes
and Ramanujam [58], Gupta and Seeborg [59], Hartog and Herbert [62], Hough and Duffy [63], Kaplan and Dickson
[68], King and Premkumar [69], and Merten and Severance [77]. These studies are not included in the present
analysis.
The list of MIS journals was taken from those commonly included in studies of research and publications
in the field [97, 33, 34]0. When specific individuals recurred in the database, a computer search on the
individuals was done in an attempt to find the full body of publications connected with a particular
survey effort. This sometimes introduced additional journals into the search (Data Management; Health,
Marketing, and Consumer Behavior; Information Age; Information Processing Management;
Microprocessing and Microprogramming; Product Inventory Management), but only for those
individuals and studies.
Each journal was reviewed by the authors, starting with the table of contents, but extending to
the abstracts and to the articles themselves. These reviews produced 133 articles. Eight other articles
were obtained through computer searches on individuals. After examination, these 141 articles were
grouped into 122 studies because some articles are based on the same survey project and research
method. In order to obtain the most accurate assessment of the surveys from which several articles were
published, all related materials were used to describe and analyze the particular research effort. The
method used to assess survey research involves three discrete steps: (1) classification of the studies by
purpose, (2) development of a framework for assessment, and (3) actual assessment of the studies.
Classification of Studies by Purpose

Survey research can be used for exploration, description, or explanation purposes. The use of
survey research for these purposes is different, however, from the use of case studies or experiments for
such purposes.
The purpose of survey research in exploration is to become more familiar with a topic and to try
out preliminary concepts about it. A survey in this context is used to discover the range of responses
likely to occur in some population of interest (end users, IS managers, Fortune 500 companies, etc.) and
to refine the measurement of concepts. The exploratory survey focuses on determining what concepts to
measure and how to measure them best. The exploratory survey also is used to discover and raise new
0
. Additional journals that were searched but for which no survey research were identified include: Accounting
Review; ACM Computing Surveys; Harvard Business Review; Sloan Management Review; Systems, Objectives, and
Solutions; and Transactions on Database Systems. As part of our continuing research in this area, we are adding new
journals such as Information Systems Research, Organizational Science, and Informatization in the Public Sector to
the database.
possibilities and dimensions of the population of interest. For example, Rockart and Flannery [86] did a
survey to help define the various types of end-user computing that were developing in organizations.
The use of survey research for exploration as an end in itself is almost never warranted.
Exploratory surveys should be used as the basis for developing concepts and methods for more detailed,
systematic descriptive or explanatory surveys [8, 41, 49]. In short, the whole purpose of an exploratory
survey is to elicit a wide variety of responses from individuals with varying viewpoints in a loosely
0
structured manner as the basis for design of a more careful survey.
The purpose of survey research in description is to find out what situations, events, attitudes or
opinions are occurring in a population. Survey research aimed at description asks simply about the
distribution of some phenomena in a population or among subgroups of a population. The researcher's
concern is simply to describe a distribution or to make comparisons between distributions. Analysis
stimulated by descriptive questions is meant to ascertain facts, not to test theory. The hypothesis is not
causal, but simply that common perceptions of the facts are or are not at odds with reality. For example,
it might examine what kind of people use computers in an organization [35], or what kind of people work
at home [94], or what kinds of applications people use at work [70].
The purpose of survey research in explanation is to test theory and causal relations. Survey
research aimed at explanation asks about the relationships between variables. It does so from
theoretically grounded expectations about how and why the variables ought to be related. The theory
includes an element of cause and effect in that it not only assumes that relations exist between the
variables, but assumes directionality (e.g., that the relationship is positive or negative, or that variable A
influences variable B). Explanatory questions may extend not only to establishing the existence of a
causal relationship but also to asking why the relationship exists. The central research question in
explanatory survey research is: "Does the hypothesized causal relationship exist, and does it exist for the
reasons posited?" An example of explanatory survey research is the Baroudi, Olson, and Ives [16] study
of whether or not user participation in systems development projects affects usage and satisfaction, and
why.
Development of Framework for Assessment
0
. Exploratory surveys are different from pilot studies. The pilot study is a small scale rehearsal of a systematic
survey aimed at testing questions, question flow, and questionnaire format with representatives of the target
population. Exploratory surveys frequently are used prior to pilot studies to determine what concepts should be
measured and how to measure them best.
Dillman [41] and Fowler [49] indicate that there are three key elements in the conduct of
surveys, and that these can be used to assess the quality of survey research. These elements include: (a)
research design, (b) sampling procedures, and (c) data collection methods. These elements, and their
related dimensions, constitute the framework used to assess survey research methodology in MIS. Table
2 presents the minimum dimensions that a study must meet for each element. Each of the elements and
selected dimensions are described next.
Research Design. A research design is the strategy for answering the questions or testing the
hypotheses that stimulated the research in the first place. Survey designs may be distinguished as cross
sectional or longitudinal, depending upon whether they exclude or include explicit attention to the time
dimension. The classic cross-sectional design collects data at one point in time from a sample selected to
represent the population of interest at that time. One can generalize safely the findings from the sample
to the population at the point in time the survey was conducted. Cross-sectional designs limit causal
inferences because the study is conducted at one point in time and temporal priority is difficult to
establish.
The classic longitudinal design collects data for at least two points in time. The underlying
principle of longitudinal designs, like that of the "one-group pretest-postest design" described by
Campbell and Stanley [26], is to measure some dimensions of interest of a given entity before and after
an intervening phenomenon to determine whether or not the phenomenon has some effects. In MIS, the
intervening variable is usually the implementation and usage of computing. The dimensions studied
range from social interaction, to organizational structure, to communication, to decision making, and to
work effects, among others. Longitudinal designs provide greater confidence for causal inferences than
cross-sectional designs because they establish temporal priority more easily.
Table 2. Minimum Dimensions of Survey Studies by Purpose
Element/Dimension Exploration Description Explanation
Research Design
Survey type Cross-sectional Cross-sectional Cross-sectional and
longitudinal
Mix of research Multiple methods Not necessary Multiple methods
methods
Unit(s) of analysis Clearly defined Clearly defined & Clearly defined &
appropriate for the appropriate for the
questions/hypotheses research hypotheses
Respondents Representative of the unit Representative of the unit Representative of the unit
of analysis of analysis of analysis
Research hypotheses Not necessary Questions or hypotheses Hypotheses clearly stated
clearly stated
Design for data Not necessary Inclusion of antecedent Inclusion of antecedent
analysis variables and time order variables and time order
of data of data
Sampling
Procedures
Representativeness Approximation Explicit, logical Explicit, logical
of sample frame argument; reasonable argument; reasonable
choice among alternatives choice among alternatives
Representativeness Not a criterion Systematic, purposive, Systematic, purposive,
of the sample random selection random selection
Sample size Sufficient to include the Sufficient to represent the Sufficient to test
range of the phenomena population of interest & categories in theoretical
of interest perform statistical tests framework with statistical
power
Data Collection
Pretest of With subsample of sample With subsample of sample With subsample of sample
questionnaires
Response rate No minimum 60-70% of targeted 60-70% of targeted
population* population*
Mix of data Multiple methods Not necessary Multiple methods
collection
methods
* Babbie [8], Dillman [41]
Another critical issue in research design is determining the unit(s) of analysis--or the unit about
which statements are being made. It may be an individual, group, department or organization.
Alternatively, it may be an application, system, or application portfolio; or it may be a development
project, or any of the phases of a development project. The point is that the unit of analysis can be
anything the researcher decides as long as the unit chosen relates to the questions and hypotheses in the
research. There also may be more than one unit of analysis in a survey, such as the individual, work
group, and organization.
A final issue is design for data analysis. When exploration or description is the aim of survey
research, analysis frequently involves no more than developing the marginal and cross-tabulations for the
variables and using simple descriptive statistics such as means and medians. Thus there are no design
issues. When explanation is the aim, analysis must employ the full logic of survey analysis [87]. That
logic is illustrated by testing hypotheses with cross-sectional data. The data produced by a survey
comprise the answers to questions which respondents of the survey have been asked, or which have been
collected through secondary sources, or both. These questions may all refer to one point in time, but
more typically, they refer to several different points in time (present, immediate past, distant past, future).
The logic of survey analysis is based on the assumption that the time order of data can be established, or
reasonably inferred.
The use of cross-sectional survey data to test causal hypotheses requires that the investigator
design the survey to include data on the independent and dependent variables and on such antecedent
variables as theory would suggest might explain the expected original relation. Analysis, then, involves
introducing these antecedent variables into the two-variable (or more) relation to test the null hypothesis.
Testing causal relationships with cross-sectional designs in this manner is only possible when very
specific factual data that can be correctly remembered by informants are used.
Sampling procedures. Sampling is concerned with drawing individuals or entities in a population
in such a way as to permit generalization about the phenomena of interest from the sample to the
population. The most critical element of the sampling procedures is the choice of the sample frame
which constitutes a representative subset of the population from which the sample is drawn. The sample
frame must adequately represent the unit of analysis. For example, the Vitalari and Venkatesh [94] study
of computing in the home has the household as the unit of analysis, and draws the sample from a list of
people who had bought computers on credit. The logic was that households that bought computers for
home use (versus office use) would likely buy them on personal credit because the average cost was
around $3,000.00. While that might not be an adequate sampling frame for home computer users today,
it was appropriate for 1983 when computers were first being adopted for home use.
Sampling is also concerned with representativeness in selection of individual respondents from
the sample frame. One aspect of representativeness in the home computing study concerns giving each
potential respondent an equal chance of being included in the sample. This requires random selection of
households from the sample frame. Another aspect of representativeness concerns selecting a specific
respondent from each household. In the household study, this requires purposive choice of the adult who
uses the home computer the most. As can be seen by this discussion, these sampling issues involve
judgment rather than simple application of technique.
Data Collection. Regardless of the unit of analysis, the units for data collection in survey
research are usually individuals.0 Individual responses are often aggregated for larger units of analysis
such as role, work group, department, or organization. Depending upon the nature of the research, it may
be sufficient to have a single individual as respondent for each of these units of analysis. More often,
however, it is necessary to have several individuals as respondents because people function in different
roles and at different levels of the hierarchy and, consequently, have differing experiences and
perceptions of the technology and its impacts in the organization. For example, when studying the
impact of computing on the work environment of organizational employees, it would be insufficient to
have managers as the only respondents. One would also need to survey operational employees and staff
personnel to obtain an appropriate understanding of the phenomenon studied. However, it would be
sufficient to have managers as respondents about the impact of computing on their work environment.
Therefore, it is not only important to determine exactly what is the unit of analysis, but also who will be
the respondents representing the unit of analysis of interest. Once this is determined, most sampling
issues are straightforward.
The choice of data collection method, such as mail questionnaire, telephone interviews, or face-
to-face interviews, is significant because it affects the quality and cost of the data collected. For
example, mail questionnaires are very good for gathering factual data, but they are less effective when
sensitive data and complex data are needed. In general, quality and cost are highest with face-to-face
interviews or telephone interviews whereas quality and cost are lower with mail questionnaires and group
administration.
Another important aspect of data collection is whether multiple methods are used. This is
particularly important because each data collection method is limited on what it can measure effectively.
Using multiple methods permits one to have more complete data on the phenomenon of interest and a
broader and richer understanding. The quality of data is also enhanced because triangulation is possible.
0
. When using administrative records as the source of data, however, the units for data collection might be
applications or development projects rather than individuals.
Assessment of the Studies
Once characterized according to purpose by the authors, each study was reviewed by the authors
and described in terms of foregoing three elements and their associated dimensions as shown in Table 1.
For about two-thirds of the articles, it was sufficient to review the articles themselves. The other third
required review of additional work as well for one of three reasons: (1) the description of methods in the
article is minimal, (2) the article refers to other similar work for more detail about method, or (3) the
article is part of a larger study and it is difficult to tell whether it is representative of the larger study.
This additional review creates some asymmetry in our treatment of MIS research, but we felt it was better
to treat projects fully and completely than to rely only on a few published articles and assume they
represent the total survey effort, when in fact they probably do not.
FINDINGS ABOUT THE QUALITY OF SURVEY RESEARCH IN MIS

Figure 1 illustrates the evolution of surveys from 1980 to 1990 by research purpose using the 122
different studies in the database. The figure clearly indicates that there has been a constant growth in
usage of survey research in MIS in the past decade (except for 1984 and 1987). The use of survey
research in any one year has grown from 3 surveys in 1980 to 30 in 1989, an increase of 900%. Also,
there has been a shift in the predominant purpose for which survey research has been used in the past
decade. While exploratory and descriptive studies constitute the great majority of surveys before 1985--
averaging 70%--they represent only 54% of the studies after 1985, a proportionate reduction of 23%.
Conversely, explanatory studies constitute only 30% of all studies between 1980 and 1984, but they
represent 46% of the studies after 1984, a proportionate growth of 53%.
------------------------------------
Insert Figure 1 about here
------------------------------------
If survey research is reflective of the state of knowledge generation and cumulation in MIS, as
we expect it is, this finding suggests there is a trend away from theory generation (exploration,
description) toward theory testing (explanatory). This suggests that there might be some cumulation of
knowledge from the various surveys in MIS in the past decade. Before 1985, major research efforts were
made to understand what computing-related phenomena occurred and how much occurred. After 1985,
the focus of research changed to address the question of how and why the phenomena occur. This
finding, suggesting cumulation of knowledge and the maturation of the MIS field, converges with similar
findings by Culnan [32, 32], Culnan and Swanson [34].
Tables 3, 4, and 5 present the characteristics of a sample of studies classified according to the
purposes of exploration, description, and explanation respectively (see appendix A for the list of articles
0
for each purpose). The findings of our assessment are generally reported by purpose except when there
is no significant difference between surveys of the three purposes. In order to get most benefits from this
analysis, we focus on the weakest aspects of survey research in MIS, that is, on those aspects which, if
improved, would provide the greatest overall improvement in the quality of survey research in the field.
------------------------------------------------
Insert Tables 3, 4, and 5 about here
------------------------------------------------
Research Design
Survey type. Table 6 presents the distribution of studies according to their type of survey as
cross-sectional or longitudinal.
Table 6. Distribution of Studies by Survey Type and Purpose
Survey Type Exploration Description Explanation Total
No. (percent) No. (percent) No. (percent) No. (percent)
Cross-sectional 35 (100%) 31 (100%) 53 (95%) 119 (98%)
Longitudinal 0 (0%) 0 (0%) 3 (5%) 3 (2%)
Total 35 (100%) 31 (100%) 56 (100%) 122 (100%)
(%) = Percent of number of studies by purpose.
Table 6 indicates that there are very few longitudinal studies (3 out of 122 studies or 2%). This is an
obvious lack in the field. Longitudinal studies are needed because most phenomena studied in MIS are
dynamic in nature, evolve over time, and produce effects that can best be observed over time. Questions
about impact, policy effects, development and implementation of computing all involve a time
dimension. They require measurement over time, and sometimes over long periods of time. For example,
several cross-sectional studies of information technology impacts suffer from the fact that the impacts
observed at introduction of a new system are different from those at routine use, and different still from
those resulting from long-term, sustained use.
0
. Because of space limitation and of the dimension of the complete tables (25 pages total), only a samples of each
table are presented in this article. The tables are available upon request to Alain Pinsonneault.
Mix of research methods. Our data base indicates that only eight studies of the 122 (or 7%) use
multiple research methods. Two studies use quasi-experiments to supplement survey research [53, 88],
and six studies use the case method [57, 65, 67, 80, 84, 85].
The use of multiple research methods is especially important in exploratory and explanatory
studies. When knowledge is crude and limited (exploratory surveys), multiple methods permit a wider
and more complete understanding of the phenomenon studied. Because each method provides a partial
perspective on reality, multiple research methods increase the validity of the data and findings. Yet, only
three exploratory surveys (5%) use other research methods in conjunction with surveys [65, 57, 85]. In
explanation studies, the use of multiple research methods is important to validate data and to provide
extensive tests of the causal model. Here again, only three explanatory studies (7%) use multiple
research methods [52, 86, 84, 88].
Units of analysis. Table 7 presents studies grouped by unit of analysis and purpose. It indicates
that most surveys use individuals (40%) or organizations (37%) as the unit of analysis, and in about equal
proportions by purpose. The work group and department are seldom used as units of analysis, accounting
for only four percent of the surveys overall.
Table 7. Studies by Units of Analysis and Purpose
Unit of Exploration Description Explanation Total
analysis* No. (percent) No. (percent) No. (percent) No. (percent)
Individual 13 (38%) 12 (40%) 24** (43%) 48 (40%)
Work group 0 (0%) 0 (0%) 1 (2%) 1 (1%)
Department 2 (6%) 0 (0%) 1 (2%) 3 (3%)
Organization 13 (38%) 14 (47%) 19 *** (34%) 44 (37%)
Application 4 (12%) 3 (10%) 5 (9%) 12 (10%)
Project 1 (3%) 1 (3%) 2 (4%) 4 (3%)
Not described 1 (3%) 0 (0%) 4 (7%) 5 (4%)
Total 34 (100%) 30 (100%) 56 (100%) 120 (100%)
* One survey that focused on methods (descriptive) as the unit of analysis, and another that focused on issues
(exploratory) are not included in the table.
** Includes one each that had the individual and role and individual and job as the units of analysis.
*** Includes one survey were the organizations were state governments.
Table 7 also shows that applications are used about three times as often as projects as the unit of
analysis, but taken together they still represent only fourteen percent of all the surveys. Very few of the
surveys involve more than one unit of analysis, but those that do involve the individual and either the
role, job or organization. Only a half dozen of the surveys involve multiple units of analysis.
Respondents. Table 8 presents the organization-level studies grouped by whether they use
single-role respondents or multiple-role respondents.
Table 8. Organization Level Studies by Single-Role vs. Multiple-Role Respondents and Purpose
Respondents Exploration Description Explanation Total
Single-Role
Respondent
User 3 (23%) 0 (0%) 0 (0%) 3 (7%)
Manager 2 (15%) 4 (40%) 4 (29%) 10 (24%)
IS professional 5 (38%) 0 (0%) 1 (7%) 6 (14%)
IS manager 2 (15%) 5 (50%) 3 (21%) 8 (19%)
Subtotal 12 (92%) 9 (90%) 8 (57%) 27 (64%)
Multiple-Role
Respondents
User & 0 (0%) 0 (0%) 2 (4%) 2 (5%)
manager
User & IS 0 (0%) 0 (0%) 1 (7%) 1 (2%)
professional
User & IS 0 (0%) 1 (10%) 2 (14%) 10 (24%)
manager
IS professional 0 (0%) 0 (0%) 0 (0%) 1 (2%)
& IS manager
Others 1 (8%) 0 (0%) 0 (0%) 1 (2%)
Subtotal 0 (0%) 1 (10%) 6 (43%) 15 (35%)
Total 13 (100%) 10 (100%) 14 (100%) 44 (100%)
Not described 2 (13%) 3 (23%) 4 (29%) 9 (18%)
As Table 8 shows, the respondents are generally representative of the unit of analysis when studies focus
on a small group of individuals (e.g. development group, application user, department). However, the
respondents are less representative in studies focusing on organizations as the unit of analysis. As shown
in Table 8, of the 44 studies using the organization as their unit of analysis, only 14 (32%) use multiple
respondents from multiple groups like end-users, line managers, DP managers, and systems analysts.
This fact is even more striking when the surveys are examined by purpose. Table 8 indicates that over
90% of the exploratory and descriptive surveys use single-role respondents in organization level studies,
whereas 57% of the explanatory studies use single-role respondents and 43% use multiple-role
respondents.
Research hypotheses. Our data base indicates that more than half of all studies (52% or 64
studies) either do not provide research hypotheses/questions or do not describe them clearly enough to
get an understanding of the study's aim. This is problematic because research hypotheses or questions
shape the sampling procedures, data collection, and data analysis. If there are no questions or
hypotheses, or if they are poorly formulated, it is unlikely that the survey effort will yield useful results
except by accident.
Sampling Procedures
Representativeness of sample frame. The selection of the sample frame--the population segment
from which a sample is drawn and which is taken as a surrogate for the real population of interest--is
seldom discussed in the articles even though it is often more important than selection of the sample itself.
Table 9 presents the distribution of surveys among the most common sample frames grouped by research
purposes. Table 9 indicates that almost one half of the studies do not report or describe the sample frame
used. Only 37% of exploratory studies and 48% of explanatory studies describe their sample frames.
Descriptive studies are much better in this respect, having 70% of the studies describing their sample
frame.
Table 9. Studies by Sample Frame and Purpose

Sample frames Exploration Description Explanation Total
Population (census) 2 (6%) 1 (3%) 6 (11%) 9 (7%)
Clients 2 (6%) 2 (6%) 1 (2%) 5 (4%)
Members of an
association or 7 (20%) 15 (48%) 14 (25%) 36 (30%)
subscribers to a
magazine
Attendees of a 2 (6%) 1 (3%) 1 (2%) 4 (3%)
conference or seminar
Geographic region 0 (0%) 3 (10%) 5 (9%) 8 (7%)
Not reported 22 (63%) 9 (29%) 29 (52%) 60 (49%)
Total 35 (100%) 31 (100%) 56 (100%) 122 (100%)
To the extent they are even discussed, sample frame issues tend to be convenience sample frames rather
than ones drawn for substantive or theoretical reasons. Most convenience sample frames are drawn from
local members of an association like the Society for Information Management or the Data Processing
Management Association, sponsors of an MIS research center, customers of a vendor, or readers of a
computer magazine. Only 7% of the studies actually use the population of interest to draw their samples.
Some articles point out that samples were selected to include different industries, but do not explain why
this is important to the study. Very few engage in lucid discussion such as that in Attewell and Rule [7]
about sample selection from "establishments" rather than "firms" when studying organizational impacts
of the information technology.
Representativeness of the sample. Table 10 present organization-level surveys grouped by the
procedures used to draw the sample from the sample frame and by purposes.
Table 10. Studies by Sampling Procedures and Purpose (Organization-level Surveys)

Sampling Exploration Description Explanation Total
procedures No. (percent) No. (percent) No. (percent) No. (percent)
Systematic
Random {0 (0%)} {2 (15%)} {4 (25%} {6 (15%)}
(1 (8%)) (1 (8%)) (2 (13%)) (4 (10%))
Stratified {0 (0%)} {0 (0%)} {1 (6%)} {1 (2%)}
(0 (0%)) (0 (0%)) (0 (0%)) (0 (0%))
Purposive {3 (25%)} {3 (23%)} {0 (0%)} {6 (15%)}
(3 (25%)) (3 (23%)) (6 (38%)) (12 (29%))
Unsystematic
Convenience {1 (8%)} {1 (8%)} {3 (19%)} {5 (12%)}
(0 (0%)) (1 (8%)) (0 (0%)) (1 (2%))
Snowball {0 (0%)} {0 (0%)} {0 (0%)} {0 (0%)}
(0 (0%)) (1 (8%)) (3 (19%)) (4 (10%))
Not described {8 (67%)} {7 (54%)} {8 (50%)} {23 (56%)}
(8 (67%)) (7 (54%)) (5 (31%)) (20 (49%))
Total {12 (100%)} {13 (100%)} {16 (100%)} {41 (100%)}
(12 (100%)) (13 (100%)) (16 (100%)) (41 (100%))
{ }: Selection of organizations
( ): Selection of respondents
Table 10 indicates that almost 70% of the surveys selected the organization using unsystematic sampling
procedures and 71% of the studies sampled the respondents within organizations using unsystematic
procedures. The use of unsystematic sampling procedures varies across purpose. Over 80% of
exploratory and descriptive studies use unsystematic sampling procedures. It is a very significant
weakness in descriptive studies because their usefulness lies in the generality of their findings, and
unsystematic sampling procedures greatly hinder this capacity. The very high usage of unsystematic
sampling procedures in exploratory studies is less significant because exploratory studies are not
intended to be generalizable to a population.
Table 11 presents the individual-level surveys by sampling procedures grouped by purpose.
Table 11. Studies by Sampling Procedures and Purpose (Individual-level surveys)

Sampling Exploration Description Explanation Total
procedures No. (percent) No. (percent) No. (percent) No. (percent)
Systematic
Random {0 (0%)} {0 (0%)} {2 (7%} {2 (4%)}
(2 (14%)) (0 (0%)) (5 (19%)) (7 (13%))
Stratified {0 (0%)} {0 (0%)} {2 (7%)} {2 (4%)}
(0 (0%)) (1 (8%)) (0 (0%)) (1 (2%))
Purposive {1 (7%)} {0 (0%)} {3 (11%)} {4 (8%)}
(1 (7%)) (3 (25%)) (11 (41%)) (15 (28%))
Unsystematic
Convenience {4 (29%)} {3 (25%)} {4 (15%)} {11 (21%)}
(0 (0%)) (1 (8%)) (2 (7%)) (3 (6%))
Snowball {0 (0%)} {0 (0%)} {0 (0%)} {0 (0%)}
(2 (14%)) (2 (17%)) (2 (7%)) (6 (11%))
Not described {9 (64%)} {9 (75%)} {16 (59%)} {34 (64%)}
(9 (64%)) (5 (42%)) (7 (26%)) (21 (40%))
Total {14 (100%)} {12 (100%)} {27 (100%)} {53 (100%)}
(14 (100%)) (12 (100%)) (27 (100%)) (53 (100%))
{ }: Selection of organizations
( ): Selection of respondents
Table 11 indicates that 57% of the studies use unsystematic sampling procedures to select individuals
and almost 90% of the studies draw individuals from organizations sampled using unsystematic
procedures. Almost 80% of the exploratory studies use unsystematic sampling procedures to draw
samples of individuals. All descriptive studies draw their sample of individuals from organizations that
were chosen using unsystematic sampling procedures. This is problematic because it jeopardizes the
very essence and usefulness of descriptive studies.
It is even more troublesome to note that in most studies, the potential bias incurred by using an
unsystematic sampling procedure is not recognized or taken into account when interpreting data and
inferring findings to the population. Table 12 shows that 90% of the surveys do not report or do not test
for sample bias.
Table 12. Studies by Sample Bias Test and Purpose

Sample Bias Test Exploration Description Explanation Total
Not reported (or 33 (94%) 30 (97%) 47 (84%) 110 (90%)
tested)
Tested and no bias 2 (6%) 1 (3%) 9 (16%) 12 (10%)
Total 35 (100%) 31 (100%) 56 (100%) 122 (100%)
Sample size. Table 13 presents the surveys grouped by unit of analysis, sample size, and
purposes. It shows the size of samples in the studies when both individuals and organizations were the
units of analysis. Although applications and projects were the units of analysis in some studies, we did
not calculate the sample size for these.
Table 13. Studies by Unit of Analysis, Sample Size, and Purpose*
Sampl Exploration Description Explanation Total
e size
Indiv. Organ. Indiv. Organ. Indiv. Organ. Indiv. Organ.
No. (%) No. (%) No. (%) No. (%) No. (%) No. (%) No. (%) No. (%)
Less 2 (11%) 4 (33%) 0 (0%) 6 (43%) 4 (14%) 4 (20%) 6 (10%) 14 (20%)
than
50
51-100 4 (22%) 5 (42%) 3 (23%) 4 (29%) 6 (21%) 5 (25%) 13 (22%) 14 (30%)
101- 5 (28%) 1 (8%) 2 (15%) 0 (0%) 3 (10%) 1 (5%) 10 (17%) 2 (4%)

150
151- 2 (11%) 1 (8%) 0 (0%) 3 (21%) 3 (10%) 4 (20%) 5 (8%) 8 (13%)
200
Over 5 (28%) 1 (8%) 8 (62%) 1 (7%) 12 (41%) 6 (15%) 25 (42%) 8 (13%)
200
Not 0 (0%) 0(0%) 0 (0%) 0 (0%) 1 (3%) 0 (0%) 1 (2%) 0 (0%)
describ
ed
Total 18 12 13 14 29 20 60 48 (100%)
(100%) (100%) (100%) (100%) (100%) (100%) (100%)
Avera 216 90 298 88 388 226 300 134
ge (N)
* This table excludes studies that use other units of analysis like applications, systems, or departments.
Table 13 indicates that one-half of surveys at the individual level have sample sizes of less than 150 and
that two-thirds of the surveys at the organization level have sample sizes of less than 150. This is
significant because the smaller the sample the less its precision. Moreover, the gains in precision
increase considerably with samples between 100 and 200, after which gains fall off [49].
Differences by purpose indicate that explanatory studies have much larger samples both at the
individual level (388) and at the organization level (226) whereas exploratory studies have smaller
samples (individual level: 216; organization level: 90) with descriptive studies somewhat in between
(individual level: 296; organization level: 88). About one-half of explanatory surveys have both
individual and organization level samples greater than 150. One-half of the descriptive surveys have
individual level samples greater than 150 and one third have organization level samples greater than 150.
One-third of the exploratory surveys at the individual level and one-tenth at the organization level have
samples greater than 150. This pattern is also apparent when looking at the average size of samples
across studies by purpose (Table 13)
Data Collection
Pretest of questionnaires. Another important aspect of the quality of survey research is whether
the questionnaires are pretested and how they are pretested. Table 14 distributes the studies by whether
or not the questionnaires are pretested and by research purposes.
Table 14. Studies by Whether the Questionnaires are Pretested and Purpose
Pretest Exploration Description Explanation Total
No. (percent) No. (percent) No. (percent) No. (Percent)
Yes 10 (29%) 5 (16%) 14 (25%) 29 (24%)
No or not 25 (71%) 26 (84%) 42 (75%) 93 (76%)
reported
Total 35 (100%) 31 (100%) 56 (100%) 122 (100%)
Table 14 indicates that ninety-three studies (76%) do not pretest the questionnaire or do not report that
pretests have been done. This very large number of studies may be inflated a little. Some studies apply
pretested questionnaires from previous studies to similar respondents of other samples, in which case,
there is no need to pretest the questionnaire again. On the other hand, several studies use questionnaires
that were tested with samples of different respondents without pretesting them again. Two studies pretest
the questionnaires with respondents other than the respondents of the sample [73, 78].
Response rate. Table 15 presents the distribution of studies by response rates and purposes.
Table 14 indicates that 90 surveys out of the 122 (or 74%) either do not report the response rate or have a
rate below 51%, which is considered inadequate in the social sciences.0 Eighty-four percent of the
exploratory surveys, 77% of the descriptive surveys and 68% of the explanatory surveys have a response
rate below the 51% margin (assuming that when the response rate is not reported, it is low).
Table 15. Studies by Response Rates and Purpose
Response rates Exploration Description Explanation Total
Above 70% 4 (11%) 3 (10%) 13 (23%) 20 (16%)
51-70% 3 (9%) 4 (13%) 5 (9%) 12 (10%)
Below 51% 14 (44%) 15 (48%) 22 (39%) 51 (42%)
Not reported 14 (40%) 9 (29%) 16 (29%) 39 (32%)
Total 35 (100%) 31 (100%) 56 (100%) 122 (100%)
0
. Babbie [8: 165] feels that, in the social sciences, a response rate of at least 50% is adequate; a response rate of at
least 60% is good; and a response rate of 70 percent or more is very good. The U.S. Office of Management and
Budget generally asks that a survey procedures be likely to yield a response rate in excess of 75 percent [49 :48].
This poor response rate is particularly troublesome for descriptive studies because their usefulness lies in
their capacity to generalize the findings to a population with high confidence. Such low response rates
jeopardize any attempt to generalize findings in an adequate way. Here again, low response rate in
exploratory studies is less significant because their findings are not intended to be generalizable to a
population. Over one third of the explanatory surveys have a response rate above 51% while only about
one fifth of the exploratory and of the descriptive studies have such a rate.
Mix of Data Collection Methods. Table 16 presents the distribution of studies among the five
data collection methods used in MIS surveys. Table 16 indicates that mail questionnaires are the most
frequently used method of data collection regardless of research purpose. Overall, ninety-four studies
use mail questionnaire (77%), twelve studies use face-to-face interview (10%), and twelve studies use
face-to-face questionnaire (10%). Quite surprisingly, there is almost no use of telephone interviewing
and computer imbedded questionnaires (where questionnaires are sent through the organization's
computer system and filled by the respondents from their terminals or personal computers).
Table 16. Studies by Data Collection Methods and Purpose
Data Collection Exploration Description Explanation Total
Method No. (percent) No. (percent) No. (percent) No. (percent)
Mail questionnaire 24 (69%) 25 (80%) 45 (80%) 94 (77%)
Face-to-face 6 (17%) 3 (10%) 3 (5%) 12 (10%)

interview*
Face-to-face 5 (14%) 3 (10%) 4 (7%) 12 (10%)
questionnaire**
Telephone 0 (0%) 0 (0%) 3 (4%) 3 (2%)
interview
Computer 0 (0%) 0 (0%) 1 (2%) 1 (1%)
embedded
questionnaire
Not described 0 (0%) 0 (0%) 1 (2%) 1 (1%)
Total 35 (100%) 31 (100%) 56 (100%) 122 (100%)
* Face-to-face interview is when researcher complete questionnaires as the respondent answers questions.
** Face-to-face questionnaire is when respondents complete questionnaires in the presence of researchers
(usually in group administrations).
The very high usage of mail questionnaire converges with the findings of an overall low response
rate observed in Table 15. Mail questionnaires are well known for their low response rates. Sixty-seven
of the ninety-four surveys using mail questionnaires, or three-fourths of the surveys, have a response rate
below 51% or do not report the response rate. When the response to mail questionnaires is examined by
purpose, the response rates are evenly poor: 84% of exploratory surveys, 77% of descriptive surveys,
and 68% of explanatory surveys have response rates below 51% or not reported. This is a weak poor
performance, both overall and by survey purpose.
Table 17 presents the studies categorized by research purposes and whether they use multiple or
single data collection methods. Table 17 indicates that there is very limited use of multiple data
collection methods. Researchers rely mainly on mail questionnaires (Table 16) and use them alone.
Table 17. Studies by Usage of Data Collection Methods and Purpose
Data collection Exploration Description Explanation Total
method No. (percent) No. (percent) No. (percent) No. (percent)
Multiple 7 (20%) 2 (6%) 9 (16%) 18 (15%)
methods
Single method 28 (80%) 29 (94%) 47 (74%) 104 (85%)
Total 35 (100%) 31 (100%) 56 (100%) 122 (100%)
One hundred and four studies (85%) use a single data collection method. Even more surprising is the
fact that 80% of the exploratory studies and 74% of the explanatory studies use a single data collection
method. The very high usage of a single data collection method in descriptive studies (94%) is normal
because they are used for social description rather than for development or testing of theory, and as such,
multiple methods cannot add as much to understanding.
It is also significant to note that more than 80% of the studies where researchers go in the field to
collect data (face-to-face interview and face-to-face questionnaire) use a single data collection method.
This is quite surprising because one might expect that when researchers are in the field, they would try to
get data from as many sources as possible to supplement and validate the questionnaire-based data.
Finally, only 2 studies of the 122 (1%) use more than two complementary data collection methods:
Rivard and Huff [85] use document analysis, observation, interview to complement mail questionnaires;
and Guimaraes [58] uses document analysis and unstructured interview to complement face-to-face
questionnaires.
DISCUSSION
Our assessment of survey research in MIS indicates that the quality of survey research
methodology is lacking overall. Table 18 summarizes the weaknesses and strengths of surveys in MIS.
Table 18. Summary Assessment of Survey Studies by Purpose
Element/Dimension Exploration Description Explanation
Research Design
Survey type Adequate Adequate Need more longitudinal
surveys
Mix of research Need more use of multiple Not necessary Need more use of multiple
methods methods methods
Unit(s) of analysis Poorly defined Poorly defined Good definition
Respondents Need to increase the Need to increase the Need to increase the
number of respondents number of respondents number of respondents
Research hypotheses Adequate Inadequately stated Inadequately stated
Design for data Not necessary Need to include time Need to include time
analysis order for hypothesis- order for hypothesis-
testing testing
Sampling
Procedures
Representativeness Adequate approximation Need better explanation Need better explanation
of sample frame and justification of and justification of
choices choices
Representativeness Adequate Need more systematic Adequate
of the sample random samples
Sample size Adequate Adequate at individual Adequate at individual
level; inadequate at level; inadequate at
organization level organization level
Data Collection
Pretest of Need more reporting of Need more reporting of Need more reporting of
questionnaires tests test tests
Response rate Poor Very poor Poor
Mix of data Need more use of multiple Not a criterion Need more use of multiple
collection methods methods
methods
As summarized in Table 18 the surveys aimed at explanation generally have better methodology than
those aimed at exploration and description. But the surveys all suffer from problems in research design,
sampling procedures, and data collection. Four dimensions are particularly weak: survey type, mix of
methods, representativeness of samples, and response rates. To some extent, each problem also contains
the seeds of its own solution and therefore we suggest solutions along with each problem.
Survey Type
The first problem with current MIS research is the lack of longitudinal studies. Only 2% of the
studies use a longitudinal design. Yet, many questions asked in MIS are process-oriented and either
cannot be answered with cross-sectional surveys or cannot be answered definitively. Questions about
impact, policy effects, development and implementation of computing all involve a time dimension.
They require measurement over time, and sometimes over long periods of time. For example, several
cross-sectional studies of information technology impacts suffer from the fact that the impacts observed
at introduction of a new system are different from those at routine use, and different still from those
resulting from long-term, sustained use. The longitudinal approach is not always preferable to the cross-
sectional approach, even for process-oriented questions, however. The longitudinal approach is
particularly well suited when changes in the dependent variable are expected to occur relatively rapidly
after the intervention of the independent variable (e.g. impact of IT on the tasks of users, impact of IT on
the time spent at different roles or on social interactions). When the time lag is long (e.g. impact of IT on
the structure of organizations, on the number of managers in organizations), many external variables
might affect the dependent variable between the intervention of the independent variable and the actual
change in the dependent variable. The causal relationship might be difficult to establish.
Mix of Methods
A second major problem with current survey research in MIS is the lack of mix of research
methods. Less than 7% of the studies use multiple research methods, and less than 10% of the studies
use multiple data collection methods. Also, less than 20% of the studies in which researchers collect data
directly in the field used multiple data collection methods. This single-method approach to research is
unfortunate because it narrows the perspectives from which the phenomena are studied and limits
possibilities for gaining understanding.
Clearly, more mixing of research methods is needed in MIS survey research. This is particularly
important for explanation and exploration studies. Surveys should be used more with case studies and
field observations in order to develop a richer, more detailed, and complete understanding of how and
why certain results occur. For example, studies of the impact of computers on secretarial/clerical
personnel and professionals frequently indicate that substantially different impacts result from
computerization. Those studies suggest that clerical work is made more demeaning while professional
work is made more satisfying as a result of computerization. However, Kling and Iacono [71] survey of
work environment effects of computerization for both types of workers reveals that both clerical and
professional workers experience an enhanced work environment and that there are no substantial
differences in the responses of clerical and professional workers.
There are several alternative explanations for the survey findings. One is that the clerical
workers are simply giving pro-social responses--i.e., saying what they think they are expected to say.
Another explanation is that something analogous occurs in the environment of both clerical and
professional workers. This is the explanation that Kling and Iacono derived from detailed case study of
the historical context and nature of changes brought by computerization in the organizations studied.
Specifically, they found that eight years ago the clerical workers were essentially operating as a central
steno pool and that since that time their work had become more varied and professionalized. It now
includes more sophisticated word processing, in-house publishing, transferring of files between remote
sites, training all departmental staff in new word processing equipment and software, and attending
technical and professional meetings to keep current with changes in the field. Thus, a real change has
taken place that actually brought the clerical workers to think of themselves in more professional terms.
The point is that this comprehension of the survey data resulted from interviews in the field
aimed at understanding the context of computing and the changes that occurred in that context. The
survey results are important because they establish that the work environment impacts of computing are
essentially similar for clerical and professional workers. However, the results obtained from the survey
would have been difficult to interpret and understand without the fieldwork.
In addition, multiple data collection methods should be used more. This is particularly easy to
implement in surveys where the main data collection method requires the researcher to go in the field
(e.g. questionnaires administered by interviews). Multiple data collection methods provide a more
complete picture of the phenomenon studied and permit to validate data. This is even more important in
exploratory and explanatory studies.
Representativeness of Sample
The third problem with current survey research in MIS is the use of inadequate sampling
procedures. More than 70% of the studies use a convenience sample or do not report the sampling
procedure. Also, more than 50% of the descriptive studies either do not describe or do not have a
systematic sampling procedure. This is particularly troublesome since the very usefulness of descriptive
studies lies in the generality of their findings. The problem is heightened by the fact that researchers
often do not recognize and take into account the limitations and peculiarities of the sample and how it
might bias the findings. Also, the population to which the findings are generalized is often not clearly
identified and defined. Another dimension of the problem associated with sampling procedures is the
weak linkage between the unit of analysis and the respondents. Several studies that focus on
organizational level phenomena use very few respondents in each organization, often only one. This
greatly limits the validity of the findings since the respondents cannot reasonably be assumed to be
representative of the organization.
Response Rates
Finally, compounding the other sampling-related problems is the very low response rates. More
than 75% of the studies have a response rate below 51% or do not report it. Here again, the descriptive
studies are particularly weak with more than 70% of them having a response rate below 51% or not
reporting it.
Clearly, the sampling procedures need to be more systematic. Reliance on convenience samples
should be kept at its minimum and there should be greater use of random and stratified sampling. Also,
generalization of findings from a sample to a population should be made more carefully and the
peculiarities of the sample should always be taken into account in making inferences. In addition, the
linkage between the respondents and the unit of analysis needs to be strengthened. The unit of analysis
should be defined more precisely, first, and then the respondents should be selected in making sure that
they truly represent the unit of analysis of interest.
The response rates need to be improved. In order to do so, the entry point into the organization
should be as high in the hierarchy as possible. Then support of top managers could be used to incite
participation at lower levels. Also, getting well known professional associations to endorse the survey
and have their logo on the questionnaires might increase participation and response rates. The
questionnaires should be kept as short and simple as possible. Other techniques include personalized
follow-up letters and distribution and collection of questionnaires by the researchers or their assistants
(rather than through the mail). Computer-imbedded-questionnaire might be another promising avenue,
especially when dealing with people working directly with computers. For example, Norman and
Nunamaker [82] had a response rate over 90% using the computer-imbedded-questionnaire method.
CONCLUSION
Our assessment indicates that the quality of surveys varies significantly among studies of
different purposes: exploratory and descriptive studies are of moderate to poor quality overall, and
explanatory studies are of good quality. The lack of rigor in descriptive and exploratory surveys is
unfortunate. Descriptive surveys, based upon proper sample frames, adequate samples, and good
response rates could be very useful for several purposes. They could be used to establish the current
state of affairs in computing development, management, or usage, e.g. the forms of end-user computing
in organizations, or the extent to which each form of end-user computing is used.
Descriptive surveys can also be used to point to areas where further study might be useful, e.g., a
pattern of differential use of end-user computing might suggest research into why are some forms of end-
user computing used more than others, or into the effects of different forms of end-user computing.
Also, descriptive surveys can be used for selecting sites for further study, e.g. if testing the
hypothesis that centralized MIS departments will tend to use centralized approaches to end-user
computing (Information Center and Computer Stores) whereas decentralized MIS departments will tend
to use decentralized approaches (user training and technical assistance), then a descriptive survey could
be used to identify centralized and decentralized IS departments. These departments could then be the
object of further study aimed at testing the hypothesized relationships between the two.
It is also unfortunate that exploratory surveys are of poor quality overall. When used correctly,
exploratory surveys can be very useful either as an independent research effort, or, more often, as the
preliminary phase of a descriptive or explanatory study. Exploratory surveys can be used to become
more familiar with a topic, to explore it, and to try out preliminary concepts about it. Survey research in
this context can also be used to discover the range of responses likely to occur in the survey of the
population of interest and to refine and complete the measurement of the concepts. Exploratory surveys
can also be used to calibrate the items and indexes used in analysis. For example, a preliminary phase of
research trying to develop and test a theory of why people engage in end-user computing could focus on
exploration of how people have come to do end-user computing, or on the organizational and personal
factors that have favored the emergence of end-user computing.
The assessment indicates that most problems of survey research pertain to non careful, and
sometimes inappropriate, application rather than to fundamental misunderstanding of the methodology
itself. The recommendations made in this paper represent a first step in solving these problems and in
promoting systematic research in MIS.
Appendix A
Survey Articles by Purpose

Exploratory Surveys
Brancheau, J.C., Vogel, D.R., & Wetherbe, J.C. 1985. An investigation of the information center from the user's perspective. DataBase. Fall,
4-17.
Brooke, G.M., & Duffy, N.M. 1986. The use of financial modeling in strategic planning. Information and Management. 11, 13-24.
Carlson, C.K. 1988. Information management approach and support to decision-making. Information and Management. 135-149.
Couger, D.J. 1986. Effect of cultural differences on motivation of analysts and programmers: Singapore vs. the United States. Management
Information Systems Quarterly. 10 (2), 189-197.
Desanctis, G. 1986. Human resources information systems: A current assessment. Management Information Systems Quarterly. 10 (1), 15-29.
Doll, W.J., & Ahmed, M.U. 1985. Documenting information systems for management: A key to maintaining user satisfaction. Information and
Management. 8, 221-226.
Drury, D.H. 1985. A survey of data processing steering committees. Information and Management. 9, 1-7.
----------.1984. An evaluation of data processing steering committees. Management Information Systems Quarterly. 8 (4), 257-265.
----------. 1983. An empirical assessment of the stages of DP growth. Management Information Systems Quarterly. 7 (2), 59-70.
Ghani, J.A., & Al-Meer, A.R. 1989. Effect of end-user computing on job satisfaction: an exploratory study. Information and Management.
191-195.
Ginzberg, M. 1981b. Key recurrent issues in the MIS implementation process. Management Information Systems Quarterly. 5 (2), 47-59.
Griese, J.,& Kurpicz, R. 1985. Investigating the buying process for the introduction of data processing in small and medium-sized firms.
Information and Management. 8, 41-52.
Guimaraes, T. 1985. A study of application program development techniques. Communications of the ACM. 28 (5), 494-499.
---------. 1983. Managing application program maintenance expenditures. Communications of the ACM. 26 (10), 739-746.
Hogue, J.T., & Watson, H.J. 1983. Management's role in the approval and administration of decision support systems. Management Information
Systems Quarterly. 7 (2), 15-26.
Howard, G.S., & Smith, R.D. 1986. Computer anxiety in management: myth or reality. Communications of the ACM. 29 (7), 611-623.
Jobber, D., Saunders, J., Gilding, B., Hooley, G., & Hatton-Smooker, J. 1989. Assessing the value of quality assurance certificate for software:
an exploratory investigation. Management Information Systems Quarterly. 13 (1), 19-32.
Kagan, A., Sobol, M.G., & Quarnstrom, K. 1987. Job roles of systems analysts in the "profit" vs. the "not for profit" sectors. Information and
Kahn, B.K. 1983. Some realities of data administration. Communications of the ACM. 26 (10), 794-799.
Kaiser, K.M., & King, W. 1982. The manager-analyst interface in systems development. Management Information Systems Quarterly. 6 (1),
49-59.
Kozar, K.A. 1989. Adopting systems development methods: an exploratory study. Journal of Management Information Systems. 5 (4), 73-86.
Lederer, A.L. & Sethi, V. 1988. The implementation of strategic information systems planning methodologies. Management Information
Lee, D.M.1986. Usage pattern and sources of assistance for personal computer users. Management Information Systems Quarterly. 10 (4),
313-327.
Lientz, B.P., & Swanson, E.B. 1980. Impact of development productivity aids on application system maintenance. DataBase. Winter-Spring,
114-120.
Lucas, H.C.Jr. 1984. Organizational power and the information services department. Communications of the ACM. 27 (1), 58-65.
Magal, S.R. 1989. The evolution of information centers: a stage hypothesis. Data Base. 20 (1), 39-46.
Magal, S.R., Houston, H.C., & Watson, H.J. 1988. Critical success factors for information center managers. Management Information Systems
Quarterly. 12 (3), 413-426.
McKeen, J.D. 1983. Successful development strategies for business application systems. Management Information Systems Quarterly. 7 (3),
47-65.
Meador, C.L., Guyote, M.J., & Keen P.G.W. 1984. Setting priorities for DSS development. Management Information Systems Quarterly. 8 (2),
117-129.
Morrell, J., & Leemon, J.1987. Assessment of an effort to integrate computer functions in an engineering design firm. Database. Winter, 17-21.
Nath, R. 1989. Aligning MIS with the business goals. Information and Management. 16, 71-79.
Nelson, R.R., & Cheney, P.H. 1987. Educating the CBIS user: A case analysis. Database. Winter, 11-16.
Olson, M.H., & Chervany, N.L. 1980. The relationship between organizational characteristics and the structure of the information service
function. Management Information Systems Quarterly. 4 (2), 57-69.
Rivard, S., & Huff, S.L. 1985. An empirical study of user as application developers. Information and Management. 8, 89-102.
Rivard, S., & Huff, S.L. 1984. User developed applications: Evaluation of success from the DP department perspective. Management
Rockart, J.F., & Flannery, L.S. 1983. The management of end-used computing. Communications of the ACM. 26 (10), 776-784.
Snitkin, S.R., & King, W.R. 1986. Determinants of the effectiveness of personal decision support systems. Information and Management. 10,
83-91.
Sumner, M., & Klepper, R. 1987. Information systems strategy and end-user application development. Database. Summer, 19-30.
Zaki, A.S., & Hoffman, R.C. 1988. Information type and its impact on information dissemination. Journal of Management Information Systems.
5 (2), 71-81.
Descriptive Surveys
Adams, D.A., & Weiss, I.R. 1989. Organizational connectivity systems: Is the function being effectively managed? Data Base. 20 (1), 16-20.
Alloway, R.M., & Quillard, J.A. 1983. User managers' systems needs. Management Information Systems Quarterly. 7 (2), 27-41.
Beheshtian-Ardekani, M., & Salchenberger, L.M. 1988. An empirical study of the use of business expert systems. Information and Management.
183-190.
Benson, D.H. 1983. A field study of end user computing: Findings and issues. Management Information Systems Quarterly. 7 (4), 35-45.
Boockholdt, J.L. 1989. Implementing security and integrity in micro-mainframe networks. Management Information Systems Quarterly. 13 (2),
135-146.
Couger, D.J., & McIntyre, S.C. 1988. Motivation norms of knowledge engineers compared to those of software engineers. Journal of Management
Information Systems. 4 (3), 82-93.
Ein-Dor, P., & Segev, E. 1981. MIS development practices. Data Management. July, 25-28.
----------. 1982. Organizational context and MIS structure: Some empirical evidence. Management Information Systems Quarterly. 6 (3), 55-68.
Ferratt, T.W. & Short, L.E. 1988. Are information systems people different? An investigation of how they are and should be managed. Management
Ghani, J.A., & Al-Sakran, S. 1988. The changing data processing environment in Saudi Arabia. Information and Management. 61-66.
Gillenson, M.L. 1985. Trends in data administration. Management Information Systems Quarterly. 8 (4), 317-325.
---------. 1982. The state of practice of data administration--1981. Communications of the ACM. 25 (10), 699-706.
Goldstein, D.K. 1989. The effects of task differences on the work satisfaction, job characteristics, and role perceptions of programmer/analysts. Journal
of Management Information Systems. 6 (1), 41-58.
Hackathorn, R.D. 1988. End-user computing by top executives. Data Base. 19 (1), 1-10.
Hoplin, H.P., & Suresh, B.A. 1989. Measurements of computer system/information system performance. Data Base. 20 (2), 278-30.
Jain, H.K., & Chaturvedi, A.R. 1989. Expert system problem selection: a domain characteristics approach. Information and Management. 245-253.
Kaiser, K.M., & Bostrom, R.P. 1982. Personality characteristics of MIS project teams: An empirical study and action-research design. Management
Kim, C. & Westin, S. 1988. Software maintainability: Perceptions of EDP professionals. Management Information Systems Quarterly. 12 (2), 167-186.
Kriebel, C.H., & Strong, D.M. 1984. A survey of the MIS and Telecommunications activities of major business firms. Management Information Systems
Quarterly. 8 (2), 171-178.
Lehman, J.A., & Murthy, V.S. 1989. Business graphics trends, two years later. Information and Management. 57-69.
Lehman, J.A., Van Wetering, J., & Vogel, D. 1986. Mainframe and microcomputer-based business graphics: What satisfies users? Information and
Lehman, J.A., Vogel, D., & Dickson, G. 1984. Business graphics trends. Datamation. November 15, 119-122.
Lu, M.T., Qiu, Y., & Guimaraes, T. 1988. A status report of the use of computer-based information systems in PRC. Information and Management.
237-242.
Mahmoud, E., & Rice, G. 1988. Database accuracy: results from a survey of database vendors. Information and Management. 243-250.
Morell, J., & Fleisher, M. 1988. Use of office automation by managers--How much, and to what purpose? Information and Management. 205-210.
Munro, M., Huff, S.L., & Moore, G. 1988. Expansion and control of end-user computing. Journal of Management Information Systems. 4 (3), 5-27.
Necco, C.R., Gordon, C.L., & Tsai, N.W. 1987a. Systems analysis and design: Current practices. Management Information Systems Quarterly. 11 (4),
461-475.
Olson, M.H. 1985. Do you telecommute? Datamation. October 15, 129-132.
Schultheis, R.A., & Klepper, R. 1988. Designing and managing local area networks: a survey of LAN personnel. Information and Management. 95-104.
Sumner, M. 1985. How should applications be developed? DataBase. Fall, 25-34.
Sumner, M., & Benson, R. 1988. The impact of fourth generation languages on systems development. Information and Management. 81-92.
Watson, H.J., Lipp, A., Jackson, P.Z., Dahmani, A., & Fredenberger, W.B. 1989. Organizational support for decision support systems. Journal of
Management Information Systems. 5 (4), 87-109.
Explanatory Surveys
Ahituv, N., Neumann, S., & Zviran, M. 1989. Factors affecting the policy for distributing computing resources. Management Information
Anderson, E.E. 1989. The implementation of information systems for workers: a structural equation model. Information and Management.
171-186.
Bailyn, L. 1989. Toward the perfect workplace? Communications of the ACM. 32 (4), 460-471.
Ball, L.D., Dambolena, I.G., & Hennessey, H.D. 1988. Identifying early adopters of large software systems. Data Base. 19 (1), 21-28.
Barki, H., & Huff, S.L. 1985. Attitude to change, and decision support systems success. Information and Management. 9, 261-268.
Baroudi, J.J. 1985. The impact of roles variables on IS personnel work attitudes and intentions. Management Information Systems Quarterly. 9
(4), 341-357.
Baroudi, J.J, & Ginzberg, M.J. 1986. Impact of the technological environment on programmer analyst job outcomes. Communications of the
ACM. 29 (6), 546-555.
Baroudi, J.J., Olson, M.H., & Ives, B. 1986. An empirical study of the impact of user involvement on system usage and information satisfaction.
Communications of the ACM. 29 (3), 232-238.
Benbasat, I., Dexter, A.S., & Mantha, R.W. 1980. Impact of organizational maturity on information system skill needs. Management Information
Bergeron, F. 1986a. Factors influencing the use of DP chargeback information. Management Information Systems Quarterly. 10 (3), 225-239.
----------. 1986b. The success of DP charge-back systems from a user's perception. Information and Management. 10, 187-195.
Bergeron, F., & Berube, C. 1988. The management of end-user environment: an empirical investigation. Information and Management.
107-113.
Carter, N.M. 1984. Computerization as a predominate technology: Its influence on the structure of newspaper organizations. Academy of
Management Journal. 27 (2), 247-270.
Cerveny, R.P., & Joseph, D.A. 1988. A study of the effects of three commonly used software engineering strategies on software enhancement
productivity. Information and Management. 243-251.
Danziger, J.N., & Kraemer, K.L. 1985. Computerized data-based systems and productivity among professional workers: The case of detectives.
Public Administration Review. January/February, 196-209.
Delone, W.H. 1981 Firm size and the characteristics of computer use. Managemetn Information Systems Quarterly, December, 65-77.
Doll, W.J. 1987. Encouraging user management participation in systems design. Information and Management. 13, 25-32.
Doll, W.J., & Torkzadeh, G. 1989. A discrepancy model of end-user computing involvement. Management Science. 35 (10), 1151-1171.
Franz, C.r., Robey, D., & Koeblitz, R.R. 1986. User response to an online information system: A field experiment. Management Information
Ginzberg, M.J. 1981a. Early diagnosis of MIS implementation failure: Promising results and unanswered questions. Management Science. 27
(4), 459-478.
Green, G.I. 1989. Perceived importance of systems analysts' job skills, roles, and non-salary incentives. Management Information Systems
Quarterly. 13 (2), 115-134.
Gremillion, L.L. 1984. Organization size and information systems use: an empirical study. Journal of Management Information Systems. 1 (2),
4-17. Gupta, Y.P., & Raghunathan, T.S. 1989. Impact of information systems (IS) steering committees on IS planning. Decision
Sciences. 20, 777-793.
Igbaria, M., Pavri, F.N., & Huff, S.L. 1989. Microcomputer applications: an empirical look at usage. Information and Management. 187-196.
Kwan, S.K., & Foley Curley, K. 1989. Corporate MIS/DP and end user computing: the emergence of a new partnership. Data Base. 20 (2),
31-37.
Laudon, K.C. 1986. Data quality and due process in large interorganizational record systems. Communications of the ACM. 29 (1), 4-11.
Lehman, J.A. 1985-86. Organizational size and information system sophistication. Journal of Management Information Systems. 2 (3), 78-86.
Leonard-Barton, D., & Deschamps, I. 1988. Managerial influence in the implementation of new technology. Management Science.
34 (10), 1252-1265.
Lind, M.R., Zmud, R.W., & Fischer, W.A. 1989. Microcomputer adoption--the impact of organizational size and structure. Information and
Management. 157-162.
Lucas, H.C., Walton, E.J., & Ginzberg, M.J. 1988. Implementing packaged software. Management Information Systems Quarterly. 12 (4),
537-550.
Magal, S.R., & Carr, H.H. 1988. An investigation of the effects of age, size, and hardware option on the critical success factors applicable to
information centers. Journal of Management Information Systems. 4 (4), 60-76.
Mahmood, M.A. 1987. System development methods--A comparative investigation. Management Information Systems Quarterly. 11 (3),
293-311.
Millman, Z., & Hartwick, J. 1987. The impact of automated office systems on middle managers and their work. Management Information
Norman, R.J., & Nunamaker, J.F. 1989. CASE productivity perceptions of software engineering professionals. Communications of the ACM. 32
(9), 1102-1109.
Nutt, P.L. 1986. Evaluating MIS design principles. Management Information Systems Quarterly. 10 (2), 139-155.
Raghunathan, B., & Raghunathan, T.S. 1989. Relationship of the rank of information systems executive to the organizational roles and planning
dimensions of information systems. Journal of Management Information Systems. 6 (1), 111-126.
Raghunathan, T.S., Gupta, Y.P., & Sundararaghavan, P.S. 1989. Assessing the impact of IS executives' critical success factors on the
performance of IS organizations. Information and Management. 17 (3), 157-168.
Raho, L.E., Belohlav, J.A., Fielder, K.D. 1987. Assimilating new technology into the organization: An assessment of McFarlan and McKenney's
model. Management Information Systems Quarterly. 11 (1), 47-57.
Raymond, L. 1985. Organizational characteristics and MIS success in the context of small business. Management Information Systems
Quarterly. 9 (1), 37-52.
Rivard, S., & Huff, S.L. 1988. Factors of success for end user computing. Communications of the ACM. 31 (5), 552-561.
Robertson, D.C. 1989. Social determinants of information systems use. Journal of Management Information Systems. 5 (4), 55-71.
Robey, D., Farrow, D.L., & Franz, C.R. 1989. Group process and conflict in systems development. Management Science, 10 (35), 1172-1191.
Rushinek, A., & Rushinek, S.F.1986a. What makes users happy? Communications of the ACM. 29 (7), 594-598.
----------. 1986b. Production and inventory management software packages related to user reactions. Production and Inventory Management. 27
(1), 75-84.
----------. 1986c. The effects of computer location on end-user satisfaction. Industrial Processing and Management. 22 (4), 345-351.
Rushinek, A., & Rushinek, S.F. 1986d. End-user satisfaction of data base management systems: An empirical assessment of mainframe, mini
and micro computer-based systems using an interactive model. Database. Winter, 17-27.
----------. 1986e. Microcomputer application diagnostic system for identification and evaluation of user satisfaction factors of purchasing
software. Microprocessing and Microporogramming. 17 (4), 185-194.
----------. 1985a. Effect of system ratings on user satisfaction. Information Age. 7 (2), 98-106.
----------. 1985b. Health marketing and sales distribution software related to computer user satisfaction. Health Marketing and Consumer
Behavior. 3 (1), 79-101.
----------. 1985c. Accounting and billing software and user reactions: An interactive diagnostic audit trail. Information and Management. 9 (1),
9-20.
Saaksjarvi, M., Heikkila, J., & Saarinen, T. 1988. The dual role of information centers: an assessment of end user computing management
strategies. Information and Management. 69-78.
Sanders, G.L., & Courtney, J.F. 1985. A field study of organizational factors influencing DSS success. Management Information Systems
Quarterly. 9 (1), 77-93.
Sato, O., & Horiuchi, M. 1988. IRM as a coordinating mechanism: a study in large Japanese firms. Information and Management. 93-103.
Sauter, S.L., Gottlieb, M.S., Jones, K.C., Dodson, V.N., & Rohrer, K.M. 1983. Job and health implications of VDT use: Initial results of the
Wisconsin-NIOSH study. Communications of the ACM. 26 (4), 284-294.
Specht, P.H. 1986. Job characteristics as indicants of CBIS data requirements. Management Information Systems Quarterly. 10 (3), 271-286.
Sproull, L., & Kiesler, S. 1986. Reducing social context cues: Electronic mail in organizational communication. Management Science. 32 (11),
1492-1512.
Tavakolian, H. 1989. Linking the information technology structure with organizational competitive strategy: A survey. Management
Ugbah, S.D., & Dewine, S. 1989. New communication technologies: the impact on intra-organizational dynamics. Information and
Management. 181-186.
Vitalari, N.P., Venkatesh, A., & Gronhang, K. 1985. Computing in the home: Shifts in the time allocations patterns of households.
Communications of the ACM. 28 (5), 512-522.
Weber, R. 1988. Computer technology and jobs: An impact assessment model. Communications of the ACM. 31 (1), 68-77.
Weitzel, J.R., & Graen, G.B. 1989. System development project effectiveness: problem-solving competence as a moderator variable. Decision
Science. 20, 507-531.
Welsh, G.M.1986. The information transfer specialist in successful implementation of decision support systems. Database. Fall, 32-40.
Yap, C.S., & Walshman, G. 1986. A survey of information technology in the U.K. service sector. Information and Management. 10 (5), 267-274.
Zmud, R.W. 1983. The effectiveness of external information channels in facilitating innovation within software development groups.
Management Information Systems Quarterly. 7 (2), 43-58.
----------. 1982. Diffusion of modern software practices: Influence of centralization and formalization. Management Science. 28 (12),
1421-1431.
36
REFERENCES
1. Adams, D.A., & Weiss, I.R. Organizational connectivity systems: Is the function being effectively managed?
Data Base. 20 (1), 1989, 16-20.
2. Ahituv, N., Neumann, S., & Zviran, M. Factors affecting the policy for distributing computing resources.
Management Information Systems Quarterly. 13 (4), 1989, 389-401.
3. Alavi, M. End-user computing: The MIS managers' perspective. Information and Management. 8, 1985, 171-178.
4. Alloway, R.M., & Quillard, J.A. User managers' systems needs. Management Information Systems Quarterly. 7
(2), 1983, 27-41.
5. Amoroso, D.L., Thompson, R.L., & Cheney, P.H. Examining the duality of I.S. executives: a study of IS issues.
Information and Management. 17 (1), 1989, 1-12.
6. Anderson, E.E. The implementation of information systems for workers: a structural equation model. Information
and Management. 1989, 171-186.
7. Attewell, P. & Rule, J.B. Survey and other methodologies applied to IT impact research: experiences from a
comparative study of business computing. In K.L. Kraemer (ed.), The Information Systems Research
Challenge: Survey Research Methods, Vol. 3. Cambridge, MA: Harvard Business School Press, 1991, pp.
299-316.
8. Babbie, E.R. Survey Research Methods. Belmont, CA: Wadsworth Publishing Company Inc. 1973.
9. Bailyn, L. Toward the perfect workplace? Communications of the ACM. 32 (4), 1989, 460-471.
10. Ball, L.D., Dambolena, I.G., & Hennessey, H.D. Identifying early adopters of large software systems. Data
Base. 19 (1), 1988, 21-28.
11. Ball, L.D., & Harris, R. SMIS members: A membership analysis. Management Information Systems Quarterly.
6 (1), 1982, 19-39.
12. Barki, H., & Huff, S.L. Attitude to change, and decision support systems success. Information and Management.
9, 1985, 261-268.
13. Barki, H., Rivard, S., & Talbot, J. An information systems keyword classification scheme. Management
Information Systems Quarterly. 12 (2), 1988, 299-324.
14. Baroudi, J.J. The impact of roles variables on IS personnel work attitudes and intentions. Management
15. Baroudi, J.J, & Ginzberg, M.J. Impact of the technological environment on programmer analyst job outcomes.
Communications of the ACM. 29 (6), 1986, 546-555.
16. Baroudi, J.J., Olson, M.H., & Ives, B. An empirical study of the impact of user involvement on system usage and
information satisfaction. Communications of the ACM. 29 (3), 1986, 232-238.
17. Baroudi, J.J., & Orlikowski, W.J. The problems of statistical power in MIS research. Management Information
Systems Quarterly. 13 (1), 1988, 87-106.
18. Benbasat, I., Goldstein, D.K., & Mead, M. The case research strategy in studies of information systems.
19. Beheshtian-Ardekani, M., & Salchenberger, L.M. An empirical study of the use of business expert systems.
Information and Management. 1988, 183-190.
37
20. Benson, D.H. A field study of end user computing: Findings and issues. Management Information Systems
Quarterly. 7 (4), 1983, 35-45.
21. Boland, R.J., & Hirschheim, R.A. eds. Critical Issues in Information Systems Research. New York: John Wiley
and Sons. 1987.
22. Boockholdt, J.L. Implementing security and integrity in micro-mainframe networks. Management Information
23. Brancheau, J.C., Vogel, D.R., & Wetherbe, J.C. An investigation of the information center from the user's
perspective. DataBase. Fall, 4-17. 1985.
24. Brancheau, J.C., & Wetherbe, J.C. Key issues in Information Systems--1986. Management Information Systems
Quarterly. 11 (1), 1987, 23-45.
25. Brooke, G.M., & Duffy, N.M. The use of financial modeling in strategic planning. Information and
Management. 11, 1986, 13-24.
26. Campbell, D.T., & Stanley, J.C. Experimental and Quasi-Experimental Designs for Research, Boston: Houghton
Mifflin Company. 1963.
27. Carlson, C.K. Information management approach and support to decision-making. Information and
Management. 135-149. 1988.
28. Cheney, P.H., & Lyons, N.R. Information systems skill requirements: A survey. Management Information
29. Couger, D.J. Key human resource issues in IS in the 1990s: views of IS executives versus human resource
executives. Information and Management. 1988, 161-174.
30. ----------. Effect of cultural differences on motivation of analysts and programmers: Singapore vs. the United
States. Management Information Systems Quarterly. 10 (2), 1986, 189-197.
31. Couger, D.J., & McIntyre, S.C. Motivation norms of knowledge engineers compared to those of software
engineers. Journal of Management Information Systems. 4 (3), 1988, 82-93.
32. Culnan, M.J. Mapping the intellectual structure of MIS, 1980-1985: A co-citation analysis. Management
33. ----------. Intellectual development of Management Information Systems 1972-1982: Co-citation analysis.
Management Science 32 (2), 1986, 156-172.
34. Culnan, M.J., & Swanson, E.B. Research in Management Information Systems, 1980-1984: Points of work and
reference. Management Information Systems Quarterly. 10 (3), 1986, 289-301.
35. Danziger, J.N., & Kraemer, K.L. Computerized data-based systems and productivity among professional
workers: The case of detectives. Public Administration Review. January/February, 1985, 196-209.
36. Danziger, J.N., & Kraemer, K.L. Survey research and multiple operationalism: The URBIS project
methodology. In K.L. Kraemer (ed), The Information Systems Research Challenge: Survey Research
Methods, Vol. 3. Cambridge, MA: Harvard Business School Press, 1991, pp. 351-372.
37. Davis, F.D. Perceived usefulness, perceived ease of use, and user acceptance of information technology.
38. Desanctis, G. Human resources information systems: A current assessment. Management Information Systems
Quarterly. 10 (1), 1986, 15-29.
38
39. Dickson, G.W., Leitheiser, R.L., Wetherbe, J.C., & Nechis, M. Key information systems issues for the 80's.
40. Dickson, G.W., & Nechis, M. Key information systems issues for the 1980's. Management Information Systems
Quarterly. 8 (3), 1984, 135-159.
41. Dillman, Richard. Mail and Telephone Surveys: Total Design Method. New York, John Wiley. 1978.
42. Doll, W.J., & Ahmed, M.U. Documenting information systems for management: A key to maintaining user
satisfaction. Information and Management. 8, 1985, 221-226.
43. Doll, W.J., & Torkzadeh, G. A discrepancy model of end-user computing involvement. Management Science.
35 (10), 1989, 1151-1171.
44. Drury, D.H. A survey of data processing steering committees. Information and Management. 9, 1985, 1-7.
45. ----------. An evaluation of data processing steering committees. Management Information Systems Quarterly. 8
(4), 1984, 257-265.
46. ----------. An empirical assessment of the stages of DP growth. Management Information Systems Quarterly. 7
(2), 1983. 59-70.
47. Farhoomand, A.D. Scientific progress of management information systems, DataBase, 18 (4), 1987, 48-56.
48. Ferrat, T.W. & Short, L.E. Are information systems people different: An investigation of motivational
differences. Management Information Systems Quarterly. 10 (4), 1986, 377-389.
49. Fowler, F.J., Jr. Survey Research Methods. Beverly Hills, CA: Sage Publications. 1984.
50. Frand, J.L., & Britt, J. Fifth annual UCLA survey of business school computer usage. Communications of the
ACM. 32 (1), 1989, 62-76.
51. Frand, J.L., & McLean, E.R. Summary of the second annual UCLA survey of business school computer usage.
Communications of the ACM. 29 (1), 1986, 18.
52. Frand, J.L., McLean, E.R., & Britt, J.A. Fourth annual UCLA survey of business school computer usage.
Communications of the ACM. 31 (7), 1988, 896-911.
53. Franz, C.r., Robey, D., & Koeblitz, R.R. User response to an online information system: A field experiment.
54. Galletta, D.F., & Lederer, A.L. Some cautions on the measurement of user information satisfaction. Decision
Science. 20, 1989, 419-438.
55. Ghani, J.A., & Al-Meer, A.R. Effect of end-user computing on job satisfaction: an exploratory study. Information
and Management. 1989, 191-195.
56. Glock, C.Y., ed. Survey Research in the Social Sciences. New York, NY: Russell Sage Foundation. 1967.
57. Guimaraes, T. A study of application program development techniques. Communications of the ACM. 28 (5),
1985, 494-499.
58. Guimaraes, T., & Ramanujam, V. Personal computing trends and problems: An empirical study. Management
39
59. Gupta, J.N.D., & Seeborg, I.S. The graduate MIS course in the schools and colleges of business. Journal of
Management Information Systems. 5 (4), 1989, 125-136.
60. Gurbaxani, V., & Mendelson, H. The use of secondary analysis in MIS research. In K.L. Kraemer (ed.), The
Management Information Systems Research Challenge: Survey Research Methods, Vol. 3. Cambridge,
MA: Harvard Business School Press. 1991.
61. Gutek, B.A. Using survey research in MIS research: one researcher's experience. In K.L. Kraemer (ed.), The
Management Information Systems Research Challenge: Survey Research Methods, Vol. 3. Cambridge,
MA: Harvard Business School Press, 1991, pp. 373-382.
62. Hartog, C., & Herbert, M. 1985 Opinion survey of MIS managers: Key issues. Management Information
63. Hough, P.K., & Duffy, N.M. Top management perspectives on decision support systems. Information and
Management. 12, 1987, 21-31.
64. Ives, B., Hamilton, S., & Davis, G.B. A framework for research in computer-based management information
systems. Management Science, 26 (9), 1980, 910-934.
65. Jobber, D., Saunders, J., Gilding, B., Hooley, G., & Hatton-Smooker, J. Assessing the value of quality assurance
certificate for software: an exploratory investigation. Management Information Systems Quarterly. 13 (1),
1989, 19-32.
66. Joshzi, K. The measurement of fairness or equity perceptions of management information systems users.
67. Kaiser, K.M., & Bostrom, R.P. Personality characteristics of MIS project teams: An empirical study and action-
research design. Management Information Systems Quarterly. 6 (4), 1982, 43-61.
68. Kaplan, B. & Duchon, D. Combining qualitative and quantitative methods in information systems research: A
case study. Management Information Systems Quarterly. 12 (4), 1988, 571-587.
69. King, W.R., & Premkumar, G. Key issues in telecommunications planning, Information and Management. 17 (5),
1989, 255-266.
70. Kling, R. Adapting survey methods to study the social consequences of computerization: a response to
Attewell and Rule based on five survey-based studies. In K.L. Kraemer (ed.), The Management
Inforamtion Systems Research Challenge: Survey Research Methods, Vol. 3. Cambridge, MA: Harvard
Business School Press, 1991, pp. 337-350.
71. Kling, R., & Iacono, S. Computer Systems as Institutions: Social Dimensions of Computing in Organizations,
University of California, Irvine. 1988.
72. Kraemer, K.L., & Dutton, W.H. Survey research in the study of management information systems. In K.L.
Kraemer (ed.), The Information Systems Research Challenge: Survey Research Methods, Vol. 3.
Cambridge, MA: Harvard Business School Press, pp. 3-58. 1991.
73. Lee, D.M. Usage pattern and sources of assistance for personal computer users. Management Information
74. Lucas, H.C.jr. Methodological issues in information systems survey research. In K.L. Kraemer (ed.), The
Information Systems Research Challenge: Survey Research Methods, Vol. 3. Cambridge, MA: Harvard
Business School Press. 1991.
75. Mason, R.O., & Mitroff, L.L. A program for research on Management Information Systems. Management
Science. 19 (5), 1973, 475-485.
40
76. McFarlan, W. The Information Systems Research Challenge. Cambridge, MA: Harvard Business School Press.
1987.
77. Merten, A.G., & Severance D.G. Data processing control: A state-of-the art survey of attitudes and concerns of
DP executives. Management Information Systems Quarterly. 5 (2), 1981, 11-33.
78. Morrell, J., & Leemon, J. Assessment of an effort to integrate computer functions in an engineering design firm.
Database. Winter, 1987, 17-21.
79. Mumford, E., Hirschheim, R., Fitzgerald, G., & Wood-Harper, T. (eds.). Research Methods in Information
Systems. New York, NY: North-Holland. 1985.
80. Nelson, R.R., & Cheney, P.H. Educating the CBIS user: A case analysis. Database. Winter, 1987, 11-16.
81. Newsted, P.R., Munro, M.C., & Huff, S.L. Data acquisition instruments in management information systems. In
K.L. Kraemer (ed.), The Information Systems Research Challenge: Survey Research Methods, Vol. 3.
Cambridge, MA: Harvard Business School Press, pp. 187-210. 1991.
82. Norman, R.J., & Nunamaker, J.F. CASE productivity perceptions of software engineering professionals.
Communications of the ACM. 32 (9), 1989 1102-1109.
83. Raymond, L. Validating and applying user satisfaction as a measure of MIS success in small organizations.
Information and Management. 12 (4), 1987, 173-180.
84. Rivard, S., & Huff, S.L. Factors of success for end user computing. Communications of the ACM. 31 (5), 1988,
552-561.
85. Rivard, S., & Huff, S.L. An empirical study of user as application developers. Information and Management. 8,
1985, 89-102.
86. Rockart, J.F., & Flannery, L.S. The management of end-used computing. Communications of the ACM. 26 (10),
1983, 776-784.
87. Rosenberg. M. The Logic of Research Methods. Beverly Hill, CA: Sage Publications. 1968.
88. Sauter, S.L., Gottlieb, M.S., Jones, K.C., Dodson, V.N., & Rohrer, K.M. Job and health implications of VDT
use: Initial results of the Wisconsin-NIOSH study. Communications of the ACM. 26 (4), 1983, 284-294.
89. Simon, H. The behavioral and social sciences. Sciences, 209, July 4, 1980, 72-78.
90. Straub, D.W. Validating instruments in MIS Research. Management Information Systems Quarterly. 13 (2),
1989, 147-169.
91. Tanur, J.M. Advances in methods for large-scale surveys and experiments, in R. Mcadams, N.J. Smelser, & D.J.
Treiman (eds.), Behavioral and Social Science Research: A National Resource, Part II. Washington, D.C.:
National Academy Press. 1982.
92. Teng, J.T.C. and Galletta, D. MIS research direction: A survey of researcher's views, Database, 22, (1/2), 1991,
53-62.
93. Torkzadeh, G. The quality of user documentation: an instrument validation. Journal of Management
Information Systems. 5 (2), 1988, 99-108.
94. Vitalari, N.P. & Venkatesh, A. An emerging distributed work arrangement: A theoretical and empirical
examination of supplemental work at home. Management Sciences. Forthcoming.
41
95. Vitalari, N.P. & Venkatesh, A. Longitudinal surveys in information systems research: an examination of issues,
methods, and applications. In K.L. Kraemer (ed.), The Information Systems Research Challenge: Survey
Research Methods, Vol. 3. Cambridge, MA: Harvard Business School Press, pp. 115-144. 1991.
96. Vitalari, N.P., Venkatesh, A., & Gronhang, K. Computing in the home: Shifts in the time allocations patterns of
households. Communications of the ACM. 28 (5), 1985, 512-522.
97. Vogel, D.R., & Wetherbe, J.C. Profile of MIS research: methodology and journal preference, Database, Fall,
1984, 3-14.
98. Zmud, R.W. & Boynton, A.C. Survey measures and instruments in MIS: inventory and appraisal. In K.L.
Kraemer (ed.), The Information Systems Research Challenge: Survey Research Methods, Vol. 3.
Cambridge, MA: Harvard Business School Press. 1991.

UC Irvine: I.T. in Government

Uploaded by

Copyright:

Available Formats

UC Irvine: I.T. in Government

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

UC Irvine: I.T. in Government

Uploaded by

Copyright:

Available Formats

UC Irvine

eScholarship.org Powered by the California Digital Library

Working Paper #URB-022

DEFINITION OF SURVEY RESEARCH

Characteristics of Survey Research

Appropriate Application of Survey Research in MIS

DATA AND METHOD

Table 1. Survey articles by journal

Journals Number of Survey Articles

Classification of Studies by Purpose

Development of Framework for Assessment

FINDINGS ABOUT THE QUALITY OF SURVEY RESEARCH IN MIS

(%) = Percent of number of studies by purpose.

Table 9. Studies by Sample Frame and Purpose

Not reported 22 (63%) 9 (29%) 29 (52%) 60 (49%)

Total 35 (100%) 31 (100%) 56 (100%) 122 (100%)

Table 10. Studies by Sampling Procedures and Purpose (Organization-level Surveys)

Table 11. Studies by Sampling Procedures and Purpose (Individual-level surveys)

Table 12. Studies by Sample Bias Test and Purpose

101- 5 (28%) 1 (8%) 2 (15%) 0 (0%) 3 (10%) 1 (5%) 10 (17%) 2 (4%)

Face-to-face 6 (17%) 3 (10%) 3 (5%) 12 (10%)

Total 35 (100%) 31 (100%) 56 (100%) 122 (100%)

Survey Articles by Purpose

Olson, M.H. 1985. Do you telecommute? Datamation. October 15, 129-132.

Sumner, M. 1985. How should applications be developed? DataBase. Fall, 25-34.

You might also like