Sociology at Surrey

University of Surrey

social
research

UPDATE

The term pilot studies refers to mini

versions of a full-scale study (also called
feasibility studies), as well as the specific pre-testing of a particular research
instrument such as a questionnaire or interview schedule.

Pilot studies are a crucial element of a

good study design. Conducting a pilot
study does not guarantee success in the
main study, but it does increase the likelihood.

Pilot studies fulfil a range of important

functions and can provide valuable
insights for other researchers. There is
a need for more discussion amongst researchers of both the process and outcomes of pilot studies.

The importance of pilot studies

Edwin R. van Teijlingen and Vanora Hundley

ISSUE

35

The term pilot study is used in two different

ways in social science research. It can refer
to so-called feasibility studies which are
small scale version[s], or trial run[s], done
in preparation for the major study (Polit et
al., 2001: 467). However, a pilot study can
also be the pre-testing or trying out of a
particular research instrument (Baker 1994:
182-3). One of the advantages of conducting a pilot study is that it might give advance
warning about where the main research
project could fail, where research protocols
may not be followed, or whether proposed
methods or instruments are inappropriate or
too complicated. In the words of De Vaus
(1993: 54) Do not take the risk. Pilot test
first. These are important reasons for undertaking a pilot study, but there are additional reasons, for example convincing
funding bodies that your research proposal
for the main study is worth funding. Thus
pilot studies are conducted for a range of
different reasons (see Table 1).
Pilot studies can be based on quantitative
and/or qualitative methods and large-scale
studies might employ a number of pilot studies before the main survey is conducted.
Thus researchers may start with qualitative
data collection and analysis on a relatively
unexplored topic, using the results to design
a subsequent quantitative phase of the study
(Tashakkori & Teddlie 1998: 47). The first
phase of a pilot might involve using in-depth
interviews or focus groups to establish the
issues to be addressed in a large-scale ques-

tionnaire survey. Next the questionnaire, e.g.

the wording and the order of the questions,
or the range of answers on multiple-choice
questions, might be piloted. A final pilot
could be conducted to test the research process, e.g. the different ways of distributing and
collecting the questionnaires. For example,
a recent study exploring nurses and midwives attitudes to research followed this pattern. In this study focus groups were used
to identify key issues from which a questionnaire could be developed, and this was then
piloted prior to the study proper (Hundley
et al. 2000). On a much larger scale, the
largest (decennial) survey in the UK, the
Census (of 29th April 2001), tested methodological and other changes to the 1991 Census questionnaire on over 100,000 households in 1997. This 1997 Census Test provided essential information on public reaction to new questions and form style as well
as assessing the success of collection and
processing methods (Office for National
Statistics, General Register Office for Scotland, Northern Ireland Statistical & Research
Agency 1999: 15).
Pilot studies may also try to identify potential practical problems in following the research procedure. For example, in a recent
Scottish study of maternity care the pilot
phase demonstrated that the proposed
means of distributing the questionnaires
would not be adhered to (van Teijlingen et
al. 2001). Without consulting the research
team, the person responsible for distribut-

Dr Edwin van Teijlingen is a

medical sociologist and Senior
Lecturer in Public Health at the
Department of Public Health,
University of Aberdeen. His research interests include the organisation of maternity care at
home and abroad, substance
misuse, and psychosocial aspects of genetics.
Dr Vanora Hundley is a midwife
and Lecturer at the Centre for
Advanced Studies in Nursing,
University of Aberdeen. Her research interests include midwifery care, consumer satisfaction and preferences, and research utilisation.

social research UPDATE

ing the questionnaires from the hospital
records department decided that it was better to distribute them through the community midwives. This was despite the fact that
the hospital itself had suggested the records
department as a means of distribution. Other
problems such as poor recording and response rates can also be identified and precautionary procedures or safety nets can be
devised.
The steps used to pilot a questionnaire on a
small group of volunteers, who are as similar as possible to the target population, are
listed in Table 2.
Pilot studies can also uncover local politics
or problems that may affect the research
process. In the study described above, the
managers of maternity services had different
perceptions of what the forthcoming
changes in the Data Protection Act (1998)
allowed them to do about the involvement
of their clients in research. In the above mentioned recent Scottish study of maternity care
one head of midwifery voiced ethical concerns about the use of follow-up or reminder
letters due to a previous local incident, where
parents of an ill baby had been sent a questionnaire which was felt to be inappropriate,
and as a result of changes to the UK Data
Protection Act. Consequently reminders
were sent out via the head of midwifery in
case there were any problems with the newborn baby.

Problems of pilot studies

It should be recognised pilot studies may also
have a number of limitations. These include
the possibility of making inaccurate predictions or assumptions on the basis of pilot
data; problems arising from contamination;
and problems related to funding. These issues are now discussed in turn.
Completing a pilot study successfully is not
a guarantee of the success of the full-scale
survey. Although pilot study findings may
offer some indication of the likely size of the
response rate in the main survey, they cannot guarantee this because they do not have
a statistical foundation and are nearly always
based on small numbers. Furthermore,
other problems or headaches may not be-

come obvious until the larger scale study is

conducted.
A further concern is that of contamination.
This may arise in two ways:
1. where data from the pilot study are included in the main results;
2. where pilot participants are included in
the main study, but new data are collected from these people.
Social scientists engaged in predominantly
quantitative research are likely to argue that:
an essential feature of a pilot study is that
the data are not used to test a hypothesis or
included with data from the actual study
when the results are reported (Peat et al.
2002: 57). The obvious concern is that if
there were problems with the research tool
and modifications had to be made in the light
of the findings from the pilot study, data
could be flawed or inaccurate. However,
where an established and validated tool is
being used and the pilot study is determining other methodological aspects such as
recruitment rates, it could be argued that
such data may be of value.
A common problem isthe inclusion of pilot
study participants in the site(s) of the main
study. Here the concern is that such participants have already been exposed to an intervention and, therefore, may respond differently from those who have not previously
experienced it. This may be positive, for example the participants may become more
adept at using a new tool or procedure. However it may also be negative with participants
showing a decline in following a protocol
because it is no longer novel. Indeed both
changes in behaviour have long been recognised and a run in period, where an intervention is introduced prior to a study, is often used for these reasons. The concern
about including participants from the pilot
study in the main study arises because only
those involved in the pilot, and not the whole
group, will have had the experience. In some
cases however it is simply not possible to
exclude these pilot-study participants because to do so would result in too small a
sample in the main study. This problem
arises in particular where the samples are
clusters, for example schools, prisons or
hospitals. In such cases one can conduct a

Table 1. Reasons for conducting pilot

studies
Developing and testing adequacy of
research instruments
Assessing the feasibility of a (fullscale) study/survey
Designing a research protocol
Assessing whether the research protocol is realistic and workable
Establishing whether the sampling
frame and technique are effective
Assessing the likely success of proposed recruitment approaches
Identifying logistical problems which
might occur using proposed methods
Estimating variability in outcomes to
help determining sample size
Collecting preliminary data
Determining what resources (finance, staff) are needed for a planned
study
Assessing the proposed data analysis techniques to uncover potential
problems
Developing a research question and
research plan
Training a researcher in as many elements of the research process as possible
Convincing funding bodies that the
research team is competent and knowledgeable
Convincing funding bodies that the
main study is feasible and worth funding
Convincing other stakeholders that
the main study is worth supporting

social research UPDATE

sensitivity analysis (or sub-group analysis)
to assess to what extent the process of
piloting influences the size of the intervention effect.
Contamination is less of a concern in qualitative research, where researchers often use
some or all of their pilot data as part of the
main study. Qualitative data collection and
analysis is often progressive, in that a second or subsequent interview in a series
should be better than the previous one as
the interviewer may have gained insights
from previous interviews which are used to
improve interview schedules and specific
questions. Some have therefore argued that
in qualitative approaches separate pilot studies are not necessary (e.g. Holloway
1997:121). For example, a qualitative interviewer conducting 15 focus group interviews will listen to the recordings or read
through the transcripts of the first three or
four in order to improve the questions, the
way of introducing the issues into the group
interview or even to add new topics. Thus,
although there is no specific pilot study,
analysis of the earlier focus groups may help
improve the later ones. However, Frankland
and Bloor (1999: 154) argue that piloting
provides the qualitative researcher with a
clear definition of the focus of the study
which in turn helps the researcher to concentrate data collection on a narrow spectrum of projected analytical topics. Piloting
of qualitative approaches can also be carried out if the researcher lacks confidence
or is a novice, particularly when using the
interview technique (Holloway 1997: 121).
Problems may also arise where a pilot study
requires a significant investment of resources, making it difficult for the study
team to call a halt to the research after an
unsuccessful pilot study. Researchers might
be tempted to make considerable changes
in the main study, rather than deciding that
the proposed study is not possible with the
available resources, time, population, etc. In
contrast, funding bodies may be reluctant
to fund a further study if the pilot has been
substantial as they may view the research as
no longer original, especially if results from
the pilot study are published.

Why are pilot studies not

reported?
Publication bias may occur because of a tendency for journals to accept only papers that
have statistically significant results and not
to report non-significant effects (Mahoney
1977; Chann 1982; Dickersin, 1990). A recent
study exploring research on passive smoking found a difference of two years in the
median time to publication between findings
from significant and non-significant studies
(Misakian & Bero 1998). It follows that papers reporting methodological issues, such
as those identified during the pilot phase of
a study, will also be less attractive to publishers.
Selective publication of research results has
been recognised as a problem. It may lead
to an overestimation of the effectiveness of
interventions, exposing patients to useless
or harmful treatments, while overestimation
of adverse effects may mean that patients are
denied effective forms of care (Oxman et al.
1994). In the past editors have recognised
the dangers of publication bias with respect
to clinical trials and have offered an amnesty
for unpublished trials in an attempt to overcome these problems (Smith & Roberts,
1997). However, it is equally important to
ensure that lessons learned with respect to
the research method are shared, otherwise
patients may be subjected to poorly developed tools or money may be wasted because
methods of recruitment failed. A consistent
selection bias favouring reports of primary
research over papers on research methods,
theoretical thinking, or secondary analysis,
can lead to many of researchers re-inventing
the wheel without having had the opportunity to learn from other peoples experience.

Conclusions
It has been said that pilot studies are likely
to be underdiscussed, underused and
underreported (Prescott and Soeken, 1989
p60). Full reports of pilot studies are rare in
the research literature (Lindquist, 1991;
Muoio et al, 1995, van Teijlingen et al. 2001).
When reported, they often only justify the
research methods or particular research tool
used. Too often research papers only refer
to one element of the pilot study, for exam-

Table 2. Pilot study procedures to improve the internal validity of a questionnaire

administer the questionnaire to pilot
subjects in exactly the same way as it will
be administered in the main study
ask the subjects for feedback to identify ambiguities and difficult questions
record the time taken to complete
the questionnaire and decide whether it
is reasonable
discard all unnecessary, difficult or
ambiguous questions
assess whether each question gives
an adequate range of responses
establish that replies can be interpreted in terms of the information that
is required
check that all questions are answered
re-word or re-scale any questions that
are not answered as expected
shorten, revise and, if possible, pilot
again
(Source: Table 3.23 in Peat et al. 2002: 123)

social research UPDATE is

distributed without charge on

request to social researchers
in the United Kingdom by the
Department of Sociology at
the University of Surrey as part
of its commitment to supporting social research training
and development.
Contributions to social research UPDATE that review
current issues in social research and methodology in
about 2,500 words are welcome. All UPDATE articles
are peer-reviewed.

social research UPDATE

ple, to the pre-testing or pilot testing of a
questionnaire (De Vaus, 1993). Such papers
simply state: the questionnaire was tested
for validity and reliability. When pilot studies are mentioned in more detail in academic
papers and reports, researchers regularly
comment that they had learned from the
pilot study and made the necessary changes,
without offering the reader details about
what exactly was learnt. Some of these processes and outcomes from both successful
and failed pilot studies might be very useful
to others embarking on projects using similar methods and instruments. This is particularly important because pilot studies can
be time-consuming, frustrating, and fraught
with unanticipated problems, but it is better
to deal with them before investing a great
deal of time, money, and effort in the full
study (Mason and Zuercher, 1998). It has
also been argued that the current research
climate demands accountability from researchers, which means that there is a need
to ensure the best possible use of research
results (Crosswaite and Curtice 1994). We
would like to go one step further and argue
that researchers have an ethical obligation
to make the best use of their research experience by reporting issues arising from all
parts of a study, including the pilot phase.
Well-designed and well-conducted pilot studies can inform us about the best research
process and occasionally about likely outcomes. Therefore investigators should be
encouraged to report their pilot studies, and
in particular to report in more detail the actual improvements made to the study design
and the research process.

References
nd

Baker, T.L. (1994), Doing Social research (2

Edn.), New York: McGraw-Hill Inc.
Bowling, A. (1997), Research Methods in
Health: Investigating Health & Health
Services, Buckingham: Open University
Press. p 232.
Burns, N. and Grove, S.K. (1999), Understanding Nursing Research (2 nd edn.), Philadelphia: W.B. Saunders Company. p 40.
Crombie, I.K. and Davies, H.T.O. (1997),
Research in Health Care: Design, Conduct
and Interpretations of Health Services
Research, London: Wiley.

Crosswaite, C. and Curtice, L. (1994), Disseminating research results-the challenge of

bridging the gap between health research
and health action. Health Promotion
International 9: 289.
De Vaus, D.A. (1993), Surveys in Social
Research (3rd edn.), London: UCL Press.
Frankland, J. and Bloor, M. (1999), Some issues
arising in the systematic analysis of focus
group material, In: Barbour, R. and
Kitzinger, J. (eds) Developing Focus Group
Research: Politics, Theory & Practice,
London: Sage
Holloway, I. (1997). Basic Concepts for
Qualitative Research, Oxford: Blackwell
Science.
Hundley, V, Milne, J, Leighton-Beck, L. et al.
(2000). Raising research awareness among
midwives and nurses: does it work? Journal
of Advanced Nursing 31 (1): 78-88.
Lindquist, R. (1991). Dont forget the pilot
work! Heart Lung 20: 91-92.
Mason, D.J. and Zuercher, S.L. (1995). Pilot
studies in clinical nursing research. Journal
of the New York State Nursing Association
26: 11.
Muoio, R., Wolcott, L., and Seigel, H. (1995). A
win-win situation: The pilot program.
Journal of Continuing Education in
Nursing. 26: 230-233.
Office for National Statistics, General Register
Office for Scotland, Northern Ireland
Statistical & Research Agency. (1999), 2001
Census Information Paper, Government
Statistical Services: London.
Peat, J., Mellis, C., Williams, K. and Xuan W.
(2002), Health Science Research: A
Handbook of Quantitative Methods,
London: Sage.
Polit, D.F., Beck, C.T. and Hungler, B.P. (2001),
Essentials of Nursing Research: Methods,
Appraisal and Utilization. 5th Ed.,
Philadelphia: Lippincott Williams & Wilkins
Prescott, P.A. and Soeken, K.L. (1989), The
potential uses of pilot work. Nursing
Research 38: 60-62.
Rosenberg, K.M. and Daly, H.B. (1993),
Foundations of Behavioural Research: A
Basic Question Approach, Fort Worth/
London: Harcourt Brace College Publishers.
p 182-184.
Tashakkori, A and Teddlie, C. (1998), Mixed
Methodology: Combining Qualitative &
Quantitative Approaches, Sage

Teijlingen van, E., Rennie, A.M., Hundley, V.,

Graham, W. (2001), The importance of
conducting and reporting pilot studies: the
example of the Scottish Births Survey,
Journal of Advanced Nursing 34: 289-295

social research UPDATE

(ISSN: 1360-7898)
is published by

Department of Sociology
University of Surrey
Guildford GU2 5XH
United Kingdom.
Tel: 01483 300800
Fax: 01483 689551
Edited by Nigel Gilbert
(e-mail: [email protected])
Winter 2001 University of Surrey