Library and Information Science Research 39 (2017) 284–294

Contents lists available at ScienceDirect

Library and Information Science Research

journal homepage: www.elsevier.com/locate/lisres

Research methods: What's in the name? T

a,⁎ b
Heting Chu , Qing Ke
a
Palmer School of Library and Information Science, Long Island University, 720 Northern Boulevard, Brookville, NY 11577, United States
b
School of Information Management, Nanjing University, 163 Xianlin Road, Nanjing 210093, China

A B S T R A C T

Research methods are applied in all kinds of studies, though no consensus exists regarding what constitutes a
research method and how research methods should be categorized. Over 1900 research articles were obtained
from three major journals published between 2001 and 2010 in library and information science (LIS). Each
selected article was coded using a schema of research methods developed in this study. The coded data, along
with related publications, were analyzed qualitatively and quantitatively. This exploration shows that research
methods comprise data collection techniques (e.g., interview, observation) and data analysis techniques (e.g.,
qualitative, quantitative). Research methods should perhaps be categorized by data collection technique, as it
makes more sense than if research methods are labeled as qualitative or quantitative. This study is one of the
many efforts to facilitate a better understanding of research methods in LIS and help scholars make more in-
formed decisions about research method selection in their endeavors. Its implications can be extended to LIS
research education, training, and advocacy. Because research methods themselves are not discipline-specific,
researchers beyond the LIS field would benefit from this study as well.

1. Introduction strategy (e.g., empirical, conceptual). The temporal criterion is some-

times employed to have cross-sectional studies and longitudinal studies
Researchers need to consider and choose one or more methods for (Chen & Hirschheim, 2004). Case study appears in many research re-
their scholarly endeavors based on what they intend to study. This is ports as one type of research method (Bernhard, 1993; Gelber, 2013)
true for all scientific disciplines, including library and information although it only denotes that the research involves one or several cases
science (LIS). Although a good number of studies have explored the (i.e., sampling units) as research subjects.
topic of research methods adopted in the LIS field in past decades, three Third, assorted lists of research methods are developed in different
kinds of variations are commonly observed in practice and discourse. studies due to the two kinds of variations depicted above. Each study
First, a variety of terms other than research methods are used in the creates its own unique array of research methods unless, for example, a
literature to refer to this concept. Sample terms include research stra- later study adopts the same classification scheme developed for an
tegies (Järvelin & Vakkari, 1990), research frameworks (White & earlier project (e.g., Kumpulainen, 1991, used a scheme developed by
Marsh, 2006), research designs (Luo & McKinney, 2015), and research Järvelin & Vakkari, 1990).
methodologies (Hildreth & Aytac, 2007; Peritz, 1977). All of these variations are results of different understandings and
Second, different criteria are applied in categorizing research interpretations of what constitutes a research method and how research
methods. Some studies (Chu, 2015) categorize research methods by methods should be categorized. They can cause confusion or even be a
data collection technique (e.g., interview, observation, questionnaire) hindrance to LIS scholars in the selection and implementation of
while others (Powell, 1999) use data analysis technique (e.g., quanti- methods for research projects. Likewise, there are implications for LIS
tative, qualitative). It should be noted that the words methods and education in research methods. The present study attempts to examine
techniques are essentially used synonymously. In the present study, for those variations and suggest some measures for handling them.
the purpose of clarification, methods is used to cover the entire research
process while techniques is used to refer to specific procedures for data 2. Problem statement
collection or analysis. In most reported research, additional criteria are
often applied when categorizing research methods. For instance, There is apparently little consensus as to what constitutes a research
Järvelin and Vakkari (1990) group research methods by research method and how methods should be categorized. Determining the

⁎
Corresponding author.
E-mail addresses: [email protected] (H. Chu), [email protected] (Q. Ke).

https://doi.org/10.1016/j.lisr.2017.11.001
Received 1 September 2016; Received in revised form 12 May 2017; Accepted 13 November 2017
Available online 23 November 2017
0740-8188/ © 2017 Elsevier Inc. All rights reserved.
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

actual connotation of the term research method would help researchers public, without grouping them by any subcategory. Research metho-
differentiate research methods from other terms (e.g., research designs, dology in Hildreth and Aytac (2007), however, was defined as con-
research methodologies) often seen in research publications. Scholars taining research type (i.e., descriptive, exploratory, explanatory, and
studying research methods usually develop their own lists by adopting evaluative), data collection methods (e.g., survey questionnaire, survey
various categorization criteria. Typically, no two inventories are iden- interview) and data analysis approaches (e.g., quantitative, qualita-
tical unless a later study shares the research method categories formed tive). Research methodology (or methodologies) was taken to mean
in an earlier investigation. It may be feasible to choose one criterion for research methods in Peritz (1977) as well as in Hildreth and Aytac
categorizing all research methods so that a uniform list of research (2007). Likewise, Palvia et al. (2003), researchers in information sys-
methods can be created. Additionally, research methods are tradition- tems, treated research methodology as a synonym for research method.
ally labeled as qualitative or quantitative to conform to the qualitative From the perspective of Järvelin and Vakkari (1990) research
and quantitative research paradigms that have received much attention methods consist of research strategy (e.g., empirical, conceptual), data
in the scholarly community. Such a division may be reasonable, but it collection method (e.g., interview, observation, questionnaire), type of
also may not be able to withstand scrutiny. analysis (e.g., qualitative, quantitative), and type of investigation (e.g.,
LIS scholars now use a greater number and wider variety of research empirical, descriptive, comparative, conceptual). The same classifica-
methods in their inquiries than they did before (Chu, 2015; Park, 2004). tion scheme was followed in Kumpulainen (1991) as well as Hider and
Performing a systematic examination of research methods applied in Pymm (2008). Almost 15 years after the Järvelin and Vakkari (1990)
the LIS field will assist LIS researchers in understanding research study, a slightly modified classification scheme was applied in
methods. It would help researchers select appropriate methods when Tuomaala et al. (2014) where, for example, IR experiment was added as
conducting scholarly efforts. It might also enable LIS educators to de- a data collection method. Redundancy occurred among the four spe-
termine which research methods should be taught in degree programs, cified facets of research methods in these reports. For instance, citation
especially at the doctoral level. analysis was listed both as a research strategy and data collection
The current study focuses exclusively on research methods in the method. The same could be said about experiment and IR experiment,
field of LIS and addresses three research questions: found respectively under research strategy and data collection method.
Other identical pairs in dimensions relating to research methods in the
1) What constitutes a research method? classification scheme included qualitative method in research strategy
2) Which criterion is appropriate for categorizing and naming research and type of analysis, and empirical or conceptual in both research
methods? strategy and type of investigation. A noticeable degree of redundancy
3) Can research methods be classified exclusively as quantitative or was also observed between research strategy and data collection
qualitative? method, as well as between research strategy and type of investigation.
Research methods in the studies by Järvelin, Vakkari, and associates
3. Literature review had broad connotations that encompassed research strategy, data col-
lection method, type of analysis, and type of investigation.
Studies of the research dimension of scholarly publications cover Luo and McKinney (2015) considered research design (e.g., time
many variables in addition to research method, including author af- dimension, research framework/paradigm), research model (e.g., plat-
filiation, research topic, and the relationship between research topic form market model), and research theories (e.g., sense of community
and method. To a large extent, some of the variables belong to the theory) as separate elements. On the other hand, Ngulube (2015) re-
domain of bibliometrics and scientometrics, where key authors, core garded research methods and techniques as being synonymous; re-
publications, top research topics, and similar variables are examined search approach and design (e.g., interview, survey), research metho-
(Peritz, 1977). This study concentrated solely on the research method dology (i.e., quantitative, qualitative, mixed), and research paradigm
component. (e.g., positivism, pluralism) were among the terms chosen for ex-
Studies on research methods have typically been conducted through amining research. Taking a different approach, Feehan, Gragg,
content analyses of a set of research publications, somewhere between Havener, and Kester (1987) focused on research methods and analytical
100 and several hundred, from selected journal titles. The time span methods, and Gelber (2013) and Greifeneder (2014) adopted a similar
involved usually covers several years either consecutively (e.g., practice.
Hildreth & Aytac, 2007) or selectively (e.g., Tuomaala, Järvelin, & Data collection techniques and data analysis techniques are the two
Vakkari, 2014). Occasionally, non-journals have been chosen as data components present in most, if not all, studies on research methods. All
sources, including textbooks (Bernhard, 1993) and dissertations (Blake, other terms were employed either as a synonym of research method
1994). Some studies focus on a single journal title (Luo & McKinney, (e.g., research strategy in Järvelin & Vakkari, 1990) or to represent
2015; Ngulube, 2015) while others cover several (Chu, 2015; dimensions other than research methods (e.g., research paradigm).
Greifeneder, 2014) or a few dozen (Peritz, 1977; Tuomaala et al.,
2014). Turcios, Agarwal, and Watkins (2014) examined 105 titles, but 3.2. Categorization of research methods
included only the latest issue of each title in the chosen time period in
their study. Multiple categorizing conventions have been applied to research
Studies of this nature have not only been done in the broadly de- methods. Data collection technique, data analysis technique, research
fined discipline of LIS, but also in specific areas such as information paradigm, research design, and more have all been used as criteria in
behavior (Greifeneder, 2014; Julien, Pecoskie, & Reed, 2011), knowl- categorizing research methods. Usually, more than one criterion has
edge management (Ngulube, 2015), technical services (Gelber, 2013), been selected in creating a taxonomy or list of research methods.
information systems (Palvia, Mao, Salam, & Soliman, 2003), and health Peritz (1977) identified 11 research methods used in a selected set
informatics (Dimitroff, 1992). of LIS documents: bibliometric and similar studies, comparative studies,
content analysis, descriptive bibliography, historical methodologies,
3.1. Research methods and associated concepts information system design, secondary analysis, surveys on the public,
surveys or experiments on libraries and others, theoretical-analytic, and
Different scholars have different understandings and interpretations other and multiple as one category for studies whose methods did not
of what constitutes a research method. Peritz (1977), in one of the early fit into one of the existing categories (p. 49). As Peritz (1977) is one of
studies on the topic, identified 11 research methodologies, including the early studies on the topic, her list of research method categories
theoretical-analytic, information system design, and surveys on the varies from that of more recent studies (Järvelin & Vakkari, 1990;

285
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

Turcios et al., 2014). Some of her research methods were named after technique, IR experiment, was introduced in Tuomaala et al. (2014).
data collection techniques (e.g., content analysis, surveys on the public) Questionnaire and interview have been grouped as one category be-
whereas others were based on research target (e.g., descriptive biblio- cause both are considered to be synonymous with survey. The "not
graphy, information system design) and data analysis technique (e.g., applicable" category accounted for 43% of all the research publications
comparative studies). Peritz uniquely divided surveys into two cate- coded by Järvelin and Vakkari (1990) and 24% coded in Tuomaala
gories: surveys on the public and surveys or experiments on libraries et al. (2014).
and other settings. Unlike many researchers, Hildreth and Aytac (2007) did not set up
About a decade after the Peritz (1977), Feehan et al. (1987) pro- research methods as a variable in their study. Instead, they recorded the
duced another list of 11 research methods: bibliometrics, content ana- following data collection methods: bibliometric and citation analysis,
lysis, Delphi method, experimental, historical, observation and de- case study, computer log analysis, content analysis, Delphi study, ex-
scription, operations research, secondary analysis, survey, multiple, and periment, observation, semi-structured interview, survey interview,
other. Most of the research methods on the list were derived from data survey questionnaire, and others. Case study, as explained before,
collection techniques (e.g., bibliometrics, Delphi method), although concerns only the number of sampling units chosen and has no direct
operations research is more a technique for data analysis. Multiple relation to data collection method. The authors also specifically placed
methods became a separate category in similar studies before re- surveys into two categories: survey interview and survey questionnaire.
searchers began counting each research method individually (Chu, Semi-structured interview is listed as a data collection method separate
2015) or as primary and secondary (Palvia et al., 2003). Both Dimitroff from the survey interview, which might serve as a synonym for struc-
(1992) and Gore, Nordberg, Palmer, and Piorun (2009) adopted the tured interview in Hildreth and Aytac (2007).
same research methods taxonomy by Feehan et al. (1987) in examining Additional taxonomies of research methods can be found in other
research publications of health informatics. studies (Bernhard, 1993; Blake, 1994; Turcios et al., 2014; Williams &
As noted earlier, the study by Järvelin and Vakkari (1990) had a Winston, 2003). Usability was categorized as a research method in
significant impact on research of a similar nature, as their classification Turcios et al. (2014). Usability, like evaluation method in Järvelin and
scheme was adopted in Kumpulainen (1991), Hider and Pymm (2008), Vakkari's (1990), is simply a research objective; other techniques ought
and Tuomaala et al. (2014). Some components of the research method to be used for collecting data in usability studies.
taxonomy Järvelin and Vakkari (1990) developed can be observed in Rather than using research publications as source data, which ap-
other studies (Hildreth & Aytac, 2007) and their taxonomy is regularly pears to be the norm in the majority of related studies, Park (2004)
referenced in related publications. Table 1 displays the lists of research compared LIS curricula for research methods in Korean and U.S. uni-
strategy and data collection methods developed by Järvelin and versities, showing that action research, bibliometrics, case study,
Vakkari (1990). comparative study, content analysis, Delphi study, desk research or
According to Järvelin and Vakkari (1990), “research strategy is an literature review, ethnography, evaluative research, experiment, field
overall approach to the study within which the decisions concerning study, focus groups, historical method, information system design, in-
data collection and the type of analysis are made” (p. 399). In this terview, observation, and survey or questionnaire were covered in
context, research strategy seems to determine research method selec- those courses. This 17-item list of research methods is the longest one
tion. In Table 1, the empirical research strategy category bears a strong among all those reviewed. Some of the research methods, such as field
resemblance to the data collection methods shown at the right side of study and focus groups, did not appear in other studies, while case
the same table. However, the former comprises methods other than study, comparative study, evaluative research, and information system
data collection techniques. For example, case research method, more design were not categorized according to data collection technique. In
commonly known as case study, implies a small number of cases or addition, desk research or literature review is considered a step in the
sampling units, as opposed to a sizeable sample or an entire population, research process, rather than a technique for data collection.
are studied in the research. Any data collection technique can theore- Most studies fall into the realm of LIS, though information systems is
tically be selected for conducting a case study. Also in Table 1, eva- a related domain in which studies on research methods are published.
luation method refers to the research objective being used for evalua- Palvia et al. (2003) came up with 13 research methods used in man-
tion. Similarly, qualitative method indicates that data collected in a agement information systems research. Another method, content ana-
study are analyzed qualitatively. The results in Table 1 were obtained lysis, was added later to the list (Palvia et al., 2004). The taxonomy of
by applying multiple criteria to categorize research methods. 14 research methods, shown in the left column of Table 2, was applied
In the list of data collection methods in Table 1, one additional in Palvia, Pinjani, and Sibley (2007). Avison, Dwivedi, Fitzgerald, and

Table 1 Table 2
Research strategies and data collection methods in Järvelin and Vakkari (1990). Research methods categorized in two studies.

Research strategy Data collection method Palvia et al. (2007) Avison et al. (2008)

Conceptual research Citation analysis Case study Action research

Concept analysis Content analysis Content analysis Case study
Verbal argumentation or critique Historical source analysis Field experiment Content analysis
Empirical research Observation Field study Ethnography
Case or action research method Other methods of collecting Frameworks and conceptual Field experiment
Citation analysis Questionnaire, interview models
Content or protocol analysis Several methods of collecting Interview Grounded theory
Evaluation method Think aloud Laboratory experiment Interview
Experiment Use of data collected earlier Library research Laboratory experiment
Historical method Not applicable Literature analysis Library research/literature analysis/
Other bibliometric method frameworks/conceptual modeling
Qualitative method Mathematical model Mathematical model
Survey method Qualitative research Secondary analysis
Other empirical method Secondary data Speculation/commentary
Literature review Speculation/commentary Survey
Mathematical/logical method Survey Multi-method
System and software analysis and design Others

286
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

Powell (2008) adapted this list of research methods in their study of social science disciplines. For example, ethnography, interview, and
information systems. The latter is displayed in the right column of historical method were commonly considered qualitative methods,
Table 2. while experiment and questionnaire were usually placed in the category
Case study, library research, literature analysis, and qualitative re- of quantitative methods. After reporting that research choosing ex-
search, from the list by Palvia et al. (2007), do not suggest any data periments and modeling as research methods was increasing, Blake
collection techniques and cannot be regarded as research methods. (1994) concluded that the assumption of equating quantitative methods
According to those authors' definitions, library research and literature and the scientific method should be re-examined. The author also in-
analysis are synonymous, although they were listed separately. Palvia dicated that “newer qualitative methods can play a significant role” in
and colleagues subdivided the category of experiment into field ex- LIS research (p. 38). Hider and Pymm (2008) observed in their study
periment and laboratory experiment in order to better reflect the that qualitative research, which was often conducted through inter-
characteristics of each method. A more suitable term for the secondary views, gained a modest rise. They further commented that “an increase
data category might be secondary analysis; data alone cannot be a in the use of more sophisticated qualitative research methods has been
technique for gathering data. balanced by the decrease in historical research and by an increase in
Although Avison et al. (2008) adapted their list of research methods experimentation” (p. 112). Interview and historical method in Hider
from Palvia et al. (2007), the modified list contained unique items that and Pymm's study were labeled as qualitative research methods. In a
represented either additional research methods, such as action re- more recent research project by Turcios et al. (2014), interview was
search, or research paradigm, such as grounded theory. The multi- again defined as a qualitative research method, based on the definition
method and others categories commonly appear in many research from Beck and Manuel (2008). Greifeneder (2014) likewise denoted
methods taxonomies. The former is a category for those studies that use that qualitative methods, including interviews, content analysis, ob-
more than one method, while the latter includes any method that does servation, and focus groups, still dominated information behavior re-
not fit into an existing category. In comparison with related studies in search.
LIS, research in information systems tends to choose conceptual mod- The present study conducts a content analysis of related research
eling or mathematical modeling as a method. Even though conceptual and documents from three major LIS journals published between 2001
or mathematical modeling is used more for data analysis, this method is and 2010 to explore the classification and naming of research methods
mostly absent from the field of LIS. adopted in LIS studies, with particular attention to finding out if it is
Collecting data from three journals spanning information systems suitable to categorize research methods as qualitative and quantitative.
and health informatics, Davies (2012) developed the following list of
data collection techniques: archive of e-mails, electronic medical re-
4. Data collection and analysis
cords (EMRs), experiments, focus groups, freely available data, inter-
views, logs of user activity, observation, questionnaires, secondary data
Research articles published between 2001 and 2010 were obtained
based on Eldredge (2004), Palvia et al. (2003) and other related re-
from Journal of Documentation (JDoc), Journal of the American Society for
search. This list is distinctive for its emphasis being placed solely on
Information Science & Technology1 (JASIS&T), and Library and Informa-
data collection techniques, along with some types of data sources, such
tion Science Research (LISR). Editorials, literature reviews, book reviews,
as archive of e-mails. All the data sources on the list (i.e., archive of e-
letters to the editor, and any other non-research articles were excluded
mails, EMRs, freely available data, and secondary data) could be used
in the data collection.
for content analysis, although that data collection technique was not
Journals were chosen as data sources for this study based on their
included on the list.
status as core research journals in LIS. They have also been selected by
Despite the different practices in categorizing research methods
previous studies on research methods (Fidel, 2008; Järvelin & Vakkari,
among the studies reviewed above, one theme clearly emerges: research
1990).
methods are usually categorized by data collection technique. Other
The data collection yielded 1981 research articles from the three
criteria, such as research objective and data analysis technique, may not
chosen journals, a number higher than that recorded in any related
be selected either individually or jointly for categorizing research
studies. Table 3 details the frequency distribution of research articles by
methods.
journal.
A coding schema of research methods used in LIS was developed
3.3. Quantitative or qualitative research methods?
(Table 4) after coding all the research articles from JDoc and LISR that
were selected for this study. The resulting schema was then revised by
In recent decades, a growing interest in qualitative research has
coding research methods reported in all the JASIS&T articles in two
developed in the scholarly world. However, qualitative research has not
time periods: 2001 to 2002 and 2009 to 2010. The grounded theory
been defined and a determination of whether research methods can be
approach (Glaser & Strauss, 1967) was used to create the coding
categorized as qualitative or quantitative has not been made. Fidel
schema; the schema development was grounded on data gathered from
(1993) conducted an extensive review of qualitative research on in-
the selected research articles rather than using any existing categor-
formation retrieval by summarizing its characteristics: open, holistic,
ization. The schema development and the coding process were inter-
flexible, noncontrolling, and case oriented. Fidel (1993) also de-
twined and iterative. The abstract and methodology section of each
termined that there was no agree-upon definition for the concept of
article were examined to see what data collection techniques were
qualitative research.
applied in the research reported. If necessary, other sections of the ar-
Researchers often categorize data analysis techniques as quantita-
ticle were consulted in the coding process. The identified data collec-
tive, qualitative, or other. Such categorizations can be found in Feehan
tion techniques were all recorded and eventually grouped into cate-
et al. (1987), Hildreth and Aytac (2007), Järvelin and Vakkari (1990),
gories that became individual research methods listed in the coding
as well as studies that were modeled after theirs. Generally speaking,
schema (Table 4). Further elaboration on the coding schema develop-
data analyses that involve statistics are labeled as quantitative, while
ment is provided in the next section.
those without statistical analysis are marked as either qualitative or
Coded results by two coders for a randomly selected sample of 30
non-quantitative (Gore et al., 2009).
articles, 10 from each of the three journals, were compared for
Previous studies frequently categorized research methods, com-
prising both data collection and analysis techniques, as qualitative or
quantitative. This tradition was well documented in Powell (1999) 1
ASIS&T changed its name from American Society for Information Science &
when he explored trends in the use of research methods in LIS and other Technology to Association for Information Science & Technology in 2012.

287
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

Table 3 such as naturalism, is more meaningful at the conceptual level than

Frequency distribution of research articles by journal 2001–2010. when directly linked to a specific data collection technique. Thus, the
criterion for categorizing and naming research methods is chosen be-
JDoc JASIS&T LISR Total
tween data collection technique and data analysis technique. It is up to
367 1373 241 1981 the researcher to determine which one is more appropriate.
When explaining the etymology of the term “method”, Glazier
(2010) indicates that method is the pursuit after some end or a proce-
Table 4 dure employed to attain a certain end. In this case, the end is the data to
Coding schema for research methods.
be collected and the method is the means (p. 32). Experiment, inter-
Bibliometrics (including citation analysis, informetrics, & scientometrics) view, observation, and questionnaire are some means researchers use to
Content analysis (including discourse analysis, & secondary analysis) achieve the end of data collection. The coded data in the present study
Delphi study uniformly show that, for instance, scholars use questionnaires for data
Ethnography/field study
collection in questionnaire research. Investigators conduct experiments
Experiment
Focus groups to gather data in experimental research. Therefore, data collection
Historical method technique is selected for categorizing and naming research methods
Interview listed in the coding schema (Table 4). Data analysis technique should be
Observation reserved to distinguish between qualitative research and quantitative
Questionnaire (including index, inventory, scale, & test)
research.
Research diary/journal
Theoretical approach (e.g., conceptual analysis, modeling, & theory building)
Think aloud protocol 5.2. Research methods in the coding schema
Transaction log analysis
Webometrics (including link analysis, cybermetrics, & altmetrics)
Other methods (e.g., action research, card sorting, information horizon)
More than 15 research methods were identified and named after
corresponding data collection techniques (Table 5). Because each
method in any multi-method study was counted once in the tally, the
consistency. The intercoder agreement rate was 86.7%, exceeding the total number of research methods in each column of Table 5 exceeds the
acceptable-to-most rate of 80% (Neuendorf, 2002, p. 143). This inter- number of research articles from each journal. All the research methods
coder agreement rate also indicates the reliability of both the coding in Table 5 are described below and presented alphabetically by their
schema and coding process. Where different coding results were ob- names for easy reading.
tained, the two coders discussed and reached a consensus. This con-
stituted the first phase of data collection. In the second phase, the au- 5.2.1. Bibliometrics
thors coded the remaining six years (i.e., 2003–2008) of JASIS&T Bibliometrics is a method used for collecting publication and cita-
publications by using the same coding schema and following the same tion data. It is commonly applied in LIS research and is ranked the
coding procedure. The intercoder agreement rate for all the coded ar- second most frequently reported method in JASIS&T. In the present
ticles between the two was 91.7%, which surpasses the acceptable-to- study, bibliometrics also includes citation analysis, informetrics, and
all rate of 90% (Neuendorf, 2002, p. 143). scientometrics because they all share the same type of data: publication
If a study used more than one method, each method was recorded in and citation data. It is the research focus that differentiates one from
the order in which it was reported in the article. If a study adopted the the other. When the research focus is on citations, it is called citation
true experimental design in the form of experimental versus control analysis. When the research focus is on information, it is called infor-
groups with a pre-test, treatment, and post-test, it usually would em- metrics. When the research focus is on sciences, it is called sciento-
ploy at least another research method, such as a questionnaire or in- metrics. This method is similar to content analysis if the data collected
terview, for performing the pre-test and post-test. That study would comprise bylines, acknowledgments, and other similar components of
then be coded once as experiment and again for the other research
methods, according to what was employed for data collection. In con- Table 5
trast, research using quasi- or pre-experiments was simply coded as Frequency distribution of research methods used in three all LIS journals.
experiment, without additional methods given the nature of such ex-
Research method JASIS&T JDoc LISR Total
periments. No weights were assigned to any of the multiple methods (N = 1373) (N = 367) (N = 241)
applied in a single study because this would introduce an element of
subjectivity to the coding process. The collected data were then ana- Experiment 482 49 21 552
lyzed quantitatively and qualitatively. Bibliometrics 270 31 14 315
Questionnaire 247 51 68 366
Content analysis 215 52 73 340
Theoretical 195 141 36 372
5. Findings approach
Interview 145 49 48 242
5.1. Coding schema of research methods: some decision points Transaction log 86 18 9 113
analysis
Observation 63 11 15 89
Few of the many lists of research methods in LIS were developed Webometrics 57 9 6 72
according to a single criterion. Using multiple criteria in classifying Think aloud 35 5 3 43
research methods is common practice. In order to follow the two es- protocol
sential principles of being collectively exhaustive and mutually ex- Focus groups 29 6 9 44
Ethnography/field 16 3 2 21
clusive in any categorization or classification efforts, a single criterion study
should be employed when categorizing research methods. Research diary/ 13 5 2 20
Additionally, as shown in the literature review and data collection, journal
research methods comprise data collection techniques and data analysis Delphi study 9 0 2 11
Historical method 8 7 2 17
techniques. All other concepts are not guaranteed to be considered in
Other 10 0 0 10
every research publication, and individually such concepts go beyond Total 1880 437 310 2627
the boundary of research methods. For instance, a research paradigm,

288
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

written publications. 5.2.5. Experiment

Experiment is an established method for collecting data by fol-
5.2.2. Content analysis lowing a procedure to test what is studied in either a laboratory or field
Content analysis refers to collecting data by conducting systematic setting, corresponding to laboratory experiments and field experiments
examination of texts or other passages in the contexts of their use described in Palvia et al.’s (2007) list of research methods. Experiment
(Krippendorff, 2004, p. 18). This study considers content analysis as a is the most often used method (a frequency of 482) in the 1373 JASIS&T
data collection technique, although it can also be used for data analysis. publications. However, experiments in JASIS&T, JDoc, and LISR usually
When it is adopted as a data collection technique, researchers analyze differed from the experimental designs Campbell and Stanley (1966)
the chosen data source in order to gather the necessary data for a re- depicted because such experiments were conducted mainly for evalu-
search project. There are two things to point out regarding the word ating new procedures (e.g., key-phrase extraction), algorithms (e.g.,
“analysis”. First, analysis includes abstraction, comparison, synthesis, search result ranking), or systems (e.g., digital libraries) reported in the
and other analytical techniques applied in the process. For this reason, publications. In LIS, experiments rarely take the classic design of ex-
comparative study is not listed as a research method separately from perimental and control groups with pre-test, treatment, and post-test.
content analysis. Second, analysis in this and other research methods, Rather, experiments are normally performed in a laboratory or simu-
such as citation analysis and transaction log analysis, means that ana- lated environment with one or more of the experiment essentials and
lysis is performed for data collection purposes. therefore fall under the quasi- or pre-experiment category. In future
Content analysis can be further divided into two types: manifest and studies, experiments should be further broken down into classic and
latent. Manifest content analysis denotes that what needs to be ana- non-classic designs rather than considered as a single research method.
lyzed is physically present, observable, and countable, which is easier This would better represent the range of experiment as a research
than latent content analysis. Analyzing what research methods are used method in LIS.
in a scholarly publication is an example of manifest content analysis. In addition, experiment is a method often employed in conjunction
Latent content analysis implies that what needs to be analyzed is hidden with other data collection techniques. Typical implementations include
(underlying), conceptual, unobservable, and uncountable. An example the use of questionnaire or interview for pre-test and post-test, ob-
would be finding out what research paradigm or theoretical origin is servation for collecting behavioral data, and think aloud protocol for
applied in a research publication. Latent content analysis, when con- gathering cognitive data during the experiment. Experiment is therefore
trasted with the manifest counterpart, is more difficult. a meta-method.
In the present research, content analysis also includes discourse
analysis and secondary analysis, since these can be employed for data 5.2.6. Focus groups
collection by examining some kind of data source. The data source As a research method, focus groups refer to data collection via
defines the specific type of analysis. For example, a research method is discussion of a research problem between a moderator and a group of
called discourse analysis if the target source is discourse data, such as participants. Several groups are usually invited to participate in one
interview transcripts. Similarly, a research method is called secondary study, hence the plural form being used to refer to the method. The
analysis if the target source is secondary data, namely data collected focus group method is used a total of 44 times in the 1981 articles from
previously for other purposes. Census data are a common example of the three journals, close to the lower end of usage frequency (Table 5).
secondary data. Meta-analysis is a type of secondary analysis or content Focus groups are sometimes characterized as focus-group interviews
analysis because it is used for gathering data, usually statistical in (Kim & Kim, 2008) or group interviews (Agosto & Hughes-Hassell,
nature, from individual studies to achieve research objectives. Content 2006). However, the emphasis of focus groups is not placed on question
analysis is widely used in LIS; it ranked first, second, and fourth re- and answer, as is the case in interviews. Instead, the emphasis is placed
spectively in LISR, JDoc, and JASIS&T in terms of usage frequency on discussion and interaction among group members.
(Table 5).
5.2.7. Historical method
5.2.3. Delphi study Historical method played a dominant role in LIS research until the
The Delphi method is generally used for collecting data with a 1980s (Feehan et al., 1987; Järvelin & Vakkari, 1990). Historical
questionnaire from a group of experts to address a research problem in method refers to collecting data by examining, synthesizing, summar-
order to reach consensus and make forecasts via several rounds of ex- izing, and interpreting existing published and unpublished materials
changes. Unlike some other research methods, including content ana- related to a historical research problem. Historical method bears some
lysis and theoretical approach, a Delphi study can be easily identified in similarity to content analysis because both methods analyze the content
the coding process of this research because of its distinctive features, of materials for data collection purposes. Historical method differs from
such as the selection of experts and the conducting of several rounds of content analysis because it deals exclusively with problems of a his-
data collection using a questionnaire. Strictly speaking, it is a con- torical nature.
tingent research method that relies upon questionnaires and cannot be
used alone. The Delphi method, although well known, appears only 5.2.8. Interview
nine times in JASIS&T and twice in JDoc. Interview is a data collection technique where individual partici-
pants are asked questions relating to a research problem. There are
5.2.4. Ethnography and field study three kinds of interviews, differentiated by how interview questions are
Ethnography and field study share many characteristics in data prepared and whether probe or follow-up questions are allowed. If the
collection. Both can be applied when collecting data using multiple researcher has a prepared list of questions for the interview, it is called
techniques, such as observation and interview, in a natural setting structured interview. Structured interview is mostly analogous to
where participants live or work. Therefore, they are grouped as one questionnaire. If the researcher does not use a prepared list of questions
type of research method in this study. In addition, there is no fixed, and formulates specific interview questions on site, it is called un-
specific set of data collection techniques for either ethnography or field structured interview. If the researcher asks some questions from the
study. Instead, researchers select individual data collection techniques prepared list, but also uses some probe or follow-up questions, it is
according to their research goal. In the three LIS journals explored in called semi-structured interview. Interview is respectively ranked the
this study, a total of 21 studies chose this research method, compared to third, fifth, and sixth most frequently used research method in LISR,
none in earlier investigations of a similar nature (Feehan et al., 1987; JDoc, and JASIS&T. Although this study does not differentiate inter-
Järvelin & Vakkari, 1990; Peritz, 1977). view method by type, semi-structured interview is the most frequently

289
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

adopted because of its flexibility in application. research strategy of conceptual analysis in their study, while Bernhard
(1993) enumerates theory development in her taxonomy of research
5.2.9. Observation methods. In the current study, in terms of usage frequency, theoretical
Observation is a method for gathering data via carefully and at- approach is the top used research method in JDoc, the fourth in LISR,
tentively watching and making notes on the subject being studied. All and the fifth in JASIS&T.
five senses of the observer need to be used in data collection (Baker, Theoretical approach is somewhat similar to content analysis,
2006). Observation can be made in a lab or in the field. Common especially latent content analysis. The distinctive feature of this ap-
subcategories of observation include obtrusive or unobtrusive and proach lies in its focus on “theoretical” in that researchers perform
participant or nonparticipant. Observation as a stand-alone method is conceptual or theoretical analysis according to existing theories,
not regularly employed, totaling 89 times of usage in the 1981 research models, and the like. In some studies that adopt the theoretical ap-
articles examined in this study. proach, the ultimate objective is to develop a theory or model in a re-
As noted earlier, observation is often performed as part of meta- spective domain. Notably, the theoretical approach is not considered a
methods, such as experiment and field study. In the case of classic ex- review because the latter only does evaluation without any further
periments, observation is a must-have technique for collecting behavior conceptual analysis, modeling, or theory building.
data before and after the treatment is administered. In order to obtain
data for comparing the before-after and old-new differences, some form 5.2.13. Think aloud protocol
of observation is necessary in non-classic experiments. Ethnography Think aloud protocol is a research method intended to collect data
and field study often use observation as one of multiple data collection about participants' cognitive activities via the verbal reports of their
techniques. Consequently, observation is often discussed in the context thoughts, called think alouds, while taking part in an experiment or
of ethnography and field study (Baker, 2006; Fox, 1998; Spradley, performing some task (Ericsson & Simon, 1980). This method origi-
1980). In addition, observation is also used when applying such data nated in cognitive psychology, but has been adopted in recent decades
collection techniques as interview and focus groups, though observa- for research in LIS, especially in information seeking. Like the Delphi
tion would not be explicitly specified as a separate method in those method, think aloud protocol cannot be used alone. It is instead con-
studies. For example, the interviewer would closely observe any non- tingent on experiments of certain kinds. This method is used a total of
verbal language of the interviewee during the interview process. 43 times for data collection in the three journals.

5.2.10. Questionnaire 5.2.14. Transaction log analysis

Questionnaire, often known as survey, is a technique for data col- Transaction log analysis, as a research method, gains momentum
lection using a predefined list of questions. A questionnaire can com- when computerized systems are used for information processing and
prise both closed-ended and open-ended questions, although in most access. It refers to researchers gathering data by analyzing transaction
cases the former make up the majority or the only type of questions in logs that are automatically captured at either the server or client side.
the data collection instrument. Questionnaire was reportedly the most Transaction logs that interest LIS researchers include search logs (e.g.,
used research method in many previous LIS studies (Blake, 1994; search terms entered) and system usage data (e.g., abstracts viewed,
Turcios et al., 2014). Questionnaire is the second most used research articles downloaded). Captured screens can sometimes be grouped with
method in LISR and the third most commonly used research method in transaction logs since the screen displays interaction data between the
both JDoc and JASIS&T (Table 5). Structured interview is considered user and system. As shown in Table 5, due to the development of data
one form of survey. It is perhaps for this reason that some researchers capturing applications, transaction log analysis increased from being a
(Järvelin & Vakkari, 1990) group questionnaire and interview as one little used research method to one with a moderate usage frequency of
type of research method: survey. Certain scales (e.g., the Library An- 113 times.
xiety Scale), indexes, inventories, and tests (e.g., personality test) can
also be regarded as a type of questionnaire and are therefore placed in 5.2.15. Webometrics
this category of research method in this study. Furthermore, ques- Webometrics is defined as bibliometrics in the web environment,
tionnaire is regularly used for pre-test and post-test when conducting an where webpages and websites are generally regarded as publications;
experiment. Questionnaire is also a prerequisite technique for carrying with inlinks (i.e., links a webpage or site receives) being considered as
out the Delphi study, which is a contingent research method. In these citations and outlinks (i.e., links a webpage or site makes to others)
two cases, questionnaire may not be considered a separate data col- being considered as references. The web has its own citation or link
lection technique in the selected research publications this study ex- indexes in the form of commercial search engines, ready for researchers
amined. to explore (Thelwall, 2008). Webometrics became a research method
for collecting data from the web via a search engine in the late 1990s
5.2.11. Research diary or journal and early 2000s. In the present study, there were 72 occurrences of the
Research diary or journal is a technique used to gather data about webometric method (Table 5).
events, activities, thoughts, reflections, or other aspects by an in- As indicated in Table 4, webometrics in the context of this study also
dividual who keeps the diary over a period of time. In recent years, includes link analysis, cybermetrics, and altmetrics. A study uses the
information seeking behavior has become a common topic in the LIS link analysis method if it collects only link data. The Internet, of which
field. Information seeking behavior sometimes relies on subjects the web is a part, is often referred to as cyberspace. In that sense, cy-
keeping a research diary for data collection. A research diary can be bermetrics encompasses webometrics, but now webometrics has su-
structured (e.g., using a form), semi-structured, or unstructured (i.e., perseded it as the preferred term. Other metric data (e.g., article
open-ended with no particular format requirement). Compared with downloads, views, mentions) are increasingly available from academic
other research methods listed in Table 5, research diary is employed social media (e.g., academia.edu, Mendeley), digital libraries, and other
less often, with a total usage frequency of 20 in this study. similar platforms. Altmetrics, short for alternative metrics, emerged in
recent years as a research method for collecting such data, differ-
5.2.12. Theoretical approach entiated from traditional bibliometric or webometric data. Although the
Theoretical approach, as a research method, is a technique for current research places altmetrics in the category of webometrics, fu-
gathering data through conceptual analysis, theoretical examination, or ture taxonomy of research methods may list altmetrics separately from
similar activities. Although few studies on research methods choose the webometrics due to a growing number of studies adopting altmetrics as
same term for this technique, Järvelin and Vakkari (1990) identify a a research method.

290
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

Table 6 data collection technique a case study may use. When using a case
Frequency distribution of other research methods. study, a researcher can choose any method that suits the study's
purpose.
•
Research method Frequency
Grounded theory: According to Bawden (2012), “grounded theory
Action research 2 is, notoriously, not a research method in itself, but rather a general
Card sorting 4 strategy” (p. 156). Grounded theory refers to a study being con-
Drawing 2
ducted without adopting an existing framework or theory, with the
Information horizon 1
Photo survey 1 intention being to ground the study's work on the data being col-
lected and analyzed. Typically, the appropriate methods for a given
study are determined by the problem it explores, but not if the study
uses the grounded theory approach. The grounded theory approach
5.2.16. Other methods does not suggest any particular research method or data collection
In addition to the 15 research methods described above, there are technique for a study.
five data collection techniques reported in 10 of the 1373 JASIS&T • Library research: Also known as desk research, library research re-
research articles this study examines. Because of their low usage fre- fers to a step or component, namely literature search, in the research
quencies (Table 6), these methods are merged into one category called process. Library research itself does not imply any research method
other methods. or data collection technique. In all studies, researchers are expected
to conduct a literature search in order to learn more about the
• Action research can be considered similar to ethnography or field history of their chosen research topic.
study in that it also uses multiple data collection techniques in a real • Longitudinal study: This refers to research where temporal coverage
setting. Nevertheless, action research always intends to solve an expands over a period of time. Its counterpart is the cross-sectional
immediate problem or to produce guidelines for best practice. It study. The longitudinal study deals solely with the temporal scope
relies on techniques for collecting data that would actuate changes or coverage of a study and provides no natural linkage to any re-
or actions in the target setting or environment. Such techniques search method. A longitudinal study can employ any technique for
include participant observation, interview, and focus groups (Berg, data collection.
2009, p. 258).
• Card sorting is a technique for collecting data by asking participants In sum, data collection technique might be the only plausible cri-
to put cards of assorted entities (e.g., concepts, website headings) terion for naming and categorizing research methods. Other criteria are
into groups. Through this technique, the researcher can gather data appropriate for categorizing research strategies or paradigms (e.g.,
about how participants categorize the given entities. grounded theory), research processes (e.g., library research), research
• Drawing, as a data collection technique, asks participants to sketch types (e.g., sampling size, temporal scope), or data analysis techniques
what they see, feel, think, and do. This technique is often used in (e.g., quantitative, qualitative). For this reason, multiple methods are
conjunction with other research methods to help the researcher not selected as a type of research method in this study.
gather visual data.
• Information horizon, a technique used mainly in information 6. Discussion
seeking behavior research, aims to gather data from participants
about their usage and ranking of individual information resources Two distinctive themes emerge from this study. One theme is that
(e.g., colleagues, friends, Internet, library, personal collection). Each data collection technique might be the most appropriate criterion for
participant first draws a large circle as the information horizon and naming and categorizing research methods. The other theme demon-
then places small circles on the large one, each representing one strates that research method cannot be labeled as purely qualitative or
information source used in information seeking. Finally, the parti- quantitative.
cipant ranks those resources to indicate their preferences for each.
• The photo survey technique serves as the researcher's visual note- 6.1. Naming and categorizing research methods by data collection technique
book during data collection. Photo survey researchers use a camera
to photographically record the scenes and surroundings of their In many studies, multiple criteria have been applied for developing
study targets. The photo survey is always supplemented by ob- a list or taxonomy of research methods. The current study proposes that
servation and other techniques to make up the complete research data collection technique might be the most appropriate criterion for
methodology. naming and categorizing research methods.
First, categorizing research methods or developing a taxonomy for
There are more research methods, such as eye tracking, than pre- them is basically an exercise of classification and should thus follow the
sented above. As this study is grounded on the data gathered from the established principles that guide all such efforts: of being collectively
three LIS journals, any research method not reported in the 1981 exhaustive and mutually exclusive. This means that a single criterion
publications is not considered. should be employed in the categorization of research methods. If re-
search methods were categorized according to more than one criterion,
5.3. What a research method is not the end results would be neither mutually exclusive nor collectively
exhaustive. Quite a few such examples were reviewed earlier in this
As demonstrated in Table 4 and according to all the findings re- study.
ported so far, the following may not be considered research methods for Second, determining what could be chosen as the criterion for ca-
this study's rationale: tegorizing research methods varies among studies. Some studies select
criteria other than the data collection technique for categorization.
• Case study: Although some prior publications list case study as a Research methods comprise data collection techniques and data ana-
research method, this study finds that case study only implies a lysis techniques. Any criterion that is not part of research methods
small number of cases or sampling units, typically one or several, as would not be a proper criterion for categorization purpose. It is also
opposed to a sizeable sample or the entire population, are studied in unsuitable to select data analysis technique as a criterion for categor-
the research. There is no indication about which research method or izing research methods for reasons given below.

291
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

Third, except in the cases of content analysis and theoretical ap- another method. Such cases only account for a small percentage of all
proach, data must be first collected before being analyzed.2 The type of the research publications considered. Many articles included in this
data collected dictates the kind of technique used for analysis purpose. study reported the use of multiple methods. This reinforces the need to
This sequence indicates why data collection technique, rather than data use data analysis technique as a classification principle.
analysis technique, is the most appropriate criterion for categorizing
research methods. 6.3. Applicability of research methods

Is there any prerequisite for the research methods explored in this

6.2. No research method is entirely qualitative or quantitative
study to be applied in LIS research? Table 8 presents a taxonomy of
research methods by applicability in order to facilitate a better under-
The data analysis technique, as one of the two components of re-
standing of them. For example, some methods (e.g., questionnaire) can
search methods, is normally categorized as qualitative or quantitative
be used alone, whereas others (e.g., think aloud protocol) have to be
at the top level. Based on the nature of data collected, qualitative and
employed in conjunction with another method.
quantitative analysis techniques can each be further classified more
Among the research methods examined, some can be regarded as
specifically, a topic beyond the scope of this study.
super-methods. These include content analysis, which is used in almost
One major reason why research methods cannot be categorized as
all research to some degree. Some studies may exclusively focus on
qualitative and quantitative is that almost all the techniques (e.g.,
analyzing the content of data sources for data collection purpose, while
questionnaire, observation) are able to be used to collect both quali-
other investigations perform content analysis as a supplemental
tative and quantitative data despite the fact that each may be oriented
method. Likewise, techniques like comparison are often used in the
toward one of the two. For example, questionnaires can contain open-
process of data collection. However, comparison is not listed as a se-
ended questions, which aim to gather qualitative data, though ques-
parate technique since comparison, like summarization and abstraction,
tionnaire is usually adopted for collecting quantitative data. Similarly,
is one of the major techniques adopted in performing content analysis.
observation is typically used for qualitative data collection (e.g., how
Meta-methods in Table 8 refer to those involving multiple techni-
social network members interact with one other), but could be em-
ques for data collection. In the classic sense, experiment is a pertinent
ployed to gather quantitative data (e.g., number of contacts for each
example of meta-method. When experiment is chosen as the research
social network member). Likewise, an experiment can be used to collect
method for a study, several individual techniques such as interview,
both qualitative (e.g., observation notes) and quantitative (e.g., us-
observation, questionnaire, and think aloud protocol need to be used in
ability performance measures of an information system) data, though
various combinations according to the experimental study's purpose. In
experiment is often considered a method for quantitative research in
fact, only when multiple methods are simultaneously employed in a
publications.
study can the experiment be conducted. Ethnography or field study is
Whether qualitative or quantitative techniques should be used for
another example of meta-methods. Focus groups, interview, observa-
data analysis in a study depends entirely on the type of data (i.e.,
tion, and research diary are among the common tools ethnographers
qualitative or quantitative) being collected. When qualitative as well as
use.
quantitative data are collected, for instance via questionnaire, both
The third category in the taxonomy of research methods concerns
qualitative and quantitative techniques could be employed for ana-
stand-alone methods, where each method can be used alone in a study if
lyzing them. Placing questionnaire into the quantitative category only
appropriate. Compared with other categories in Table 8, this one is the
makes it unclear as to how the qualitative data gathered via open-ended
largest. Some of the research methods (e.g., bibiliometrics, interview,
questions should subsequently be qualitatively analyzed. It is therefore
questionnaire) are widely applied, while others (e.g., focus groups, re-
unsuitable to categorize and name research methods by data analysis
search diary) are used less frequently.
technique. For the same reason, for example, questionnaire cannot be
As a group, contingent methods have to be used together with an-
solely called a quantitative method or observation considered a purely
other method in order to be functional. Research participants, for in-
qualitative one.
stance, must conduct test searches if think aloud protocol is chosen for
Plural methodology use appears to be trending in LIS research.
gathering data about their cognitive activities in the search process.
Table 7 shows how many times each research method identified in the
Such test searches are one type of experiment and provide the basis for
current study is used in multi-method studies, indicating both the fre-
the participants to think aloud while searching. Otherwise, the parti-
quency and corresponding order in which a particular method is re-
cipants would not be able to verbalize anything meaningful in relation
ported in the data set. For example, in the case of JDoc, content analysis
to their search activities. The same is true with Delphi study, in which a
was chosen as a research method in a total of 52 studies, of which 37
questionnaire must be developed and used for data collection purpose.
listed content analysis as the first or only research method, 14 as the
The last class of research methods in Table 8 is called pseudo-
second, zero as the third (merely as a position holder when no study
methods. These methods cannot be used for data collection in research
adopts content analysis as a third research method in a multi-method
and, therefore, are not qualified as research methods. They only denote
investigation), and one as the fourth. This is presented as
the scope of research. Case study, as indicated earlier, means that the
37 + 14 + 0 + 1 after the total frequency in Table 7. In a multi-
research examines one or more cases (i.e., sampling units) rather than
method study, some methods (e.g., questionnaire) are good for gath-
the entire population. Longitudinal study implies that a period of time
ering quantitative data, while others (e.g., interview) seem suitable for
(e.g., 2001–2010) is covered in the research. Cross-sectional study, on
collecting qualitative data. This demonstrates another reason why re-
the other hand, is the opposite of longitudinal because it looks at data
search methods should not be categorized as qualitative or quantitative.
collected at one temporal point (e.g., 2001).
As shown in Table 7, only three methods (i.e., ethnography, his-
torical method, and webometrics) are not used in conjunction with
6.4. Further research

2
This is true in the cases of content analysis and theoretical approach because data It might be useful to further break down the experiment category
analysis is an integral part of the data collection process for both methods. When content into classic and non-classic because most experiments in LIS research fit
analysis or theoretical approach is chosen for collecting data, analysis has to be performed
into the latter. Also, altmetrics might be listed as a separate method in
in order to undertake the data collection task. Even though it is still legitimate to indicate
that data collection technique is the criterion used for naming and categorizing content
future coding schema of research methods as more studies adopt this
analysis and theoretical approach as research methods, the data analysis technique em- method in response to the emergence and rapid growth of social media
ployed for collecting data is inseparable from the data collection technique in either case. and other similar platforms. Finally, the coding schema of research

292
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

Table 7
Frequency and order of research methods in the data set.

JASIS&T (N = 1373) JDoc (N = 367) LISR (N = 241

Method Freq (1st + 2nd + …) Method Freq (1st + 2nd + …) Method Freq (1st + 2nd + …)

Experiment 482 (418 + 45 + 15 + 4) Theoretical approach 141 (133 + 8) Content analysis 73 (61 + 10 + 1 + 0 + 1)
Bibliometrics 270 (258 + 11 + 0 + 1) Content analysis 52 (37 + 14 + 0 + 1) Questionnaire 68 (64 + 3 + 1)
Questionnaire 247 (139 + 78 + 16 + 14) Questionnaire 51 (38 + 10 + 3) Interview 48 (26 + 16 + 4 + 2)
Content analysis 215 (149 + 34 + 18 + 14) Experiment 49 (45 + 2 + 2) Experiment 21 (14 + 4 + 3)
Theoretical approach 195 (191 + 4) Interview 49 (35 + 9 + 5) Observation 15 (6 + 6 + 3)
Interview 145 (51 + 51 + 30 + 13) Bibliometrics 31 (29 + 2) Bibliometrics 14 (13 + 1)
Transaction log analysis 86 (45 + 15 + 17 + 9) Transaction log analysis 18 (14 + 2 + 2) Transaction log analysis 9 (8 + 1)
Observation 63 (15 + 26 + 17 + 5) Theoretical approach 141 (133 + 8) Focus groups 9 (5 + 3 + 0 + 1)
Webometrics 57 (46 + 10 + 1) Observation 11 (6 + 4 + 1) Webometrics 6
Think aloud protocol 35 (3 + 14 + 11 + 7) Webometrics 9 (8 + 1) Think aloud protocol 3 (0 + 1 + 2)
Focus groups 29 (7 + 8 + 10 + 4) Historical method 7 (6 + 1) Ethnography 2
Ethnography 16 (13 + 1 + 0 + 2) Focus groups 6 (4 + 2) Historical method 2
Research diary 13 (6 + 6 + 1) Research diary 5 (1 + 3 + 0 + 1) Delphi study 2 (1 + 0 + 0 + 1)
Delphi study 9 (6 + 1 + 0 + 2) Think aloud protocol 5 (0 + 4 + 1) Research diary 2 (0 + 2)
Historical method 8 Ethnography 3
Card sorting 4 (3 + 0 + 1)
Drawing 2 (0 + 1 + 1)
Action research 2 (0 + 0 + 2)
Information horizon 1 (0 + 1)
Photo survey 1 (0 + 1)

Table 8 extend to LIS research education, training, and advocacy. In addition,

Taxonomy of research methods in LIS by applicability. because research methods themselves are not discipline-specific, the
findings of this research could also benefit researchers beyond the LIS
Applicability Method
field.
Super-methods Content analysis
Meta-methods Experiment, ethnography/field study References
Stand-alone Bibliometrics, focus groups, historical method, interview,
methods observation, questionnaire, research diary, theoretical
Agosto, D. E., & Hughes-Hassell, S. (2006). Toward a model of the everyday life in-
approach, transaction log analysis, webometrics
formation needs of urban teenagers, part 1: Theoretical model. Journal of the
Contingent Delphi study, think aloud protocol
Association for Information Science and Technology, 57, 1394–1403.
methods Avison, D. E., Dwivedi, Y. K., Fitzgerald, G., & Powell, P. (2008). The beginnings of a new
Pseudo-methods Case study, longitudinal study, cross-sectional study era: Time to reflect on 17 years of the ISJ. Information Systems Journal, 18(1), 5–21.
Baker, L. M. (2006). Observation: A complex research method. Library Trends, 55,
171–189.
methods could be applied to data collected beyond the 2001–2010 time Bawden, D. (2012). On the gaining of understanding syntheses, themes and information
analysis. Library and Information Research, 36(112), 147–162.
period from the three selected journals, as well as from other LIS Beck, S., & Manuel, K. (2008). Practical research methods for librarians and information
journal titles. professionals. New York, NY: Neal-Schuman.
Berg, B. L. (2009). Qualitative research methods for the social sciences (7th ed.). Boston, MA:
Allyn and Bacon.
7. Conclusion Bernhard, P. (1993). In search of research methods used in information science. Canadian
Journal of Information and Library Science, 18(3), 1–35.
Blake, V. L. P. (1994). Since Shaughnessy: Research methods in library and information
Data collection and analysis techniques are the two major compo- science dissertation, 1975–1989. Collection Development, 19(1/2), 1–42.
nents of any research method. Research methods, however, should be Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for
research. Chicago, IL: Rand McNally.
categorized and named by data collection technique. For example, a
Chen, W., & Hirschheim, R. (2004). A paradigmatic and methodological examination of
study would be named as observation research if observation was used information systems research from 1991 to 2001. Information Systems Journal, 14(3),
as a technique for data collection. Similarly, experimental research 197–235.
Chu, H. (2015). Research methods in library and information science: A content analysis.
suggests experiment as the data collection technique. The type of data
Library & Information Science Research, 37, 36–41.
each technique collects determines whether qualitative or quantitative Davies, K. (2012). Content analysis of research articles in information systems (LIS)
techniques should be used for analysis. Statistical or quantitative journals. Library and Information Research, 36(112), 16–28.
techniques (e.g., central tendency measures, significance tests) will be Dimitroff, A. (1992). Research in health sciences library and information science: A
quantitative study. Bulletin of the Medical Library Association, 80, 340–346.
applied to analyze numeric or quantitative data. By contrast, qualitative Eldredge, J. D. (2004). Evidence-based librarianship: An overview. Bulletin of the Medical
techniques (e.g., qualitative coding, content analysis) are employed to Library Association, 88, 289–302.
analyze textual (e.g., words, phrases, passages, discourses) or qualita- Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3),
215–251.
tive data. Feehan, P. E., Gragg, L. W., Havener, M. W., & Kester, D. D. (1987). Library and in-
The coding schema of research methods developed in this study formation science research: An analysis of the 1984 journal literature. Library &
could shed light on categorization of research methods, and could lead Information Science Research, 9, 173–185.
Fidel, R. (1993). Qualitative methods in information retrieval research. Library &
to the common use of classifying research method by data collection Information Science Research, 15, 219–247.
technique rather than other criteria. Using the taxonomy of research Fidel, R. (2008). Are we there yet?: Mixed methods research in library and information
methods by applicability (Table 8) would enable LIS researchers to science. Library & Information Science Research, 30, 265–272.
Fox, N. (1998). How to use observations in a research project. Sheffield, England: Trent
approach each method by considering its suitability in exploring the
Focus Group. Retrieved from http://www.webcitation.org/5ZndSmDde.
problem at hand. This study is one of the many efforts to promote a Gelber, N. (2013). Five years of empirical research in the area of technical services: An
better understanding of research methods and subsequently to help LIS examination of selected peer-reviewed journals, 2007–2011. Technical Services
Quarterly, 30, 166–186.
scholars make more informed decisions about research method selec-
Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative
tion and implementation. The implications of the present study can

293
H. Chu, Q. Ke Library and Information Science Research 39 (2017) 284–294

research. Chicago, IL: Aldine. universities. Library & Information Science Research, 26, 501 − 510.
Glazier, J. D. (2010). Domain assumptions of research. In L. S. Connaway, & R. R. Powell Peritz, B. C. (1977). Research in library science as reflected in the core journals of the pro-
(Eds.). Basic research methods for librarians (pp. 28–43). (5th ed). Westport, CT: fession: A quantitative analysis (1950–1975) (Unpublished doctoral dissertation)
Libraries Unlimited. Berkeley, CA: University of California.
Gore, S. A., Nordberg, J. M., Palmer, L. A., & Piorun, M. E. (2009). Trends in health Powell, R. R. (1999). Recent trends in research: A methodological essay. Library &
sciences library and information science research: An analysis of research publica- Information Science Research, 21, 91–119.
tions in the Bulletin of the Medical Library Association and Journal of the Medical Library Spradley, J. P. (1980). Participant observation. New York: Holt, Rinehart and Winston.
Association from 1991 to 2007. Journal of the Medical Library Association, 97, Thelwall, M. (2008). Bibliometrics to webometrics. Journal of Information Science, 34(4),
203–211. 605–621.
Greifeneder, E. (2014). Trends in information behaviour research. Information Research, Tuomaala, O., Järvelin, K., & Vakkari, P. (2014). Evolution of library and information
19(4), Retrieved from http://www.informationr.net/ir/19-4/isic/isic13.html#. science, 1965-2005: Content analysis of journal articles. Journal of the Association for
WfTjpxOPLRg. Information Science & Technology, 65, 1446–1462.
Hider, P., & Pymm, B. (2008). Empirical research methods reported in high-profile LIS Turcios, M. E., Agarwal, N. K., & Watkins, L. (2014). How much of library and in-
journal literature. Library & Information Science Research, 30, 108–114. formation science literature qualifies as research? Journal of Academic Librarianship,
Hildreth, C. R., & Aytac, S. (2007). Recent library practitioner research: A methodological 40, 473–479.
analysis and critique. Journal of Education for Library & Information Science, 48, White, M. D., & Marsh, E. (2006). Content analysis: A flexible methodology. Library
236–258. Trends, 55, 22–45.
Järvelin, K., & Vakkari, P. (1990). Content analysis of research articles in library and Williams, J. F., II, & Winston, M. D. (2003). Leadership competencies and the importance
information science. Library & Information Science Research, 12, 395–421. of research methods and statistical analysis in decision making and research and
Julien, H., Pecoskie, J., & Reed, K. (2011). Trends in information behavior research, publication: A study of citation pattern. Library & Information Science Research, 25,
1999–2008: A content analysis. Library & Information Science Research, 33, 19–24. 387–402.
Kim, Y. H., & Kim, H. H. (2008). Development and validation of evaluation indicators for
a consortium of institutional repositories: A case study of dCollection. Journal of the Heting Chu is a professor in the Palmer School of Library and Information Science at
American Society for Information Science and Technology, 59, 1282–1294. Long Island University, New York. She obtained her PhD degree in information studies
Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed). from Drexel University, Philadelphia, Pennsylvania, in 1991. Her teaching and research
Thousand Oaks, CA: Sage. interests include the use of information technology in library and information science,
Kumpulainen, K. (1991). Library and information science research in 1975. Libri, 41(1), especially in the areas of information representation and retrieval, library and informa-
59–76. tion science education, research methods, and scientific communication. Dr. Chu has
Luo, L., & McKinney, M. (2015). JAL in the past decade: A comprehensive analysis of published over 40 research articles in journals such as Journal of Academic Librarianship,
academic library research. The Journal of Academic Librarianship, 41, 123–129. Journal of Documentation, Journal of Education for Library and Information Science, Journal
Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage. of the Association for Information Science and Technology, and Library and Information
Ngulube, P. (2015). Trends in research methodological procedures used in knowledge Science Research. Her book Information Representation and Retrieval in the Digital Age (2nd
management studies. African Journal of Library, Archives and Information Science, ed., Information Today, 2010) has been translated into Chinese, Korean, and Arabic, and
25(2), 125–143. was also published in India. She is working on a book manuscript titled Research Methods
Palvia, P., Leary, D., Mao, E., Midha, V., Pinjani, P., & Salam, A. F. (2004). Research and Design: An Interdisciplinary Approach.
methodologies in MIS: An update. Communications of the Association for Information
Systems, 14, 526–542.
Palvia, P., Mao, E., Salam, A. F., & Soliman, K. S. (2003). Management information sys- Qing Ke is a professor in the School of Information Management at Nanjing University,
tems research: What's there in a methodology. Communications of the Association for China. She holds a PhD in information management from Nanjing University and a
Information Systems, 11, 289–309. master's degree from Wuhan University, China. She has published more than 30 research
Palvia, P., Pinjani, P., & Sibley, E. J. (2007). Editorial: A profile of information systems articles in library and information science journals in Chinese and English. Her current
research concerns research methods, information behavior, and human computer inter-
research published in the information and management. Information Management,
44(1), 1–11. action.
Park, S. (2004). The study of research methods in LIS education: Issues in Korean and U.S.

294