Users and technology: are we doing research now?

Users and technology: are we doing research now? by Elke Greifeneder Berlin School of Library and Information Science Humboldt-Universität zu Berlin [email protected] Writing about the way we do research is a dangerous task: in describing standards for research designs, authors invite criticism for their own work. While examinations of well and less well designed studies may surprise some readers, studies about research are an important part of Library and Information Science. In November 2010, Library Hi Tech has issued a call for papers entitled “user research and technology”. The editors aimed at both user research about technology and technology for user research. The present short article links the papers published within that issue to discussions about user research in recent years. The paper makes the following assumptions: first, the diversity of methods to study users is growing. Second, a confidence about labeling and using methods is becoming manifest. And last, the number of research questions that a single study answers is both diminishing and at the same time growing more diverse. Diversity of methods There is an obvious need for user studies. Libraries are facing new technologies within a rapidly changing digital environment. New services need to be developed and tested. Until lately, librarians primary received money for building something, such as a digital library. Whether that service actually met users needs often seemed secondary, and research about users, their behavior and their needs were an add-on at the end of a project. In the last two years, this situation has changed. The topic of user experience is suddenly omnipresent at conferences and in journal articles. Librarians seem to have realized that the user's involvement in as many development steps as possible is a real need – and a real plus. Since the rise of digital environments, researchers had difficulty in finding appropriate methods to study users in digital library environments. Many drew on surveys or log file analyses. Both methods collect data online and require no direct researcher involvement the way usability testing in laboratories or interviews do. Not surprisingly the number of studies that use surveys was always high, but this number seems to be declining. Julien, Pecoskie and Reed (2011) analyzed the methods that were used to study information behavior. They report that for the period 1984-1998 about 58.1% of all methods were questionnaires and interviews, which were jointly labeled as survey methods. (see also Julien & Dugan, 2000) For the period 1998-2008, this number declined to 44.7%. Greifeneder (2010) analyzed publications on user research in digital library environments for the period 1998-2009 and concludes that “in nearly half of all the studies, researchers used a survey. Log file analysis follows in popularity with 18 %, followed by interview with 12 %.” Surveys are still a popular method, but other methods appear in the user research field. In this issue on user research and technology, only three theme articles used a survey. Each of these surveys were also combined with other methods. Denton (2011) and Noh (2011) combined it with usability testing. A third article on mobile technology used additional focus groups. This article will be published in the following issue. Yang (2011) draws on a checklist for heuristic evaluation, while Lindquist and Long (2011) adopt a qualitative approach with in-depth interviews. Bond and O'English (2011) presented a familiar case study of their library and explain the development process for their services. Hegarty and Wusteman (2011) examine mobile phone use with usability testing in a laboratory; two methodological supplements will likely be published in the next Library Hi Tech issue: one article examines remote usability testing, the other uses virtual ethnography. Held, Fischer and Schrepp (2011) introduce a less well known method, the complete paired comparison approach, which helps in making decisions when a variety of feasible alternatives is available. Labeling and using methods Library and Information Science education does not always offer librarians an in-depth methodological education in social science, psychology, ethnography, mathematics or computer science. Librarian’s knowledge about surveys or interviews tends to be very basic and often self-taught. For example, few really understand the intricacies of statistics. Studies proudly present diagrams based on certain statistical tests because software products like SPSS serves as a miracle cooking machine, where researchers throw in a variety of variables in and get out nicely prepared diagrams. Fagan (2010) observes user research on faceted browsing and states that “these studies [by librarians] used fewer participants than the information science studies above, followed less rigorous methods, and were not subjected to statistical tests”. Lyons (2011) analyzed studies by national and international library organizations which in his opinion should be influential and states that “[t]oo often they employ deficient research methods or promote unjustifiable interpretations of data they have collected.” Knowing how to use methods appropriately is only part of user research. Being able to label them is another. Friese (2009) studied the methods used for case studies by analyzing the abstracts of articles from five Emerald journals (Journal of documentation, Library Management, Library Hi Tech and Reference Service Review) during the period 1999-2009. In her research design her initial plan was to analyze the method section of structured abstracts. To her surprise this design did not work, because the author’s description in the abstracts method section tended to be superficial or essentially missing. In the end, she had to read the articles as a whole in order to decipher the methods being used. Within this issue, this methodological lack seems to been overcome or at least the situation has improved. All articles showed accuracy in labeling methods, describing in detail the research design and manifested a well funded knowledge about the interpretation of research output. Reviewers today base their judgment more explicitly on these parts and require changes if papers offer no adequate answers. Less research questions, diverse areas Reviewers now also question articles that use a single study to answer several diverse research questions. An example of the latter is Heinz (2003), who examined three different aspects within one study: “User behavior during the search process, overall usability issues, satisfaction of the users with the system” [bullet points in the original]. Devakos (2006) picks up on this in her study about repositories and gives the sensible advice: “You do not have to (and should not) do and solve every issue at once”. Looking at the articles published in this issue, the number of research questions is getting smaller. The authors have concentrated on one major research question per study. Another examination of user research publications discovered that the “use” of something is the preferred purpose in user research studies. This could broadly be the analysis of the use of a digital library, the use of mobile technology or the use of library services. A content analysis revealed that in 41% researchers declare “use” to be their main purpose. (Greifeneder 2009) Within this issue, authors offered more concrete and more diverse purposes. The theme articles can be grouped into three major areas: The first area discusses technology to make resources accessible for users. Bond and O’English (2011) describe their experience in providing online access to historic films at the Washington State University Libraries. Lindquist and Long (2011) question what technology tool students need in order to facilitate their engagement with primary sources. An article that will be published later examines whether and how poorly funded libraries can use remote usability tools. The second area focuses on users (future) needs and technology: Noh (2011) analyzes user’s needs for Web-based reference resources. A second article, which will also be published in the next issue, examines student attitudes toward mobile library services for Smartphones. Yang (2011) analyzes current ACRL (Association of College and Research Libraries) OPACs and questions whether these are next or current generation catalogs. Articles in the third and last area examine the design itself and ask how researchers can design technology for users and what design changes must be made. The early and iterative integration of users in the development process is reflected by three articles: Denton (2011) tests VuFind at an early stage in an academic library and Hegarty and Wustemann (2011) evaluate EBSCO Host Mobile. Held, Fischer and Schrepp (2011) use a simple pair comparison approach to show how researchers can use the method to decide on alternative interfaces. Conclusion This paper is not in-depth research that offers evidence that a change in user research has taken place. Its arguments build on papers submitted to this special issue on that topic. Clearly, more research needs to be done. But the high number of submissions to the call for papers and the high quality of the papers is an indication that user research, especially on users interacting with technology, has become an important part of what we do in library and information science research. These papers show that the quality of user research in our field is rising, that researchers know how to label and use methods appropriately, and that they are using a greater variety of methods. Finally, researchers seem to acknowledge that user research requires one small step after another. Instead of painting a big picture with a single user study that has many research questions, they do multiple smaller in-depth research projects, which can be interconnected like one big picture puzzle that might, in the end, give a better impression of how our users actually behave and what they really need. References: Bond, T., and O'English, M. 2011. Providing online access to historic films at the Washington State University Libraries. Library Hi Tech 29, 2. Devakos, R. 2006. Towards user responsive institutional repositories. a case study. Library Hi Tech 24, 2, 173–182 Denton, W. 2011. Usability Testing of VuFind at an Academic Library. Library Hi Tech 29, 2. Fagan, J.C. 2010. Usability Studies of Faceted Browsing. A Literature Review. Information Technology and Libraries June, 58–66: 61 Friese, Y. 2009. Die Verwendung von Methoden in Fallbeispielen. Eine Untersuchung anhand der Emerald-Abstacts von 1999-2009. Bachelorarbeit, Humboldt-Universität zu Berlin - Institut für Bibliotheks- und Informationswissenschaft. Greifeneder, E. 2010. A content analysis on the use of methods in online user research. In Digital library futures. User perspectives and institutional strategies. I. Verheul, A. M. Tammaro and S. Witt, Eds. De Gruyter Saur, Berlin. Greifeneder, E. 2009. Purposeful Data Collection. An analysis of publications on digital. Presentation at the conference on Digital Library Futures, user perspectives and institutional strategies user research. Slides available at http://www.athenaeurope.org/getFile.php?id=347 Hegarty, R., and Wusteman, J. 2011. Evaluating EBSCO Host Mobile. Library Hi Tech 29, 2. Held, T., Fischer, P., and Schrepp, M. 2011. Scaling of input forms by a simple pair comparison approach. Library Hi Tech 29, 2. Heinz, S., Mandl, T., and Womser-Hacker, C. 2003. Implementation and Evaluation of a Virtual Library Shelf for Information Science Content. In 5th Russian Conference on Digital Libraries RCDL 2003, St.Petersburg (Russia). Julien, H., Duggan, L. J. 2000. A longitudinal analysis of the information needs and uses literature. Library and Information Science Research, 22, 291−309 Julien, H., Pecoskie, J.(.L.)., and Reed, K. 2011. Trends in information behavior research, 1999–2008. A content analysis. Library & Information Science Research 33, 19–24. Lindquist, T, Long, H. 2011. How Can Educational Technology Facilitate Student Engagement with Primary Sources? A User Needs Assessment. Library Hi Tech 29, 2. Lyons, R. 2011. Statistical correctness. Library & Information Science Research 33, 92–95. Noh, Y. 2011. A Study on Metadata Elements for Web-based Reference Resources System Developed through Usability Testing. Library Hi Tech 29, 2. Yang, S. 2011. Next Generation or Current Generation? A Study of the OPACs of 260 Academic Libraries in the United States and Canada. Library Hi Tech 29, 2.