Reducing speeding in web surveys by providing immediate feedback

Survey Research Methods (2017) c European Survey Research Association ISSN 1864-3361 http://www.surveymethods.org Vol. 11, No. 1, pp. 45-61 doi:10.18148/srm/2017.v11i1.6304 Reducing speeding in web surveys by providing immediate feedback Frederick G. Conrad Roger Tourangeau Institute for Social Research Ann Arbor, USA Westat, Inc. Rockville, USA Mick P. Couper Chan Zhang Institute for Social Research Ann Arbor, USA School of Journalism Fudan University Shanghai, China It is well known that some survey respondents reduce the effort they invest in answering questions by taking mental shortcuts – survey satisficing. This is a concern because such shortcuts can reduce the quality of responses and, potentially, the accuracy of survey estimates. This article explores “speeding,” an extreme type of satisficing, which we define as answering so quickly that respondents could not have given much, if any, thought to their answers. To reduce speeding among online respondents we implemented an interactive prompting technique. When respondents answered faster than a minimal response time threshold, they received a message encouraging them to answer carefully and take their time. Across six web survey experiments, this prompting technique reduced speeding on subsequent questions compared to a no prompt control. Prompting slowed response times whether the speeding that triggered the prompt occurred early or late in the questionnaire, in the first or later waves of a longitudinal survey, among respondents recruited from non-probability or probability panels, or whether the prompt was delivered on only the first or on all speeding episodes. In addition to reducing speeding, the prompts increased response accuracy on simple arithmetic questions for a key subgroup. Prompting also reduced later straightlining in one experiment, suggesting the benefits may generalize to other types of mental shortcuts. Although the prompting could have annoyed respondents, it was not accompanied by a noticeable increase in breakoffs. As an alternative technique, respondents in one experiment were asked to explicitly commit to responding carefully. This global approach complemented the more local, interactive prompting technique on several measures. Taken together, these results suggest that interactive interventions of this sort may be useful for increasing respondents’ conscientiousness in online questionnaires, even though these questionnaires are self-administered. Keywords: web surveys; speeding; survey satisficing; interactivity; data quality 1 Introduction survey respondents cut corners, it has been called “survey satisficing” (e.g. Krosnick, 1991). It is well established that some survey respondents take shortcuts when answering survey questions, providing responses that are acceptable but not optimal (see, for example, Cannell, Miller, & Oksenberg, 1981). This is reminiscent of the more general phenomenon that Simon (1956) dubbed “satisficing” in which people solve everyday problems using incomplete information, producing solutions that are good enough to largely achieve their goals but not as good as if they used all the information available to them. Thus, when Web survey respondents are prone to take certain shortcuts more often than respondents in other modes. For example, web survey respondents have been observed to produce more “Don’t Know” responses, more missing data, and more nondifferentiation than face-to-face respondents (e.g. Heerwegh & Loosveldt, 2008). This is likely related to the absence of an interviewer in web surveys. When respondents participate in survey interviews with a live interviewer, there is social pressure to invest effort in the task and avoid being seen as lazy. In addition, the web (not just web surveys) seems to promote superficial and hurried processing by users. For example, Nielsen and Loranger (2006) report that web users are more likely to scan rather than read text online relative to reading text on paper. While this general picture does not bode well Contact information: Frederick G. Conrad, Institute for Social Research, 426 Thompson Street, Ann Arbor, MI 48104-2321 ([email protected]) 45 46 FREDERICK G. CONRAD, ROGER TOURANGEAU, MICK P. COUPER AND CHAN ZHANG for the quality of web survey responses, there may be a silver lining. The interactive character of web questionnaires potentially enables designers to promote more conscientious and thoughtful responding by providing feedback when respondents seem to be taking shortcuts. The current article focuses on “speeding,” i.e., answering so quickly that it is unlikely the question has been adequately processed. In this study, we define speeding as responding so quickly that it is unlikely respondents have read the question, let alone given much thought to their answer. Speeding is a phenomenon frequently observed in web surveys (e.g. Zhang & Conrad, 2014) but less so in intervieweradministered modes. As we see it, when respondents speed they are likely to provide “throw-away” answers: they feel compelled to enter something but not necessarily an accurate response. Thus, when respondents speed, it is unlikely accuracy will be above what can be expected by chance. We define speeding as an absolute rather than relative phenomenon: response time below a psychologically based threshold rather than responses that are simply faster than other responses (as, for example, Greszki, Meyer, and Schoen, 2015 and Malhotra, 2008 define speeding)1 . The “speeding threshold” probably differs across people: A fastthinking respondent might well be able to answer carefully in a period of time that would be too brief for a slower respondent to consider the question much at all. And a respondent with a ready-made, familiar answer would also be much faster than one who is thinking about the issue for the first time. But technically, it is hard to distinguish a legitimate fast response from a response that is fast because of speeding. Therefore, in the experiments reported here we use one speeding threshold for all respondents. As we will demonstrate, the threshold we selected seems to discriminate between more and less conscientious responses and between more and less conscientious respondents. Speeding in web surveys seems to be prevalent enough to be of concern to survey researchers. It is identified as a concern in the AAPOR task force report on online panels (Baker et al., 2010, p. 32). Zhang and Conrad (2014) reported that respondents in a Dutch probability panel sped on about 15 out of 54 questions on average and, if they sped at all, were likely to speed throughout the questionnaire. In the six experiments we report in this article, between 37 and 85 percent of respondents sped at least once in answering seven critical questions in the control condition (in which there was no intervention for speeding). While these speeding statistics depend on exactly how speeding is defined, the phenomenon is common by any definition of speeding. Thus, a method to reduce speeding could help potentially improve the quality of web survey data and increase confidence in the mode for high stakes applications like social scientific and government surveys. We explore one such method here – an interactive prompting technique. 2 Current Experiments We report six web survey experiments that explore the effectiveness of an intervention triggered when respondents answered faster than a fixed threshold (see Table 1). The main question is whether the intervention reduced subsequent speeding. The intervention was an on-screen, textual prompt indicating that speeding had been detected and encouraging the respondent to give adequate thought to his or her answers. It was administered when respondents sped in the course of providing numerical answers to seven questions about autobiographical quantities (Experiments 1–3 and 5) or simple mathematical problems (Experiment 4); we refer to these seven items in each experiment as the “prompting items.” Respondents were randomly assigned to either a control condition in which there was no prompting for speeding on these items or an experimental condition in which respondents were prompted when they sped. In Experiments 1–3, we manipulated the prompting “dose” so that speeders were either prompted every time or just the first time they sped on the prompting items. Each experiment was embedded in a larger omnibus questionnaire. The types and content of items varied widely. We were concerned only with performance on the seven prompting items and (in Experiments 1–4) two grid questions that came after the prompting items; the remainder of each questionnaire was administered for purposes unrelated to the current study. Each questionnaire varied in length; because of skip patterns, different respondents answered different numbers of questions within each survey. Table 1 presents completion times for the entire questionnaire as an indication of overall effort. The table also includes key attributes about the design of the six experiments. Respondents were recruited from volunteer panels in Experiments 1–4 and from a probability panel in Experiment 5. Our focus is on the effects of the speeding prompt, comparing performance of respondents in the experimental to control groups. The speeding prompt read: “You seem to have responded very quickly. Please be sure you have given the question sufficient thought to provide an accurate answer. Do you want to go back and reconsider your answer?” We developed four hypotheses where we had clear directional predictions. In addition, we have three research questions for which we do not have theoretically based predictions. Because the prompts were tightly coupled with speeding on a particular question, we expected the prompts to give respondents the impression that their behavior was being monitored, increasing their sense of accountability. To the extent 1 Related to this relative notion of speeding is the common practice of removing answers with the slowest and fastest response times (e.g. Heerwegh, 2003). The general assumption is that these outliers are of low quality and, in the case of very fast responses, are the result of speeding. 47 REDUCING SPEEDING IN WEB SURVEYS BY PROVIDING IMMEDIATE FEEDBACK Table 1 Overview of Experiments. Type of Questions Mean Complet. Durationa Sample Size Type of Sample Field Period Prompting Dose 1 2 3 4 Frequency, Grids Frequency, Grids Frequency, Grids Numeracy, Grids 20.1 27.6 21.3 25.3 2463 2453 3046 2565 Volunteer Volunteer Volunteer Volunteer Aug–Sep, 2007 Apr–May, 2008 Sep 2008 Jun–Jul, 2010 Once, Every Time Once, Every Time Once, Every time Every Time 5a 5b Frequency Frequency 42.9 47.4 929 913 Probability Probability Jul 2009 Aug 2009 Every Time Every Time Experiment Distinguishing Features Replicat. of Exp. 1 Prompting early/late Commitment crossed with prompting Longitudinal design The seven frequency questions and seven numeracy questions enable measurement of speeding and its reduction with prompting. The numeracy questions also enable measurement of response accuracy. Grids allow measurement of straightlining and its reduction with prompting. Mean completion duration provides an estimate of the effort required to finish the full questionnaire, which varied in length between experiments and, due to skip patterns, between respondents. Two prompting doses are used in experiments 1–3 to evaluate how much the number of prompts matters. Measuring the effect of dose was not a goal of experiments 4-5b, so only the “larger” dose – Every Time – was used. a in minutes that speeding results from the absence of such accountability, prompting speeders in this way should reduce subsequent speeding. Hypothesis 1: Prompting will reduce speeding on subsequent questions. When respondents are first prompted they might reason “I am being monitored and do not want to be seen as lazy” and so pay more careful attention to the task. Thus, it is possible that the administration of a single prompt would be as effective at reducing speeding as prompting respondents whenever they were caught speeding. Alternatively, it’s possible that some respondents would not be deterred from speeding by a single prompt but would slow down when prompted multiple times, because multiple prompts may emphasize the importance of responding carefully. In extreme cases, some respondents might be so focused on finishing quickly that they will not slow down until the repeated prompts actually delay completion. Because we do not have a directional prediction we ask the following research question: Research Question 1: Does a single prompt, delivered the first time a respondent is caught speeding, reduce speeding as much as a prompts delivered each time the respondent speeds? It is likely that as respondents advance through any questionnaire more than a few items long, they become more fatigued. This might compromise their ability to control and adjust how they produce their answers, i.e., it may be more difficult for fatigued respondents to slow down after being prompted than it is for less fatigued respondents. Thus, prompting later in the questionnaire may be less effective than prompting early. Alternatively, the fatigue created by completing many questions may not affect respondents’ ability to control how they answer questions so that after being prompted they should be able to attend to the task more carefully and work harder. In this case, prompts should be equally effective early and late in the questionnaire. Because there is reason to believe that prompting later in the questionnaire will moderate the effectiveness of prompting but also that it will not, we ask: Research Question 2: Will the effectiveness of prompts vary throughout the questionnaire or remain constant, even as speeding increases? In Experiment 3, the questions on which speeders were prompted appeared early in the questionnaire (Q4–10) for a random subsample of about two thirds of respondents and later (Q60–66) for the remaining third. Because speeding is assumed to reflect minimal effort and, as a result, low quality data, reducing speeding helps the survey enterprise to the extent that slower responses increase data quality2 . We examine the relationship between reduced speeding and data quality in Experiment 4 by measuring speeding on questions for which the true value is known, enabling us to determine response accuracy. In particular we administered simple probability or arithmetic questions designed to assess numerical literacy (“numeracy”), for which there are clear right and wrong answers. While respondents 2 Of course very long response times can also indicate reduced quality. Specifically, they might reflect difficulty for respondents who are trying to answer in good faith (e.g. Conrad, Schober, & Coiner, 2007; Draisma & Dijkstra, 2004; Yan & Tourangeau, 2008). In such cases, simplifying and improving the offending questions should lead to responses that are both faster and of improved quality. 48 FREDERICK G. CONRAD, ROGER TOURANGEAU, MICK P. COUPER AND CHAN ZHANG might not be more accurate each time they answer after being prompted – they might answer more carefully without necessarily answering correctly – if they slow down, their chances of answering any given question accurately should increase. Hypothesis 2: Prompting respondents when they speed will increase response accuracy. Other measures of data quality might potentially be informative. For example, prompting speeding might increase the length of open-ended responses, reduce the prevalence of primacy effects, and reduce the number of missing observations on subsequent questions. However, we focus here on response accuracy, as it is generally the most direct measure of response quality for factual questions (but see Zhang, 2013 and Zhang and Conrad, 2016 for evidence that prompting respondents for speeding can increase the length of open-ended responses). This raises the question of whether the effects of prompting are confined to the item on which a respondent has been caught speeding or whether the effects can transfer to other items in the questionnaire. There is some evidence that speeding is related to other types of survey satisficing. Malhotra (2008) observed more primacy among low education respondents who answered most quickly (fastest tercile). Similarly, Callegaro, Yang, Bhola, Dillman, and Chin (2009) reported that “satisficers” (identified by exogenous factors) answered faster than “optimizers” (who had more reason to respond accurately), suggesting that the satisficers were speeding or were at least less conscientious. Although these authors define fast responding in relative terms, unlike our threshold-based definition, their findings suggest that speeding and satisficing in general may have a common origin. If so, it is plausible that speeding and straightlining (non-differentiation) are related. If that is the case, reducing speeding with interactive prompts may also reduce later straightlining. Hypothesis 3: Speeding prompts will reduce subsequent straightlining. To test this, we investigated whether there was less straightlining in the prompting than control conditions for several grid questions (three grid questions were presented in Experiments 1 and 2, and two grid questions were presented in Experiments 3 and 4; see Appendix C) that followed the prompting items. If Hypothesis 3 is supported, this would suggest that prompting can improve data quality beyond the item on which the prompt was delivered. It is possible that the prompts tested here are only effective when they are novel. More specifically, it may be that respondent’s prompted in more than one wave of a longitudinal survey will dismiss the later prompts, having recognized in a previous wave that there is little cost to speeding aside from receiving the potentially annoying message. Respondents in Experiment 5 were members of the Face-to-Face Recruited Internet Survey Platform (FFRISP), an area probability sample representative of the US population (e.g., Villar, Malka, & Krosnick, 2010). The FFRISP data for the current study were collected in two waves (July and August, 2009) allowing us to test the effectiveness of the prompts across waves of an online survey. We randomly assigned respondents to a No Prompt control or Prompt Every Time condition; the respondents remained in the same condition in both waves. We asked them the same autobiographical quantity questions that we asked respondents in Experiments 1–3. On the one hand it is possible that prompting will become less effective after the first wave of data collection because respondents recall that there is little cost to being prompted. On the other hand, prompting after the first wave may remind respondents that the questionnaire software is tracking their actions. Research Question 3: Is the effectiveness of prompting reduced over waves of a longitudinal survey? To the extent prompting reduces speeding it may work by shaming respondents into being more conscientious. An alternative method that appeals to respondents’ better nature and is not linked to particular response behaviors is to simply ask respondents at the start of the questionnaire to provide high quality information. Charles Cannell and his associates (e.g., Cannell et al., 1981; Oksenberg, Vinokur, & Cannell, 1979) pioneered an approach along these lines for face-to-face interviews; they asked respondents to commit to providing thoughtful responses by signing a printed commitment statement. Cannell and his colleagues generally found that this improved data quality. However, the respondents made a face-to-face commitment to a human interviewer so it is possible that requesting a commitment from respondents will not be as effective when it is made online. We tested the impact of respondents’ commitment on their speeding behavior by asking half of the respondents in Experiment 4 to commit to answering thoughtfully and carefully at the start of the questionnaire. Half of the respondents were asked to explicitly commit by clicking an on-screen option; the other half of the respondents received a filler statement the same length as the commitment statement. Hypothesis 4: Respondents who commit to conscientious responding will speed less than uncommitted respondents. We crossed prompting with commitment and randomly assigned respondents to one of the four resulting groups. The respondents in the commitment groups were presented the following statement at the start of the questionnaire: It is very important that you read each question carefully and think about your answer before you give it. We rely on REDUCING SPEEDING IN WEB SURVEYS BY PROVIDING IMMEDIATE FEEDBACK our respondents being thoughtful and taking this task seriously. Respondents were then asked to select a radio button to register their commitment: I commit to reading each question carefully and thinking about my answer before I give it. # Yes # No, but I will participate anyway The filler statement presented to the other respondents at the start of the questionnaire was: Thank you very much for participating in our study. We are grateful that you are willing to contribute to our project and hope you find it a stimulating and worthwhile experience. No action was required by the respondent. 2.1 Experimental Materials Respondents’ answers were timed (the difference between the time the respondents pressed the Next button after answering the previous question and the time the respondent pressed the Next button after answering the current question) for seven questions requiring numerical answers. For respondents in the experimental conditions, the software prompted them when a response time fell below a fixed threshold, either the first time or any time this happened depending on the dosage condition. The seven items either concerned autobiographical quantities (e.g., “Overall, how many overnight trips have you taken in the PAST 2 YEARS?”) or, in Experiment 4, simple arithmetic questions (e.g., “If the chance of getting a disease is 10%, how many people out of 100 would be expected to get the disease: 1, 10 or 20?”). The arithmetic items were adapted from items developed to assess numerical literacy (numeracy) (L. M. Schwartz, Woloshin, Black, & Welch, 1997). For all of these items (see Appendices A and B), an accurate answer required at least some thought so speeding is likely to signal incomplete thinking at best and, thus, poor data quality. The prompt was triggered when respondents answered more quickly than 350 milliseconds (msec) per word, which means that the speeding threshold for a ten-word question would have been 3500 msec (i.e., three and a half seconds). The thinking behind this particular threshold is that if a respondent enters an answer faster than most people can read the question (Carver, 1992), let alone think about the answer, he or she should probably slow down and answer subsequent questions more carefully. But one could choose other group-based thresholds, e.g., Taylor, Frankenpohl, and Pettee (1960) report a somewhat faster median reading speed of about 215 msec per word for US college students, or individualized thresholds (e.g. Jackson & McClelland, 1979). Our threshold is a coarse measure of speeding but the approach is simple and inexpensive to implement; to the extent that intervening when response times fall below this threshold slows 49 respondents down and leads them to invest more effort in the response task, it may be well-suited to production web surveys. The first and third grid questions (Appendix C) each involved one reverse-worded statement so that straightlining, if observed, would be particularlly likely to reflect low effort responding. One of these concerned dietary practices and the other two concerned the strength and extremity of the respondents’ opinions and the manner in which they carry out different mental tasks. 3 Results We begin by examining the prevalence and correlates of speeding. Next, we examine the effect of prompts (single and multiple) on speeding, response accuracy and later straightlining. Then, we examine the impact of prompting over two waves of data collection. We next evaluate the impact of the prompts when respondents have committed to answering carefully and, finally, we examine a potential negative effect of prompting – whether prompts promote breakoffs. 3.1 Prevalence and correlates of speeding In all six experiments, there was a substantial amount of speeding in the control condition (where there was no prompting), although the exact amounts varied. The percentages of control respondents who sped (responded in less than 350 msec per word) at least one time in the six experiments were 85%, 82%, 42%, 74%, 37%, and 53%. Many factors differ between the experiments so the specific percent of respondents who sped in any one experiment is not by itself meaningful. The point is that speeding prevalence can be quite high, as in Experiments 1, 2 and 4 and, even where less prevalent, it is far from absent. Not only is speeding more common than one would like, but the percentages of speeders who sped multiple times was quite high, tailing off only for 6 and 7 speeding episodes across the 7 prompting questions. This can be seen in Figure 1, which presents the percent of speeders by the number of questions on which they sped in the control condition of each experiment. The general pattern is that most speeders sped just once or twice but between 11.5 and 19.0 percent of the respondents sped four times, i.e., on more than half of the prompting items, across the experiments. And in Experiment 4, 11.2 percent of the speeders sped on every question. Both the number of respondents who ever sped and the number of questions on which they sped argue for some measure to reduce the prevalence of speeding. Who are the speeders? Across all the experiments, younger respondents (especially in the 18–34 year old age group) sped on significantly more questions than did older respondents, controlling for other demographic variables. In 50 0 % Speeders Speeding 10 20 30 40 FREDERICK G. CONRAD, ROGER TOURANGEAU, MICK P. COUPER AND CHAN ZHANG 1 2 3 4 5 6 Number of Questions on which Speeding Detected Exp 1 Exp 4 Exp 2 Exp 5a 7 Exp 3 Exp 5b Figure 1. Percent of speeders (respondents who sped at least once) in the control condition, distributed over the number of questions on which they sped, for each of the six experiments. separate multivariate models for each experiment, controlling for gender, education, income, employment status, and race and in Experiments 1–4 for web panel membership and self-reported “Internet ability,” only age was consistently related to speeding.3 Yan and Tourangeau (2008) found that age is generally related to response times for web survey questions, with older respondents taking longer to answer than younger ones. Thus, it is not surprising that older respondents were less likely to meet our threshold for speeding. How much of the age effect reflects slowing from cognitive aging (e.g., Park & Reuter-Lorenz, 2009; Park & Schwarz, 2000; Park et al., 1996; Salthouse, 1991, 1996) versus greater conscientiousness among older respondents is hard to disentangle in the current data. 3.2 Impact of intervention on subsequent speeding Our first hypothesis is that prompting respondents when they speed will reduce subsequent speeding. To test this, we compare speeding prevalence in the control and the prompting conditions. Although respondents who were prompted were invited to revisit the question on which they had just sped, we look only at the times for the initial response in order to promote comparability between the experimental and control conditions. Similarly, we restrict the analysis to those respondents who sped at least once during the seven prompting questions, whether in the control or prompting conditions. If respondents did not speed at all, they were treated the same way (i.e., no prompt) regardless of the condition to which they were assigned because their performance is not germane to our evaluation of the prompting procedure. Because response times for the initial answer to the first question cannot be affected by prior prompting, we examine the effect of prompting on speeding for questions 2–7. We observed less speeding in the prompting than control conditions in all six experiments, as can be seen in Table 2. Respondents who sped at least once sped on fewer questions in the experimental conditions than in the control condition. The reduction ranges in size from .2 to .6 questions, that is, reductions in speeding prevalence of 10% to 24%. Considering that the number of questions on which control respondents sped ranged from 2.0 to 3.1, these reductions due to prompting are substantial. Our first research question asked whether a single prompt was as effective as multiple prompts. As can be seen in the rightmost column of Table 2, the two prompting doses do not differently affect the amount of speeding. This suggests that a single prompt can be as effective as more prompts in reducing the amount of speeding. The analysis, thus far, examines the global effect of prompting speeders. While this approach indicates that prompting has the intended effect overall, it does not directly measure the effect of prompting on subsequent speeding question by question. The next analysis does just this by comparing speeding on questions 2–7 among respondents who were prompted for speeding on an earlier question to speeding by control respondents on the same questions. We conducted these analyses for each question individually for each experiment. A summary is presented in Table 3. The right-most column presents the number of questions (out of the six) for which there was significantly less speeding after prompting for prior speeding than without prompting (i.e. in the control condition). By this analytic approach, prompting reduced subsequent speeding in all of the experiments; the number of questions for which reductions were observed ranged between 2 and 5 questions out of 6. A more detailed presentation of the data appears in Appendix D. 3.3 Effectiveness of prompts throughout the questionnaire Our second research question was whether the intervention was any less effective later than early in the questionnaire. There was in fact more speeding when the questions appeared late in the questionnaire than early, F(1, 1157) = 7.12, p < .05, suggesting that speeding increased when respondents were fatigued. However, prompting appeared to be equally effective early and late in the questionnaire, interaction of prompt condition and position in the questionnaire, F(2, 1155) < 1. Thus, it seems that when they were prompted, respondents slowed down and answered more conscientiously irrespective of fatigue. 3 The effects of age were significant in 4 out of 5 experiments (treating 5a and 5b as one experiment), with p-values ranging from 0.024 to < .0001. 51 REDUCING SPEEDING IN WEB SURVEYS BY PROVIDING IMMEDIATE FEEDBACK Table 2 Mean number of speeding episodes and F values, by condition and experiment Experiment No Prompt Prompt Prompt Every Time Once 3.1 3.1 2.0 2.7 2.5 2.6 2.8 2.8 1.8 2.1 1.9 2.1 1 2 3 4 5a 5b 2.9 2.9 1.9 - ANOVA (omnibus) ANOVA (Prompt Every Time vs. Prompt Once) F df1, df2 p F df1, df2 p 6.47 5.57 2.64 50.00 9.12 7.46 2, 2038 2, 2045 2, 1158 1, 1860 1, 313 1, 453 0.0016 0.0039 0.0716 < 0.001 0.0027 0.0066 2.31 <1 1.91 - 1, 1329 1, 1326 1, 777 - n.s. n.s. n.s. - Table 3 Percent of respondents who sped again after speeding on one or more previous question(s) # of Qs for which difference is significant Condition Experiment 1 2 3 4 5a 5b a 3.4 No Prompt 62.8 59.8 33.9 47.5 50.6 59.9 Prompt Every Time 52.3 51.9 31.2 35.9 45.1 42.8 Prompt Once Prompta vs. No Prompt (Control) 54.5 52.9 32.9 - 5 out of 6 4 out of 6 3 out of 6 6 out of 6 2 out of 6 4 out of 6 In Experiments 1–3, the two prompting conditions are collapsed. Response Accuracy We consider respondents to have been accurate if they correctly answered six or seven of the seven numeracy questions, and we based this on final answers in order to take account of changes made after respondents returned to the question, post-prompting, potentially changing their answers (although speeders who changed their answers were not more accurate than those who did not). Overall, prompting did not affect accuracy: 22.6% percent of respondents were accurate in the prompt condition while 22.0% were accurate in the control condition, X 2 (1) = 0.10, ns. The fact that prompting can lead to slower responses apparently does not guarantee that these responses will be more accurate. But answering more slowly should increase the chances of answering accurately, especially if the task is neither too easy nor too difficult. If response accuracy is already near the ceiling without prompting then prompting will not noticeably increase accuracy. Similarly, if response accuracy is near the floor, at least some of that inaccuracy is more likely due to task difficulty than to speeding; pushing respondents to try hard through prompting is unlikely to matter. But if the task is moderately difficult, the effects of prompting on accuracy should be more visible. The numeracy ques- tions used in Experiment 4 involve basic mathematical skills so we reasoned that respondents’ education level could affect the difficulty of the questions and therefore the impact of prompting on response accuracy. More specifically, we suspected that the most educated respondents (college graduate or higher) would find the numeracy questions easy and so would be quite accurate overall, despite some speeding. At the other extreme, the least educated respondents (high school or less) might find the numeracy tasks so difficult that increased effort would not increase accuracy. Respondents with intermediate levels of education (some college or an Associate’s degree), in contrast, could find the task to be of modest difficulty and thus their accuracy should benefit most from prompting. This is largely what was observed (see Figure 2). The percent of those with a high school degree or less who were accurate was not affected by prompting: 5.9% with prompting, 8.9% without, X 2 = 1.09, n.s.. Similarly prompting did not affect accuracy for those with at least a Bachelors Degree: 30.9% with prompting, 34.3% without, X 2 = 0.87, n.s.. However significantly more respondents with some college or an Associate’s degree accurately answered the numeracy items: 22.2% with prompting, 16.0%, without, X 2 = 4.21, 52 Some College or Assoc. Degr. BA or More 40 % Straightlining 20 30 30 10 20 0 10 0 0 % Respondents Who Answered Accurately 40 High School or Less 50 FREDERICK G. CONRAD, ROGER TOURANGEAU, MICK P. COUPER AND CHAN ZHANG No Prompt Prompt Every Time No Prompt Prompt Every Time No Prompt p = 0.04. We see this as at least modest support for the prediction that prompting can improve response quality, particularly when the task is neither too difficult nor too easy. For other questions and response tasks, more thought may well improve accuracy for all respondents, but here the effects were concentrated in this broad group – about 40% of all respondents – with intermediate levels of education. 3.5 Speeding and straightlining In all four experiments, we observed a positive association between speeding and straightlining: the more respondents sped on the seven prompting items, the more likely they were to straightline in later grid questions (see Figure 3, which presents the percentage of control respondents who straightlined on at least one of the two grids by the number of times they sped previously). This suggests that speeding and straightlining may share an underlying tendency to minimize effort. This is consistent with findings reported by Zhang and Conrad (2014) and Greszki et al. (2015). This raises the question of whether speeding prompts reduced later straightlining as predicted by Hypothesis 3? We found mixed support for the hypothesis. In Experiment 4, 9.5% of respondents straightlined if they had been prompted previously whereas 12.8% of control respondents straightlined, X 2 (1) = 4.94, p < 0.05. However the effect was not significant in Experiments 1–3; it was in the predicted direction for one of the prompting conditions but in the opposite direction in the other. There were no grids in Experiment 5. It may be that the effects of speeding prompts have to be large to be carried over to the subsequent grid questions; the effect of prompting on speeding in Experiment 4 was as great as or greater than the effects in the other experiments (see Tables 2 and 3). There is evidence that speeding prompts can reduce 2 3 4 Speeding Incidences on Previous Questions Exper 1 Exper 3 Prompt Every Time Figure 2. Percent accurate respondents at three levels of education 1 5 6 or 7 Exper 2 Exper 4 Figure 3. Proportion of respondents who straightlined at least once by number of speeding incidents, Experiments 1– 4. straightlining from a study by Zhang (2013), and Zhang and Conrad (2016) who prompted both straightlining and speeding within grid questions and reported that either type of prompt reduced both behaviors on subsequent grid questions (see also Kunz & Fuchs, 2014a, 2014b). In addition, after they were prompted for straightlining or speeding, respondents provided longer answers to an open-ended question. These findings suggest that the act of prompting may be more important than the particular behavior triggering the prompt in reducing satisficing behaviors. 3.6 Longitudinal effectiveness of speeding prompts Our third research question asks whether there is any reduction in the prompt’s effectiveness after the first wave of a longitudinal survey. As indicated earlier, 37% and 53% of the control respondents sped at least once in the two waves of Experiment 5. Clearly, speeding is common among respondents in the representative sample used in the study, and so is not restricted to volunteer web panels. Speeding was more common in the second wave; this difference was significant in a MANOVA comparing number of speeding episodes in the seven prompting items across waves: F(1, 894) = 69.2, p < 0.001. Some respondents may have answered faster in the second wave because they recognized the questions from the first wave and did not need to read them as carefully the second time. However, despite the difference in speeding prevalence, prompting was equally effective in both waves: in a MANOVA, prompting reliably reduced speeding episodes compared to no prompt (F(2, 893) = 3.07, p = 0.05) and, most importantly, did not interact with the wave F(1, 894) = 0.01, p = 0.91 (see Figure 4). REDUCING SPEEDING IN WEB SURVEYS BY PROVIDING IMMEDIATE FEEDBACK Wave B 2.5 2 1.5 # of Speeding Episodes 3 Wave A No Prompt Prompt Every Time No Prompt Prompt Every Time Figure 4. Impact of prompts on mean number of speeding episodes in two waves of Experiment 5 with a probability sample. 3.7 Commitment and speeding Our fourth hypothesis is that by committing to careful participation, respondents will speed less. In order to test this, we analyzed the speeding behavior of all respondents in the commitment and control conditions in Experiment 4, not just the speeders (as in the analyses of the prompt’s effectiveness in this experiment), and we looked at the effect of commitment on all seven prompting items. We did this because (1) almost all (99%) of those asked to commit did so, and (2) the commitment or filler request was presented to all respondents at the start of the questionnaire making it possible to assess its effects from the first item onward in contrast to prompting whose assessment required at least one prompt to be administered and so could not begin until the second item. Commitment did, in fact, reduce speeding as predicted in the fourth hypothesis. Respondents who were asked to commit to thoughtful and careful responding sped 2.0 times, on average; respondents who were not asked to commit sped on 2.4 questions, F(1, 2447) = 15.4, p < 0.001. Commitment and prompting both reduced speeding but their effects seemed to be independent of each other. For comparability to the commitment analyses, we look at the effects of prompting for all respondents and over all seven items: Respondents in the prompt group sped on 1.98 questions in contrast to respondents in the no prompt control group who sped 2.44 times on average (F(1, 2447) = 28.15, p < 0.001); prompting and commitment did not interact (F(2, 2447) = 0.11, n.s.) We also examined the effects of commitment on response accuracy for the numeracy items. Commitment improved accuracy (i.e., answering 6 or 7 questions correctly) for respondents with a college degree or higher (38.8% vs. 26.8%, X 2 (1) = 11.15, p < 0.001) but not significantly for those with 53 a high school education or less (9.0% vs. 5.8% X 2 (1) = 1.23, n.s.), or some college or an Associates degree, 18.8% vs. 19.2%, X 2 (1) = 0.02, n.s; the interaction of commitment and education level was marginally significant, X 2 (2) = 5.23, p = 0.08. This interaction contrasts with the interaction of prompting and education where it was respondents with intermediate levels of education (some college or an Associate’s degree) whose accuracy was improved by the intervention. When appropriately motivated (committed) it seems that more educated respondents may shift into a higher mental gear that is not available to respondents with less education, and which they do not engage when they are simply prompted. Finally, commitment strengthened the effect of prompting in reducing later straightlining. As discussed before, there was less straightlining in the prompt than control condition but straightlining was particularly rare when committed respondents were also in the prompt condition: 6.2% of respondents straightlined in the Commitment-Prompting condition compared to 12.4%, 11.9%, and 12.8% in the No Prompt-Commitment, Prompt-No Commitment, and No Prompt-No Commitment conditions, X 2 (3) = 9.67, p = 0.022. One possibility is that having committed to careful performance, respondents were more sensitive to prompting and thus maintained their increased conscientiousness when completing subsequent items, including the grids. In summary, asking respondents to commit to answering with care led to fewer speeding episodes than did a neutral statement. The effect was independent of the effect of prompting. Commitment also improved response accuracy for respondents with at least a college degree. Given the independent effects of commitment and prompting on speeding, and their impact on different subgroups’ accuracy, joint use of the two techniques would likely have additive benefit. 3.8 Breakoffs Although it reduced speeding and improved response accuracy, the prompting intervention could have annoyed respondents, leading some to break off. Overall, the percentage of speeders who broke off in Experiments 1–4 was small, ranging from none to 2.4%. (The Experiment 5 data set included only completed cases so we could not compute the breakoff rate for that study.) Breakoff data are presented by experimental condition in Table 4 (comparisons are evaluated with Fisher’s exact test because of the small number of breakoffs). Prompting respondents once did not significantly increase breakoffs compared to the No Prompt control condition in any of the experiments (rightmost column), suggesting that, in these studies, an intervention does not discourage respondents from completing the task. Prompting every time also did not reliably affect the number of breakoffs compared to the no prompt control condition in Experiments 2, 3 and 4 54 FREDERICK G. CONRAD, ROGER TOURANGEAU, MICK P. COUPER AND CHAN ZHANG Table 4 Percent and number of breakoffs, by condition and experiment Breakoffs Fisher’s Exact Test for Difference No Prompt Prompt Every Time Prompt Once Experiment % n % n % 1 2 3 4 0.0 0.1 1.0 0.3 0 1 4 3 1.1 0.6 1.7 0.3 7 4 7 3 0.2 0.7 2.4 - but it did increase breakoffs slightly in Experiment 1 (second column from right) where there were seven more breakoffs in the prompt every time than no prompt conditions. Although the difference in Experiment 1 is significant, the actual increase in breakoffs (parenthesized numbers in the table) is small compared to the overall sample size. It could be that prompting every time for a larger number of questions will increase breakoffs for chronic speeders. Given that a single prompt reduced speeding as much as multiple prompts, and there is no compelling evidence that a single prompt increases breakoffs, the one prompt design seems like the sensible approach. 4 Discussion and Conclusions Speeding was common across the six web survey experiments we conducted: 40 to 80% of the control respondents answered at least one question so quickly it is unlikely they read the entire question and even less likely they thought carefully about their answers. But, the lesson from these experiments is that, by incorporating interactive feedback into online questionnaires, survey designers may be able to reduce the prevalence of speeding and thus improve the quality of responses. Prompting respondents immediately after they were caught speeding reduced the prevalence of this behavior in all of the experiments, whether the prompting occurred early or late, whether in a single wave or across two monthly waves. The reduction in speeding was associated with some evidence of improved response quality: the prompts increased response accuracy on a set of seven numeracy items for the group of respondents with moderate educational attainment and they reduced straightlining across all respondents in the same experiment, although not consistently in the other experiments. The intervention had very little impact on breakoffs, suggesting that its benefits substantially outweigh its costs. Moreover, prompting complemented the benefits of commitment: the two techniques improved response accuracy for different, non-overlapping subgroups, and when respondents had both committed to careful responding and were in the prompt condition they were Every Time vs. No Prompt Once vs. No Prompt n p p 1 5 9 - < 0.01 0.21 0.40 1.00 0.49 0.12 0.26 - especially unlikely to straightline. By exploiting the interactivity of web surveys to reduce speeding, designers can potentially improve response quality beyond what is observed in other modes4 . For example, speeding is likely to be very common in paper questionnaires, although it is typically impossible for researchers to detect or to intervene to reduce its prevalence. And in spoken interviews there is pressure to speak (respond) after more than about a second of silence, suggesting that respondents in theses modes may rush their answers to fill the silence (see Schober et al., 2015). But designers of online questionnaires can easily measure response times and deliver feedback accordingly. Although we observed increased accuracy with slower responses for a subset of respondents, we cannot say for sure that speeding always leads to less accurate responses or that slowing respondents necessarily increases their accuracy. Greszki et al. (2015) removed responses they flagged as “too fast” and observed virtually no effect on response distributions. But they examined items in a single domain – politics and elections. In our studies, the experimental items required numerical responses that may call for memory- and arithmetic-based processes. Taking more time could well improve the accuracy of the answers and change the response distributions. More generally, the relationship between speeding and accuracy may depend on item type, difficulty, and domain. In the absence of clear information about the quality of very fast answers to particular questions, it seems preferable to reduce speeding rather than to adjust for it analytically. There may be ways to increase the benefits of interactive prompting. For example, customizing the speeding thresholds could result in prompts for respondents whose response times are above the generic threshold we used, but who are 4 Other examples of online, interactive prompts include (1) DeRouvray and Couper (2002) who prompted respondents after answering “Don’t know” to provide a substantive response, and (2) Holland and Christian (2009) who prompted respondents to provide more content in open responses. REDUCING SPEEDING IN WEB SURVEYS BY PROVIDING IMMEDIATE FEEDBACK speeding nevertheless. The speeding threshold could be customized for individuals based, for example, on their reading times, or for subgroups whose cognitive function or tendency to speed is known a priori, such as younger versus older respondents. We see this as an important next step in this line of research. But the effectiveness of approaches like this, that try to motivate respondents to exert more effort or at least avoid the stigma of being caught taking shortcuts, have limits: hardcore satisficers seem to experience none of the negative psychological consequences that “maximizers” do when their performance is sub-optimal (e.g. B. Schwartz et al., 2002). Those who are strongly disposed to take shortcuts are likely to be unaffected by prompting or any similar technique. This suggests that knowing something about the characteristics of respondents who are both likely to speed and likely to respond to a prompt could allow more targeted interventions. Exploring this type of issue would also be an important extension of the current research. A possible downside of interactive prompting is that the sense of being monitored – and the likely increase in accountability – may create a sense of social presence (the feeling that someone is there) which may discourage candid responding about sensitive topics. Zhang (2013), and Zhang and Conrad (2016) report that prompting reduced speeding but increased socially desirable responding for two subsequent questions on sensitive topics (marijuana use and exercise frequency). Online data collection affords survey designers many options to improve response quality that are not available to designers of paper questionnaires, in particular interactive techniques such as the prompting technique tested here. Yet, in practice, web questionnaires do not exploit the interactive capabilities of the mode very often. The results presented here demonstrate that progress can be made with relatively modest changes to current practice. 5 Acknowledgments Replication materials are published on OpenICPSR (Conrad, Zhang, Tourangeau, & Couper, 2017) and – as usual – as supplementary materials alongside this article on SRM’s Website. We thank Reg Baker, Courtney Kennedy, and Stephanie Stern for advice and assistance. In addition, we thank the National Institute of Child Health and Development (NIH Grants R01 HD041386-04A1 and R01 HD04138601A1) and the National Science Foundation (NSF Grant SES0106222) for financial support. Any opinions, findings and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the National Science Foundation or the National Institutes of Health. 55 References Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., . . . Lavrakas, P. J. (2010). Research synthesis: AAPOR report on online panels. Public Opinion Quarterly, 74(4), 711–781. Callegaro, M., Yang, Y., Bhola, D. S., Dillman, D. A., & Chin, T. Y. (2009). Response latency as an indicator of optimizing in online questionnaires. Bulletin de Méthodologie Sociologique, 103, 5–25. Cannell, C. F., Miller, P. V., & Oksenberg, L. (1981). Research on interviewing techniques. Sociological Methodology, 12, 389–437. Carver, R. P. (1992). Reading rate: theory, research, and practical implications. Journal of Reading, 36, 84–95. Conrad, F. G., Schober, M. F., & Coiner, T. (2007). Bringing features of human dialogue to web surveys. Applied Cognitive Psychology, 21, 165–188. Conrad, F. G., Zhang, C., Tourangeau, R., & Couper, M. P. (2017). Reducing speeding in web surveys by providing immediate feedback [data set]. Ann Arbor: Inter-university Consortium of Political and Social Research. doi:10.3886/E100431V1 DeRouvray, C. & Couper, M. P. (2002). Designing a strategy for reducing “no opinion” responses in web-based surveys. Social Science Computer Review, 20, 3–9. Draisma, S. & Dijkstra, W. (2004). Response latency and (para) linguistic expressions as indicators of response error. In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (pp. 131–149). Hoboken, NJ: Wiley. Greszki, R., Meyer, M., & Schoen, H. (2015). Exploring the effects of removing “too fast” responses and respondents from web surveys. Public Opinion Quarterly, 79(2), 471–503. Heerwegh, D. (2003). Explaining response latencies and changing answers using client-side paradata from a web survey. Social Science Computer Review, 21, 360–373. Heerwegh, D. & Loosveldt, G. (2008). Face-to-face versus web surveying in a high-internet-coverage population: differences in response quality. Public Opinion Quarterly, 72(5), 836–846. Holland, J. L. & Christian, L. M. (2009). The influence of topic interest and interactive probing on responses to open-ended questions in web surveys. Social Science Computer Review, 27(2), 196–212. Jackson, M. D. & McClelland, J. L. (1979). Processing determinants of reading speed. Journal of Experimental Psychology: General, 108(2), 151. Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3), 213–236. 56 FREDERICK G. CONRAD, ROGER TOURANGEAU, MICK P. COUPER AND CHAN ZHANG Kunz, T. & Fuchs, M. (2014a). Instant interactive feedback in grid questions: insights from eye tracking data. Paper presented at the sixth European Congress of Methodology, Utrecht, Netherlands. Kunz, T. & Fuchs, M. (2014b). Instant interactive feedback in grid questions: reminding web survey respondents of speeding and nondifferentiation. Paper presented at annual conference of the American Association for Public Opinion Research, Anaheim, CA. Malhotra, N. (2008). Completion time and response order effects in web surveys. Public Opinion Quarterly, 72(5), 914–934. Nielsen, J. & Loranger, H. (2006). Prioritizing web usability. Berkeley, CA: New Riders. Oksenberg, L., Vinokur, A., & Cannell, C. F. (1979). Effects of commitment to being a good respondent on interview performance. In C. F. Cannell, L. Oksenberg, & J. M. Converse (Eds.), Experiments in interviewing techniques (pp. 74–108). Ann Arbor: Survey Research Center, University of Michigan. Park, D. C. & Reuter-Lorenz, P. (2009). The adaptive brain: aging and neurocognitive scaffolding. Annual Review of Psychology, 60, 173–196. Park, D. C. & Schwarz, N. (Eds.). (2000). Cognitive aging: a primer. Philadelphia, PA: Psychology Press. Park, D. C., Smith, A. D., Lautenschlager, G., Earles, J. L., Frieske, D., Zwahr, M., & Gaines, C. L. (1996). Mediators of long-term memory performance across the life span. Psychology and aging, 11(4), 621. Salthouse, T. A. (1991). Theoretical perspectives on cognitive aging. Hillsdale, NJ: Lawrence Erlbaum Associates. Salthouse, T. A. (1996). The processing-speed theory of adult age differences in cognition. Psychological review, 103(3), 403. Schober, M. F., Conrad, F. G., Antoun, C., Ehlen, P., Fail, S., Hupp, A. L., . . . Zhang, C. (2015). Precision and disclosure in text and voice interviews on smartphones. PLOS ONE, 10(6). doi:10.1371/journal.pone.0128337 Schwartz, B., Ward, A., Monterosso, J., Lyubomirsky, S., White, K., & Lehman, D. R. (2002). Maximizing versus satisficing: happiness is a matter of choice. Journal of Personality and Social Psychology, 83(5), 1178– 1197. Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, H. G. (1997). The role of numeracy in understanding the benefit of screening mammography. Annals of internal medicine, 127(11), 966–972. Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological review, 63(2), 129. Taylor, S. E., Frankenpohl, H., & Pettee, J. L. (1960). Grade level norms for the components of the fundamental reading skills. EDL Research and Information Bulletin #3. New York: EDL/McGraw Hill. Villar, A., Malka, A., & Krosnick, J. (2010). Assessing the accuracy of the face-to-face recruited internet survey platform: a comparison of behavioral and healthrelated findings from the FFRISP with those of major national surveys. Paper presented at the annual conference of the American Association for Pubic Opinion Research, Chicago, IL. Yan, T. & Tourangeau, R. (2008). Fast times and easy questions: the effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22(1), 51–68. Zhang, C. (2013). Satisficing in web surveys: implications for data quality and strategies for reduction. Unpublished doctoral dissertation. University of Michigan, Ann Arbor, MI. Zhang, C. & Conrad, F. G. (2014). Investigation of speeding in web surveys: tendency to speed and its association with straightlining. Survey Research Methods, 8(2), 127–135. Zhang, C. & Conrad, F. G. (2016). Intervention as a strategy to reduce satisficing behaviors in web surveys: evidence from two experiments on how it works. Social Science Computer Review. Online first. doi:10.1177/ 0894439316683923 REDUCING SPEEDING IN WEB SURVEYS BY PROVIDING IMMEDIATE FEEDBACK Appendix A Autobiographical Quantity Questions 1. Overall, how many overnight trips have you taken in the PAST 2 YEARS? 2. In the past year, about how many calories did you consume in a typical day? 3. In an average month, how much money do you spend on alcohol? (Please enter 0 if you spend nothing on alcohol) 4. During the last 7 days, how many drinks of any kind of alcoholic beverages did you drink? 5. During the past 30 days, how many times did you have 5 or more drinks of alcoholic beverage on one occasion? 6. On how many days in the past year did you stay in bed because of illness or injury? 7. During the past 10 years, how many times did you get any traffic tickets, including speeding tickets and parking tickets? 57 58 FREDERICK G. CONRAD, ROGER TOURANGEAU, MICK P. COUPER AND CHAN ZHANG Appendix B Numeracy Questions Now for some questions on your perception of health related risk . . . 1. Different sports vary in the risk of injury. Which of the following numbers represents the biggest risk of getting injured while playing sport? 2 1 in 200 2 1 in 10 2 1 in 50 2 1 in 100 2. Heart disease is the leading cause of death in the United States. If Person A’s chance of getting a particular type of heart disease is 1% in ten years, and person B’s risk is half that of A’s, what is B’s risk? 2 2% 2 5% 2 0.2% 2 0.5% 3. Viruses cause some of the most familiar infectious diseases, such as the common cold and the flu. If the chance of getting a particular type of viral infection is 0.0005, about how many out of 10,000 are expected to get infected? 2 0.5 25 2 50 2 500 4. Chronic diseases, such as heart disease, stroke, cancer, diabetes, and arthritis, are among the most common, costly, and preventable of all health problems in the U.S. If the estimated prevalence of a particular type of chronic disease is 1 out of 1,000, what percent of people would get the disease? 2 1% 2 0.1% 2 0.01% 2 0.001% 5. If Vaccine A’s chance of causing serious side effects is 0.1% and Vaccine B’s chance is double that of A’s, what percent of people are not expected to experience serious side effects after taking Vaccine B? 2 98% 2 99% 2 99.9% 2 99.8% 6. Vision loss is a public health problem in the U.S. If Persons A’s chance of vision loss is 1 in 100 in twenty years, and person B’s risk is double that of A’s, what is B’s risk? 2 2 out of 200 2 1 out of 200 2 2 out of 50 2 None of the above 7. Iron deficiency is a condition resulting from too little iron in the body. If the risk of iron deficiency in a certain demographic group is 0.2%, how many people out of 1,000 in that group would be expected to have iron deficiency? 2 0.2 22 2 20 2 200 59 REDUCING SPEEDING IN WEB SURVEYS BY PROVIDING IMMEDIATE FEEDBACK Appendix C Grid questions 1. Indicate how much you favor or oppose each of the following statements. Strongly oppose Somewhat oppose Neither favor nor oppose Somwhat favor Strongly favor Avoiding “fast food?” # # # # # Maintaining a healthy diet? # # # # # Monitoring cholesterol levels closely? # # # # # Emphasizing the taste of food rather than its nutritional value? # # # # # Paying close attention to the nutritional information on food packaging? # # # # # Limiting the amount of red meat in your diet? # # # # # Balancing one’s diet across the key food groups? # # # # # 2. (Experiment 1 and 2 only) The next few questions ask about the style or manner you use when carrying out different mental tasks. Your answers to the questions should reflect the manner in which you typically engage in each of the tasks mentioned. There are no right or wrong answers; we only ask that you provide honest and accurate answers. Strongly disagree # Moderately disagree # Neither agree nor disagree # Moderately agree # Strongly agree # When listening to someone describing their experiences, I try to mentally picture what was happening. # # # # # I do a lot of reading. # # # # # I find it helps to think in terms of mental pictures when doing many things. # # # # # I enjoy learning new words. # # # # # I enjoy work that requires the use of words. 60 FREDERICK G. CONRAD, ROGER TOURANGEAU, MICK P. COUPER AND CHAN ZHANG 3. For each of the following statements, please rate to what extent it characterizes you. Extremely uncharacteristic Somewhat uncharacteristic Somewhat characteristic Extremely characteristic I prefer to avoid taking extreme positions # # # # I want to know exactly what is good and bad about everything # # # # If something does not affect me, I do not usually determine if it is good or bad # # # # There are many things for which I do not have a preference # # # # I like to have strong opinions even when I am not personally involved # # # # I would rather have a strong opinion than no opinion at all # # # # I only form strong opinions when I have to # # # # 61 REDUCING SPEEDING IN WEB SURVEYS BY PROVIDING IMMEDIATE FEEDBACK Appendix D Percent respondents speeding on each question after speeding on one or more earlier questions in each experiment. No Prompt Prompt Every Time Prompt Once % % % Experiment 1 Q2 37.1 Q3 63.7 Q4 85.8 Q5 75.0 Q6 55.3 Q7 59.6 20.1 53.6 75.8 64.4 44.9 55.0 Experiment 2 Q2 24.2 Q3 64.8 Q4 83.9 Q5 72.0 Q6 47.1 Q7 66.5 Prompt vs. No Prompt Prompt Every Time vs. Once χ2 p χ2 p 15.0 52.6 79.9 68.6 50.7 60.3 20.96 6.29 9.68 12.12 9.23 0.70 < 0.01 0.01 < 0.01 < 0.01 < 0.01 0.40 1.42 < 1.00 1.80 2.03 3.99 3.67 0.23 0.84 0.18 0.15 0.05 0.06 21.1 56.7 78.1 59.3 41.5 55.0 15.3 53.9 80.6 62.4 45.0 60.5 2.60 5.27 3.38 20.56 2.52 14.20 0.11 0.02 0.07 < 0.01 0.11 < 0.01 1.99 < 1.00 < 1.00 1.16 1.49 3.93 0.16 0.56 0.40 0.28 0.22 0.05 Experiment 3 Q2 0.7 Q3 43.9 Q4 40.3 Q5 58.8 Q6 17.1 Q7 42.5 2.0 49.7 39.5 49.1 13.7 33.4 2.1 55.9 36.8 52.3 14.5 36.0 Experiment 4 Q2 55.2 Q3 36.9 Q4 63.2 Q5 37.7 Q6 45.4 Q7 46.7 42.7 23.0 56.1 24.5 35.2 34.0 - 17.09 31.41 7.53 33.65 18.44 29.07 < 0.01 < 0.01 0.01 < 0.01 < 0.01 < 0.01 - - Experiment 5a Q2 50 Q3 62.1 Q4 61.9 Q5 30.5 Q6 54.0 Q7 45.2 85.71 49.3 55.1 18.1 26.1 36.3 - too few cases 1.34 0.25 0.55 0.46 3.72 0.05 15.52 < 0.01 1.76 0.19 - - Experiment 5b Q2 100.0 Q3 58.3 Q4 74.1 Q5 27.3 Q6 44.1 Q7 55.5 70.6 42.4 48.2 21.5 31.1 43.2 - too few cases 3.58 0.06 11.70 < 0.01 1.20 0.27 5.08 0.02 4.49 0.03 - - too few cases 3.17 0.08 0.33 0.56 5.06 0.02 1.63 0.20 6.13 0.01 too few cases 1.17 0.28 < 1.00 0.54 < 1.00 0.45 < 1.00 0.75 < 1.00 0.48