With the rise of Web-based applications, it is both important and feasible for human-computer int... more With the rise of Web-based applications, it is both important and feasible for human-computer interaction practitioners to measure a product’s user experience. While quantifying user attitudes at a small scale has been heavily studied, in this industry case study, we detail best Happiness Track-ing Surveys (HaTS) for collecting attitudinal data at a large scale directly in the product and over time. This method was developed at Google to track attitudes and open-ended feedback over time, and to characterize products ’ user bases. This case study of HaTS goes beyond the design of the ques-tionnaire to also suggest best practices for appropriate sam-pling, invitation techniques, and its data analysis. HaTS has been deployed successfully across dozens of Google’s prod-ucts to measure progress towards product goals and to in-form product decisions; its sensitivity to product changes has been demonstrated widely. We are confident that teams in other organizations will be able to embrace ...
Researchers increasingly utilize online tools to gather insights. We show how privacy comfort as ... more Researchers increasingly utilize online tools to gather insights. We show how privacy comfort as measured by questionnaires differs across various survey sample providers. To investigate potential differences depending on provider, we fielded a small set of privacy-related benchmark questions regarding past experience, present and future concerns to six major US survey providers. We found substantial differences depending on privacy benchmark and provider population, illustrating that privacy-related research may yield different insights depending on provider choice. 1.
ACM 978-1-4503-1952-2/13/04. Change aversion is a natural response, which technology often exacer... more ACM 978-1-4503-1952-2/13/04. Change aversion is a natural response, which technology often exacerbates. Evolutionary changes can be subtle and occur over many generations. But Internet users must sometimes deal with sudden, significant product changes to applications they rely on and identify with. Despite the best intentions of designers and product managers, users often experience anxiety and confusion when faced with a new interface or changed functionality. While some change aversion is often inevitable, it can also be managed and minimized with the right steps. This case study describes how our understanding of change aversion helped minimize negative effects for the transition of the Google Docs List to Google Drive, a product for file storage in the cloud. We describe actions that allowed for a launch with no aversion. Author Keywords Case study; satisfaction; happiness; change aversion; UI design; usability; improvement; user-centered design; user experience; user research; ...
In this paper, we introduce a new construct for measuring individuals’ privacy-related beliefs an... more In this paper, we introduce a new construct for measuring individuals’ privacy-related beliefs and understandings, namely their perception of the frequency with which information about individuals is gathered and used by others for advertising purposes. We introduce a preliminary instrument for measuring this perception, called the Ad Practice Frequency Perception Scale. We report data from a survey using this instrument, as well as the results of an initial clustering of participants based on this data. Our results, while preliminary, suggest that this construct may have future potential to characterize and segment individuals, and is worthy of further exploration.
Survey research is widely used in human-computer interaction (HCI) to measure users’ attitudes an... more Survey research is widely used in human-computer interaction (HCI) to measure users’ attitudes and collect product feedback. At a glance, survey research seems straightforward: it’s easy to write questions, and there are many inexpensive tools to field surveys. However, there is a wide gap between quickand- dirty surveys and surveys that are properly planned, constructed, and analyzed. This interactive full-day tutorial explains all stages of the survey research life cycle, and examines approaches for surveys to meet specific HCI goals. The tutorial is divided into two parts: During the first half, we discuss the use of surveys in HCI research, and review survey fundamentals (including survey types and examples, a short history of survey research, survey appropriateness in the context of HCI, an overview of the survey life cycle, research goals & constructs, and population & sampling). In the second half, we will examine elements of high quality questionnaire design, including when ...
As the influence and use of artificial intelligence (AI) have grown and its transformative potent... more As the influence and use of artificial intelligence (AI) have grown and its transformative potential has become more apparent, many questions have been raised regarding the economic, political, and social implications of its use. Public opinion plays an important role in these discussions, influencing product adoption, commercial development, research funding, and regulation. In this paper we present results of a survey of public opinion of artificial intelligence conducted with 10,005 respondents spanning eight countries and six continents. We report widespread perception that AI will have significant impact on society, and highlight public optimism about applications for healthcare as well as public concerns about privacy and job loss. We also identify four key sentiments (exciting, useful, worrying, and futuristic) whose prevalence distinguishes response to AI in different countries.
Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society
As the influence and use of artificial intelligence (AI) have grown and its transformative potent... more As the influence and use of artificial intelligence (AI) have grown and its transformative potential has become more apparent, many questions have been raised regarding the economic, political, social, and ethical implications of its use. Public opinion plays an important role in these discussions, influencing product adoption, commercial development, research funding, and regulation. In this paper we present results of an in-depth survey of public opinion of artificial intelligence conducted with 10,005 respondents spanning eight countries and six continents. We report widespread perception that AI will have significant impact on society, accompanied by strong support for the responsible development and use of AI, and also characterize the public's sentiment towards AI with four key themes (exciting, useful, worrying, and futuristic) whose prevalence distinguishes response to AI in different countries. CCS CONCEPTS • Social and professional topics → Computing / technology policy; • Human-centered computing → Human computer interaction (HCI); • Computing methodologies → Artificial intelligence.
Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, 2015
Online surveys are widely used in human-computer interaction (HCI) to gather feedback and measure... more Online surveys are widely used in human-computer interaction (HCI) to gather feedback and measure satisfaction; at a glance many tools are available and the cost of conducting surveys appears low. However, there is a wide gap between quick-and-dirty surveys, and surveys that are properly planned, constructed, and analyzed. This course examines survey research approaches that meet HCI goals, selecting the appropriate sampling method, questionnaire design best practices, identifying and avoiding common survey biases, and questionnaire evaluation. Attendees will gain an appreciation for the breadth and depth of surveys in HCI, combined with keys to conducting valid, reliable, and impactful survey research themselves.
With the rise of Web-based applications, it is both important and feasible for human-computer int... more With the rise of Web-based applications, it is both important and feasible for human-computer interaction practitioners to measure a product's user experience. While quantifying user attitudes at a small scale has been heavily studied, in this industry case study, we detail best Happiness Tracking Surveys (HaTS) for collecting attitudinal data at a large scale directly in the product and over time. This method was developed at Google to track attitudes and open-ended feedback over time, and to characterize products' user bases. This case study of HaTS goes beyond the design of the questionnaire to also suggest best practices for appropriate sampling, invitation techniques, and its data analysis. HaTS has been deployed successfully across dozens of Google's products to measure progress towards product goals and to inform product decisions; its sensitivity to product changes has been demonstrated widely. We are confident that teams in other organizations will be able to embrace HaTS as well, and, if necessary, adapt it for their unique needs.
Researchers increasingly utilize online tools to gather insights. We show how privacy comfort as ... more Researchers increasingly utilize online tools to gather insights. We show how privacy comfort as measured by questionnaires differs across various survey sample providers. To investigate potential differences depending on provider, we fielded a small set of privacy-related benchmark questions regarding past experience, present and future concerns to six major US survey providers. We found substantial differences depending on privacy benchmark and provider population, illustrating that privacy-related research may yield different insights depending on provider choice. Represe ntation US pop. US Mturker US pop. US pop. US pop. US pop.
CHI '14 Extended Abstracts on Human Factors in Computing Systems, 2014
Surveys are a commonly used method within HCI research. While it initially appears easy and inexp... more Surveys are a commonly used method within HCI research. While it initially appears easy and inexpensive to conduct surveys, overlooking key considerations in questionnaire design and the survey research process can yield skewed, biased, or entirely invalid survey results. Fortunately decades of academic research and analysis exist on optimizing the validity and reliability of survey data, from which this course will draw. To enable the creation of unbiased surveys, this course demonstrates questionnaire design biases and pitfalls, provides best practices for minimizing these, and reviews different uses of surveys within HCI.
Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems - CHI EA '14, 2014
This case study presents a critical analysis of microsurveys as a method for conducting user expe... more This case study presents a critical analysis of microsurveys as a method for conducting user experience research. We focus specifically on Google Consumer Surveys (GCS) and analyze a combination of log data and GCSs run by the authors to investigate how they are used, who the respondents are, and the quality of the data. We find that such microsurveys can be a great way to quickly and cheaply gather large amounts of survey data, but that there are pitfalls that user experience researchers should be aware of when using the method.
With the rise of Web-based applications, it is both important and feasible for human-computer int... more With the rise of Web-based applications, it is both important and feasible for human-computer interaction practitioners to measure a product’s user experience. While quantifying user attitudes at a small scale has been heavily studied, in this industry case study, we detail best Happiness Track-ing Surveys (HaTS) for collecting attitudinal data at a large scale directly in the product and over time. This method was developed at Google to track attitudes and open-ended feedback over time, and to characterize products ’ user bases. This case study of HaTS goes beyond the design of the ques-tionnaire to also suggest best practices for appropriate sam-pling, invitation techniques, and its data analysis. HaTS has been deployed successfully across dozens of Google’s prod-ucts to measure progress towards product goals and to in-form product decisions; its sensitivity to product changes has been demonstrated widely. We are confident that teams in other organizations will be able to embrace ...
Researchers increasingly utilize online tools to gather insights. We show how privacy comfort as ... more Researchers increasingly utilize online tools to gather insights. We show how privacy comfort as measured by questionnaires differs across various survey sample providers. To investigate potential differences depending on provider, we fielded a small set of privacy-related benchmark questions regarding past experience, present and future concerns to six major US survey providers. We found substantial differences depending on privacy benchmark and provider population, illustrating that privacy-related research may yield different insights depending on provider choice. 1.
ACM 978-1-4503-1952-2/13/04. Change aversion is a natural response, which technology often exacer... more ACM 978-1-4503-1952-2/13/04. Change aversion is a natural response, which technology often exacerbates. Evolutionary changes can be subtle and occur over many generations. But Internet users must sometimes deal with sudden, significant product changes to applications they rely on and identify with. Despite the best intentions of designers and product managers, users often experience anxiety and confusion when faced with a new interface or changed functionality. While some change aversion is often inevitable, it can also be managed and minimized with the right steps. This case study describes how our understanding of change aversion helped minimize negative effects for the transition of the Google Docs List to Google Drive, a product for file storage in the cloud. We describe actions that allowed for a launch with no aversion. Author Keywords Case study; satisfaction; happiness; change aversion; UI design; usability; improvement; user-centered design; user experience; user research; ...
In this paper, we introduce a new construct for measuring individuals’ privacy-related beliefs an... more In this paper, we introduce a new construct for measuring individuals’ privacy-related beliefs and understandings, namely their perception of the frequency with which information about individuals is gathered and used by others for advertising purposes. We introduce a preliminary instrument for measuring this perception, called the Ad Practice Frequency Perception Scale. We report data from a survey using this instrument, as well as the results of an initial clustering of participants based on this data. Our results, while preliminary, suggest that this construct may have future potential to characterize and segment individuals, and is worthy of further exploration.
Survey research is widely used in human-computer interaction (HCI) to measure users’ attitudes an... more Survey research is widely used in human-computer interaction (HCI) to measure users’ attitudes and collect product feedback. At a glance, survey research seems straightforward: it’s easy to write questions, and there are many inexpensive tools to field surveys. However, there is a wide gap between quickand- dirty surveys and surveys that are properly planned, constructed, and analyzed. This interactive full-day tutorial explains all stages of the survey research life cycle, and examines approaches for surveys to meet specific HCI goals. The tutorial is divided into two parts: During the first half, we discuss the use of surveys in HCI research, and review survey fundamentals (including survey types and examples, a short history of survey research, survey appropriateness in the context of HCI, an overview of the survey life cycle, research goals & constructs, and population & sampling). In the second half, we will examine elements of high quality questionnaire design, including when ...
As the influence and use of artificial intelligence (AI) have grown and its transformative potent... more As the influence and use of artificial intelligence (AI) have grown and its transformative potential has become more apparent, many questions have been raised regarding the economic, political, and social implications of its use. Public opinion plays an important role in these discussions, influencing product adoption, commercial development, research funding, and regulation. In this paper we present results of a survey of public opinion of artificial intelligence conducted with 10,005 respondents spanning eight countries and six continents. We report widespread perception that AI will have significant impact on society, and highlight public optimism about applications for healthcare as well as public concerns about privacy and job loss. We also identify four key sentiments (exciting, useful, worrying, and futuristic) whose prevalence distinguishes response to AI in different countries.
Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society
As the influence and use of artificial intelligence (AI) have grown and its transformative potent... more As the influence and use of artificial intelligence (AI) have grown and its transformative potential has become more apparent, many questions have been raised regarding the economic, political, social, and ethical implications of its use. Public opinion plays an important role in these discussions, influencing product adoption, commercial development, research funding, and regulation. In this paper we present results of an in-depth survey of public opinion of artificial intelligence conducted with 10,005 respondents spanning eight countries and six continents. We report widespread perception that AI will have significant impact on society, accompanied by strong support for the responsible development and use of AI, and also characterize the public's sentiment towards AI with four key themes (exciting, useful, worrying, and futuristic) whose prevalence distinguishes response to AI in different countries. CCS CONCEPTS • Social and professional topics → Computing / technology policy; • Human-centered computing → Human computer interaction (HCI); • Computing methodologies → Artificial intelligence.
Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, 2015
Online surveys are widely used in human-computer interaction (HCI) to gather feedback and measure... more Online surveys are widely used in human-computer interaction (HCI) to gather feedback and measure satisfaction; at a glance many tools are available and the cost of conducting surveys appears low. However, there is a wide gap between quick-and-dirty surveys, and surveys that are properly planned, constructed, and analyzed. This course examines survey research approaches that meet HCI goals, selecting the appropriate sampling method, questionnaire design best practices, identifying and avoiding common survey biases, and questionnaire evaluation. Attendees will gain an appreciation for the breadth and depth of surveys in HCI, combined with keys to conducting valid, reliable, and impactful survey research themselves.
With the rise of Web-based applications, it is both important and feasible for human-computer int... more With the rise of Web-based applications, it is both important and feasible for human-computer interaction practitioners to measure a product's user experience. While quantifying user attitudes at a small scale has been heavily studied, in this industry case study, we detail best Happiness Tracking Surveys (HaTS) for collecting attitudinal data at a large scale directly in the product and over time. This method was developed at Google to track attitudes and open-ended feedback over time, and to characterize products' user bases. This case study of HaTS goes beyond the design of the questionnaire to also suggest best practices for appropriate sampling, invitation techniques, and its data analysis. HaTS has been deployed successfully across dozens of Google's products to measure progress towards product goals and to inform product decisions; its sensitivity to product changes has been demonstrated widely. We are confident that teams in other organizations will be able to embrace HaTS as well, and, if necessary, adapt it for their unique needs.
Researchers increasingly utilize online tools to gather insights. We show how privacy comfort as ... more Researchers increasingly utilize online tools to gather insights. We show how privacy comfort as measured by questionnaires differs across various survey sample providers. To investigate potential differences depending on provider, we fielded a small set of privacy-related benchmark questions regarding past experience, present and future concerns to six major US survey providers. We found substantial differences depending on privacy benchmark and provider population, illustrating that privacy-related research may yield different insights depending on provider choice. Represe ntation US pop. US Mturker US pop. US pop. US pop. US pop.
CHI '14 Extended Abstracts on Human Factors in Computing Systems, 2014
Surveys are a commonly used method within HCI research. While it initially appears easy and inexp... more Surveys are a commonly used method within HCI research. While it initially appears easy and inexpensive to conduct surveys, overlooking key considerations in questionnaire design and the survey research process can yield skewed, biased, or entirely invalid survey results. Fortunately decades of academic research and analysis exist on optimizing the validity and reliability of survey data, from which this course will draw. To enable the creation of unbiased surveys, this course demonstrates questionnaire design biases and pitfalls, provides best practices for minimizing these, and reviews different uses of surveys within HCI.
Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems - CHI EA '14, 2014
This case study presents a critical analysis of microsurveys as a method for conducting user expe... more This case study presents a critical analysis of microsurveys as a method for conducting user experience research. We focus specifically on Google Consumer Surveys (GCS) and analyze a combination of log data and GCSs run by the authors to investigate how they are used, who the respondents are, and the quality of the data. We find that such microsurveys can be a great way to quickly and cheaply gather large amounts of survey data, but that there are pitfalls that user experience researchers should be aware of when using the method.
Uploads
Papers by Aaron Sedley