CHI '13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA '13, 2013
ABSTRACT User Experience (UX) leaders and managers are required to continually adapt to changes i... more ABSTRACT User Experience (UX) leaders and managers are required to continually adapt to changes in: organizational strategies and re-structuring, resources, technology, economic pressures, and other factors. Simultaneously, more companies are realizing that they need UX expertise to ensure that they are competitive in today's marketplace. This panel is comprised of UX leaders who have created strategies and tactics to succeed both in spite of and with the aid of the past and current trends. The panel will focus on the current trends, what strategies and tactics have and have not worked in addressing these trends, and also discuss which future trends they think will impact UX departments, companies, and the field, and how they are preparing for these future trends. The panel will be of interest to managers, practitioners and those who work closely with these teams, including developers, project managers, market researchers, test managers, and executives.
Questions of fairness, robustness, and transparency are paramount to address before deploying NLP... more Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests. We argue that reliability testing -- with an emphasis on interdisciplinary collaboration -- will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.
The term “field study” encompasses a category of activities that can include contextual inquiry, ... more The term “field study” encompasses a category of activities that can include contextual inquiry, on-site interviews, simple observations, and ethnography. During a field study, a researcher visits end users in their own environments (e.g., home or workplace) and observes them while they are going about their daily tasks. Field studies can last anywhere from a couple of hours to several days to even longer depending on the goals and resources of your study. Using this technique, a UX professional gains a better understanding of the environment and context. By observing users in their own environment, you can capture information that affects the use of a product (e.g., interruptions, distractions, other task demands) and additional context that cannot be captured or replicated in a lab environment. Field studies can be used at any point during the product development life cycle but are typically most beneficial during the conceptual stage.
Interviews are one of the most frequently used user research techniques. In the broadest sense, a... more Interviews are one of the most frequently used user research techniques. In the broadest sense, an interview is a guided conversation in which one person seeks information from another. There are a variety of different types of interviews you can conduct ranging from structured to unstructured depending on your constraints and needs. Interviews are incredibly flexible and can be conducted as a solo activity or in conjunction with another user experience activity (e.g., following a card sort). The end result of a set of interviews is an integration of perspectives from multiple users. If you conduct interviews with multiple user types of the same process/system/organization, you can obtain a holistic view. Finally, interviews can be used to guide additional user research activities.
This new and completely updated edition is a comprehensive, easy-to-read, "how-to" guid... more This new and completely updated edition is a comprehensive, easy-to-read, "how-to" guide on user research methods. You'll learn about many distinct user research methods and also pre- and post-method considerations such as recruiting, facilitating activities or moderating, negotiating with product developments teams/customers, and getting your results incorporated into the product. For each method, you'll understand how to prepare for and conduct the activity, as well as analyze and present the data - all in a practical and hands-on way. Each method presented provides different information about the users and their requirements (e.g., functional requirements, information architecture). The techniques can be used together to form a complete picture of the users' needs or they can be used separately throughout the product development lifecycle to address specific product questions. These techniques have helped product teams understand the value of user experience...
Questions of fairness, robustness, and transparency are paramount to address before deploying NLP... more Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests. We argue that reliability testing — with an emphasis on interdisciplinary collaboration — will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.
Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
The study of fairness in machine learning applications has seen significant academic inquiry, res... more The study of fairness in machine learning applications has seen significant academic inquiry, research and publication in recent years. Concurrently, technology companies have begun to instantiate nascent program in AI ethics and product ethics more broadly. As a result of these efforts, AI ethics practitioners have piloted new processes to evaluate and ensure fairness in their machine learning applications. In this session, six industry practitioners, hailing from LinkedIn, Yoti, Microsoft, Pymetrics, Facebook, and Salesforce share insights from the work they have undertaken in the area of fairness, what has worked and what has not, lessons learned and best practices instituted as a result. • Krishnaram Kenthapadi presents LinkedIn's fairness-aware reranking for talent search. • Julie Dawson shares how Yoti applies ML fairness research to age estimation in their digital identity platform. • Hanna Wallach contributes how Microsoft is applying fairness principles in practice. • Lewis Baker presents Pymetric's fairness mechanisms in their hiring algorithm. • Isabel Kloumann presents Facebook's fairness assessment framework through a case study of fairness in a content moderation system. • Sarah Aerni contributes how Salesforce is building fairness features into the Einstein AI platform. Building on those insights, we discuss insights and brainstorm modalities through which to build upon the practitioners' work. Opportunities for further research or collaboration are identified, with the goal of developing a shared understanding of experiences and needs of AI ethics practitioners. Ultimately, the aim is to develop a playbook for more ethical and fair AI product development and deployment.
Now that you have gotten stakeholder buy-in, learned about your products and users, considered le... more Now that you have gotten stakeholder buy-in, learned about your products and users, considered legal and ethical considerations, and arranged any facilities you may need, you are ready to choose the user experience research method that is best suited to answer your research question. In this chapter, we walk you through what it takes to do research well including how to get the right people involved, how to ask the right questions, and how to make sure that you know the limits of your project. With those things in mind, we teach you how to choose the right user research method for your research question. This chapter concludes by providing an overview of the methods presented in this book and a decision aid for choosing which methods will work best in situations you will encounter.
When starting work on a new project, your first objectives are to learn about the product (if it ... more When starting work on a new project, your first objectives are to learn about the product (if it already exists), domain, and (target) users. It is key for you to ascertain as much as possible about any existing products and domain in terms of functionality, competitors, and customers, so you do not duplicate work or spend time generating knowledge that already exists. This is done via using your product yourself, customer support comments, social sentiment analysis, log files and web analytics, speaking with your marketing department, competitive analysis, and getting feedback from early adopters or partners (e.g., trusted testers). In addition, you need to assess what is currently understood about the users and begin to create user profiles. This information will enable you to choose appropriate user research activities so that you can ultimately improve your product. In this chapter, we detail how to collect product information from a variety of sources and how to make sense of the information readily available to you. We will also discuss how to create user profiles, personas, scenarios, guiding principles, and antiprinciples and how to use these as design tools so that you can maximize the impact of your research. Finally, we discuss special user types that you should keep in mind when designing your product: international users, challenged users, children, and older adults.
In this chapter, we cover the fundamentals of executing a successful user research activity. Beca... more In this chapter, we cover the fundamentals of executing a successful user research activity. Because user research activities can occur in the lab or in the field, in person, or via e-mail, audio chat, or video chat, how much you, as the researcher, have to do “during” an activity can differ greatly depending on which activity you have chosen. We discuss who should be invited to observe user research activities and how you should prepare them to be good observers; how to welcome participants, warm-up strategies; how and when to provide incentives and how to develop rapport; and how to moderate your activity, use a think-aloud protocol, and debrief participants. We then describe strategies on how to record a session and provide tips for good notetaking. We conclude the chapter with recommendations for how to deal with awkward situations that may come up during the activity such as when a participant arrives late to a session.
Once you have completed all the research you can before involving your end users and you have ide... more Once you have completed all the research you can before involving your end users and you have identified the user requirements activity necessary to answer your open questions, you need to prepare to plan and execute your activity. These steps are critical to ensuring that you collect the best data possible from the real users of your product. We cover everything that happens or that you should be aware of prior to collecting your data, including sample size and creating a proposal, recruiting your participants, and piloting your activity.
Now that you have decided to pursue user experience research, you may be wondering whether a perm... more Now that you have decided to pursue user experience research, you may be wondering whether a permanent space needs to be built to conduct these activities. The answer is no. It is great if you have the budget, but it is not absolutely necessary. If you want to observe users in their own environment (e.g., field studies) or collect data from a lot of users (e.g., surveys, diary studies), then a usability lab is unnecessary. However, labs are excellent for card sorting, interviews, and focus groups. In this chapter, we discuss the options for setting up facilities to conduct user research. We look at the pros and cons for each option, as well as what you should be aware of when making your selection.
In earlier chapters, we have presented a variety of user requirements activities to fit your need... more In earlier chapters, we have presented a variety of user requirements activities to fit your needs. After conducting a user requirements activity, you have to effectively relay the information that you have collected to the stakeholders in order for it to impact your product. If your findings are not communicated clearly and successfully, you have wasted your time. In this, the concluding chapter, we show you how to prioritize and report your findings, present your results to stakeholders, and ensure that your results get incorporated into the product.
We’ve stated repeatedly how critical it is to understand your users and their needs in order to d... more We’ve stated repeatedly how critical it is to understand your users and their needs in order to design a successful product or service; however, at some point you need to evaluate what you have created to see if you got it right. You will see places you nailed it and others that you missed so you know where to go back and conduct additional research. Just like with other research methods described in this book, none of the evaluation methods here are meant to stand alone. Each uncover different issues and should be used in combination to develop the ideal user experience.
The term “field study” encompasses a category of activities that can include contextual inquiry, ... more The term “field study” encompasses a category of activities that can include contextual inquiry, on-site interviews, simple observations, and ethnography. During a field study, a researcher visits end users in their own environments (e.g., home or workplace) and observes them while they are going about their daily tasks. Field studies can last anywhere from a couple of hours to several days to even longer depending on the goals and resources of your study. Using this technique, a UX professional gains a better understanding of the environment and context. By observing users in their own environment, you can capture information that affects the use of a product (e.g., interruptions, distractions, other task demands) and additional context that cannot be captured or replicated in a lab environment. Field studies can be used at any point during the product development life cycle but are typically most beneficial during the conceptual stage.
Abstract Before conducting any kind of user activity there are a number of ethical and legal cons... more Abstract Before conducting any kind of user activity there are a number of ethical and legal considerations that you must be aware of. You are responsible for protecting the participants, your company, and the data you collect. These are not responsibilities that should be taken lightly. This chapter applies to all readers. Even if you are doing “just a little research,” if it involves human participants, this chapter applies to you. In this chapter, we will inform you of what you need to know before conducting a user activity to make sure you collect valuable data without legal or ethical harm. These are general guidelines but you also need to be aware of your local laws, practices, and regulations.
A card sort is most often used to inform or guide the development of the information architecture... more A card sort is most often used to inform or guide the development of the information architecture of a product. For example, it can help determine the hierarchy in applications. It can also provide information when deciding how to organize displays and controls on an interface. To conduct the technique, each participant sorts physical or virtual cards describing objects or concepts in a product into meaningful groups. By aggregating the grouping created by several users, we can learn how closely related each of the concepts are. This method tells us how a product’s features should be structured to match the users’ expectations about how those features are related. This technique can be conducted with individuals or with a group of users working individually.
CHI '13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA '13, 2013
ABSTRACT User Experience (UX) leaders and managers are required to continually adapt to changes i... more ABSTRACT User Experience (UX) leaders and managers are required to continually adapt to changes in: organizational strategies and re-structuring, resources, technology, economic pressures, and other factors. Simultaneously, more companies are realizing that they need UX expertise to ensure that they are competitive in today's marketplace. This panel is comprised of UX leaders who have created strategies and tactics to succeed both in spite of and with the aid of the past and current trends. The panel will focus on the current trends, what strategies and tactics have and have not worked in addressing these trends, and also discuss which future trends they think will impact UX departments, companies, and the field, and how they are preparing for these future trends. The panel will be of interest to managers, practitioners and those who work closely with these teams, including developers, project managers, market researchers, test managers, and executives.
Questions of fairness, robustness, and transparency are paramount to address before deploying NLP... more Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests. We argue that reliability testing -- with an emphasis on interdisciplinary collaboration -- will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.
The term “field study” encompasses a category of activities that can include contextual inquiry, ... more The term “field study” encompasses a category of activities that can include contextual inquiry, on-site interviews, simple observations, and ethnography. During a field study, a researcher visits end users in their own environments (e.g., home or workplace) and observes them while they are going about their daily tasks. Field studies can last anywhere from a couple of hours to several days to even longer depending on the goals and resources of your study. Using this technique, a UX professional gains a better understanding of the environment and context. By observing users in their own environment, you can capture information that affects the use of a product (e.g., interruptions, distractions, other task demands) and additional context that cannot be captured or replicated in a lab environment. Field studies can be used at any point during the product development life cycle but are typically most beneficial during the conceptual stage.
Interviews are one of the most frequently used user research techniques. In the broadest sense, a... more Interviews are one of the most frequently used user research techniques. In the broadest sense, an interview is a guided conversation in which one person seeks information from another. There are a variety of different types of interviews you can conduct ranging from structured to unstructured depending on your constraints and needs. Interviews are incredibly flexible and can be conducted as a solo activity or in conjunction with another user experience activity (e.g., following a card sort). The end result of a set of interviews is an integration of perspectives from multiple users. If you conduct interviews with multiple user types of the same process/system/organization, you can obtain a holistic view. Finally, interviews can be used to guide additional user research activities.
This new and completely updated edition is a comprehensive, easy-to-read, "how-to" guid... more This new and completely updated edition is a comprehensive, easy-to-read, "how-to" guide on user research methods. You'll learn about many distinct user research methods and also pre- and post-method considerations such as recruiting, facilitating activities or moderating, negotiating with product developments teams/customers, and getting your results incorporated into the product. For each method, you'll understand how to prepare for and conduct the activity, as well as analyze and present the data - all in a practical and hands-on way. Each method presented provides different information about the users and their requirements (e.g., functional requirements, information architecture). The techniques can be used together to form a complete picture of the users' needs or they can be used separately throughout the product development lifecycle to address specific product questions. These techniques have helped product teams understand the value of user experience...
Questions of fairness, robustness, and transparency are paramount to address before deploying NLP... more Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests. We argue that reliability testing — with an emphasis on interdisciplinary collaboration — will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.
Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
The study of fairness in machine learning applications has seen significant academic inquiry, res... more The study of fairness in machine learning applications has seen significant academic inquiry, research and publication in recent years. Concurrently, technology companies have begun to instantiate nascent program in AI ethics and product ethics more broadly. As a result of these efforts, AI ethics practitioners have piloted new processes to evaluate and ensure fairness in their machine learning applications. In this session, six industry practitioners, hailing from LinkedIn, Yoti, Microsoft, Pymetrics, Facebook, and Salesforce share insights from the work they have undertaken in the area of fairness, what has worked and what has not, lessons learned and best practices instituted as a result. • Krishnaram Kenthapadi presents LinkedIn's fairness-aware reranking for talent search. • Julie Dawson shares how Yoti applies ML fairness research to age estimation in their digital identity platform. • Hanna Wallach contributes how Microsoft is applying fairness principles in practice. • Lewis Baker presents Pymetric's fairness mechanisms in their hiring algorithm. • Isabel Kloumann presents Facebook's fairness assessment framework through a case study of fairness in a content moderation system. • Sarah Aerni contributes how Salesforce is building fairness features into the Einstein AI platform. Building on those insights, we discuss insights and brainstorm modalities through which to build upon the practitioners' work. Opportunities for further research or collaboration are identified, with the goal of developing a shared understanding of experiences and needs of AI ethics practitioners. Ultimately, the aim is to develop a playbook for more ethical and fair AI product development and deployment.
Now that you have gotten stakeholder buy-in, learned about your products and users, considered le... more Now that you have gotten stakeholder buy-in, learned about your products and users, considered legal and ethical considerations, and arranged any facilities you may need, you are ready to choose the user experience research method that is best suited to answer your research question. In this chapter, we walk you through what it takes to do research well including how to get the right people involved, how to ask the right questions, and how to make sure that you know the limits of your project. With those things in mind, we teach you how to choose the right user research method for your research question. This chapter concludes by providing an overview of the methods presented in this book and a decision aid for choosing which methods will work best in situations you will encounter.
When starting work on a new project, your first objectives are to learn about the product (if it ... more When starting work on a new project, your first objectives are to learn about the product (if it already exists), domain, and (target) users. It is key for you to ascertain as much as possible about any existing products and domain in terms of functionality, competitors, and customers, so you do not duplicate work or spend time generating knowledge that already exists. This is done via using your product yourself, customer support comments, social sentiment analysis, log files and web analytics, speaking with your marketing department, competitive analysis, and getting feedback from early adopters or partners (e.g., trusted testers). In addition, you need to assess what is currently understood about the users and begin to create user profiles. This information will enable you to choose appropriate user research activities so that you can ultimately improve your product. In this chapter, we detail how to collect product information from a variety of sources and how to make sense of the information readily available to you. We will also discuss how to create user profiles, personas, scenarios, guiding principles, and antiprinciples and how to use these as design tools so that you can maximize the impact of your research. Finally, we discuss special user types that you should keep in mind when designing your product: international users, challenged users, children, and older adults.
In this chapter, we cover the fundamentals of executing a successful user research activity. Beca... more In this chapter, we cover the fundamentals of executing a successful user research activity. Because user research activities can occur in the lab or in the field, in person, or via e-mail, audio chat, or video chat, how much you, as the researcher, have to do “during” an activity can differ greatly depending on which activity you have chosen. We discuss who should be invited to observe user research activities and how you should prepare them to be good observers; how to welcome participants, warm-up strategies; how and when to provide incentives and how to develop rapport; and how to moderate your activity, use a think-aloud protocol, and debrief participants. We then describe strategies on how to record a session and provide tips for good notetaking. We conclude the chapter with recommendations for how to deal with awkward situations that may come up during the activity such as when a participant arrives late to a session.
Once you have completed all the research you can before involving your end users and you have ide... more Once you have completed all the research you can before involving your end users and you have identified the user requirements activity necessary to answer your open questions, you need to prepare to plan and execute your activity. These steps are critical to ensuring that you collect the best data possible from the real users of your product. We cover everything that happens or that you should be aware of prior to collecting your data, including sample size and creating a proposal, recruiting your participants, and piloting your activity.
Now that you have decided to pursue user experience research, you may be wondering whether a perm... more Now that you have decided to pursue user experience research, you may be wondering whether a permanent space needs to be built to conduct these activities. The answer is no. It is great if you have the budget, but it is not absolutely necessary. If you want to observe users in their own environment (e.g., field studies) or collect data from a lot of users (e.g., surveys, diary studies), then a usability lab is unnecessary. However, labs are excellent for card sorting, interviews, and focus groups. In this chapter, we discuss the options for setting up facilities to conduct user research. We look at the pros and cons for each option, as well as what you should be aware of when making your selection.
In earlier chapters, we have presented a variety of user requirements activities to fit your need... more In earlier chapters, we have presented a variety of user requirements activities to fit your needs. After conducting a user requirements activity, you have to effectively relay the information that you have collected to the stakeholders in order for it to impact your product. If your findings are not communicated clearly and successfully, you have wasted your time. In this, the concluding chapter, we show you how to prioritize and report your findings, present your results to stakeholders, and ensure that your results get incorporated into the product.
We’ve stated repeatedly how critical it is to understand your users and their needs in order to d... more We’ve stated repeatedly how critical it is to understand your users and their needs in order to design a successful product or service; however, at some point you need to evaluate what you have created to see if you got it right. You will see places you nailed it and others that you missed so you know where to go back and conduct additional research. Just like with other research methods described in this book, none of the evaluation methods here are meant to stand alone. Each uncover different issues and should be used in combination to develop the ideal user experience.
The term “field study” encompasses a category of activities that can include contextual inquiry, ... more The term “field study” encompasses a category of activities that can include contextual inquiry, on-site interviews, simple observations, and ethnography. During a field study, a researcher visits end users in their own environments (e.g., home or workplace) and observes them while they are going about their daily tasks. Field studies can last anywhere from a couple of hours to several days to even longer depending on the goals and resources of your study. Using this technique, a UX professional gains a better understanding of the environment and context. By observing users in their own environment, you can capture information that affects the use of a product (e.g., interruptions, distractions, other task demands) and additional context that cannot be captured or replicated in a lab environment. Field studies can be used at any point during the product development life cycle but are typically most beneficial during the conceptual stage.
Abstract Before conducting any kind of user activity there are a number of ethical and legal cons... more Abstract Before conducting any kind of user activity there are a number of ethical and legal considerations that you must be aware of. You are responsible for protecting the participants, your company, and the data you collect. These are not responsibilities that should be taken lightly. This chapter applies to all readers. Even if you are doing “just a little research,” if it involves human participants, this chapter applies to you. In this chapter, we will inform you of what you need to know before conducting a user activity to make sure you collect valuable data without legal or ethical harm. These are general guidelines but you also need to be aware of your local laws, practices, and regulations.
A card sort is most often used to inform or guide the development of the information architecture... more A card sort is most often used to inform or guide the development of the information architecture of a product. For example, it can help determine the hierarchy in applications. It can also provide information when deciding how to organize displays and controls on an interface. To conduct the technique, each participant sorts physical or virtual cards describing objects or concepts in a product into meaningful groups. By aggregating the grouping created by several users, we can learn how closely related each of the concepts are. This method tells us how a product’s features should be structured to match the users’ expectations about how those features are related. This technique can be conducted with individuals or with a group of users working individually.
Uploads
Papers by Kathy Baxter