Papers by Nadia Berthouze
Chronic pain is a prevalent condition where fear of movement and pain interferes with everyday fu... more Chronic pain is a prevalent condition where fear of movement and pain interferes with everyday functioning. Yet, there is no open body movement dataset for people with chronic pain in everyday settings. Our EmoPain@Home dataset addresses this with capture from people with and without chronic pain in their homes, while they performed their routine activities. The data includes labels for pain, worry, and movement confidence continuously recorded for activity instances for the people with chronic pain. We explored two-level pain detection based on this dataset and obtained 0.62 mean F1 score. However, extension of the dataset led to deterioration in performance confirming high variability in pain expressions for real world settings. We investigated activity recognition for this setting as a first step in exploring the use of the activity label as contextual information for improving pain level classification performance. We obtained mean F1 score of 0.43 for 9 activity types, highligh...
14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Sep 17, 2022
Interacting with a car was once a tactile experience, which is on the decline with the rise of ca... more Interacting with a car was once a tactile experience, which is on the decline with the rise of car assistants, where the dominant form of interaction is through screen displays and voice recognition. These interaction modalities within a car are not the only options available. In this paper, we discuss reintroducing tactility into the automotive experience. This work presents a tactile embodiment of an intelligent car system, different from previous studies, to improve engagement and emotional connection between users and future intelligent cars. A prototype tool was designed to embody an intelligent car system. It was used to investigate how to interact with and control a smart-comfort system to improve user comfort. The tool invited users to interact through touch. Users could use their hands to physically agree or disagree with changes made by the system with the system moving in response, creating a bi-directional interaction symbiosis that reprioritises tactility.
Multisensory Research
In this review, we discuss how specific sensory channels can mediate the learning of properties o... more In this review, we discuss how specific sensory channels can mediate the learning of properties of the environment. In recent years, schools have increasingly been using multisensory technology for teaching. However, it still needs to be sufficiently grounded in neuroscientific and pedagogical evidence. Researchers have recently renewed understanding around the role of communication between sensory modalities during development. In the current review, we outline four principles that will aid technological development based on theoretical models of multisensory development and embodiment to foster in-depth, perceptual, and conceptual learning of mathematics. We also discuss how a multidisciplinary approach offers a unique contribution to development of new practical solutions for learning in school. Scientists, engineers, and pedagogical experts offer their interdisciplinary points of view on this topic. At the end of the review, we present our results, showing that one can use multi...
Chronic pain causes substantial disability, and people living with chronic pain often use protect... more Chronic pain causes substantial disability, and people living with chronic pain often use protective behaviours and movements to minimize pain and worry about exacerbating pain during everyday activities such as loading the washing machine. We present work in progress on ubiquitous interactive sonification of body movement to help people with chronic pain to understand and positively modify their movements in clinical and functional situations. The sonification blends informational and aesthetic aspects and is intended for daily use.
Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 2019
Research shows that exposure to nature has benefits for people's mental and physical health and t... more Research shows that exposure to nature has benefits for people's mental and physical health and that ubiquitous and mobile technologies encourage engagement with nature. However, existing research in this area is primarily focused on people without visual impairments and is not inclusive of blind and partially sighted individuals. To address this gap in research, we interviewed seven blind people (without remaining vision) about their experiences when exploring and experiencing the outdoor natural environment to gain an understanding of their needs and barriers and how these needs can be addressed by technology. In this paper, we present the three themes identified from the interview data; independence, knowledge of the environment, and sensory experiences.
Proceedings of the International Conference on New Interfaces for Musical Expression, Jun 1, 2020
The expressive control of sound and music through body movements is well-studied. For some people... more The expressive control of sound and music through body movements is well-studied. For some people, body movement is demanding, and although they would prefer to express themselves freely using gestural control, they are unable to use such interfaces without difficulty. In this paper, we present the P(l)aying Attention framework for manipulating recorded music to support these people, and to help the therapists that work with them. The aim is to facilitate body awareness, exploration, and expressivity by allowing the manipulation of a pre-recorded 'ensemble' through an interpretation of body movement, provided by a machine-learning system trained on physiotherapist assessments and movement data from people with chronic pain. The system considers the nature of a person's movement (e.g. protective) and offers an interpretation in terms of the joint-groups that are playing a major role in the determination at that point in the movement, and to which attention should perhaps ...
Frontiers in Computer Science, 2022
Posterior Cortical Atrophy is a rare but significant form of dementia which affects people's ... more Posterior Cortical Atrophy is a rare but significant form of dementia which affects people's visual ability before their memory. This is often misdiagnosed as an eyesight rather than brain sight problem. This paper aims to address the frequent, initial misdiagnosis of this disease as a vision problem through the use of an intelligent, cost-effective, wearable system, alongside diagnosis of the more typical Alzheimer's Disease. We propose low-level features constructed from the IMU data gathered from 35 participants, while they performed a stair climbing and descending task in a real-world simulated environment. We demonstrate that with these features the machine learning models predict dementia with 87.02% accuracy. Furthermore, we investigate how system parameters, such as number of sensors, affect the prediction accuracy. This lays the groundwork for a simple clinical test to enable detection of dementia which can be carried out in the wild.
In this paper we introduce weDRAW, a project to support primary school children in the exploratio... more In this paper we introduce weDRAW, a project to support primary school children in the exploration of mathematical concepts, through the design, development and evaluation of multisensory serious games, using a combination of sensory interactive technologies. Working closely with schools, using participatory design techniques, the games will be embedded into the school curricula, and configurable by teachers. Besides application to typically developing children, a major goal is to explore the benefits of this multisensory approach with visually impaired and dyslexic children.
Participants answers to the following study: Evaluating Saliency Map Explanations for Convolution... more Participants answers to the following study: Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study.
IEEE Transactions on Parallel and Distributed Systems, 2022
Breakthroughs in unsupervised domain adaptation (uDA) can help in adapting models from a label-ri... more Breakthroughs in unsupervised domain adaptation (uDA) can help in adapting models from a label-rich source domain to unlabeled target domains. Despite these advancements, there is a lack of research on how uDA algorithms, particularly those based on adversarial learning, can work in distributed settings. In real-world applications, target domains are often distributed across thousands of devices, and existing adversarial uDA algorithms-which are centralized in nature-cannot be applied in these settings. To solve this important problem, we introduce FRuDA: an end-to-end framework for distributed adversarial uDA. Through a careful analysis of the uDA literature, we identify the design goals for a distributed uDA system and propose two novel algorithms to increase adaptation accuracy and training efficiency of adversarial uDA in distributed settings. Our evaluation of FRuDA with five image and speech datasets show that it can boost target domain accuracy by up to 50% and improve the training efficiency of adversarial uDA by at least 11×.
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020
This paper introduces a new dataset, Libri-Adapt, to support unsupervised domain adaptation resea... more This paper introduces a new dataset, Libri-Adapt, to support unsupervised domain adaptation research on speech recognition models. Built on top of the LibriSpeech corpus, Libri-Adapt contains 7200 hours of English speech recorded on mobile and embeddedscale microphones, and spans 72 different domains that are representative of the challenging practical scenarios encountered by ASR models. More specifically, Libri-Adapt facilitates the study of domain shifts in ASR models caused by a) different acoustic environments, b) variations in speaker accents, c) previously unexplored factors such as heterogeneity in the hardware and platform software of the microphones, and d) a combination of the aforementioned three shifts. We also provide a number of baseline results quantifying the impact of these domain shifts on the Mozilla DeepSpeech2 ASR model.
Companion Publication of the 2021 International Conference on Multimodal Interaction, 2021
Pain is a ubiquitous and multifaceted experience, making the gathering of ground truth for traini... more Pain is a ubiquitous and multifaceted experience, making the gathering of ground truth for training machine learning system particularly difficult. In this paper, we reflect on the use of voice-based Experience Sampling Method (ESM) approaches for collecting pain self-reports in two different real-life case studies: long-distance runners, and people living with chronic pain performing housework activities. We report on the reflections emerging from these two qualitative studies in which semi-structured interviews were used to exploratively gather initial insights on how voice-based ESM could affect the collection of self-reports as ground truth. While frequent ESM questions may be considered intrusive, most of our participants found them useful, and even welcomed those question prompts. Particularly, they found that such voice-based questions facilitated in-the-moment self-reflection, and stimulated a sense of companionship leading to richer self-reporting, and possibly more reliable ground truth. We will discuss the ways in which participants benefitted from subjective self-reporting leading to an increased awareness and self-understanding. In addition, we make the case for the possibility of building a chatbot with ESM capabilities in order to gather more enhanced, refined but structured ground truth that combines pain ratings and their qualification. Such rich ground truth can provide could be seen as more reliable, as well as contributing to more refined machine learning models able to better capture the complexity of pain experience.
Proceedings of the Fourth International Workshop on Social Sensing, 2019
We present our vision and key research directions for next generation audio and speech-sensing sy... more We present our vision and key research directions for next generation audio and speech-sensing systems, to make them robust against variabilities in sensing hardware and operating conditions.
Developing computational methods for bodily expressed emotion understanding can benefit from know... more Developing computational methods for bodily expressed emotion understanding can benefit from knowledge and approaches of multiple fields, including computer vision, robotics, psychology/psychiatry, graphics, data mining, machine learning, and movement analysis. The panel, consisting of active researchers in some closely-related fields, attempts to open a discussion on the future of this new and exciting research area. This paper documents the opinions expressed by the indi-
Despite the recent breakthroughs in unsupervised domain adaptation (uDA), no prior work has studi... more Despite the recent breakthroughs in unsupervised domain adaptation (uDA), no prior work has studied the challenges of applying these methods in practical machine learning scenarios. In this paper, we highlight two significant bottlenecks for uDA, namely excessive centralization and poor support for distributed domain datasets. Our proposed framework, MDDA, is powered by a novel collaborator selection algorithm and an effective distributed adversarial training method, and allows for uDA methods to work in a decentralized and privacy-preserving way.
The 23rd International ACM SIGACCESS Conference on Computers and Accessibility, 2021
First author's affiliation, an Institution with a very long name Figure 1: Factors that influence... more First author's affiliation, an Institution with a very long name Figure 1: Factors that influence self-efficacy belief of blind and partially sighted people Orientation and mobility (O&M) training provides essential skills and techniques for safe and independent mobility for blind and partially sighted (BPS) people. The demand for O&M training is increasing as the number of people living with vision impairment increases. Despite the growing research on O&M assistive technologies (AT), few studies have examined the experiences of BPS people during O&M training, including the use of technology to aid O&M training. To address this gap, we conducted semistructured interviews with 20 BPS people and 8 Mobility and Orientation Trainers (MOT). The interviews were thematically analysed and organised into four overarching themes: Tools and Strategies for O&M training, Technology Use in O&M Training, Changing Personal and Social Circumstances, and Social Influences. Our findings show that the self-efficacy belief evolves over time and across different circumstances, therefore, requiring ongoing O&M support for BPS people. We discuss opportunities for accessibility research in multimodal technologies to increase access to and effectiveness of O&M training. CCS CONCEPTS •Human-centered computing ~ Accessibility ~ Accessibility technologies
Proceedings of the 2021 International Conference on Multimodal Interaction, 2021
Lecture Notes in Computer Science, 2005
The conveyance and recognition of human emotion and affective expression is influenced by many fa... more The conveyance and recognition of human emotion and affective expression is influenced by many factors, including culture. Within the area of user modeling, it has become increasingly necessary to understand the role affect can play in personalizing interactive interfaces using embodied animated agents. Currently, little research focuses on the importance of emotion expression through body posture. Furthermore, little research aims
Workshop Paper * accumulated over 20 seconds, sample taken from middle of gaming session ** scale... more Workshop Paper * accumulated over 20 seconds, sample taken from middle of gaming session ** scale: 1 (low)-5 (high)
Many journeys in urban environments are short and could be conveniently carried out on foot or bi... more Many journeys in urban environments are short and could be conveniently carried out on foot or bike. However, many people use public transport or cars and this places pressure on urban transport infrastructures. Motivating people to change their transport habits is a wicked problem and challenging to address. We outline our current approach that involves a long term study of FitbBit users to identify the bright spots: the factors that enable people to successfully change their habits in the long term.
Uploads
Papers by Nadia Berthouze