Bridging the Gap Between Analog and Digital

Delft University of Technology Physical fights back Introducing a model for bridging analog digital interactions Heijboer, Stefan; Tempelman, Erik; Schumann, Josef; Groen, Pim DOI 10.1145/3349263.3351510 Publication date 2019 Document Version Final published version Published in Adjunct Proceedings - 11th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2019 Citation (APA) Heijboer, S., Tempelman, E., Schumann, J., & Groen, P. (2019). Physical fights back: Introducing a model for bridging analog digital interactions. In Adjunct Proceedings - 11th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2019 (pp. 93-98). (Adjunct Proceedings - 11th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2019). Association for Computing Machinery (ACM). https://doi.org/10.1145/3349263.3351510 Important note To cite this publication, please use the final published version (if applicable). Please check the document version above. Copyright Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons. Takedown policy Please contact us and provide details if you believe this document breaches copyrights. We will remove access to the work immediately and investigate your claim. This work is downloaded from Delft University of Technology. For technical reasons the number of authors shown on this cover page is limited to a maximum of 10. AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands Physical Fights Back: Introducing a Model for Bridging Analog Digital Interactions Stefan Heijboer Erik Tempelman Abstract BMW Group Faculty of Industrial Design Engineering Knorrstrasse 147, Delft University of Technology 80937 München, Landbergstraat 15, 2628 CE Delft Germany The Netherlands [email protected] [email protected] Josef Schumann Pim Groen Current transformational developments in automotive user interface (UI) technology are causing a shift in emphasis from safety and efficiency to emotion and flexibility. The many factors to consider in parallel make this a difficult process, in which technological affordances all too easily push the user to the background. To address this issue, this paper introduces an interaction model linking the different tangible control elements, including smartphone functionality, and shows how non-driving-related activities (e.g. climate control, multimedia access) can be represented physically. Next, a working prototype is presented that supports the design and development of novel tactile UIs. By integrating layers of sensors and actuators, a flexible UI is created that pushes technology to the background, giving proper attention to the user again and enabling effective research on how to make the digital world tangible for users. BMW Group Faculty of Aerospace Engineering Knorrstrasse 147, Delft University of Technology 80937 München, Kluyverweg 1, 2629 HS Delft Germany The Netherlands [email protected] [email protected] Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for thirdparty components of this work must be honored. For all other uses, contact the Owner/Author. AutomotiveUI '19 Adjunct, September 21–25, 2019, Utrecht, Netherlands © 2019 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-6920-6/19/09. https://doi.org/10.1145/3349263.3351510 Author Keywords Tangible user interface; smart materials; shapechanging interfaces; haptic feedback; ACM Concepts Human-centered computing → Interaction design theory, concepts and paradigms 93 AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands Introduction Although natural user interfaces (NUI), such as speechbased interactions combined with large information displays in the dashboard and consumer electronic devices (CED) are seen as a key players in future user interface (UI) systems, several studies show that speech can result in higher workloads [1, 2], and the increasing usage of a phone decreases safety [3], thus advocating for the tangible ease of pressing a button to turn on simple in-car comfort functions such lights, climate etc. In fact, not only controlling safety- and effortlessness profit as Ishii [4] states that tactile feedback also prevents user frustration. In here, action possibilities instead of controls interact with the user on an aesthetic basis and thus form, interaction and function are combined into one interface element. Within the automotive industry several concepts with novel TUI have recently been presented, such as the “shy-tech” fabric media and wooden touchpad controls of the BMW iNEXT [8]. Although it could be argued that these UI surfaces are purely used for aesthetical reasons, Norman argues that beauty also contributes to a better understanding of a product [9]. Contribution Statement Preference of input and output devices during highly automated driving activities have been studied by Pfleging et al. [5], showing that smartphones are favored the most, followed by in-vehicle information systems. However, the study does not take basic yet essential comfort related non-driving-related activities (NDRA) into account, such as controlling temperature, nor does it acknowledge novel tangible user interfaces (TUI) modalities that enable hybrid versions of digitalanalogue controls, such as active haptic feedback or shape changing surfaces. According to Ishii [4], TUIs have always been special purpose whereas graphical UIs are general purpose and in that sense are flexible to adapt to most use-cases. With new technological- and material advancement however, such as printed electronics, smart materials [6] and smart textiles, potential interfaces could contribute to what Frens [7] calls rich user interaction. 94 While it is still relevant to measure effectiveness and distraction of new TUI [10], based on the shift towards autonomous driving, the importance/emphasis of TUI is turning towards a) how users can be surprised by novel TUI again instead of having black smooth surfaces that resemble CEDs, and b) how novel TUI can support the needed flexibility for future UI concepts. Digital content from CEDs can be represented digitally into in-car screens in many ways, ranging from completely integrating physical CED objects into car UI objects to mirroring displays [11]. In addition to the two previous challenges, the relation between NDRAs and their enablers concerning comfort and entertainment functions poses a third challenge: c) how can “shy-tech” enable the CED to have physical roots, while maintaining flexibility, regardless of future CED development. AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands This paper reports on 1) a basic model how NDRAs, such as comfort and multimedia functions, could be represented physically, 2) the translation to a preliminary prototype that permits UI to be designed independently from the technology, and 3) next steps Table 1: Simplified physical varieties of 2.5D surfaces. Note that modalities can also be combined. 1. Passive search haptics [16] with integrated sensing and lighting can provide surprising UI in combination with materials generally not associated with UI. 2. Transforming surfaces communicate information to the user via changes in temperature or texture (e.g. electroadhesion). 3. Active haptics can fuse general- and special purpose according to use-case (the prototype presented in the next chapter of this paper includes this modality). 4. With morphing shapes, the aesthetics are inherently connected with the function’s state (e.g. when a function is active, the physical appearance can change). UI Model for NDRA’s Figure 1: Graphical representation of media control Figure 2: Graphical representation of positional interconnection with CED: the interface moves along with the CED Surface Modalities In order for TUIs to continue to contribute valuably in future transportation interior’s human machine interfaces systems, the interface has to provide flexibility by a) maintaining the appeal of “real materials” while blending digital expectations and content, and b) supporting the variety of positions the user will adopt in future interiors. Many 2D UI surfaces can be found in automotive TUI, ranging from traditional resistive or capacitive sensing displays and trackpads to the iNEXT’s material integrated UI surfaces. According to Alexander et al. [12] shape changing interfaces can solve many relevant challenges. Various classification models exist on shape-changing interfaces [10, 13, 14, 15] but because of high abstraction levels, none of the models are applicable to transportation interiors. Therefore, table 1 introduces the physical varieties of relevant UI surfaces, or 2.5D surfaces (surfaces with fixed boundaries -fixed in a window- that are capable of expanding elastically within that boundary). 2.5D Surface Modalities Invisible physical Visibly physical interface shapes interface shapes Non-physically actuated (2) Transforming surfaces (1) Passive search haptics Physically actuated (3) Active haptic feedback (4) Morphing shapes 95 Interface Hierarchy In order to give face to the modalities in coherence to Ishii’s division of purposes, this work-in-progress paper presents a three-row model to control NDRAs and tertiary comfort and entertainment functions. As seen in table 2, each row increases in fidelity of information, thus creating a three-step hierarchy. Row Function Example Enabler 1 Basic selection Media control Fixed buttons 2 Main controls Pause/scroll/skip Hybrid system 3a Extensive control Equalizer CED 3b NUI/CED support Notifications Low-res display AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands Table 2: Hierarchical model bridging the gap between analog and digital TUI In here, the position of a CED is detected in row 3b and row two moves with CED accordingly. Also, different use-cases are provided with different haptic feedback patterns in row one and two. By offering such a flexible system, NDRAs that are embedded into the car and CED focused NDRAs (e.g. adjusting seating positions and browsing social media) can happen simultaneously. Embodiment of UI Model Figure 3: Close-up of possible graphical representation of volume use-case through woven fabric Row Technology n.a. Local piezo haptics for confirmation haptics 1 Side-fire RGB LEDs for symbol lighting Linear array of piezo’s for search haptics 2 Flexible 25-PPI passive matrix mini-LED display Stitched conductive yarn for capacitive sensing position finger 3a 3b Stitched conductive yarn for capacitive sensing position CED Flexible 5-PPI smart LED display Table 3: Overview of used technologies and materials. Although the final objective is to investigate and develop UI possibilities with all 2.5D modalities, as a second delivery this paper combines a first set of technologies into a preliminary prototype. Following up on Heijboer et al. [17], this prototype will help to investigate active haptic feedback (2.5D surface modality 3) use-cases for digital interactions represented in the physical world. As a technological basis, the Light.Touch.Matters project [18] provided guidelines in a first attempt to combine various smart materials into a “one-layer” flexible UI material. With the ambition to include materials such as RESi [19], conductive carbon fibers [20] or PEDOT-pTS dyed yarn [21] to pursue the elegance of thin layers, table 3 lists the technologies and materials that have been used. Next to the two benefits mentioned in surface modalities, this prototype presents a first attempt to integrate TUI with freedom of top surface material (as seen in figure 3) and freedom of form (as seen in figure 4). 96 Figure 4: Functional prototype with 3 interaction areas behind a fabric top surface Conclusion and Future Work This work-in-progress paper explores how NDRAs in connection to mobility interiors can be fulfilled by novel TUI. The prototype has provided the authors with initial insights on the influence that interfaces, technologies, materials and shapes have on each other. Insights include possible relationships between a CED and corresponding TUI position, influence of display resolutions on graphical interfaces, the impact of search haptics on top surface material, and transmission effects of low- and medium resolution displays (5-30 pixels per inch) on yarn-based material. For next steps, various iterations on integrating other technologies and materials will be carried out. Updated prototypes will enable further research on the liking and exploring of novel TUIs while providing feedback to the technology side. Acknowledgement We thank our colleagues from the prototyping facility UnternehmerTUM Makerspace GmbH, especially Moritz Neuberger and Oliver Schrank for their vital contribution to the prototype. AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands References [1] N. Martelaro, J. Teevan, and S. T. Iqbal, “An Exploration of Speech-Based Productivity Support in the Car,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2019, pp. 264:1– 264:12. [2] J. Lee, T. Brown, B. Caven, S. Haake, and K. Schmidt, “Does a Speech-Based Interface for an In-Vehicle Computer Distract Drivers?,” in Proc. World Congress on Intelligent Transport System, 2000. [3] V. Lewis, T. Dingus, S. Klauer, and J. Sudweeks, “An overview of the 100-car naturalistic study and findings,” Proc Int Tech Conf Enhanc. Saf. Veh., vol. 19, 2005. [4] H. Ishii, “Tangible Bits: Beyond Pixels,” in Proceedings of the 2Nd International Conference on Tangible and Embedded Interaction, New York, NY, USA, 2008, pp. xv–xxv. [5] B. Pfleging, M. Rang, and N. Broy, “Investigating User Needs for Non-driving-related Activities During Automated Driving,” in Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia, New York, NY, USA, 2016, pp. 91–99. [6] A. Minuto, D. Vyas, W. Poelman, and A. Nijholt, “Smart Material Interfaces: A Vision,” in Intelligent Technologies for Interactive Entertainment, 2012, pp. 57–62. [7] J. Frens, “Research through Design: a Camera Case Study,” in Design Research Now: Essays and Selected Projects, R. Michel, Ed. Basel: Birkhäuser Basel, 2007, pp. 135–154. [8] “BMW’s Vision iNext concept serves as tech ‘incubator,’” Automotive News, 15-Sep-2018. 97 [Online]. Available: https://www.autonews.com/article/20180915/OEM 04/180919764/bmw-s-vision-inext-conceptserves-as-tech-incubator. [Accessed: 14-Jun2019]. [9] D. A. Norman, The design of everyday things, Revised and Expanded edition. New York, NY: Basic Books, 2013. [10] Y. Forster, S. Hergeth, F. Naujoks, M. Beggiato, J. Krems, and A. Keinath, “Learning to use automation: Behavioral changes in interaction with automated driving systems,” Transp. Res. Part F Traffic Psychol. Behav., vol. 62, pp. 599–614, 2019. [11] S. Diewald, A. Möller, L. Roalter, and M. Kranz, “Mobile Device Integration and Interaction in the Automotive Domain,” presented at the AutoNUI: Automotive Natural User Interfaces Workshop at the 3rd International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI 2011), 2011. [12] J. Alexander et al., “Grand Challenges in ShapeChanging Interface Research,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2018, pp. 299:1–299:14. [13] M. K. Rasmussen, E. W. Pedersen, M. G. Petersen, and K. Hornbæk, “Shape-changing Interfaces: A Review of the Design Space and Open Research Questions,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2012, pp. 735–744. [14] M. Sturdee and J. Alexander, “Analysis and Classification of Shape-Changing Interfaces for Design and Application-based Research,” ACM AutomotiveUI '19 Adjunct, September 21-25, 2019, Utrecht, Netherlands Comput Surv, vol. 51, no. 1, pp. 2:1–2:32, Jan. 2018. [15] I. P. S. Qamar, R. Groh, D. Holman, and A. Roudaut, “HCI Meets Material Science: A Literature Review of Morphing Materials for the Design of Shape-Changing Interfaces,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2018, pp. 374:1–374:23. [16] S. J. Breitschaft, S. Clarke, and C.-C. Carbon, “A Theoretical Framework of Haptic Processing in Automotive User Interfaces and Its Implications on Design and Engineering,” Front. Psychol., vol. 10, p. 1470, 2019. [17] S. Heijboer, S. J. Breitschaft, and C.-C. Carbon, “Characterization of Active Haptic Feedback for User Interface Design and Development,” in 2019 IEEE World Haptics Conference (WHC), Tokyo, Japan, 2019. [18] E. Tempelman, “Design Driven, Materials Anchored: How Design Input Shaped the LTM Materials Stream,” TU Delft, Delft, Project report Methodological reflection, May 2016. [19] P. Parzer et al., “RESi: A Highly Flexible, PressureSensitive, Imperceptible Textile Interface Based on Resistive Yarns,” in Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, New York, NY, USA, 2018, pp. 745– 756. [20] B. Gebben, “Highly conductive and versatile Fibers from CNT’s.” [Online]. Available: https://horizon2020.zenit.de/fileadmin/Horizon202 0/01_Veranstaltungen/05-11-2015__Successful_2015/Day2/10_Highly%20conductive %2C%20robust%20and%20versatile%20CNT%20f 98 ibers%20-%20Dr.%20Bert%20Gebben.pdf. [Accessed: 26-Jul-2019]. [21] “for Developer AI SILK CORPORATION   .” [Online]. Available:
 http://www.ai-silk.com/english/for_deveroper/. [Accessed: 26-Jul-2019].