The sound of touch

The Sound of Touch: On-body Touch and Gesture Sensing Based on Transdermal Ultrasound Propagation Adiyan Mujibiya1,2, Xiang Cao1,4, Desney S. Tan1, Dan Morris1, Shwetak N. Patel1,3, Jun Rekimoto2 1 Microsoft Research Beijing, China; Redmond, USA {desney,dan}@microsoft.com 2 The University of Tokyo Tokyo, Japan {adiyan, rekimoto}@acm.org 3 University of Washington Seattle, USA [email protected] 4 Lenovo Research & Technology Beijing, China [email protected] ABSTRACT Recent work has shown that the body provides an interesting interaction platform. We propose a novel sensing technique based on transdermal low-frequency ultrasound propagation. This technique enables pressure-aware continuous touch sensing as well as arm-grasping hand gestures on the human body. We describe the phenomena we leverage as well as the system that produces ultrasound signals on one part of the body and measures this signal on another. The measured signal varies according to the measurement location, forming distinctive propagation profiles which are useful to infer on-body touch locations and on-body gestures. We also report on a series of experimental studies with 20 participants that characterize the signal, and show robust touch and gesture classification along the forearm. Author Keywords On-body sensing; Gestures; Skin; Ultrasound propagation; ACM Classification Keywords H.5.2. Information interfaces and presentation (e.g., HCI): User Interfaces - Input devices and strategies; INTRODUCTION We propose a novel on-body touch and gesture sensing technology based on active ultrasound signal transmission. It provides rich contextual information of on-body touch and gesture such as: 1) continuous indication of touch or gesture presence, 2) continuous localization of touch (potentially supporting slider-like interaction), 3) pressuresensitive touch (potentially supporting a touch-and-click event), and 4) an arm-grasping hand gesture (Figure 1). This paper highlights an active system with transducers that resonate the skin surface with low-frequency ultrasound and a receiver that measures the signal at some other point on the body. Significant portions of the signal actually transform into a surface wave when placed in perpendicular conPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. ITS '13, October 06 - 09 2013, St Andrews, United Kingdom. Copyright 2013 ACM 978-1-4503-2271-3/13/10…$15.00. http://dx.doi.org/10.1145/2512349.2512821 Figure 1. We proposed a novel on-body touch sensing method based on transdermal ultrasound propagation, and we explore the design space using two exemplary sensor configurations: (a) A wearable transmitter/receiver pair to detect location, duration, and applied pressure. (b) One armband that combines the transmitter and receiver to detect handgrasping gestures. This method can be immediately useful in: (1) information access and user input for the visually impaired; (2, 3) user input for users with limited access to an input device. tact with human skin (Figure 2). Complex body properties such as muscle mass, geometry, and bone structures attenuate the signal in relatively distinct ways and provide reasonably good signal differentiation. We measure the received signal at a multitude of frequencies. Our proposed sensing principle is by nature able to continuously detect touch and gestures, by which we mean that – unlike previous work – we can detect both the onset and offset of a touch event. We can also continuously sense the location of a touch event, by measuring the signal amplitude, which degrades as the function of the distance between the transmitter and receiver. When pressing harder against the skin, the amplitudes of the measured signal dramatically increases across our transmitting frequencies (in contrast to more subtle changes related to location), therefore applying an adaptive threshold is sufficient to sense changes in pressure. To sense more complex on-body touch and gesture events such as discrete touch location sensing and hand gesture sensing, we leverage the feature-distinctive propagation profile of the measured signal, and then use machine learning classification techniques. One crucial factor in instrumenting the body to add touch and gesture is to make the sensing hardware configuration as simple and as minimally intrusive as possible. Our proposed sensing method requires as few as two ultrasound transducers acting as transmitter and receiver to be in perpendicular contact with the skin. Furthermore, both of the transducers can be implemented with low-profile sound cards already existing in popular handheld devices, such as smart phones and music players. The sensing unit is also safe, low-power, and inexpensive in both hardware and software cost. The sensing technology presented is effective for various body parts. Users may attach the designated active signal transmitter and the receiver on their area of interest to add touch and gesture sensitivity. In this work, we focused our sensing experiment on the forearm, considering user convenience and affordance of the supported interaction modalities. However, our signal propagation experiments suggest that the sensing method could be extended to various body parts. Hence the potential input space is more diverse than what could be achieved with e-textiles, finger tap body acoustics [6], or capacitive sensing [14]. Despite past research exploring feasibility of using ultrasound range sensors propagating through air and hitting the forearm [7], we are not aware of previous work exploring on-body propagation of the signal for touch and gesture interaction. The specific contributions of this paper are: 1) Proposal of novel on-body touch and gesture sensing, using ultrasound signal transmission and acquisition. 2) Description of a wearable system to demonstrate the usability and reliability of the proposed method. 3) Presentation of results from a series of experiments aimed at profiling on-body ultrasound wave propagation as well as exploring the capabilities, robustness, and limitations of the system. 4) Description of the interaction design supported using the proposed method. We also note on-body sensormounting options for specific interaction modality. RELATED WORK Recently, multiple researchers have explored using wearable sensors that measure signals passing through the human body to turn it into the core interaction device. For example, researchers have studied approaches leveraging electromagnetic (EM) propagation on the human body to create Personal Area Networks [21] as well as their uses in HCI work [22]. Cohn et al. [2, 3] leverage power-line noise picked up by the human body acting as an antenna to recognize touched locations on uninstrumented walls. Saponas et al. [13] demonstrated the feasibility of classifying tapping and lifting gestures across all five fingers by sensing the signals of muscle activation through electromyography (EMG). This approach typically requires high-cost amplification systems and is limited to finger gestures. Our sensing method utilizes cheap ultrasound transducers and can be applied to various body parts, and senses a wider interaction area with the sensor placed on the forearm. Sato et al. proposed Touché [14], which leveraged swept frequency capacitive sensing to add touch and gesture sensitivity to an object with a single electrode. While this work also notes the feasibility of the sensing method to be deployed on the human body, the supported touch gestures are limited to body parts that meet the requirements of spatial separation between the source and sensor electrodes (such as between fingers in different hands). Therefore, more detailed interaction on the user’s forearm offering more affordable and subtle interactions are missed. Our sensing method further adds the possibility to sense continuous touch location (slider), touch pressure, and arm-grasping gestures. Point Upon Body (PUB) [7] and SonarWatch [8] proposed a method to explore human factors with respect to how users can interact with their forearms, and explored the design space using an ultrasound rangefinder to detect fingertapped positions on an arm based on the time of flight of air-propagated ultrasound. This setup is limited to sensing distances roughly along a single direction, and requires line of sight from the sensor. In contrast, our method is based on transdermal propagation that allowed us to enable touch sensing for positions surrounding the arm, as well as pressure-sensitive touch and hand-grasp gesture sensing. Takemura et al. [18] reported their early work on the usage of active signal injection (bone conduction sound at 800 Hz), but measure only elbow angle. In separate work, Harrison et al. present Skinput, a system that utilizes passive bio-acoustic sensors placed on the body to detect the presence and locations of taps on the body [6]. This system relies on the fact that these taps generate lowfrequency signals that propagate through the body and “sound” quite different depending on the composition of body parts they pass through on the way to the sensor. While it was interesting to conceive of interfaces that utilize the body as a tap surface, our work attempts to extend the capabilities of Skinput to sense temporally and spatially continuous touch points, adding gesture sensing capability, and to sense touch on a larger range of body parts. In early explorations, we found that low-frequency (between 20 and 100 kHz) ultrasound propagated quite well through the human body, but in a non-uniform manner that might allow us to differentiate touches. In fact, medical researchers have shown that low-frequency ultrasound is effective in enhancing transdermal transport of various molecules [1, 10, 11, 16, 17, 19]. Within this work, they have demonstrated that the frequencies around 40 kHz have excellent transdermal propagation with minimal heating [17]. Each of these transducers was set to a power output of 0.2 W, and we used the stereo I/O from a computer’s sound card to drive them. It would be relatively easy to embed the system into a smaller device using smaller form-factor ultrasound sensors (e.g. surface-mount devices) and substituting the signal-processing unit with a general-purpose microcontroller and analog-to-digital converter (ADC). This would allow the entire sensing unit to be embedded on a mobile/handheld device. Figure 2. The properties of ultrasound signal propagation along the skin. A transducer placed perpendicular to the skin results in surface wave propagation. Currently, we are not aware of previous attempts to exploit transdermal ultrasound propagation in designing on-body touch and gesture sensing. TRANSDERMAL ULTRASOUND PROPAGATION When ultrasound energy penetrates the body tissues, biological effects can be expected to occur if the tissues absorb the energy [20]. As illustrated in Figure 2, we leverage the surface mechanical wave of low-frequency ultrasound. Consequent with Surface Acoustic Wave (SAW) phenomenon [9], when a sound wave traveling through a medium encounters a boundary with a dissimilar medium that lies perpendicular to the direction of the wave, a portion of the wave energy will be reflected (with some portions transforming into a surface wave) and a portion will continue straight ahead (forming a shear wave). The percentage of reflection versus transmission is related to the acoustic impedances of the two materials. In the case of a boundary between aluminum (acoustic impedance of 40.6 to 45.7) and human skin (1.53 to 1.68) [12], the calculated reflection coefficient is 92%. Consequently, in our studies, we put ultrasound transducers in direct perpendicular contact with the skin. Exciting the skin with ultrasound produces a surface wave whose signal strength depends on the transmission power; more transmission power allows a larger sensing area. The non-EM nature of the signal also adds to the reliability of the sensing result compared to capacitive-sensing-based approaches such as Touché [14], which are highly affected by whether user’s body is connected to ground, or whether there are noise sources in the surrounding environment. IMPLEMENTATION Sensor Design We chose to work near 40 kHz for a number of reasons. First, it is inaudible. Second, there is very little ambient noise in the environment in this range, as energy is lost relatively rapidly as waves propagate through air. Third, these frequencies are already popular for tasks like range sensing, and many low-powered, cheap commodity transducers exist on the market. In fact, transducers with a center frequency between 25 and 60 kHz cost less than US$1 each. For our prototype, we used 14 mm diameter aluminum housing transducers with a center frequency of 40 kHz. In our approach, preventing the receiver from catching the signal propagated through the air is highly important, because it will affect the robustness of the system. Fortunately, there is less noise in the low-frequency ultrasound range; reflection and/or refraction from the transmitters are highly unlikely to disperse through the air, due to the nature of our mounting design (perpendicular skin contact). We used sealed, waterproof ultrasound transducers to provide an acoustic pathway between the transducer and the skin. This method minimizes any air coupling and adapts the contours of the probe to the skin. While we could have further enhanced this coupling by using sonic conduction gel, we chose not to as we do not believe this is practical in realworld scenarios. Based on early trials, we applied 10 V RMS driving voltage and a sound pressure level (SPL) of 20.4 dB. The transmitter’s SPL was measured at the center part of the transducer. Canada, Japan, Russia, and the International Radiation Protection Agency recommend safe levels of 110 dB SPL ceiling operation for frequencies from 25 to 50 kHz [15]. Therefore, our signal is significantly below this threshold. To sample the signal seen by the receiver, we also used the same conventional computer’s sound card, which has a sampling rate of 192 kHz at 24-bit precision. The Nyquist frequency was 96 kHz, which provided enough headroom for our designated transmitting frequency. This kind of audio codec chip has been widely used in PC motherboards, as well as high-end mobile audio players and mobile phone; therefore, deploying both the transmission and reception parts of our approach should be relatively easy. Sensor Configurations There are many sensor configurations possible using our sensing approach. In this paper, we narrowed our explorations down to two basic sensing configurations based on the usability and the contextual information we hoped to achieve. First, we formed a wearable pair, consisting of armband mounted ultrasound transmitters and a ringmounted receiver (Figure 3). We hypothesized that the spatial mapping between the transmitters and body parts with its complex geometry would exhibit variance in signal propagation, helpful in disambiguating touch location (Figure 8 shows our varied signal reading). In this configuration, user can perform interactions (as described in Figure 1a) on the body part that is attached to the transmitter with the minimally intrusive ring-mounted receiver. Figure 3. Wearable pair configuration: 4 ultrasound transmitters mounted on an armband; and a receiver mounted on a ring. We separate transmitters based on frequency. The second configuration is a combination of transmitters and receivers in a single armband (Figure 4). We aimed to recognize simple arm-grasping gestures (as illustrated in Figure 1b), while alleviating the need for finger-mounted receiver. Since these gestures displace external tissues and form different acoustic paths, we believed we could sense meaningful signals with only a single armband. Figure 4. Combination of transmitters and receivers in an armband. Using this configuration, system complexity is reduced and allows direct interaction between user’s hands. heights and body types was important so that we can appropriately generalize the results. Study 1: Measuring Signal Propagation on Body Parts The first experiment examined the profile of on-body ultrasound propagation, and to gain the profile of which body parts possess consistent signal amplitude deterioration that may further be utilized to detect spatially continuous touch gesture along the skin (e.g. potentially used as slider). Furthermore, we demonstrate effects of the applied pressure on the receiver’s signal. While the separate mounting requirement seems cumbersome, in the future, transmitters may be embedded into various accessories people are already wearing (wristwatches, sport armbands, headsets, or clothes). On the other hand, the receiver is by default embedded in a minimally-intrusive ring form factor for finger touch, but may also be embedded in other everyday devices such as pens or mobile phones for using them as a pointing tool on the body. In this study, we aimed to generally characterize transdermal ultrasound propagation; therefore we transmitted a single 40 kHz of sinusoidal ultrasound signal when the transmitter unit was placed on the following extensive body parts selection (with summaries of why each was selected): EVALUATION • We used aforementioned sensing configurations and conducted a set of controlled experiments to profile the signal propagation through various body parts, as well as to evaluate the performance and robustness of our system. Procedure • • Participants We recruited 20 participants (6 female), divided into two groups. The first group of 10 participants (2 female) completed our first three studies. They were 25 to 35 years old, 165 to 180 cm tall, and weighing between 54 and 85 kg. Participant’s Body Mass Index (BMI) ranged from 18.69 (normal-underweight) to 28.73 (obese as defined in Asia, where the work was conducted), with an average of 21.56 (SD=3.27). A second group of 10 participants (4 female) completed the 4th study of arm grasping gesture recognition. They were 21 to 28 years old, 159 to 178 cm tall, with BMI from 15.64 to 23.82. The data collection was distributed over approximately 5 days span for the first group, and 7 days span for the second group. The time span adds the real-world parameters variation, such as temperature and humidity. The diversity of • • • Forearm: anterior (body part #1) and posterior (#2). Large and convenient interaction surface. Upper arm: anterior (#3) and posterior (#4). Large and convenient (sports armband mounting strategy). Forehead (#5). Boney part of human body with relatively large area (headphone mounting strategy). Back-of-neck (#6). Collar mounting strategy. Foot: anterior (#7). Profiling uniformed boney structure. Foot: posterior (#8). Profiling firm muscular structure. We measured the signal amplitude with the finger-mounted receiver; by subsequently perform skin touch at 5 cm, 10 cm, and 15 cm away from the transmitters, as well as no skin touch (no contact between skin and receiver). For body parts where armband transmitter mounting was difficult (such as forehead and back-of-neck), we used a single transducer acting as a transmitter placed perpendicularly on the skin surface; e.g. position near the ear on backof-neck, and position above the eye on forehead. The participants were seated in a conventional chair while the experimenter measured the received signal at the desig- Figure 5. Experiment scene using wearable pair configuration (study 1, 2, and 3). nated position (transmitter and receiver placement was consistent across participants). The experimenter also controlled touch pressure application. See Figure 5 for sensor configuration and on-body sensor placement example. Figure 6. Signal propagation results for various body parts. Note the deterioration of the signal according to the distance. Error bars show standard deviation. Results We calculated the average signal power across participants for each measurement position. The results are compiled in Figure 6. We measured the propagation along the skin, with attenuation based on distance. We also compared the propagation for different body parts where fleshy parts of the body tend to propagate the signal better than boney parts. Also, body parts with firm muscular volume seem to propagate the signal better, as experiment results show that the posterior side of forearm yield significantly higher amplitudes than the associated anterior areas. Back-of-neck and forehead measurement results were particularly interesting due to the inconsistency of the correlation between distance and amplitude. In the back-of-neck, amplitude counter-intuitively increases with distance from the transmitter, while forehead exhibited slight amplitude decrease for measurement position near the center of the forehead. These phenomenon were actually consistent with the underlying physics on transdermal ultrasound propagation, where a portion of the signal will form surface wave that propagates along the skin, and another portion will form shear wave, which exhibit reflections from the muscular tissue underneath the skin, thus affect signal amplitude on circular measurement positions (i.e. neck and forehead). These results encouraged us to adopt machine learning to sense on-body touch on radial positions of the body, where the signal amplitude often do not decrease with distance from the transmitter. To check the effects of BMI in signal propagation, we divide our participants into two groups, participants with BMI over or below 23 (borderline for normal to overweight/obese). The group with higher BMI exhibited weaker signal, especially for positions such as forearm and upper arm. However, signals were present in all. As an example of the measurement results regarding to the BMI effects, we compiled forearm and upper arm signal propagation results in Figure 7. Figure 7. Signal propagation results on forearm and upper arm for participants with BMI over and below 23. Note that overall the group with higher BMI exhibited weaker signal, except posterior side of upper-arm where the effects of firm muscular volume were considered more significant than BMI effects. Error bars show standard deviation. In explorations on the effects of the applied touch pressure, we found that the signal amplitude increased considerably when pressing the receiver harder against the skin. This could potentially be leveraged as a click signal. This contextual information will further enrich the usability of skin as an interactive surface that also pressure sensitive. Study 2: Space-domain Continuous Touch Position and Touch Pressure Detection To achieve space-domain continuous touch sensing, we leverage the measured signal amplitude that generally degrades as the function of the distance from the transmitter. When pressing harder against the skin, the amplitudes of the measured signal significantly increased across all of our transmitting frequencies (in contrast to more subtle and non-uniform change in spatially continuous touch events). We further apply adaptive threshold to sense these events. Based on the results of our prior study and potential interaction application, we focused on the forearm as it showed consistent deterioration of the measured signal amplitudes, and its convenience for providing interaction affordance [6,13]. Procedure We adopted the same sensing configuration as our previous study, using wearable pair of transmitter and receiver transducers (Figure 3 & 5). However, in this study the transmitters are injecting 4 transmitting frequency (35, 40, 45, and 50 kHz of sinusoidal wave) on the user’s forearm. The usage of multiple transmitting frequencies enriches the feature set usable for inferring touch and gesture, because different frequencies propagate differently [6], thus different frequency response profiles are created at different positions (Figure 8). the system to perform interactive sliding gestures. In this study, we used the same gesture recognition frameworks as the prior study. However, instead of performing post hoc cross-validation, we mapped the following gestures to realtime interactive events: moving away from or closer to the transmitter mapped to sliding events (forward or backward respectively, where sliding velocity depends on moving speed), and pressing gesture (without slide) mapped to selection event. We designed a simple GUI of a sliding menu with 11 selectable items (as shown in Figure 11). Participants were seated in front of a computer displaying the GUI, while instructed to select 5 randomly assigned items, by performing pressing gesture to confirm selection events. In the beginning of the experiment and after each selection event, the menu was designed to slide back to middle (6th) item. We gain an average successful select rate of 92%, with an average selection time of 2.28 seconds (to select one item within 11 selectable menu). This result seems encouraging to design real-time application for on-body interactions. Study 3: Discrete Touch Position Classification Position Set for Classification Figure 8. Signal captured from a participant during (a) nogesture, (b, c, d) touch gesture on positions along the forearm gradually away from transmitter. Our signal visualization highlighted raw signal, spectrum (nearby 40 kHz), and it’s linear average. Transducer’s center frequency of 40 kHz is highlighted in red. Note the variance of the signal measured in different locations, which forms the basis of our classification approach. We selected 7 classification sets from the multitude of possible test position combinations (Figure 9). These represent logical sets of locations on the forearm, and test the limit of our sensing and classification capability. All of classification sets below include no-touch gesture. 1. 3 Positions (palm, and back-of-palm) 2. 4 Positions (3 positions along the anterior) 3. 5 Positions (4 positions surrounds the wrist) 4. 5 Positions (4 positions surrounds middle part) 5. 9 Positions (4 positions along the anterior and posterior) 6. 9 Positions (4 positions along the sides of anterior and posterior) 7. Gesture and no-gesture segmentation across all classification sets Participants were asked to perform four gestures: 1) Continuously sliding the ring-mounted receiver away from the transmitter and 2) towards the transmitter, and 3-4) are same gestures as 1 and 2 but with considerable pressure applied. Each participant performed 5 rounds of the designated 4-gesture set. Each gesture’s signal is automatically detected and segmented, with one-second-gesture window length. In a recent study conducted by Lin et al. [7], users were generally able to distinguish up to 6 points in eyes-free situations. We expect in scenarios where visual attention is allowed users can achieve higher resolution, thus we experimented with above resolutions. Our sensing method also allows radial sensing, i.e. sensing positions surrounding the arm, not limited to a single side along the arm. Results Participants were seated in a chair in front of a desktop computer showing the experiment steps and directions, as well as visualization of the signals (Figure 5). The armband was adjusted to ensure good contact between the sensor and the skin. The armband was placed on participant’s nondominant arm, and measurements were done with participant’s dominant hand. Similar to study 2, transmitter frequencies were adjusted to 35, 40, 45, and 50 kHz in sinusoidal wave, configured as wearable pair (Figure 3). Using this configuration, we were able to aggregate richer feature Averaged across participants, our real-time adaptive threshold correctly detects 98.21% (SD=4.52, chance=25%) of the aforementioned gestures. This provides evidence that by simply observing the behavior of amplitude spikes across the designated frequencies, our system is able to reliably detect sliding and pressing gestures. Interactive study: mapping for continuous gestures We conducted a study on event mapping for our spatially continuous touch sensing. We aim to clarify the usability of Procedure Figure 9. Classification accuracy for the designated 7 position sets. Labels show accuracy value, chance level, and standard deviation. Error bars indicate standard deviations. Position sets 1-4 demonstrate immediate usable accuracy for interaction, while position set 5 and 6 show the classifier’s scalability when increasing the number of positions. Position set 7 shows our system’s robustness to segment gesture and no-gesture across all the previous 6 position sets. set by leveraging the variability of the frequency response profiles created at different positions, as well as spatial separation between transmitters. To calculate the classification accuracy for various conditions, we ran five-fold cross-validations. Each fold consisted of a single round through all measurement positions in the classification set. Each was separated by several minutes to avoid the possibilities of over-fitting. We pick one round as test, and use the rest (four other rounds) as a training set. We repeat this condition for each round (fold), and we aggregate our classification results. Signal Processing and Classification To train the system, we collected 100 samples (a sample consists of the averages for every FFT bin in a 25 ms window) for each position. Samples were separated by 2 ms. Including software overhead, sample collection for each of the locations took about 3 seconds. We calculated a 2048-point FFT, resulting in 1024 spectral samples with 93.75 Hz frequency resolutions, to which we applied a Hamming window for smoothing. We generated 120 features for each sample. Our features are inspired by previous work on bio-acoustics (Skinput [6]), and we use a fairly exhaustive feature set to provide readers with a complete palette for subsequent implementations. In detail, our feature space consists of: 1. Acoustic power (84): The amplitudes of each transmitting frequency and their surrounding bands (±1 kHz). 2. Amplitude ratio between each transmitting frequency (6): Signal amplitude fluctuates when pressure applied, but the ratios between amplitudes were frequently stable. 3. Average amplitude (4): The average of acoustic power from each transmitter frequency with ±1 kHz bandwidth. This will essentially eliminate temporal fluctuations. 4. Average amplitude ratio between each transmitting frequency (6): We also considered ratios between average amplitude described above as feature. 5. Standard deviation (4): The different signal spectrum form (valley and peak) can be quantified in standard deviation between each transmitter frequency’s 21 bands. 6. Linear average (10): We calculated linear average grouped for 32 bands (Figure 4, bottom). This has essentially wider bandwidth and potentially useful for capturing harmonic signals as well as outlier removal. We included frequency range from 28.5 to 55.5 kHz 7. Log average (5): We also included log average for 28 to 72 kHz. This represents the broader band, as well as considering combined transmitting frequency amplitude and their surrounding bands as feature. 8. Zero crossings (1): We included total amount of zero crossed value of raw signal stream. We adopted the Sequential Minimal Optimization (SMO) implementation of Support Vector Machine (SVM) within the WEKA [5] for our classification. This is a supervised machine learning technique that constructs a classification hyperplane in our high dimensional feature space. Results Classification results are shown in Figure 9. Classification sets 1 to 4 demonstrate useable accuracy readily applicable for real-time interaction systems. Classification sets 5 and 6 pushed the limit for our sensing method to a usable 79.96% and 81.87% in average for all participants. Classification set 7 shows the robustness of our sensing method to segment gesture and no-gesture signal across all 6 classification sets. The inclusion of the no-touch condition in position sets 1 to 6 reflects our system’s target scenarios, where differentiating touch from no-touch is as important as position classification. To get more insights on our system’s capability to classify on-body touch gestures, we also run our classification without including no-touch gesture (i.e. only classify the actual touch locations). Averaged across all 10 participants, we aggregated slightly lower accuracy (e.g. for set 1 at 93.81% (SD=10.1%), set 2 at 98.29% (SD=2.57%), set 3 at 84.20% (SD=10.72%), set 4 at 89.8% (SD=11.54%), and set 5 at 88.55% (SD=7.37%)). Overall accuracy decrease are insignificant, therefore show promising real-world applicability of our classifier. We also examine the effects of participant’s BMI on our classification results (compiled in Figure 10). Overall, there system to classify simple arm-grasping gestures as an example of what our sensing method is able to support. Based on early trials, we found that applying significant pressure on the forearm exhibits significant fluctuations on the signals sensed by receivers that are collocated with transmitters in a single armband. Performing these gestures displace external and internal tissues, thus form different acoustic paths for our signal. To evaluate our system’s capability on sensing armgrasping gestures, we defined a gesture set consisted of five gestures: no touch, one, two, three, and four finger grasping (See Figure 1b for gesture illustration). Procedure Figure 10. Classification accuracy for the designated 7 position sets divided between participants with BMI over and below 23. Note that there is no clear indication of accuracy degradation relative to participant’s BMI. Error bars show standard deviation. is no indication of accuracy degradation based on participant’s BMI, which is encouraging. In PUB [7] the accuracies for discrete touch sensing were 84%, 66.6% and 65.7% for 5, 6 and 7 points, respectively. With our approach, we aggregate 98.75% (set 2 with 4 points) and 81.87% (set 6 with 9 points). This clearly shows the advantage of our transdermal ultrasound approach compared to previous work. Classification set 1, 3, 4, and 6 also showed the advantage of our sensing method, which enables touch sensing for positions surrounding the arm (no limitation in sensing along a single axis on the forearm). To explore our feature space, we ranked features by the square of the weight assigned by the SVM. Among the consistently best features we observed across all the classification sets were standard deviations, transmitter frequency amplitudes, and their ratios. Note that standard deviations indicate strong correlation between measurement position and the dispersion of the received signal amplitudes. We repeated our classification test for the whole sets using highly ranked features only (standard deviations (4), transmitter’s frequency amplitudes (4) and their ratios (6), average amplitude for 2 kHz bandwidth (4) and their ratios (6)); shrinking the total number of features to 24. The classification results only decreased within an average of 0.63% while the classification time were noticeably faster (1.89 times faster for training and 4.36 times faster for classification). On average, we obtain 30 classified samples per second, which is sufficient for real-time recognition. Study 4: On-body Gesture Classification The ability to sense on-body touch gestures is also important in supporting eyes-free and subtle interaction modalities. In this study, we demonstrate the capability of our The participants placed the armband of ultrasound transducers on their upper-forearm, and performed arm-grasping gestures on the forearm. The sensing configuration used was the combined one-armband transmitter and receiver (Figure 5). Participants were instructed to apply maximal pressure while maintaining comfortable feeling and avoid pain, when performing arm-grasping gestures. We inherit the procedures and techniques of our discrete touch position classification (Study 3). Results The average accuracy for 10 participants was 86.24% (SD=6.72%, chance=20%). The result seems promising considering that the classification was conducted with baseline machine learning techniques with plenty of headroom for fine-tuning the parameters. INTERACTION DESIGN & ENVISIONED APPLICATIONS Based on the studies described in previous section, the following design implications are derived: 1. Per-user training is most desirable. There are significant individual differences of how user performs touch gestures. Fortunately, the training phase of our method only took considerably short time (±3 seconds per- gesture). 2. Appropriate sensor configuration for specific interaction is crucial. We used different sensor configuration for study 1-3 and study 4. In our current setup, fingermounted receiver is required to robustly infer touch locations. However, as we discussed in study 4, different arm-grasping gestures exhibits significant variety in signal propagation profiles, therefore the requirement of finger-mounted receiver can be mitigated. Similar approach can be taken when designing interaction based on this sensing method on other body parts. 3. Leverage the sensing capability to perform legato touch sensing. Participants have the tendency to locally shift their finger to adjust the final position. Our approach’s ability to continuously sense touch can help determining the time threshold for a touch event to be finalized. 4. Sensing on fleshy and firm muscular volume body parts. Based on our study, those body parts gain better propagation, good for higher power-to-sensing area ratio. We intend to perform further studies on amplifying propagation of the low-frequency ultrasound mechanical wave for skin-to-skin contact (specially for wearable pair sensing configuration), so that users can interact with their fingers rather than with the sensing ring. Exploring the application of this sensing method for capturing rhythmic pattern of user’s gestures as input method is also feasible [4]. CONCLUSION Figure 11. We developed a proof of concept sliding menu application, which also served as our experiment platform in our interactive study (as a part of study 2). In this study, we investigated event mapping for continuous and pressureaware touch gestures. Note that in the future, the sensor unit (transmitter and receiver transducers) and the display contents can be embedded into wearable devices enabling truly mobile experience. We addressed on-body sliding menu user input in Study 2 (also presented in Figure 11). Furthermore, we envisioned two exemplary application domains in which our approach can be immediately useful: 1) supporting interaction modalities for visually impaired users (who can accurately touch their own body parts using proprioception, as illustrated by Figure 1.1), and 2) for users with limited access to an input device (illustrated by Figure 1.2 and 1.3, e.g. when driving, biking, running, etc.). Other interactive application examples uniquely enabled by this technique includes: a) Wrist mounted mobile phone will embed the sensing unit and appropriates the forearm to be an input surface, b) Headphones that simultaneously plays music and transmits ultrasound signals through the skull, and s/he may touch different parts of the head/neck/face to control the music, c) Multiple sensors placed in various body parts can be used to sense wide-body gestures. Exploring and developing these applications remains future work. DISCUSSION AND FUTURE WORK Our proposed approach has demonstrated that signal amplitude on the receiver deteriorates according to the distance from the transmitter on certain body parts such as forearm, upper arm, and foot. However, during our sample collection we noticed random fluctuation in the signal reading when sliding the receiver on the skin. This produces potential error-prone samples, greatly reducing the robustness of our classification. To address this limitation, we suggest smoothing over longer windows, so as to negate the temporal spikes. Also, in the current design of our prototype, we do not measure phase changes or effects of Doppler shift in response to user interaction. These aspects remain future work. We have demonstrated the feasibility, discussed interaction design as well as present proof-of-concept applications of a novel on-body touch and gesture sensing approach by leveraging low-frequency ultrasound wave propagation along the skin. This sensing method enables a breadth of viable applications using the human body as an input surface. We discussed the signal propagation across different body parts representing boney, fleshy, and firm-muscular tissues. By examining the variations of the signal propagation across different positions on the skin, we have shown that we can robustly classify a series of touch gestures performed at different locations on the forearm. We also demonstrated the feasibility of spatially continuous touch sensing along the skin, pressure-sensitive touch sensing, and arm-grasping gesture sensing. The sensor unit we used in our study consists of off-the-shelf components which are inexpensive in both bill of materials and computing power. Furthermore, it can be replaced with smaller form factor, allowing the whole setup including the signal-processing unit to be embedded into a mobile device. ACKNOWLEDGEMENTS We are thankful to anonymous reviewers for their thoughtful comments and subjects who participated in the experiments. REFERENCES 1. Cui, J., Wei, Y., & Wang, H. The Study of Lowfrequency Ultrasound to Enhance Transdermal Drug Delivery. Complex Medical Engineering, 2007, 1221– 1224. 2. Cohn, G., Morris, D., Patel, S. N., & Tan, D. S. Humantenna: Using the Body as an Antenna for Real-Time Whole Body Interaction. CHI ‘12, 1901–10. 3. Cohn, G., Morris, D., Patel, S. N., & Tan, D. S. Your Noise is My Command: Sensing Gestures Using the Body as an Antenna. CHI ‘11, 791–800. 4. Ghomi, E., Faure, G., Huot, S., Chapuis, O., and Beaudouin-Lafon, M. Using Rhythmic Patterns as an Input Method. CHI ‘12, 1253–1262. 5. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., and Witten, I. H. The Weka Data Mining Software: an Update. SIGKDD Explor. Newsl. 11, 1 (Nov. 2009), 10–18. 6. Harrison, C., Tan, D. S., & Morris, D. Skinput: Appropriating the Body as an Input Surface. CHI ‘10, 453– 462. 7. Lin, S.-Y., Su, C.-H., Cheng, K.-Y., Liang, R.-H., Kuo, T.-H., and Chen, B.-Y. PUB - Point Upon Body: Explor- ing Eyes-Free Interaction and Methods on an Arm. UIST ‘11, 481–488. 8. Liang, R.-H., Lin, S.-Y., Su, C.-H., Cheng, K.-Y., Chen, B.-Y., and Yang, D.-N. 2011. SonarWatch: appropriating the forearm as a slider bar. In SIGGRAPH Asia 2011 Emerging Technologies (SA ‘11). Article 5, 1 page. 9. Lord Rayleigh. On Waves Propagated along the Plane Surface of an Elastic Solid. Proc. London Math. Soc. s117 (1)(1885), 4–11. 10. Mitragotri, S., Blankschtein, D., and Langer, R. Transdermal Drug Delivery Using Low-frequency Sonophoresis. Pharmaceutical Research 13 (1996), 411–420. 11. Mitragotri, S., Farrell, J., Tang, H., Terahara, T., Kost, J., and Langer, R. Determination of Threshold Energy Dose for Ultrasound-induced Transdermal Drug Transport. Controlled Release 63, 1-2 (2000), 41–52. 12. Ogura, I., Kidikoro, T., Iinuma, K., Takehara, Y., Tanaka, K., and Matsuda, A. Measurement of Acoustic Impedance of Skin. Ultrasound in Medicine 4, RC 78.7, U4 A 5a (1978), 535. 13. Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan, R., Turner, J., and Landay, J. A. Enabling Always-available Input with Muscle-Computer Interfaces. UIST ‘09, 167– 176. 14. Sato, M., Poupyrev, I., Harrison, C. Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects. CHI ‘12, 483–492. 15. Non-Ionizing Radiation Section Bureau of Radiation and Medical Devices Department of National Health and Welfare. Guidelines for the safe use of ultrasound: Part II - Industrial and Commercial applications. Available online at: http://www.hc-sc.gc.ca/ewhsemt/pubs/radiation/safety-code_24securite/index_e.html. Accessed at April 5th, 2013. 16. Shung, K. K., Thieme, G. A., and Dunn, F. Ultrasonic Scattering in Biological Tissues. Journal of the Acoustical Society of America 94 (1993), 3033. 17. Suchkova, V., Siddiqi, F. N., Carstensen, E. L., Dalecki, D., Child, S., & Francis, C. W. Enhancement of Fibrinolysis with 40-khz Ultrasound. Circulation 98, 10, 1030–1035. 18. Takemura, K., Ito, A., Takamatsu, J., and Ogasawara, T. Active bone-conducted sound sensing for wearable interfaces. UIST ‘11 Posters. 53–54. 19. Terahara, T., Mitragotri, S., Kost, J., and Langer, R. Dependence of Low-frequency Sonophoresis on Ultrasound Parameters; Distance of the Horn and Intensity. International Journal of Pharmaceutics 235, 1-2 (2002), 35–42. 20. Wells, P. Physics of ultrasound. Ultrasonic Exposimetry. Ziskin, M. and Lewin, P., eds). 35. 21. Zimmerman, T. G. Personal Area Networks: Near-Field Intrabody Communication. IBM Systems Journal 35, 3.4, 609–617. 22. Zimmerman, T. G., Smith, J. R., Paradiso, J. A., Allport, D., and Gershenfeld, N. Applying Electric Field Sensing to Human-Computer Interfaces. CHI ‘95, 280–287.