IM6D

IM6D: Magnetic Tracking System with 6-DOF Passive Markers for Dexterous 3D Interaction and Motion Jiawei Huang (a) Tsuyoshi Mori Kazuki Takashima Shuichiro Hashi Yoshifumi Kitamura Research Institute of Electrical Communications, Tohoku University (b) (c) (d) (e) Figure 1: Our system is feasible for dexterous 3D motion and interaction. (a) System Overview, (b) Markers fit finger tracking tasks, (c bottom): Marker design, (c top, d), (e) Usage in example tasks Abstract We propose IM6D, a novel real-time magnetic motion-tracking system using multiple identifiable, tiny, lightweight, wireless and occlusion-free markers. It provides reasonable accuracy and update rates and an appropriate working space for dexterous 3D interaction. Our system follows a novel electromagnetic induction principle to externally excite wireless LC coils and uses an externally located pickup coil array to track each of the LC coils with 5DOF. We apply this principle to design a practical motion-tracking system using multiple markers with 6-DOF and to achieve reliable tracking with reasonable speed. We also solved the principle’s inherent dead-angle problem. Based on this method, we simulated the configuration of parameters for designing a system with scalability for dexterous 3D motion. We implemented an actual system and applied a parallel computation structure to increase the tracking speed. We also built some examples to show how well our system works for actual situations. CR Categories: I.3.1 [Computer Graphics]: Hardware Architecture—Input Devices I.3.7 [Computer Graphics]: ThreeDimensional Graphics and Realism—Virtual Reality; Keywords: motion capture, sensor, virtual reality, augmented reality, 3D user interface 1 Introduction Motion-tracking technology plays a critical role in computer animation, virtual reality, and human-computer interaction. The in- ACM Reference Format Huang, J., Mori, T., Takashima, K., Hashi, S., Kitamura, Y. 2015. IM6D: Magnetic Tracking System with 6-DOF Passive Markers for Dexterous 3D Interaction and Motion. ACM Trans. Graph. 34, 6, Article 217 (November 2015), 10 pages. DOI = 10.1145/2816795.2818135 http://doi.acm.org/10.1145/2816795.2818135. Copyright Notice Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. SIGGRAPH Asia ‘15 Technical Paper, November 02 – 05, 2015, Kobe, Japan. Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-3931-5/15/11 ... $15.00. DOI: http://doi.acm.org/10.1145/2816795.2818135 novations of tracking systems have accelerated both research and industrial applications in such fields. Over the decades, numerous projects have developed various motion tracking systems (e.g., [Sutherland 1968] [Zimmerman et al. 1987] [Wang and Popovic 2009] [Wang et al. 2013] [Raab et al. 1979]) to make motion tracking applicable for as many areas as possible. However, the tracking of dexterous motions, which means complicated motions for small spaces, remains difficult. Dexterous 3D motion tracking requires the tracking of the subtle and complex movements of small and easily occluded multiple targets in a 3D space. Typical examples are multiple-finger-based gestural interactions. To achieve such tasks, the markers used in a feasible tracking approach must be multiple, identifiable, small, lightweight, wireless, and capable of 6-DOF. Existing tracking approaches, which partially satisfy these requirements, are used with designs that compromise and limit the essential part of dexterous motion tracking, including relatively large wired magnetic markers that are fastened to the fingers, a combinational formation of optical markers for identification, and use of many cameras and sophisticated algorithms to avoid occlusion of targets from cameras. With this background, we propose our new magnetic motion tracking system that features identifiable, tiny, lightweight, wireless, and occlusion-free multiple 6-DOF magnetic passive markers with reliable accuracy and reasonable speed. These outstanding features will contribute to opening new fields for 3D applications. Our system’s key idea is designing a marker that consists of three magnetically coupled LC resonant magnetic coils for which we use the very promising principle proposed by [Yabukami et al. 2004]; however, their idea is limited to 5-DOF measurements, except for rotation around the coils axis, and inherently has a dead-angle problem where the position and orientation cannot be measured. Moreover, reasonable computation speed cannot be achieved with the necessary number of coils. To solve the dead-angle problem and achieve 6-DOF tracking, we designed markers containing three coils in different directions. In addition, we developed a multi-thread computation structure to make our system reliable and fast enough for real-time dexterous motion tracking and interaction sensing with a sufficient number of markers. This paper offers two types of contributions. As a scientific contribution, we propose a novel approach to achieving stable 6-DOF tracking for multiple, identifiable, small, lightweight, wireless, and ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015 217:2 • J. Huang et al. occlusion-free markers. As an engineering contribution, we develop an actual implementation based on our approach, achieve reasonable computation speed with a sufficient number of markers by applying a parallel computation structure using GPU, and demonstrate a potential example for dexterous 3D interactions based on our approach. 2 Related Work designed a system based on this principle to provide high performance for dexterous 3D motion tracking and interaction sensing, with reasonable hardware. Our approach to designing markers is inspired by the work of [Paradiso et al. 2003] (although this is not a 3D tracking system) and [Raab et al. 1979]. We also combine three LC coils into one marker; however, our approach does not require the LC coils to be perpendicular to each other. In addition, we achieve reliability and solve the dead-angle problem using this new design. 2.1 3D Motion-tracking Systems 2.3 Finger-based Dexterous Interaction 3D motion-tracking systems have been actively researched, and not a few of them have been available as commercial products. One such approach is that of [Foxlin et al. 1998], which combines an acoustic tracking array with an inertial sensing device ([Foxlin and Durlach 1994]) by anticipating mutual compensation for VR and 3DUI systems; however, this has difficulty in dexterous interactions because acoustic tracking can be affected by occlusion and other external factors (e.g., reflection, air flow, temperature, and so on) and the limited data-update rate at the speed of sound. Optical tracking systems are widely applied in such areas as VR and fullbody motion capture ([Ribo et al. 2001]). Regardless of whether it is active or passive, a single marker can only provide 3D positions. The identification of multiple tracking points and calculation of orientation often require somewhat awkward approaches, such as a unique combinational use of several markers for each tracking point. Furthermore, it inherently suffers from the problem of occlusion. The popularity of image-based tracking systems, which resolve the spatial positional data of the tracking object by applying an image processing algorithm (e.g., [Shotton et al. 2011]) to the depth image, continues to increase. [Wang and Popovic 2009] proposed a real-time method for tracking hand motions with high accuracy, but it requires the user to wear a colorful glove to track the entire hand while assuming a simple background. [Shiratori et al. 2011] proposed an approach of wearing video cameras on the human body and computed motion based on the captured video; however, this requires a large amount of computation. Many other interesting systems have been designed for motion tracking in computer graphics, vision, and signal processing such as [Daniel Vlasic and Popovic 2007], [Zhang et al. 2014], and [Sharp et al. 2015]; however, they still pose difficulties for dexterous 3D interactions where occlusion often occurs. 2.2 Magnetic systems Magnetic tracking systems ([Raab et al. 1979]) are widely used due to their high accuracy and occlusion-free features. However, in general, they require wired or battery-operated markers to emit a magnetic field as well as synchronization, data transfer, while introducing disturbance to the motion of tracking targets (e.g., [Ascension ] and [Polhemus ]). In contrast with these existing magnetic tracking systems, [Yabukami et al. 2004] and [Hashi et al. 2008] proposed a principle that conducts magnetic induction and sensing outside the marker so that it can be tiny, lightweight, battery-less, and wireless. We believe this idea is a promising approach for dexterous 3D motion tracking and interaction sensing because its markers can be occlusion-free and identifiable. However, it provides only incomplete 5-DOF tracking, since its inherent dead-angle problem (discussed in detail later) makes tracking reliability uncertain; consequently, its usage remains quite limited. [Yabukami et al. 2007] achieved a speed of 20 fps for a single LC coil; however, the prototype has to use one computer for each LC coil. Therefore, we 3D finger-based interaction is one of the most common topics of VR and AR. For example, [Grossman et al. 2004] and [Song et al. 2008] used a single or stereo camera to track user hands, and [Hilliges et al. 2012] used a depth camera, while other examples can be found in [Tompson et al. 2014], [Zhao et al. 2012] and [Tang et al. 2013]. However, most of these systems suffer from the occlusion problem while tracking the fingers. Glove-based systems are also widely used (e.g. [Zimmerman et al. 1987], [Virtual Realities ], [CyberGlove ], [5DT ]). Such systems can be worn on the user’s hand and employ flex sensors to measure the joint angles. They provide highly accurate joint tracking, but it is difficult to apply them in conventional motion-tracking tasks other than for the hand and fingers. Moreover, to precisely measure each joint angle, cumbersome calibration is required for each joint and each unique user [Wang and Neff 2013], and the abrasion of the flex sensors might damage the hardware through long-time use. Users cannot conduct dexterous tasks while wearing gloves. There are some magnetic field sensing systems for finger interactions, such as [Chen et al. 2013], [Harrison and Hudson 2009], and [Chan et al. 2013]; however, these achievements were only 2D or limited-3D interactions like pinch gestures. Much research has investigated the possibility of dexterous motion capture with existing tracking systems. [Sandilands et al. 2012] proposed an integrated system to capture the motion of interaction between a human and small objects, such as bottlecaps. However, it requires multiple tracking systems including depth sensors and active magnetic tracking systems, and the magnetic markers have to be wired to keep them small for hand motions. This research also indicates that optical systems are infeasible for this task since occlusion is a big challenge in such scenarios. [Wang et al. 2013] proposed a video-based method to capture the close-range motion of hands and real objects. Even though this method shows high accuracy, it requires a large amount of computation and is impossible for real-time applications. 3 System Structure Our method attempts to apply the tracking principle proposed by [Yabukami et al. 2004] to practical tracking tasks. Therefore, this section first briefly explains the basic workflow of the principle and then gives our method’s algorithm. 3.1 Tracking Principle As shown in Fig. 2, once a varying electromagnetic field is generated by the driving coil through which modulated current passes, the LC coil inside the electromagnetic field is induced, generating a resonant magnetic flux. Then the pick-up coils in several different locations select the magnetic field from the LC coil. Signals are measured by AD converters, and finally the program computes the 5-DOF pose of the LC coil; x, y, z of position, θ and φ of orientation. However, ψ (rotation around this coil’s axis) cannot be ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015 IM6D: Magnetic Tracking System with 6-DOF Passive Markers for Dexterous 3D Interaction and Motion • 217:3 Figure 3: A model of marker with three LC coils Figure 2: Tracking principle computed. The position and orientation of the LC coils are computed by solving an inverse problem; assuming that the flux density generated by the LC coil can be regarded as a magnetic dipole field, more than six values (in our system, the number of values is the number of pick-up coils) of the flux density at the known positions are required to calculate the six parameters of the LC coil: position (x, y, z), orientation (θ and φ), and the magnetic moment. To solve this inverse problem, we apply the following equations (1)-(3) and the nonlinear method of least squares with the effect optimization of the Gauss-Newton method [Nakagawa and Koyanagi 1982]: S (~ p) = n X (i) ~ meas ~ (i) (~ B −B cal p) 2 → M inimum (1) i=1 ~ (i) (~ B cal p) = 1 4πµ0 ( − ~ · r~i ) · r~i ~ 3( M M + 3 ri ri5 ) (2) factors are appropriate. During tracking, once the LC coil is in a range of pose, it cannot be driven. This is called the dead-angle problem. Actually, this happens when the axis of the LC coil falls within ± 15 degrees against the plane of the pick-up coil array, and it is quite usual during practical tasks. In such a pose, the position and orientation of the LC coil cannot be properly computed. As a consequence, the initial value of the next time instant is corrupted, and the system loses the ability to track the LC coil. This critical problem prevents the principle from succeeding in practical longterm use. To solve this problem, we propose a geometrical approach that contains three LC coils with different poses in one marker, in which relative positions and orientations of the three LC coils are known. We define the situation of being driven as “available” and the other situation as “unavailable.” In our design, the system ensures that in any situation, for any marker, at least one LC coil is available. Thus, it gains tracking continuity. Another ability gained through this design is 6-DOF tracking, since generally in practical use two or more LC coils are available in one marker. The information of two incomplete 5-DOF LC coils helps to indicate the 6-DOF pose of the marker, and it is expected to improve the tracking accuracy. Here, S (~ p) is the objective function (residual sum of squares), i is the pick-up coil number, n is the total number of pick-up coils, (i) ~ meas ~ (i) is the theoretical flux B is the measured flux density, B cal density that takes into account the magnetic dipole field, p ~ is the ~ is the magnetic moment. Vector parameter of the LC coil, and M ~r = ( x, y, z ) shows the position of the LC coil, and θ and φ are expressed directions of the LC coil. After the above process, the spatial pose (5-DOF) and the magnetic moment can be obtained. This method requires a set of initial values. For each time instant, the latest computed result is used as the initial value, and the first initial value is pre-set. Here, we simplify the shape of the marker into a rigid cube as shown in Fig. 3; however, it can be applied to LC coil combinations of any shape. The system first selects from the three LC coils the one that has the strongest resonant flux, which is designated as the “main” coil (assumed to be LC coil A in the figure). This LC coil is expected to provide accurate 5-DOF information of its position and direction. However, that information does not include rotation around the coils axis (φ), and so this missing single-DOF information is compensated by information provided by another LC coil. Assuming that LC coil C in the figure has the second-strongest resonant flux among the three (called “complementary”), the positional and directional information of LC coil C is transformed by the known spatial relationship between these two LC coils and thus used to provide the 6th DOF information of LC coil A. A superposed wave including all of the resonant frequency components of the LC coils is used to realize simultaneous excitation. The induced voltage wave measured by the pick-up coils is analyzed in each frequency spectrum by FFT analysis ([Hashi et al. 2008]). When only one LC coil is available, the system uses the orientation of the last time instant for computation. This compromise approach works well during practical use, since such situations are relatively rare and discrete. p ~ = ( x, y, z, θ, φ, M ) (3) Since LC coils resonate with inducing signals in specific frequencies, multiple LC coils can be designed with different resonant frequencies as unique IDs. Therefore, a modulated signal consisting of sine waves in different frequencies is applied to the driving coil, and each LC coil is induced in its own frequency. 3.2 Reliable Tracking and 6-DOF Computation The tracking principle indicates an important and serious problem: unreliable coil tracking. The computation of the inverse problem requires a proper initial value and a strong enough resonant flux; however, it is impossible for the principle to ensure that these two 4 System Design 4.1 Marker Design The goal of the marker design is to increase the opportunities for two or more LC coils in a marker to be available during practical use, thus avoiding the dead-angle problem and improving reliability. In addition, for dexterous-motion-tracking tasks, it is essential that the markers be as small as possible. On the other hand, a layout where the LC coils are too close would cause interference among the LC coils. Therefore, throughout our design process we care- ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015 217:4 • J. Huang et al. Table 1: Parameters for simulation fully considered how to find a well-balanced feasible design. Our initial idea of designing markers is to make the three LC coils orthogonal to each other, which is a cube with LC coils embedded in each edge as shown in Fig.4(a). This is easy to produce, but it contains much redundant space, hence its big size. Moreover, its shape induces more opportunities where only one LC coil among three is available, e.g., when it is placed in a stable pose. Figures 4 (b) and (c) show other marker designs, i.e., dodecahedron and arch. The idea is to improve the cube design by reducing redundant space and to achieve irregular LC coil layouts. However, such designs are still big, and we realized that designs with sharp edges could not avoid the waste of space. A ring-shaped marker (Fig.4(d)) is an efficient design to put on fingers and to achieve a reasonably small size. The three LC coils are fixed with different orientations on the ring, making sure their related pose would not change during the finger motion. Parameter Radius of LC Coil (mm) Length of LC Coil (mm) Radius of pick-up coil (mm) Interval of pick-up coil (mm) 4.2.2 Symbol r l Rp Li Small 2 10 70 10 Large 4 20 130 15 Pick-up coil array and tracking space size With more pick-up coils, the system can achieve better accuracy; however, it also requires more complicated hardware and expensive calculation. According to the hardware and the requirement of reasonable speed for dexterous 3D motion tracking and interaction, the appropriate number of pick-up coils is 32, since to cover such space 32 pick-up coils can support enough accuracy (millimeter level as described above), and the computation load is limited to the minimum. For ideal calculation, there is no limitation on the tracking space, and for practical use with noise, the effective tracking space of a 32-pick-up-coil array can be described as a semi-sphere area inside the pick-up coil array with volume Va : (a) cube (b) dodecahedron (c) arch (d) ring Va = Figure 4: Marker Designs 4.2 Hardware Configuration for Designing Tracking Field The system performance greatly varies based on the configuration of each component. As described above, there are several requirements that a tracking system for dexterous 3D motion and interaction must satisfy. We addressed a key design problem: How to choose the configuration that achieves the best performance and to make the system satisfy the requirements of dexterous 3D motion and interaction. We first describe the trade-offs and then how to determine the configuration. 4.2.1 Marker size and volume of tracking space A strong enough driving signal is needed to operate an LC coil. The following formula describes the relationship between the strength of the driving signal and the parameters: M ∝ HQS (4) Here, M is the magnetic moment as the induced strength by the magnetic field, H is the exciting magnetic field passing through the marker, Q is the quality factor of the marker determined by its coil part, and S is the size of its cross-section. Since M is reduced when the coil goes far from the plane of the driving coil and the pick-up coil array, the formula indicates that the system can improve the range of the tracking space by increasing H, Q, and S. However, for practical use there are limitations to the three factors. First, H should be limited to a low level for human safety, or otherwise the tracking space might burn the body inside it, and the driving coil will become so hot that it will break. Second, Q and S are determined by the size of the coils; according to the above requirements, the marker size should be limited to a reasonable level. 3π( 5Rp + 8 5Li 2 )3 (5) Here, Rp is the radius of a single pick-up coil, and Li is the interval between each of the two adjacent pick-up coils. This shows the scalability of our system; by arranging larger pick-up coils with larger intervals, the system can achieve a bigger tracking space. However, according to the principle, the system error increases when Rp and Li become larger. Thus we want to identify a balance between the tracking space and the tracking accuracy to satisfy the dexterous 3D motion and interaction requirements. 4.2.3 Theoretical error Theoretical error exists due to the calculation assumptions discussed in [Hashi et al. 2009], and the error depends on the designed parameters. To demonstrate how parameter changes affect tracking accuracy, we built a simulator of our system design to find the best implementation configuration. The simulator sets simulative signals from a marker in specific position P~s for all pick-up coils, and simulative tracking result P~r can be calculated by putting the simulative signals into the inverse problem. The tracking error is defined as e: e = |P~s − P~r | (6) We built a visualizer to show the simulation results. An example image of them is shown in Fig. 5. The parameters are shown in Table 1. The color of each capsule represents the accuracy of the tracking result when the coil is placed at the position of the capsules. Green means perfect accuracy (error ≤ 1mm), while red means poor (error ≥ 5mm). First, we simulate the configuration for high accuracy. When it is (r = 2 mm, l = 20 mm, Rp = 45 mm, Li = 45 mm), the coil is big enough to generate a strong magnetic field. On the other hand, the intervals of the pick-up coils and their sizes are so small that the error can be reduced. The simulation result in this case is shown in Fig. 5(a). The system error is small, since almost all of the results are perfect and the capsules are all green; however, the tracking space is small (260 mm×260 mm×120 mm), and since the coil is big (20 mm×4 mm), it will become heavy for dexterous interaction and motion uses. In contrast, another set of configurations target wide usage, i.e., small coils and large tracking space that can be applied in a large space of use (r = 1 mm, l = ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015 IM6D: Magnetic Tracking System with 6-DOF Passive Markers for Dexterous 3D Interaction and Motion • 217:5 Table 2: Maker specifications (a) high accuracy (b) wide tracking space Figure 5: Simulation and results 10 mm, Rp = 60 mm, Li = 60), shown in Fig. 5(b). By this configuration, the coil’s size is only 10 mm×2 mm, and the tracking space is large. However, the theoretical error is huge. As shown in the figure, the coil signs are either yellow or red, and generally the accuracy is worse than the first configuration. Marker ID Z A B C D E F G H I J K L M N Coil turn 600 600 500 384 350 295 247 194 193 186 189 142 146 147 105 Condenser (pF) 1500 680 560 560 470 470 560 680 560 470 390 680 560 390 820 Peak Resonance Frequencies (kHz) 61.5–62.2 88.9–89.9 119.0–120.2 153.3–154.8 181.1–182.9 208.5–210.4 242.2–244.6 269.8–272.3 302.4–305.1 328.7–331.6 359.7–362.8 388.3–391.8 419.7–423.4 445.5–449.4 482.3–486.3 Since a trade-off exists between the spatial design (i.e., size of tracking space and LC coil) and the tracking accuracy, a configuration that balances these two factors might be the best choice for general interactive uses. Based on this, we identified a reasonable configuration with the implementation parameters. 5 Parallel Computation Structure The computation of each LC coil, which mainly contains FFT and inverse problem computation, takes a single-thread program 100 milliseconds to complete the data of one time instant for one LC coil. Simply separating the computation into multiple threads requires Nf (number of pickup-coils) + Nc (number of LC coils) threads, which exceeds the limitation of the CPU cores. Hence, we designed a computation structure involving both CPU and GPU. In our structure, the GPU computes 32 (number of pick-up coils) FFTs simultaneously because it contains many cores, which is suitable for FFT computations. Since the inverse problem computation requires linear enumeration, a high-speed CPU is suitable. The FFT and the inverse problem computation are executed asynchronously to minimize the time of each time instant. This computation structure with a single LC coil to compute 5-DOF measurements is implemented in our first prototype [Huang et al. 2014]. We also propose an improved algorithm for the inverse problem. In [Yabukami et al. 2007], 5-DOF of each LC coil is computed, where the initial value in the inverse problem is first set as the origin, and the computation result of each time instant becomes the initial value of the next time instant. Since this iteration is not corrupted as soon as the LC coil becomes unavailable, our approach focuses on how to get a better initial value. Our new structure first chooses available LC coils, using the marker’s position to indicate its initial value and get its result. Then based on the availability of the evaluation results, if two LC coils are available, one will be computed using the information of the other. Finally, the marker’s 6-DOF information can be achieved. 6 Figure 6: Different resonant frequencies for different markers (PXIe-1075, NI Ltd.), an amplifier (HSA4011, NF Ltd.) + function generator (PXIe-6124, NI Ltd.) for generating the signals, and a conventional computer (Xeon E3×2, Intel Ltd., 16G RAM, Titan Black×1, NVidia Ltd.) for multi-thread computation. System and layout images of the pick-up coils are shown in Fig. 1(a). The system can simultaneously track a maximum of 15 LC coils (i.e., five markers) at 30 fps. The coils’ Ni-Zn ferrite cores are 4-mm radius, 15-mm long cylinders. The turns of the Polyester Enameled Copper Wire (PEW) for each LC coil range from 100 to 600 (details in Table 2), making their resonant frequencies different for unique identifications (Fig. 6). Fig. 7 shows the LC coil and designs of markers. A transparent heat-shrink tube covering protects the LC coil, which weighs about 1 g. We used a 3D printer with high accuracy to produce the markers with a cage design. The rings are about the size of human fingers, and the radius of the tube used to fix the LC coils is kept to the minimum. Our implementation is made with the following major features: 1. Small and lightweight: Since the markers are simply 3D printed cages with three LC coils, their volume is only about 4 cm3 , and they weigh 10 g, even in the case of the cube. This PROTOTYPE IMPLEMENTATION: IM6D We implemented a prototype based on our approach. To create a semi-sphere tracking area with a 150-mm radius, the system uses 32 pick-up coils with a 15-mm radius and 60-mm intervals between the coils. The application scenario uses a plane layout with a driving coil set at the same horizontal plane, an AD converter set (a) LC coil (b) cube (c)arch (d) dodecahedron (e) ring Figure 7: LC coil and markers ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015 217:6 • J. Huang et al. (a) Proposed approach (b) Yabukami’s principle Figure 8: Tracking result of rotating moving marker (a) Raw results on XY plane (b) Regression results on XY plane (c) Raw results on XZ plane (d) Regression results on XZ plane feature enables natural and non-intrusive user experience. 2. Wireless: Markers are driven by a signal provided by an external driving coil and require no wired connections or batteries. With this feature there is no limitation on motion or occlusion in the tracking area introduced by wires. 3. Identification: Each LC coil is identified by its own unique resonant frequency to allow individual identification. This feature is critical for dexterous motion and interaction; without it, the system would easily yield chaos ids for targets, since they are so close to each other. 4. Occlusion-free: The system is based on an electro-magnetic theory so the markers can be tracked even if they are occluded. This feature solves the problem of occlusion, which frequently occurs when using popular optical tracking systems for dexterous motion and interaction. 7 Figure 9: Static accuracy 5. High accuracy: The system can provide millimeter-level accurate position tracking. Detailed evaluation of the accuracy is described later in the evaluation section. least one coil is always available. Note that the tracking is accurate enough and no jitter can be found in our approach, even though the motion of the marker is very slow at the starting and ending points. 6. 6-DOF: Since each marker is tracked as a rigid body, each has 6-DOF and can remain small. 7.2 Static Accuracy 7. Reasonable speed: Even when using five markers simultaneously (with 15 LC coils), the calculation time for all of the markers at each time instant is about 30 milliseconds in total, which satisfies the requirements of real-time applications. Our static accuracy experiment focuses on the accuracy of each single LC coil because the 6-DOF markers’ accuracies completely rely on its accuracy. We put an LC coil at different locations inside the designed workspace using a 3-axis mechanical position controller with a high-accuracy laser range finder and took 100 samples to investigate its performance. We used a common coordinate system in which XZ shows a plane of the pick-up coil array and Y is vertical to this plane, where the origin is the center of the pick-up coil array’s plane. This evaluation only occupies a quarter of the whole tracking space, and since the layout of pick-up coils is symmetrical, the remaining part will yield exactly the same results. System Evaluation We conducted several experiments to evaluate IM6D’s reliability, speed, and accuracy. Other features of the markers, such as identification and occlusion-free, are not evaluated because they can be easily inferred from the tracking principle, and the example of actual use in the next section also shows them. 7.1 Reliability Since the most critical feature of our system is stability, we compared our approach and the single LC coil approach of [Yabukami et al. 2007]. We measured the flux of the three coils in a marker in real time when a user moves a marker spatially by hand and then computed their positions with both our approach and that of [Yabukami et al. 2004] (i.e., only inverse problem computation). The results are shown in Fig. 8. Small spheres indicate the location of the tracking results, and the order of transparency indicates the order of time. The small axes on the spheres indicate the orientations of the marker at those time instants. The results clearly show that our new approach (Fig. 8(a)) has high reliability for tracking dexterous motions. With Yabukami’s method (Fig. 8(b)), tracking was soon lost after the LC coil rotation changed, while our new method can still track the marker because at As shown in Fig. 9, our results imply two interesting points. First, the tracking error, defined as the variance of the result in each location, appears to be low. The results also show that, even at the edge of the designed tracking space (a 150-mm radius semi-sphere), the original error between the raw data and the real position (white rectangles) remains less than 5 mm as shown in Fig. 9(a)(c). This result proves that our system is feasible for dexterous-motion-tracking tasks. Second, the tracking result shows a bias approaching the origin. To prove this, we continued to take samples out of the designed tracking space, and the result matches our expectation. This can be explained by signal attenuation due to the distance, and such non-uniformity of accuracy actually exists in almost every tracking system. We believe that this bias can be corrected by regression methods, and so we did a machine-learning-based regression with limited samples. We trained a simple 3-layer artificial neural network (ANN) with half of the tracking results and later used it to process the rest. As shown in Fig. 9(b)(d), the corrected result (blue points) is satisfactory, since within the tracking space, the distance between the actual and computed locations is as close as 1 ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015 IM6D: Magnetic Tracking System with 6-DOF Passive Markers for Dexterous 3D Interaction and Motion (a) θ rotational measurements • 217:7 (b) θ rotational accuracies Figure 12: Finger tracking (c) φ rotational measurements (d) φ rotational accuracies Figure 10: Rotational measurements and accuracies Figure 13: Tracking balls in a cloth bag CPU and GPU) and the system’s hardware has the potential to support up to ten markers (with 30 LC coils) without decreasing the speed. 8 Example in Use We also built several actual examples to prove the utility of our system in practical use, since we believe its features will enable new possibilities in the research of dexterous motion tracking and interaction sensing. Figure 11: FPS evaluation mm. This method can be easily applied to a larger scale of samples, given the system’s ability to correct the bias of the coils in any pose. Similarly, the rotational accuracy of a single LC coil is measured by rotating the coil around an axis using a rotatable mechanical structure set on the plane of the pick-up coils. Results obtained from four typical locations in the tracking space are shown in Fig. 10. Here, the average errors are 2.21 degrees in φ and 1.89 degrees in θ; however, larger errors can be seen around the dead angles (0 and 180 degree), where the axis of the LC coil comes closer to parallel to the plane of the pick-up coil array. Our marker design with three LC coils solves this problem because at least one LC coil is always available; thus the system can reduce the rotational error. 7.3 Speed We experimentally investigated the tracking speed of our system. We ran a graphics application that indicated the marker poses and took 30-second bits of speed data during a normal tracking task to randomly track the moving fingers. As shown in Fig.11, our results show that even with five markers (with 15 LC coils), our system is still faster than 30 Hz, and variance exists due to the different time cost for convergence in different situations. The number of markers only slightly affects the speed of the system. This is because our computation structure handles each part of the computation in different computing units (i.e., The first example shows a finger-tracking result. Five markers were attached to all five of the user’s fingers on his right hand (Fig. 12), and their positions and orientations were tracked during operation. Each of the colored arches indicates the position and orientation of the markers, and the white cylinders connected to the arches indicate the fingers. We found the accuracy does not change when the markers are moving; however, when the marker gets far away from the pick-up coil array, the accuracy does decrease. Another example of capturing finger motion during a sewing task (Fig. 1(d)) shows our system’s potential to capture dexterous finger motion in a complex environment, where the targets (finger tips) are often occluded by objects, such as cloth. Other examples of such tasks are surgical operations, traditional handcrafts, and so on. Although we have not yet obtained the best design of markers suitable for this task, this example clearly shows the effectiveness of our IM6D system: multiple identifiable, wireless, occlusion-free markers. Apart from the finger-motion tracking, (Fig. 13) shows an example of tracking five balls inside a cloth bag, in which a marker is installed in each of the balls. Although the bag occludes the balls, our system adequately tracks their positions and orientations. We also built an example for using our system to track toys in water. To achieve such tasks, high accuracy and the ability to track 6-DOF poses in 3D space are required. With our system, it is easy to track not only the position and orientation of toys but also the drift that characteristically happens on water as shown in (Fig. 14). The final example is virtual object manipulation by the fingers of both hands (Fig. 1(e)). Two markers are attached to the thumb and the index finger of the right hand, and one is attached to the ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015 217:8 • J. Huang et al. balance and have taken steps to ensure that the current output stays far below the level needed to create such a situation. Figure 14: Tracking objects on water left hand’s index finger. Here, the fingers of the right hand operate selection, translation, and rotation tasks. The index finger of the left hand controls a tool that can cut the virtual object in 6-DOF. Because each of the markers is identified, different functions can be assigned to different fingers. 9 General Discussion As we expected, our examples in the previous section show the potential of our system for dexterous 3D motion tracking and interaction sensing. In the virtual object manipulation example, users can easily cut the cube into any desired shape because their fingers can move freely, and the system’s accuracy is high enough for them to precisely choose the position. Furthermore, the identification feature allows users to easily perform different methods of manipulation with different fingers. Other tracking tasks, which are usually challenging for other tracking systems, show our system’s potential for enabling new possibilities in dexterous motion tracking. As shown in the previous section, it is possible to apply our system in typical motion-tracking tasks other than those for the hand and fingers. Note that since the ring part of the marker is only used for containing and fixing each LC coil, our approach can also be used in non-container tracking applications. With the known relative positional and directional relations, multiple (≥3) LC coils can be embedded in a rigid object for tracking. We have to admit that the current designs of markers are not perfect and might not be feasible for some specific tasks. However, considering the priority of our goal, we think the current design offers high usability and efficiency. Furthermore, we will try different and potentially better designs, as well as apply 3D-printing technology to make the marker firmer. One of the most important factors influencing the accuracy of this system is the calibration of a marker, i.e., minimizing the difference between the designed ideal layouts and the actual locations of the LC coils on the marker. Insufficient calibration may cause a jitter of the measurements. Reasons considered to cause error in our system include molding shape error by a 3D printer, thickness variations of LC coils by different wire turns, and the mounting gap of an actual LC coil on a marker, among other causes. It is difficult to achieve an ideal calibration because this is currently done manually; however, we should still pursue approaches to obtain better calibration results. We think there is still room to improve the implementation. First, because we used a conventional computer in our current prototype, faster computation could be achieved using higher-performance computers with special hardware. In addition, the signal generator for the driving coil supports a maximum of 12-V output, and it is not possible to amplify the driving signal without amplifying the noise. When the number of LC coils is increased, the contribution of each frequency decreases, the ratio of the noise increases, and the tracking accuracy is reduced. However, simply using more sophisticated amplifiers would cause a high output power, which poses a risk of danger to the human body. We have considered this It is possible to apply our technology at different scales for different purposes. For example, a dense pick-up coil array with a small interval will gain high accuracy while the tracking space becomes small. On the other hand, by expanding the interval of pick-up coils, a larger tracking space can be obtained. Moreover, a room-sized tracking system might be possible by embedding a large number of pick-up coils under the floor, although we have to give careful attention to the power of the generated magnetic field in order to avoid damage to the human body. Another possibility is to embed smaller pick-up coils in a rooms furniture, to make a “smart” environment with the ability to sense human behaviors. 10 Conclusion and Future Work We proposed a novel magnetic dexterous-motion-tracking system with small, light, wireless, occlusion-free, and identifiable 6-DOF markers. We presented our design of a tracking system and implemented an actual system. With the use of a marker with three LC coils, we solved several key problems of the original tracking principle, including the dead-angle problem and the limitation of incomplete 5-DOF. Also, we achieved simultaneous 15 coils for real-time use, make the tracking principle actually available for a variety of practical tracking tasks. We experimentally evaluated the system’s performance and showed actual examples in use. From our results we proved that our system suits dexterous 3D interaction well because the occlusion-free and identification features enable accurate tracking. The markers support 6-DOF tracking and simultaneously maintain small-sized and lightweight hardware. These features allow the markers to be easily fastened to small tracking targets, such as fingers and actual small objects. We believe that these features also enable many new possibilities in 3D interaction areas. Currently, the initial value of the inverse problem is chosen by experience or is based on the previous time instant’s calculation results, making it not fully reliable. We will investigate a data-driven method that roughly chooses an approximate initial value based on the sensed flux. Combined with such a method, our system’s tracking continuity will be further improved. The marker now requires fixed relative positions and directions for LC coils. We are also considering an alternative design for the marker using soft materials as well as creation of a calibrating system that would give the marker yet more flexibility. Our system could also gain a larger workspace by reducing noise and enhancing the driving signal with better hardware components. Furthermore, we will design a 3D layout for the pick-coil array, since this might be another approach to solving the dead-angle problem. We are creating more applications to validate this system’s possibilities in the area of dexterous 3D motion research. Acknowledgments This work was supported in part by JSPS KAKENHI Grant Number 15H01697. References 5DT. http://www.5dt.com/. A SCENSION. http://www.ascension-tech.com/. C HAN , L., L IANG , R.-H., T SAI , M.-C., C HENG , K.-Y., S U , C.H., C HEN , M. Y., C HENG , W.-H., AND C HEN , B.-Y. 2013. ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015 IM6D: Magnetic Tracking System with 6-DOF Passive Markers for Dexterous 3D Interaction and Motion Fingerpad: Private and subtle interaction using fingertips. In Proceedings of the Symposium on User Interface Software and Technology, ACM, 255–260. C HEN , K.-Y., LYONS , K., W HITE , S., AND PATEL , S. 2013. utrack: 3D input using two magnetic sensors. In Proceedings of the Symposium on User Interface Software and Technology, ACM, 237–244. C YBER G LOVE. http://www.cyberglovesystems.com/. DANIEL V LASIC , ROLF A DELSBERGER , G. V. J. B. M. G. W. M., AND P OPOVIC , J. 2007. Practical motion capture in everyday surroundings. In Proceedings of SIGGRAPH, 1932– 1936. F OXLIN , E., AND D URLACH , N. 1994. An inertial headorientation tracker with automatic drift compensation for use with hmd’s. In Proceedings of the Symposium on Virtual Reality Software and Technology, ACM, 159–174. F OXLIN , E., H ARRINGTON , M., AND P FEIFER , G. 1998. Constellation: a wide-range wireless motion-tracking system for augmented reality and virtual set applications. In Proceedings of SIGGRAPH, 371–378. G ROSSMAN , T., W IGDOR , D., AND BALAKRISHNAN , R. 2004. Multi-Enger gestural interaction with 3D volumetric displays. In Proceedings of the Symposium on User Interface Software and Technology, ACM, 61–70. H ARRISON , C., AND H UDSON , S. E. 2009. Abracadabra: Wireless, high-precision, and unpowered finger input for very small mobile devices. In Proceedings of the Symposium on User Interface Software and Technology, ACM, 121–124. H ASHI , S., T OYODA , M., YABUKAMI , S., I SHIYAMA , K., O KAZAKI , Y., A RAI , K. I., AND K ANETAKA , H. 2008. Wireless magnetic motion capture system using multiple LC resonant magnetic markers with high accuracy. Sensors and Actuators A: Physical 142, 2, 520–527. H ASHI , S., YABUKAMI , S., K ANETAKA , H., I SHIYAMA , K., AND A RAI , K.-I. 2009. Numerical study on the improvement of detection accuracy for a wireless motion capture system. IEEE Transactions on Magnetics, 45, 6 (June), 2736–2739. H ILLIGES , O., D. K IM , S. I., W EISS , M., AND W ILSON , A. 2012. Holodesk: direct 3D interactions with a situated seethrough display. In Proceedings of the Conference on Human Factors in Computing Systems, ACM, 2421–2430. H UANG , J., TAKASHIMA , K., H ASHI , S., AND K ITAMURA , Y. 2014. IM3D: Magnetic motion tracking system for dexterous 3D interactions. In SIGGRAPH Emerging Technologies, 12:1. NAKAGAWA , T., AND KOYANAGI , Y. 1982. Experimental Data Analysis by the Least Square Method. The University of Tokyo Press. PARADISO , J. A., PARDUE , L. S., H SIAO , K.-Y., AND B EN BASAT, A. Y. 2003. Electromagnetic tagging for electronic music interfaces. Journal of New Music Research 32, 395–409. P OLHEMUS. http://polhemus.com/. R AAB , F., B LOOD , E., S TEINER , T., AND J ONES , H. 1979. Magnetic position and orientation tracking system. IEEE Transactions on Aerospace and Electronic Systems AES-15, 5 (Sept.), 709–718. R IBO , M., P INZ , A., AND F UHRMANN , A. L. 2001. A new optical tracking system for virtual and augmented reality applications. • 217:9 In Proceedings of the IEEE Instrumentation and Measurement Technology Conference, 1932–1936. S ANDILANDS , P., C HOI , M., AND KOMURA , T. 2012. Capturing close interactions with objects using a magnetic motion capture system and a rgbd sensor. In Motion in Games, M. Kallmann and K. Bekris, Eds., vol. 7660 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, 220–231. S HARP, T., K ESKIN , C., ROBERTSON , D., TAYLOR , J., S HOTTON , J., K IM , D., R HEMANN , C., L EICHTER , I., V INNIKOV, A., W EI , Y., F REEDMAN , D., KOHLI , P., K RUPKA , E., F ITZGIBBON , A., AND I ZADI , S. 2015. Accurate, robust, and flexible real-time hand tracking. In Proceedings of the Conference on Human Factors in Computing Systems, ACM, 3633– 3642. S HIRATORI , T., PARK , H. S., S IGAL , L., S HEIKH , Y., AND H OD GINS , J. K. 2011. Motion capture from body-mounted cameras. ACM Transaction on Graphics. 30, 4 (July), 31:1–31:10. S HOTTON , J., F ITZGIBBON , A., C OOK , M., S HARP, T., F INOC CHIO , M., M OORE , R., K IPMAN , A., AND B LAKE , A. 2011. Real-time human pose recognition in parts from single depth images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1297–1304. S ONG , P., Y U , H., AND W INKLER , S. 2008. Vision-based 3D finger interactions for mixed reality games with physics simulation. In Proceedings of ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, 7:1–7:6. S UTHERLAND , I. 1968. A head-mounted three dimensional display. In Proceedings of Fall Joint Computer Conference, 1, 757– 764. TANG , D., Y U , T.-H., AND K IM , T.-K. 2013. Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In Proceedings of IEEE International Conference on Computer Vision, 3224–3231. T OMPSON , J., S TEIN , M., L ECUN , Y., AND P ERLIN , K. 2014. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transaction on Graphics, 33, 5 (Sept.), 169:1–169:10. V IRTUAL R EALITIES. http://www.vrealities.com/. WANG , Y., AND N EFF , M. 2013. Data-driven glove calibration for hand motion capture. In Proceedings of the Symposium on Computer Animation, ACM, 15–24. WANG , R. Y., AND P OPOVIC , J. 2009. Real-time hand-tracking with a color glove. ACM Transactions on Graphics, 28, 63 (Aug.), 63:1–63:8. WANG , Y., M IN , J., Z HANG , J., L IU , Y., X U , F., DAI , Q., AND C HAI , J. 2013. Video-based hand manipulation capture through composite motion control. ACM Transactions on Graphics, 43, 32 (July), 43:1–43:13. YABUKAMI , S., H ASHI , S., T OKUNAGA , Y., A RAI , K., AND O KAZAKI , Y. 2004. Development of a position-sensing system for a wireless magnetic marker. Journal of the Magnetics Society of Japan 28, 877–885 (in Japanese). YABUKAMI , S., O GASAWARA , K., S AITOH , H., H ASHI , S., T OYODA , M., O KAZAKI , Y., AND A RAI , K. 2007. Development of a fingertip position-sensing system using LC resonated markers. Journal of the Magnetics Society of Japan 31, 439–444 (in Japanese). ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015 217:10 • J. Huang et al. Z HANG , P., S IU , K., Z HANG , J., L IU , C. K., AND C HAI , J. 2014. Leveraging depth cameras and wearable pressure sensors for full-body kinematics and dynamics capture. ACM Transaction on Graphics, 33, 6 (Nov.), 221:1–221:14. Z HAO , W., C HAI , J., AND X U , Y.-Q. 2012. Combining markerbased mocap and rgb-d camera for acquiring high-fidelity hand motion data. In Proceedings of the SIGGRAPH/Eurographics Symposium on Computer Animation, 33–42. Z IMMERMAN , T. G., L ANIER , J., B LANCHARD , C., B RYSON , S., AND H ARVILL , Y. 1987. A hand gesture interface device. In Proceedings of the Conference on Human Factors in Computing Systems and Graphics Interface, ACM, 189–192. ACM Transactions on Graphics, Vol. 34, No. 6, Article 217, Publication Date: November 2015