WEARABLE SENSORS AND HEALTH MONITORING SYSTEMS

Received 28 October 2019; accepted 5 February 2020. Date of publication 12 March 2020; date of current version 23 March 2020.
Digital Object Identifier 10.1109/JTEHM.2020.2978842

3-D Space Visualization System Using Ultrasonic

Sensors as an Assistive Device for the Blind
JUNG-HUN KIM, JI-EUN PARK , AND JONG-MIN LEE
School of Electronics Engineering, Kyungpook National University, Daegu 41566, South Korea
CORRESPONDING AUTHOR: J.-M. LEE ([email protected])
This work was supported by the Technology Transfer and Commercialization Program through the INNOPOLIS Foundation funded by the
Ministry of Science, ICT and Future Planning (Development of a Multisensor-based Smart Glasses for Providing
Visual Information) under Grant 2015DG0008.

ABSTRACT This study proposes a new assistive device for the blind that uses more than one-dimensional
data to draw objects. The study aims to convert three-dimensional (3-D) spatial information into sound
information using 6-axis and ultrasonic sensors, and to draw a 3-D depiction of the space ahead for the
user. Fourteen participants were involved in testing, wherein 4 were visually impaired. Moreover, the male
to female ratio was 7:3, with the average age of participants at 28.8 years. An initial sound recognition
experiment was designed to assess the device’s accuracy through participant use. Recognition rates were
70% for normal participants and 88% for the blind participants. Additional experiments expanded the
environmental conditions by requiring participants to discern the distances of 10 objects, positioned at both
high and low locations. Two different scenarios were employed: stationary and walking scenarios. The
stationary distance measurement participants scored an average of 96 points, while the walking participants
averaged 81 points. Under the given conditions, this study found that its assistive device for the visually
impaired can draw a 3-D space with 88.5% accuracy. This probability promises a basic level of utility that
can assist those with visual impairment in controlled environments, such as hospitals and homes.

INDEX TERMS Auditory signal, visual disturbance, visual reconstruction, vision system.

I. INTRODUCTION Another employs radio-frequency identification (RFID) to

Development efforts for assistive devices for people with provide the location and information of objects [8]. Inspired
disabilities (PWD) have become more active than ever [1]. by Microsoft’s Kinect system, one device attaches to canes
Various ancillary equipment enabling PWDs to engage in and acquires visual three-dimensional (3-D) information to
physical and social activity, and overcome daily obstacles detect front-laying objects through the use of a vibration
are in development. The propagation of these devices has sensor [9], [10].
had positive effects on the lives of PWDs who need assistive However, existing assistive devices are limited to
devices [2]. The recent technological advances have allowed one form of information transmission in their obstacle
the research and development of assistive devices to progress, recognition—making a sound when there is an object in front
both domestically and internationally, steadily [3]–[5]. of it. There has yet to be one that addresses their fundamen-
This study offers one potential alternative by propos- tal problem: their inability to describe a 3-D environment
ing a method to compensate for any visual impairment. precisely. Thus, in this study, 3 octaves Do, 4 octaves Do,
As an essential bodily function, one’s vision is responsi- 5 octaves Do, high sound, middle sound, and low sound are
ble for both spatial and object recognition, also contribut- all used to draw a 3-D space in front of the user. These
ing to the individual’s sense of balance. Some studies have sounds are used to determine the distance and location of
tested infrared-based obstacle detection systems; these are objects by coming as feedback to the user, adding an element
developed for indoor use, and they improve user mobil- of depth to the sensor’s feedback, and indicating position
ity by signaling when obstacles are ahead of the user [6]. with corresponding sounds. As no other device uses such
Others explored the utility of the Global Positioning Sys- feedback to cue spatial information to users, we evaluate a
tem (GPS) in an assistive device, supplemented by ultrasonic newly patented differentiated assistive device that relays the
sensors, notifying users through signaled vibrations. [7]. spatial information with sound-based feedback.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 8, 2020 2700305
 J.-H. Kim et al.: 3-D Space Visualization System Using Ultrasonic Sensors as an Assistive Device for the Blind

FIGURE 1. Recognition method of the 3-D space around the object (from
left to right: top, 60 cm distance; bottom, 90 cm. ∗ Do refers to Do of the
diatonic scale 3C = 3 octaves Do 4C = 4 octaves Do 5C = 5 octaves Do of
diatonic H = high sound at 60-cm distance M = middle sound at 90-cm
FIGURE 2. Method of acquisition information in three-dimensional space
distance L = low sound at 120-cm distance.
through three ultrasounds (TOP, CENTER, BOTTOM). TOP: TOP indicates the
position of the ultrasonic sensor. The distance of the ultrasonic sensor is
II. MATERIALS AND METHODS 200 cm. 200 cm is the distance of the building. CENTER: CENTER indicates
the position of the ultrasonic sensor. The distance of the ultrasonic sensor
A. MATERIALS is 100 cm. 100cm is the distance of the front door of the car. BOTTOM:
The study was approved by the Institutional Review Board BOTTOM indicates the position of the ultrasonic sensor. The distance of
the ultrasonic sensor is 50 cm. 50cm is the distance of the back of
(IRB 2017-0079) of the Kyungpook National University, the car.
compliant with regulations. Experiments were conducted
using auxiliary devices for the blind as glasses. To ensure the system sounds a 5-step volume of 5C octaves. This system
participant safety, a security agent was placed in front of the is designed to help users recognize 3-D spaces by adjusting
participant, and a nurse was always present in the event of the pitch and volume corresponding to the varying distances
an emergency. Out of the 14 participants, 4 were visually of obstacles.
impaired, while the remaining 10 had no abnormalities in The roll, pitch, and yaw values are outputs of the attitude
hearing and sound recognition. The participants’ gender ratio and heading reference systems (AHRS) 6-axis sensor. Roll
was 7:3 (male to female), with an average age of 28.8. The indicates horizontal values (X-axis coordinates) while yaw
participants consented to have their eyes covered for specific represents the slope values; pitch measures the vertical values
parts of the experiment. used to obtain the Y-axis coordinate of the frontal object.
Using the distance value of the ultrasonic sensor and the
B. METHODOLOGY Y-axis coordinate of the AHRS 6-axis sensor allows the
Spatial awareness is crucial for walking; for assistive devices construction of a distance map.
with ultrasound, determining objects at a distance from each Fig. 2 shows how the distance information of the 3-D
other is an essential cognitive factor. The maximum range space was obtained using the collected data from the top,
was set to 210 cm to eliminate superimpositions due to the middle, and bottom ultrasonic sensors. The collected distance
radiation angle of the ultrasonic sensor. To have this system information from the ultrasound has 10 steps as subdivi-
map 210 cm of the room ahead, the ultrasonic wave was sions: 30, 50, 70, 90, 110, 130, 150, 170, 190, and 210 cm
subdivided into three levels (i.e., top, middle, and bottom) to (depending on the distance). The actual detection distance
precisely locate an object. When the system detects an object of the ultrasound reaches a maximum of 700 cm. However,
across all three levels, each would simultaneously transmit if the detection range of the object widens, unnecessary sen-
the distance information. Fig. 1 shows how the device recog- sory information of various objects not within the immediate
nizes 3-D space: proximity of 210 cm (e.g., cars, people, animals, and so on)
In a mapped 3-D space, the object is recognized using 3C, would register. Therefore, the scope of detection was set
4C, or 5C octaves; a different pitch sounds depending on to 210 cm.
the distance and position of the object (Fig. 1). The volume As demonstrated above, specific sounds were generated
of those notes’ ranges from levels 1–10, depending on the by changing the amplitude A and the frequency f from the
distance of the user to the object. At 0 cm, the volume is at fundamental sinusoidal wave:
level 1; at the 210 cm limit, the volume reaches level 10. For S(t) = A × sin(2π ft)
example, if a bottom-lying object is 30 cm away, the system
S(t): sound
will sound a 2-step volume of 3C octaves; if set at center
height and 60 cm away, the system will sound a 3-step volume A : amplitude
of 4C octaves; if found at the top height and 90 cm away, sin(2π ft):sine wave (1)

2700305 VOLUME 8, 2020

J.-H. Kim et al.: 3-D Space Visualization System Using Ultrasonic Sensors as an Assistive Device for the Blind

FIGURE 4. Assistive device model. (It consists of a power button,

earphone, and three (TOP, CENTER, BOTTOM) ultrasonic sensors).

FIGURE 3. Transformation computation of 3-D information to auditory

information. f : f represents the frequency and from the high frequency f
to the low frequency (f N ) A: Indicates the magnitude of the frequency.
At a distance of 200 cm, the frequency is small amplitude. 50cm has a
large amplitude-frequency when the distance is close. Sin: A sine wave
was used.

The distance measured via ultrasound can be expressed by FIGURE 5. To prevent the interference of ultrasonic waves, ultrasonic
profiles were made. To prevent the interference of the ultrasonic wave is
gauging the intensity of the sound: manufactured by tilting 23◦ .
S(d, t) = A(d) × sin(2π ft)
S(d,t): sound
A(d) : distance information value
sin(2π ft):sine wave (2)
The frequency according to the height change Y of the
object is expressed as follows:
XN
SS (t) = A(d) × sin(2π fy t)
y=1
SS (t): sound
XN
A(d) : 2-D distance information value
y=1
sin(2π fy t):sine wave with y-axis information (3) FIGURE 6. Participants determined whether they could recognize several
sounds at the same time. Three octaves (3C, 4C, 5C) were tested with
Finally, the 3-D space information is converted into audi- varying volumes. ∗ Do refers to Do of the diatonic scale 3C = 3 octaves Do
4C = 4 octaves Do 5C = 5 octaves Do of diatonic H = high sound at 60-cm
tory information by changing frequencies using the difference distance M = middle sound at 90-cm distance L = low sound at 120-cm
in sound volume according to the signal of the distance and distance.
the three levels of the ultrasound.
Figure 4 illustrates the visor-like device designed for this the experiment orientation. Section III-A discusses the test
experiment. Its visor design was done to match conventional results.
eyewear, eliminating bias and mimicking the general user While seated, participants were instructed to pinpoint the
interface. The top, middle, and bottom ultrasounds are located presented objects through the device and list them from near-
at the left side of the device, while the distance information est to farthest. After measuring, the distance of the object
is collected at the front of the visor. would be changed. From an initial score of 10 points, 1 point
This layout prevents signals from interfering with each was deducted for every wrong answer. The participants were
other’s transmission and reception. According to the exper- given 30 minutes to practice after orientation before the
iment result of Fig. 5, the slope of the TOP, CENTER, and experiment proper. The test was subsequently performed
BOTTOM ultrasonic sensor tilted 23◦ . 10 times. Section III-B discusses the scores of these trials.
Fig. 6 distinguishes the object position via the assigned The distance recognition experiment proceeded as follows:
octaves and volume. Twenty-seven sound recognition tests first, the participant was oriented on the different sounds the
were conducted on the 10 participants 90 minutes after device emitted. Then, the participant equipped the device

VOLUME 8, 2020 2700305

 J.-H. Kim et al.: 3-D Space Visualization System Using Ultrasonic Sensors as an Assistive Device for the Blind

TABLE 1. Distance measurement of objects in 3-D space using sound

volume.

TABLE 2. Experimental study of 3-D space using sound volume.

FIGURE 7. Indoor distance measurement experiment. The experiment

was conducted to recognize the distance of each object in the room.
Participants identified the objects one to one, from 1 to 4.

scores imply that the device and its mechanics are intuitive,
and the different sounds emitted are easily distinguishable.

C. EXPERIMENTAL METHOD FOR INTERIOR OBSTACLES

EN ROUTE
The results of the experiment using upper and lower obsta-
cles have lower results compared to the other experiments.
However, despite that gap with other experiments, the dis-
crimination of distance and location in the 3-D space has
approximately 81 % recognition rate

IV. DISCUSSION
FIGURE 8. The participant was instructed to move past the obstacles
using the assistive device. While moving, the participants had to locate While previous studies used auxiliary devices with ultrasonic,
the objects by hand. camera, or infrared sensors to analyze surrounding obstacles,
the assistive devices designed here use converted data to
and moved forward, pinpointing obstacles from the initial help the blind navigate past peripheral obstacles [11]–[15].
location. As the participant drew closer to the object, they Although there are viable data among the different sensors
pointed at it to demonstrate obstacle recognition. At the end available, ultrasound proves most useful in discriminating
of the course, they removed the blind assistive device with distances. Certain external circumstances (i.e., the cam-
the help of a security guard. Section III-C discusses the test era sensor commits errors depending on the shape/pattern,
results. infrared rays are not suitable for outdoor use) limit cameras
and infrareds; thus, ultrasonic sensors are the recommended
tool for developing these assistive devices.
III. RESULTS
However, ultrasound can only discriminate distance, and
A. SOUND RECOGNITION EXPERIMENT
ultrasonic sensors cannot recognize such overall space. The
The 10 normal participants averaged 19 out of the 27 device developed for this study creates a two-dimensional
(70.37%) objects. On average, when a participant perceives plane coordinate using a 6-axis sensor and draws that missing
sound, they needed more 180 minutes to distinguish between 3-D space, taking a step above existing models.
all 27 sounds and achieve satisfactory results. Based on these With normal participants attaining recognition rates
results, 3-D space recognition can work with the assigned of 70% respectively and the distance measurement experi-
sounds for position information. ment scoring 96 points and the combined indoor walking
test scoring 81 points implies that 81% of the given 3-D
B. DISTANCE MEASUREMENT EXPERIMENT IN NORMAL space is recognizable through the device. However, as the dis-
PERSON tance is limited to a 210-cm range, the obtained information
The distance measurement results (Table 1) did not dis- of 3-D space is restricted in turn. Longer distances might be
play any significant difference between each other. The high measured using a higher frequency ultrasonic sensor with a

2700305 VOLUME 8, 2020

J.-H. Kim et al.: 3-D Space Visualization System Using Ultrasonic Sensors as an Assistive Device for the Blind

smaller radiation angle. Nevertheless, the proposed assistive ACKNOWLEDGMENT

device employing a 6-axis sensor, a high-frequency ultrasonic (Jung-Hun Kim and Ji-Eun Park are co-first authors.)
sensor, and GPS to locate positions is expected to perform
more accurately compared to other assistive devices for the REFERENCES
blind. [1] C.-T. Shih, C.-H. Shih, and N. Ren Guo, ‘‘Development of a computer
assistive input device using screen-partitioning and mousekeys method for
Visually impaired people have more sensitive hearing than people with disabilities,’’ in Proc. Int. Conf. Comput. Sci. Service Syst.
individuals without any impairment [16]. By using these (CSSS), Piscataway, NJ, USA, Jun. 2011, pp. 2201–2204.
echolocation functions, the device makes it possible for per- [2] I. Kim, W. Cho, G. Yuk, H. Yang, B.-R. Jo, and B.-H. Min, ‘‘Kinematic
analysis of sit-to-stand assistive device for the elderly and disabled,’’ in
sons with this particular disability to identify better their Proc. IEEE Int. Conf. Rehabil. Robot., Piscataway, NJ, USA, Jun. 2011,
surrounding environment and the obstacles that lay before pp. 1–5.
them. Especially useful is the distance discrimination the [3] Y. K. Meena et al., ‘‘EMOHEX: An eye tracker based mobility and
hand exoskeleton device for assisting disabled people,’’ in Proc. IEEE
device employs, allowing users to distinguish the location Int. Conf. Syst., Man, Cybern. (SMC), Piscataway, NJ, USA, Oct. 2016,
and distance of objects ahead. If a person with any visual pp. 2122–2127.
impairment is equipped with this device, based on our find- [4] I. Ulrich and J. Borenstein, ‘‘The GuideCane-applying mobile robot tech-
nologies to assist the visually impaired,’’ IEEE Trans. Syst., Man, Cybern.
ings, they will be more likely to recognize and adapt to any A, Syst. Hum., vol. 31, no. 2, pp. 131–136, Mar. 2001.
front-laying obstacles; this may prove especially useful in [5] R. Farcy, R. Leroux, A. Jucha, R. Damaschini, C. Grégoire, and
controlled environments such as hospitals and homes with its A. Zogaghi, ‘‘Electronic travel aids and electronic orientation aids for
blind people: Technical, rehabilitation and everyday life points of view,’’ in
portability and relatively compact functionality. Proc. Conf. Workshop Assistive Technol. People Vis. Hearing Impairments
Future studies might compare the effectivity of such a Technol. Inclusion, 2006, p. 12.
device versus the more traditional equipment to assist persons [6] K. Sudhakar, P. Saxena, and S. Soni, ‘‘Obstacle detection gadget for
visually impaired peoples,’’ Int. J. Emerg. Technol. Adv. Eng., vol. 2,
with any visual impairment. Also, these devices might be pp. 2250–2459, Dec. 2012.
adapted to more real-life simulations by adding the recogni- [7] D.-H. Kim and H.-G. Ryu, ‘‘Obstacle recognition system using ultrasonic
tion of moving obstacles as objects of study, or a combination sensor and duplex radio-frequency camera for the visually impaired per-
son,’’ in Proc. 13th Int. Conf. Adv. Commun. Technol. (ICACT), Piscataway,
of object behaviors in their testing conditions. NJ, USA, Feb. 2011, pp. 326–329.
[8] A. Dionisi, E. Sardini, and M. Serpelloni, ‘‘Wearable object detection
V. CONCLUSION system for the blind,’’ in Proc. IEEE Int. Instrum. Meas. Technol. Conf.
This new device addresses issues previous iterations of simi- Proc., Piscataway, NJ, USA, May 2012, pp. 1255–1258.
lar assistive devices faced by utilizing a different data acqui- [9] H. Takizawa, S. Yamaguchi, M. Aoyagi, N. Ezaki, and S. Mizuno, ‘‘Kinect
cane: Object recognition aids for the visually impaired,’’ in Proc. 6th
sition method. The 6-axis and ultrasonic sensors gather data Int. Conf. Hum. Syst. Interact. (HSI), Piscataway, NJ, USA, Jun. 2013,
that, when used together, allows a seemingly more accurate pp. 473–478.
communicable signal to reach users. These signals, ranging [10] C. Vigneshwari, V. Vimala, and G. Sumithra, ‘‘Sensor based assis-
tance system for visually impaired,’’ Int. J. Eng. Trends Tech., vol. 4,
across 3–5 octaves of sound, are distinguishable enough for pp. 4338–4343, Oct. 2013.
those without visual impairment, so it is also likely they will [11] C.-G. Kim and B.-S. Song, ‘‘Design of a wearable walking-guide system
serve equally as well, if not better, for those who possess any for the blind,’’ in Proc. 1st Int. Conv. Rehabil. Eng. Assistive Technol.
Conjunct. 1st Tan Tock Seng Hospital Neurorehabil. Meeting (i-CREATe),
visual impairment. Their enhanced auditory sense could lead New York, NY, USA, 2007, pp. 118–122.
to higher accuracy in distinguishing the 3–5-octave sounds. [12] D. J. Calder, ‘‘Assistive technology interfaces for the blind,’’ in Proc.
The experimental results showed that the sound recogni- 3rd IEEE Int. Conf. Digit. Ecosystems Technol., Piscataway, NJ, USA,
Jun. 2009, pp. 318–323.
tion result was 70.37%, the distance recognition result was [13] A. I. Nour Alshbatat, ‘‘Automated mobility and orientation system for blind
96%, and the indoor obstacle recognition result was 81%. The or partially sighted people,’’ Int. J. Smart Sens. Intell. Syst., vol. 6, no. 2,
device’s functionality exhibits an awareness of distance that pp. 568–582, 2013.
[14] M. F. Saaid, I. Ismail, and M. Zikrul Hakim Noor, ‘‘Radio frequency
can be furthered in more complex scenarios. The developed identification walking stick (RFIWS): A device for the blind,’’ in Proc. 5th
assistive device will produce different sounds depending on Int. Colloq. Signal Process. Its Appl., Piscataway, NJ, USA, Mar. 2009,
the distance and position of the obstacle in front of it. With pp. 250–253.
[15] A. A. Nada, M. A. Fakhr, and A. F. Seddik, ‘‘Assistive infrared sensor based
this device, you will be informed of the distance and location smart stick for blind people,’’ in Proc. Sci. Inf. Conf. (SAI), Piscataway, NJ,
of obstacles within 200 cm in uncontrolled environments. USA, Jul. 2015, pp. 1149–1154.
This can add to the growing knowledge in assistive technol- [16] M. E. Nilsson and B. N. Schenkman, ‘‘Blind people are more sensi-
tive than sighted people to binaural sound-location cues, particularly
ogy for the daily living activities of people with visual impair- inter-aural level differences,’’ Hearing Res., vol. 332, pp. 223–232,
ment as well as enhance their productivity in routine works. Feb. 2016.

VOLUME 8, 2020 2700305