Chapter Five

Augmented Reality [AR]

The fundamental idea of AR is to combine, or mix, the view of the real environment with
additional, virtual content that is presented through computer graphics. Its convincing effect
is achieved by ensuring that the virtual content is aligned and registered with the real objects.
As a person moves in an environment and their perspective view of real objects changes, the
virtual content should also be presented from the same perspective.
Augmented reality (AR) is a form of emerging technology that allows users to overlay
computer-generated content in the real world. AR refers to a live view of a physical real-
world environment whose elements are merged with augmented computer-generated images
creating a mixed reality.
The augmentation is typically done in real-time and in semantic context with environmental
elements. By using the latest AR techniques and technologies, the information about the
surrounding the real world becomes interactive and digitally usable. Through this augmented
vision, a user can digitally interact with and adjust information about their surrounding
environment.
Augmented Reality (AR) is a real-time direct or indirect view of a physical real-world
environment that has been enhanced/augmented by adding virtual computer-generated
information to it.
Augmented reality is the integration of digital information with the user's environment in
real-time. Unlike virtual reality, which creates an artificial environment, augmented reality
uses the existing environment and overlays new information on top of it. A live direct or
indirect view of a physical, real-world environment whose elements are augmented by
computer-generated sensory input such as sound, video, graphics, or GPS data.
5.2. Virtual reality (VR), Augmented Reality (AR) vsMixed reality (MR)
With constant development in computer vision and the exponential advancement of computer
processing power, virtual reality (VR), augmented reality (AR), and mixed reality (MR)
technology is becoming more and more prominent. With some overlap in the applications and
functions of these emerging technologies, sometimes these terms get confused or are used
incorrectly. The main differences between them are explained below.

Emerging Technologies Page 1

5.2.1. Virtual Reality (VR)
VR is fully immersive, which tricks your senses into thinking you’re in a different
environment world apart from the real world. Using a head-mounted display (HMD) or
headset, you'll experience a computer-generated world of imagery and sounds in which you
can manipulate objects and move around using haptic controllers while tethered to a console
or PC. It is also called a computer-simulated reality. It refers to computer technologies using
reality headsets to generate realistic sounds, images, and other sensations that replicate a real
environment or create an imaginary world. Advanced VR environment will engage all five
senses (taste, sight, smell, touch, sound), but it is important to say that this is not always
possible (See Figure 5.2).

Figure 5.2 Example of Immersive Technology

Using VR devices such as HTC Vive, Oculus Rift googles Cardboard, users can be
transported into several real-world and imagined environments. The most advanced VR
experiences even provide freedom of movement – users can move in a digital environment
and hear sounds. Moreover, special hand controllers can be used to enhance VR experiences.
Most VR headsets are connected to a computer (Oculus Rift) or a gaming
console(PlayStationVR) but there are standalone devices (Google Cardboard is among the
most popular) as well. Most standalone VR headsets work in combination with smartphones
– you insert a smartphone, wear a headset, and immerse in the virtual reality (see Figure 5.3).

Emerging Technologies Page 2

Figure 5.3 VR Case that Inserts a Smartphone

5.2.2. Augmented Reality (AR)

In augmented reality, users see and interact with the real world while digital content is added
to it. If you own a modern smartphone, you can easily download an AR app and try this
technology.
There’s a different way to experience augmented reality, though – with special AR headsets,
such as Google Glass, where digital content is displayed on a tiny screen in front of a user’s
eye.AR adds digital elements to a live view often by using the camera on a smartphone.
Examples of augmented reality experiences include Snapchat lenses and the game Pokemon
Go. augmented reality (AR) is a live, direct or indirect view of a physical, real-world
environment whose elements are augmented (or supplemented) by computer-generated
sensory input such as sound, video, graphics, or GPS data.

5.2.3. Mixed Reality (MR)

Mixed Reality (MR), sometimes referred to as hybrid reality, is the merging of real and
virtual worlds to produce new environments and visualizations where physical and digital
objects co-exist and interact in real-time. It means placing new imagery within a real space in
such a way that the new imagery can interact, to an extent, with what is real in the physical
world we know (see figure 5.5 and Figure 5.6). For example, with MR, you can play a virtual
video game, grab your real-world water bottle, and smack an imaginary character from the
game with the bottle. Imagination and reality have never been so intermingled. The key
characteristic of MR is that the synthetic content and the real-world content can react to each
other in real-time.

Emerging Technologies Page 3

Figure 5.5 Mixed Reality in Engineering and Medicine

In mixed reality, you interact with and manipulate both physical and virtual items and
environments, using next-generation sensing and imaging technologies. MR allows you to
see and immerse yourself in the world around you even as you interact with a virtual
environment using your own hands—all without ever removing your headset.

Figure 5.6 Mixed Reality in Entertainment

It provides the ability to have one foot (or hand) in the real world, and the other in an
imaginary place, breaking down basic concepts between real and imaginary, offering an
experience that can change the way you game and work today. One of the most obvious
differences among augmented reality, virtual reality, and mixed reality is the hardware
requirements and also VR is content that is 100% digital and can be enjoyed in a fully
immersive environment, AR overlays digital content on top of the real world. and MR is a
digital overlay that allows interactive virtual elements to integrate and interact with the real-
world environment. Numerous augmented reality apps and games can run on almost every
smartphone on the market.

On the other hand, virtual reality programs require specialized VR headsets, noise-canceling
headphones, cameras to track room space and boundaries, and sometimes even motion

Emerging Technologies Page 4

capture technology. Some of the biggest names in VR tech today are the Oculus Rift, HTC
Vive, and PlayStation VR. For the enjoyment of simple VR videos, there are affordable
makeshift VR headsets like the Google Cardboard, which work by running a video in 360
modes on your smartphone and inserting the phone into the headset. Mixed reality hardware
is still emerging and hasn’t quite broken into the mainstream consumer market, most likely
due to the price. The consumer releases of the Microsoft HoloLens and MagicLeap One retail
for over USD 2000, which is 3 to 4 times the cost of the PlayStation VR and HTs Vive VR
headsets. However, mixed reality applications sometimes require exponentially more
processing power and thus require more powerful hardware. For example, the Microsoft
HoloLens includes a built-in microphone array, binaural sound capabilities, a built-in camera
for recording, a depth sensor, head-tracking cameras, and an inertial measurement unit that
helps track your head movement. On top of the traditional CPU and GPU, Microsoft also
created a Holographic Processing Unit to help track where the user is looking and understand
command gestures.

5.3. The architecture of AR Systems

The first Augmented Reality Systems (ARS) were usually designed with a basis on three
main blocks, as is illustrated in Figure 5.7: (1) Infrastructure Tracker Unit, (2) Processing
Unit, and (3)Visual Unit. The Infrastructure Tracker Unit was responsible for collecting data
from the real world, sending them to the Processing Unit, which mixed the virtual content
with the real content and sent the result to the Video Out module of the Visual Unit. Some
designs used a Video In, to acquire required data for the Infrastructure Tracker Unit.

Emerging Technologies Page 5

Figure 5.7 Augmented Reality Systems (ARS) standard architecture
The Visual Unit can be classified into two types of system, depending on the followed
visualization technology:
1. Video see-through: It uses a Head-Mounted Display (HMD) that employs video-mixing
and displays the merged images on a closed-view HMD.
2. Optical see-through: It uses an HMD that employs optical combiners to merge the images
within an open-view HMD.HMDs are currently the dominant display technology in the AR
field. However, they lack in several aspects, such as ergonomics, high prices, and relatively
low mobility due to their sizes and connectivity features. An additional problem involving
HMD is the interaction with the real environment, which places virtual interactive zones to
the user, making the collision with these zones hard due to the difficulty to interact with
multiple points in different depths. Alternative approaches to developing ARS involve the use
of monitors and tablets.
Monitors are used as an option for the indirect view since the user does not look directly into
the mixed world. Tablets are used in direct view since the user points the camera to the scene
and looks directly into the mixed world. Both approaches still have difficulties in getting a
collision.

Emerging Technologies Page 6

5.4. Applications of AR Systems
Technology is ever-changing and ever-growing. One of the newest developing technologies
is augmented reality (AR), which can be applied to many different disciplines such as
education, medicine, entertainment, military, etc. Let us see some of its applications.
5.4.1. AR In education
Augmented reality allows flexibility in use that is attractive to education. AR technology can
be utilized through a variety of mediums including desktops, mobile devices, and
smartphones. The technology is portable and adaptable to a variety of scenarios. AR can be
used to enhance content and instruction within the traditional classroom, supplement
instruction in the special education classroom, extend content into the world outside the
classroom, and be combined with other technologies to enrich their applications. More
importantly, the following reasons for using augmented reality in education:
• Affordable learning materials - posters, digital illustrations, physical models, prototypes are
very expensive and schools can’t find enough money to purchase all
the supplementary materials they would like to. Using AR technology allows for avoiding
investments in physical materials. Besides, students can get access to learning materials and
interact with them anytime.
• Interactive lessons - when AR technology is used in classrooms, students can view models
on their smartphones and get a better idea of the concepts they are studying. That increases
engagement and reinforces learning.
• Higher engagement - when teachers integrate augmented reality into their lectures, they
attract the attention of their students and make lessons more effective. When students are
interested, it is much easier to make them work more productively.
• Higher retention - using the AR app, students can get access to augmented models that
represent any real objects from a famous monument or work of art to a molecule. Besides,
students can get access to a website with specific information. When learning with AR
technology, students use different senses and retain more knowledge for a long time.
• Boost intellectual curiosity - augmented reality makes students more excited about learning
certain subjects. Modern students were born in a digital era so they will always be excited
with innovative technologies that can help them learn new ideas and develop their critical-
thinking skills.

Emerging Technologies Page 7

5.4.2. AR In Medicine
Augmented reality is one of the current technologies changing all industries, including health
care and medical education. The purpose of any invention and technology is to simplify our
life. Augmented reality has the potential to play a big role in improving the healthcare
industry. Only a few years since the first implementations of augmented reality in medicine,
it has already filled an important place in doctors’ and nurses’ routine, as well as patients’
lives.
This new technology is enhancing medicine and healthcare towards more safety and
efficiency. For now, augmented reality has already made significant changes in the following
medical areas:
• surgery (minimally invasive surgery);
• education of future doctors;
• diagnostics;
• AR tools may also aid to detect the signs of depression and other mental illnesses by
reading from facial expressions, voice tones, and physical gestures. In medicine, AR has the
following applications:
1) Describing symptoms - Have you ever been in a situation when it was hard to describe to
the doctor what was bothering you? It is a common problem for all of us, the roots of which
extend to overreacting and lack of knowledge. And what is most important, it impacts on
finding out the accurate diagnosis. The first steps to find the solutions are already made. To
increase patients’ education, the medical app AyeDecide is using augmented reality to show
the simulation of the vision, harmed by the different diseases. It helps patients to understand
their conditions and describe correctly their symptoms.
2) Nursing care - About 40% of the first intravenous injections fail, and this ratio is even
higher in the case of children and elderly patients. AccuVein uses augmented reality to cope
with this negative statistic. A handheld scanner projects on the skin and shows the patient's
veins. It increases the successful finding of the vein from the first try in 3,5times. That is why
this invention got the greatest recognition among the general public and medical staff.
3) Surgery - In no sphere augmented reality does not have such practical application as in
medicine, especially in surgery, where it helps to save lives.
Three-dimensional reconstructions of organs or tumors will help surgeons become more
efficient at surgery operations. For example, spinal surgery, as usual, is a long and difficult
process.

Emerging Technologies Page 8

But with the use of AR, can reduce the time, cut the risks, and improve the results. The Israeli
startupAugmedics had created an augmented reality headset for spine surgeons. This
technology overlays a 3D model of the CT scan on the spine, so, the surgeon gets some kind
of “X-ray” vision.
4) Ultrasounds - Some time ago ultrasound made a small revolution in medicine. Today, it
has another chance to make the same with using augmented reality. Already a few AR
software companies developed handy ultrasound scanners, which with the help of smart
glasses works as a traditional ones. It is hard to overestimate the usefulness of this
technology. Especially when we talk about using it in developing countries, in military
medicine (on the battlefields), and even in the ambulance.
5) Diabetes management - In 2017, the number of people who struggle with diabetes reached
up to 425 million adults worldwide. And the amount of diagnosed people is increasing every
year. In 2014, Google revealed the plans for creating a smart contact lens (GoogleContact
Lens), in which the main function will be to measure the glucose levels in theaters.
It will help people with this disease to live the life they used to, without permanent worries
about sugar levels in the blood.
6) Navigation - Using AR in navigation apps has already become a “traditional” way. By
pointing your phone to the city landscape, you get information about nearby objects of your
interest (museums, hotels, shops, metro stations, etc.). In the same way, AR can be useful to
provide information about the nearest hospitals. For example, the EHBOapp helps to find the
nearest to you AEDs (automated external defibrillators). Generally, AR provides the
following benefits to patients and healthcare workers:
• Reduce the risks associated with minimally invasive surgery.
• Better informed decisions about the right treatment and illness prevention.
• Make procedures more tolerable.
• Better aftercare
• Medical training and education.
• Assistance in medical procedures and routine tasks.
5.4.3. AR In Entertainment

Augmented reality can be used in various “entertainment” industries as entertainment covers

quite a several different industries – music, movies, live shows, games – and all of them can
benefit from using augmented reality.

Emerging Technologies Page 9

• AR in games - the AR games were praised for increasing physical activity in people – you
have to move around to find your target, for example, Pokémon. At the same time, there are
complaints that players could cause various incidents and accidents being too engrossed in
the game. In any case, Pokémon GO has rightfully earned its popularity and opened the world
of AR games to us.

• AR in music - music is not only about listening to favorite tracks and putting together
playlists. When we like a piece, we often want to find out more about its background: the
performer's bios, the lyrics of the song, the making of the recording, or the music video.
Augmented reality can do all that and much more providing complete information on the
track or its performer. Augmented reality can enhance live performances by illustrating the
story told by a track or displaying the way it was created by the band.

• AR on TV - this may seem a bit far-fetched, as television already shows a virtual world,
thus adding augmented reality will raise it to the second power. However, some experiments
of fusing augmented reality on TV are already being made with the promise of future
enhancements. One way of integrating augmented reality in television is adding
supplementary information to what is going on the TV screen – such as match scores, betting
options, and the like.

• AR in eSports- recently, the industry of eSports has been gaining popularity in all parts of
the globe. Competitive online gaming has become as fascinating as real sports, and the
technology is following it closely with new solutions and unusual implementations.
Augmented reality turns eSports shows into interactive experiences allowing the watchers to
become participants.

• AR in the theater- in this sector, augmented reality can serve not only for entertainment
purposes but also for accessibility. The possibility to overlay virtual objects over the real
environment can be used, for example, for subtitling in various theater shows. Now, many
theaters use LED displays either to provide subtitles for translation or to assist hearing-
impaired visitors. However, LED equipment is not available in each theater and even when it
is, it can distract the viewers from the show.

Emerging Technologies Page 10