Metaverse and

Immersive
Interaction
Technology
 Immersive Interaction Technology
• Immersive interaction technology can make the metaverse more realistic, thus allowing people
to experience the world better; therefore, immersive technology is the core technology of the
metaverse.

• Immersive interaction technologies include many aspects, and currently commonly used
technologies include Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and
Extended Reality (XR).

• VR can realize the input and output of information in the metaverse by taking over the
complete sense of sight, hearing, touch, and motion capture to bring people an all-around
immersive experience.

• AR technology superimposes a layer of virtual information on top of the real world, but
interaction is not yet possible at this stage.

• MR mixes the virtual with the real, creating virtual objects that can interact with the real
environment by projecting a light field on the retina, enabling partial retention of the virtual
image and the ability to switch freely with reality.

• XR includes all three kinds of “reality” (AR, VR, and MR), and it is also increasingly mentioned.

https://www.youtube.com/watch?v=_WHYJBM
Stages of Mental Expansion to Real World
 Virtual Reality (VR)
Virtual Reality (VR) technology involves mainly computer, electronic information, and simulation
technologies. VR creates an entirely virtual environment by isolating all real-world images through the
device.

Users can experience 3D virtual immersion through VR head-mounted displays. Virtual reality
technology uses computers to generate simulated environments and immerse the user in them. It uses
electronic signals generated by computer technology, combined with various output devices, to
transform real-life data into virtual images that people can see.

These images can be real objects in reality or substances that are invisible to our naked eyes, and 3D
models will represent these images. Because these images are not something we can see directly but
are simulations of the real world through computer technology, they are called virtual reality.

Virtual reality enables users to be fully immersed in a virtual world. A proper virtual environment should
be able to simulate the five human senses (taste, sight, smell, touch, and hearing), but this is not yet
entirely possible with current technology.
 Virtual Reality (VR)
VR devices have undergone several iterations of updates:

CardBoard: To allow the left and right eyes to see the standard picture, the image is split on the left and
the right, and the images seen by the two eyes are slightly different after the split screen, which can
produce a sense of three-dimensional distance, and the screen can be pushed up to a distance of one
meter through a convex lens, but doing so distorts the picture and creates colored edges. This negative
effect is well offset by making the image anti-distortion and the edge anti-color. To solve the problem of
other objects in the field of view, a cardboard box can be used to cover the lens to obscure the excess
vision. Based on the above principle, Google launched the first generation of simple VR devices,
CardBoard.
https://www.youtube.com/watch?v=9PQeQDL0ndo&t=16s

VR All-in-One Headset: Although the CardBoard has met the basic needs of a VR, this simple box does
not meet the needs of consumers to wear. For this reason, some researchers began to improve the
CardBoard, replacing the cardboard box with a sponge and rubber. Due to the different facial structures
of men and women and to meet the needs of near-sighted people, the new box added the eye distance
adjustment function. But this box requires the phone as part of the VR, also known as all in ones, such
as Xiaomi VR eyes and Storm Magic Mirror.

https://www.youtube.com/watch?v=BYWFhnU0ssM
 Virtual Reality (VR)
VR Splitter: Some manufacturers detached the phone from the VR device and this improved the
portability of the device, but in terms of performance, it still belonged to the lightweight products, as its
play and VR all-in-one machine are almost the same. Some examples are products such as the Piconeo,
GearVR of Samsung, etc.
https://www.youtube.com/watch?v=-gnvQS2xhRg

PC VR: VR all-in-one and VR splitters are based on the mobile processor; to get a higher resolution,
refresh rate, and more accurate tracking accuracy, it is necessary to add a high-performance PC, not
only that but also need to use multiple high-speed cameras to track the player’s movements, and the
most famous products are HTC VIVE.
https://www.youtube.com/watch?v=x_i5cUiB_Vw

Home Mainframe: VR Microsoft and Nintendo have not launched products in the field of home consoles,
Sony relies on years of technical accumulation, and the game ecology launched PS VR. With the support
of the performance of the game console, the camera monitors the colored light ball held by the player’s
hand to track its position and movement. But unlike the PC platform, the independence of Sony’s game
ecosystem dictates that game content is not as rich as PC VR.
Virtual Reality (VR)
 Augmented Reality (AR)
Augmented reality (AR) technology combines virtual objects with the real world by superimposing
images on the real world and using holograms.
Google first developed AR glasses, and each lens has a built-in projector that combines reality with the
virtual. It functions similarly to a smartphone placed on glasses. Essentially, AR and VR are similar; in
the long run, they are products that overlap and interact with reality and the virtual. Unlike VR, AR is an
augmentation of the real world. On the other hand, the core structure of VR is mainly based on the
environmental simulation system, which completely abandons the real world and strives to construct
the perfect virtual world that can deceive human senses.

However, it is not yet possible to completely confuse the virtual world with the real world at the optical
level. However, they cannot be confused at the optical level, and there is a transition phase. From a
technical point of view, in the roadmap of VR and AR, it can be seen that the underlying chip is
Qualcomm, the algorithm is a combination of visual technology AI, and the content engine is Unity. AR
has higher requirements for optics and algorithms. Optically, lens semiconductor optical technology
must be implemented, and algorithms require technologies such as large-scale positioning. Popular AR
applications include Pokemon Go and Snapchat’s AR bit-emojis. Compared to VR, which is a
cumbersome device to wear, AR has a more extensive user base and more application scenarios.
Various practical applications in medicine, education, and industry have proven that AR has a more
profound impact on humans as a tool.
 Augmented Reality (AR)
AR phones and AR smart glasses

Apple has been in the AR space for a long time, acquiring Metaio in 2015 and developing it into ARKit.
At the WWDC 2017 developer conference, the WWDC2017 developer conference, Apple provided an
interface for iOS 11 through ARKit. Its software solution using computer vision technology enables
intelligent understanding of the real environment by combining cameras, gyroscopes, and
accelerometers. After two iterations, a live demo of the Minecraft AR game was completed at
WWDC2019 on ARKit3, which supports live capture and full-body motion capture. In a subsequent
update, LiDAR was added to reduce the hardware configuration requirements.

AR smart glasses, such as ODG, Darqri, Vuzix, Epson, and Snapchat, have launched their products in the
enterprise or consumer space. Osterhout Design Group is one of the major U.S. military companies,
founded in 1999 as a technology incubator. It currently focuses on AR headsets with Snapdragon
processor’s 1080 pixel OLED displays. Its stand-alone computer glasses and “see-through” 3D displays
have a broad market. The navigation system and inertial sensing technology allow users to experience
telepresence, remote maintenance, and repair and perform well in industrial production

https://www.youtube.com/shorts/nEsP55HFwGM
Augmented Reality (AR)
 Mixed Reality (MR)
Mixed reality (MR) is a new visualization environment resulting from the fusion of real and virtual
worlds, where real and data entities coexist while being able to interact in real-time. In other words,
“images” are placed in real space, and these “images” can interact with familiar objects to some
extent. MRs key feature is that virtual and real objects can interact in real-time.

MR is between augmented and virtual reality, blurring the boundary between virtual and reality,
integrating digital virtual objects into the real world for interaction. In the virtual world, real objects
appear in the form of virtual reality.

Conceptually, MR is similar to AR, but MR allows interaction with the real world and instant access to
information. Traditional AR technology mainly uses prism optics to refract real images, but the
perspective is not large, and the clarity is not high enough. The new MR technology, to bring a better
immersive interactive experience, may choose helmets, mirrors, transparent devices, etc., as the
carrier of its technology in addition to glasses and projectors.

One of the more technologically advanced and versatile MR devices available today is Microsoft’s
HoloLens line, which includes HoloLens 1, released in 2015, and HoloLens 2, released in 2019. Users
can interact with holograms through gaze, voice, and gestures. In addition to Microsoft’s HoloLens
family of MR devices, there are well-known MR devices such as the Samsung Gentoo MR and HP MR.

https://www.youtube.com/watch?v=pIsjVaqdNpc
Mixed Reality (MR)
Extended
Reality
(XR)
•Extended reality (XR) refers
to the combination of real and
virtual through computers to
create a virtual environment
that can be interacted with by
humans and machines, which
is also the collective name for
various technologies such as
AR, VR, and MR. XR is a
rapidly growing field and can
be used in many applications
such as environmental,
marketing, real estate,
education and training, and
remote work.
https://www.youtube.com/watch?v=m_o99
 Application of Immersive Interaction Technology
Digital Twins
A digital twin is a digital dynamic twin that creates things from the real world in virtual space. With
the help of sensors, the operating state of the body and data from the external environment can be
mapped to the digital twin in real-time. The essence is to create a digital version of a “clone,” also
known as a “digital twin.” The digital twin realizes the feedback from the real physical system to the
digital model
in virtual space. Various simulations, analysis, data accumulation, exploration, and even artificial
intelligence applications based on the digital model apply to the real physical system. The digital
twin can replicate the physical elements of the real world, and its end product is a “clone universe”
that mirrors the real world. The metaverse is a copy and modification of the real world based on the
logic of reality or illusion (e.g., surrealism, science fiction, etc.).

https://www.youtube.com/watch?v=GAkB98e
 Digital Twins & its characteristics
Interoperability: The digital twin’s physical objects and digital spaces can be mapped, dynamically
interacted, and connected in both directions. Thus, the digital twin can map physical entities to
various digital models and transform and fuse between different digital models.

Real-Time: Because the digital twin is going to reproduce physical entities that change with the
timeline, the data needs to be managed so that computers can recognize and process, namely,
digitized.

Scalability: Digital twin technology can integrate, add, and replace digital models and extend model
content.

Fidelity: The digital twin requires virtual objects to maintain a high degree of simulation not only in
terms of solid geometry but also in terms of state, phase, and temporal state, making every effort to
ensure the similarity between the digital virtual model and the physical entity.

Closed Loop: The digital virtual body in the digital twin is used to describe the visual model and
internal mechanisms of the physical entity, thus monitoring the state data of the physical entity,
performing analytical reasoning, optimizing process parameters and operational parameters, and
implementing decision-making functions, in other words, using a closed-loop system for both the
virtual body and the physical entity.
 Digital Twins & its Applications
Digital Design: The digital twin technology is used to create a digital twin of the product design and
perform system simulation in the virtual space to achieve feedback design, iterative innovation, and
continuous optimization. Currently, digital design practices in prototype design, process design,
engineering design, and digital prototyping have been commonly carried out in automotive, marine,
aerospace, and precision equipment manufacturing.

Virtual Factory: Virtual factory refers to the digital virtual workshop and digital factory based on the
combination of digital twin technology and MES. It can realize the dynamic data interaction between
physical entities and digital virtual entities and forecast production in real-time according to the changes
in the virtual space.

Equipment Maintenance: Develop and design the devices digital twin and interact with the physical entity
synchronously to achieve digital management of the device lifecycle.

Smart City: By building a digital twin of the city, we simulate the interaction of weather, infrastructure,
population, land, industrial transportation, and other elements in the digital world in a combination of
quantitative and qualitative forms and draw a “city portrait” to help decision-makers improve urban
planning in the physical world.

Smart Healthcare: The digital twin is combined with medical services to achieve dynamic monitoring,
simulation, and modeling of the human body’s operating mechanism and medical devices, accelerate the
transformation of scientific research innovation into clinical practice, improve the efficiency of
medical diagnosis, and optimize the control and supervision of medical device quality.
Some other Applications
Some other Applications
Some other Applications
Some other Applications