A VIRTUAL MOUSE

Submitted for the partial fulfillment for the award of the

degree of

B.Sc., Computer Science

Submitted by

RAMYA. V
REG.NO : 222102912

Under the guidance of

Dr. PRIYA N, M.Sc., M.Phil., Ph.D.

Assistant Professor
Department Of Computer Science

HINDUSTAN COLLEGE OF ARTS & SCIENCE

DEPARTMENT OF COMPUTER SCIENCE

April – 2024
 HINDUSTAN COLLEGE OF ARTS & SCIENCE
(Affiliated to University of Madras | Reaccredited by NAAC | 2(f) Status by UGC)
Rajiv Gandhi Road(OMR), Padur, Kelambakkam, Chennai – 603 103.

DEPARTMENT OF COMPUTER SCIENCE

CERTIFICATE

This is to certify that the project entitled “A VIRTUAL MOUSE” is being

submitted to the University of Madras, Chennai by RAMYA V, Register No:
222102912 for the fulfillment for the award of degree of Bachelor of Science in
Computer Science is a bonafide record of work done by her under my guidance and
supervision during the academic year 2021-2024.

GUIDE Head of the Department

Place :
Date :

Submitted for the Viva-Voce examination on …… / …….. / …………….

Examiners

1). _____________________________ 2). _____________________________

 DECLARATION

This is certified that the Mini project work carried by RAMYA. V at Hindustan Collage Of
Arts And Science, Chennai under the direct supervision and the proper guidance of the
Dr. PRIYA N for the partial fulfillment of the requirement for the award of the degree in B.Sc.
Computer Science, under Hindustan College of Arts & Science.

I hereby declare that this project is a bonafide work done in VI semester in B.Sc. Computer
Science as a Mini Project and completed successfully.

Signature of Candidate

Place:
Date:
 ACKNOWLEDGEMENT

First I would like to thank the Almighty, for showering his blessings and supporting hands on
all instances of this project and making it a complete success.

I wish to record my profound and sincere gratitude to the Director

Dr. SUSAN VERGHESE, Deputy Director Dr. V.J. PHILIP, of our institution and the Principal
Dr. S. THIRUMAGAN, for extending their full support and co-operation in boosting my skills by
providing me with all latest facilities in the lab.

I would like to put forward my hearty gratitude to our Dean & Head of the Department
Dr. Y. ANGELINE CHRISTOBEL, for her valuable assistance in the project.

I am indebted to Dr. PRIYA N, M.sc, M.Phil., Ph.D., in the Department of Computer

Science, my project guide for her constant words of encouragement and suggestions.

I also thank all the other STAFF of the Department of Computer Science for their support in
my project.

Last but not least I would like to thank my PARENTS AND FRIENDS for all my endeavors
during the project and for always providing me with the necessary motivation to complete this
project.
 TABLE OF CONTENTS

CHAPTER CONTENT NAME PAGE NO.

NO.

1 INTRODUCTION 1
1.1 About the Project 1
2
1.2 Architecture Diagram
3
1.3 Need for the project 4
7
1.4 Existing work
9
1.5 Proposed work 10

1.6 Advantages of proposed work

1.7 Problem statement

12
2 LITERATURE SURVEY 13
2.1 Review of Literature survey

3 METHODOLOGY 15
3.1 System design
3.2 virtual mouse model 16
3.3 Process model 18
3.4 DFD
3.5 Er diagram 19
3.6 Dataset Description 20
3.7 pseudo code
3.8 system requirements
21
3.9 Hardware requirements 22
24
25
 28
4 RESULT ANALYSIS

5 SAMPLE COADING 33

6 CONCLUSION 45
46
7 FUTURE ENHANCEMENENT
 CHAPTER 1

INTRODUCTION

A virtual mouse is software that allows users to give mouse inputs to a system without using
an actual mouse. To the extreme it can also be called as hardware because it uses an ordinary
web camera. A virtual mouse can usually be operated with multiple input devices, which may
include an actual mouse or a computer keyboard. Virtual mouse which uses web camera
works with the help of different image processing techniques. In this the hand movements of
a user is mapped into mouse inputs. A web camera is set to take images continuously. Most
laptops today are equipped with webcams, which have recently been used insecurity
applications utilizing face recognition. In order to harness the full potential of a webcam, it
can be used for vision based CC, which would effectively eliminate the need for a computer
mouse or mouse pad. The usefulness of a webcam can also be greatly extended to other HCI
application such as a sign language database or motion controller. Over the past decades there
have been significant advancements in HCI technologies for gaming purposes, such as the
Microsoft Kinect and Nintendo Wii. These gaming technologies provide a more natural and
interactive means of playing videogames. Motion controls is the future of gaming and it have
tremendously boosted the sales of video games, such as the Nintendo Wii which sold over 50
million consoles within a year of its release. HCI using hand gestures is very intuitive and
effective for one to one interaction with computers and it provides a Natural User Interface
(NUI). There has been extensive research towards novel devices and techniques for cursor
control using hand gestures. Besides HCI, hand gesture recognition is also used in sign
language recognition, which makes hand gesture recognition even more significant.

1.1About the project

A virtual The user may use additional hand gestures and draw point-based lines with their
hand to move the computer cursor around. The recommended framework involves a low-goal
camera as a sensor and can show hued point based lines while following a client's hand in
two aspects. Python is utilized to carry out the framework.

mouse typically refers to a software-based or simulated mouse pointer used on a computer or

mobile device. It allows users to interact with the graphical user interface by moving the
pointer on the screen, clicking, and performing various mouse-related actions without the

1
need for a physical mouse. Virtual mice are commonly used in touchscreen devices, remote
desktop applications, and virtual machine environments to navigate and control the operating
system and applications. If you have specific questions or need more information about
virtual mice, please feel free to ask

The Al Virtual Mouse uses computer vision techniques to track hand movements and
translates them into cursor movements on the screen. The system is designed to be intuitive
and user-friendly, allowing users to interact with their computer without the need for a
physical mouse.

1.2Architecture Diagram

A virtual mouse doesn't have a complex architecture like a physical device, but it involves
software components and interactions within the operating system and software applications.
Here's a simplified architecture diagram of how a virtual mouse works:

 Input Devices: Physical input devices, such as a touchpad, touchscreen, or even

external mice, provide input signals to the computer.

 Operating System Input Stack: The operating system's input stack processes input
signals from various devices, including those from virtual mice.

 Virtual Mouse Driver: In cases where a virtual mouse is emulated or required, a

virtual mouse driver interprets and generates virtual mouse input.

 Graphics Hardware and Drivers: The virtual mouse input is sent to the graphics
hardware, which is responsible for rendering the visual elements on the screen. The
graphics drivers facilitate this communication.

 Operating System Cursor Control: The operating system uses the virtual mouse input
to control the position of the cursor on the screen.

 Applications: Applications running on the computer interact with the cursor position
to respond to user input, including clicking, dragging, and more.

2
  User Interface: The user interacts with the virtual mouse through the user interface of
the applications and the operating system.

It's important to note that the exact architecture may vary depending on the operating system
and the specific software or hardware configurations in use. The diagram above represents a
simplified overview of how a virtual mouse integrates into the broader system

1.3Need for the project

The virtual mouse is developed using Python and OpenCV libraries. The project includes the
implementation of various image processing algorithms, such as hand segmentation, feature
extraction, and classification. Moreover, it is robust to various lighting conditions,
backgrounds, and hand sizes.

The need for a virtual mouse project can arise in various scenarios where traditional physical
mice or touch input may not be practical or accessible. Here are some common use cases and
reasons for developing a virtual mouse project:

 Accessibility: Virtual mice can greatly benefit individuals with disabilities, such as
those with limited mobility who may have difficulty using physical mice or
touchscreens. It provides an alternative way to interact with a computer.

3
  Remote Desktop: In remote desktop or virtual desktop environments, users may need
to control a computer from a different location. Virtual mice facilitate this control
without the need for physical peripherals.

 Embedded Systems: In embedded systems, such as IoT devices or kiosks, a virtual

mouse can provide a user interface for configuration and control when a physical
mouse is not available.

 Virtual Machines: Virtualization technology often requires virtual mice to interact

with guest operating systems running in virtual machines.

 Gaming: Some gaming scenarios may involve virtual mice for controlling characters
or objects in virtual environments.

 Testing and Simulation: Developers and testers might use virtual mice to simulate
user interactions for testing software applications.

 Mobile Devices: Virtual mice can be used in mobile devices to enhance user
experience when navigating through apps and websites.

 Presentations and Education: Virtual mice can be handy during presentations, online
training, and educational applications where the presenter or instructor needs to
highlight or interact with content.

 Gesture Recognition: In gesture-based control systems, a virtual mouse can be used to

interpret and respond to hand or motion gestures.

 Custom Interfaces: Sometimes, unique or specialized interfaces may require virtual

mice to create tailored user experiences.

The need for a virtual mouse project ultimately depends on the specific requirements and
challenges of a given application or situation. Developing a virtual mouse solution can
improve accessibility, convenience, and usability in various computing environments
1.3.1Existing work
This type of virtual mouse system is typically developed using the Python programming
language, and computer vision projects also employ the OpenCV package. The system makes
4
use of well-known Python libraries like MediaPipe for the real-time tracking of hands and
finger movements.

Creating a virtual mouse using Python typically involves simulating mouse movements and
clicks programmatically. Here are some common libraries and methods to achieve this:

Pyautogui: PyAutoGUI is a popular Python library for automating mouse and keyboard
interactions. You can use it to move the mouse, click, and perform various tasks. Here's a
basic example:
import pyautogui

# Move the mouse to coordinates (x, y)

pyautogui.moveTo(x, y)

# Click the mouse at the current location

pyautogui.click()
pynput: The pynput library allows you to control and monitor input devices, including the
mouse. You can use it to control mouse movements and clicks. Here's a simple example:

from pynput.mouse import Controller

mouse = Controller()

# Move the mouse to coordinates (x, y)

mouse.position = (x, y)

# Click the mouse at the current location

mouse.click()
AutoIt: While not a Python library, AutoIt is a scripting language and automation tool that
can be used to control the mouse and keyboard on Windows systems. You can execute AutoIt
scripts from Python using the subprocess module.

pyMouse: pyMouse is a simple Python library for simulating mouse events. It's a bit less
feature-rich than some other options but can be easy to use for basic mouse control.

5
Please note that to use these libraries, you may need to install them using pip. Also, some
operating systems might require elevated privileges to simulate mouse movements, especially
if your application needs to interact with other programs running on the system.
Keep in mind that simulating a virtual mouse can be used for legitimate purposes like
automation and accessibility, but it can also be misused. Always follow ethical guidelines and
respect the terms of service of the applications and systems you are interacting with
Creating a virtual mouse involves a combination of hardware and software solutions to
emulate mouse input. Here are some existing work and technologies related to virtual mice:
Touchscreen Technology: Many modern devices, such as smartphones and tablets, use
touchscreen technology to create a virtual mouse-like experience. Users can touch, swipe,
and tap on the screen to control the cursor and interact with applications.

Stylus and Graphics Tablets: Artists and designers often use stylus and graphics tablet
combinations to create a virtual pen or mouse. These devices offer precise control for
drawing and design work.

 Trackpads :Laptop computers often come with built-in trackpads that act as virtual
mice. Users can swipe and tap on the trackpad to move the cursor and click.

 Wireless Presenters: Wireless presenters, often used in presentations, come with built-
in laser pointers that can act as virtual mice for controlling slides and navigating
presentations.

 Gesture Control: Technologies like Microsoft's Kinect and Leap Motion allow users
to control the cursor and interact with applications using hand and gesture
movements, creating a virtual mouse experience.

 VR and AR Controllers: Virtual Reality (VR) and Augmented Reality (AR) systems
often include handheld controllers that can function as virtual mice within virtual
environments. They enable users to interact with the virtual world.

 Voice Commands: Some virtual assistant systems, such as Siri, Google Assistant, and
Amazon Alexa, allow users to control certain actions by voice commands, reducing
the need for traditional mouse input.

6
  Accessibility Software: Various accessibility software tools and solutions cater to
individuals with physical disabilities. These may include head-tracking systems, eye-
tracking devices, and more to create virtual mouse experiences for those who cannot
use traditional mice.

 Remote Desktop and Screen Sharing Software :Applications like TeamViewer and
Any Desk allow users to control a remote computer's mouse and keyboard over the
internet, essentially creating a virtual mouse for remote control.

 Gaming Controllers: In the realm of gaming, various controllers, joysticks, and

gaming mice are designed to offer precise and customizable control for gamers,
creating a virtual mouse experience for gaming purposes.

These are some of the existing technologies and solutions related to virtual mice. They serve
a variety of purposes, from general computer use to specific applications like accessibility
and gaming.
1.3.2 Proposed work
Proposed work in the field of virtual mice can focus on improving existing technologies,
addressing new use cases, and enhancing the user experience. Here are some potential areas
of research and development for virtual mouse technology:
Gesture Recognition Develop more advanced gesture recognition systems that allow users to
control the virtual mouse through hand or body movements with higher precision and
reliability. This can be applied in gaming, virtual reality, and other interactive applications.
Natural Language Control Explore natural language processing and voice recognition for
controlling the virtual mouse. This can make computer interactions more accessible and user-
friendly.
Machine Learning and AI Integration Integrate machine learning and artificial intelligence to
create smart virtual mice that adapt to user preferences and behavior. This can improve user
efficiency and enhance the user experience.
Cross-Platform Compatibility Develop virtual mouse solutions that work seamlessly across
different operating systems and devices, promoting interoperability and reducing user
friction.
Accessibility Enhancements Focus on improving virtual mouse technology for individuals
with disabilities, such as advanced eye-tracking systems or neurotechnology-based control
methods.

7
Virtual Reality and Augmented Reality Integration Enhance virtual mouse control within VR
and AR environments, enabling users to interact more naturally with virtual objects and
interfaces.
Haptic Feedback Integrate haptic feedback into virtual mouse solutions to provide users with
tactile sensations, enhancing the sense of touch and realism in virtual interactions.
Security and Privacy Research and implement security features to protect virtual mouse
systems from unauthorized use or malicious manipulation.
Collaborative Virtual Mice Develop virtual mice that can be used collaboratively by multiple
users simultaneously, enabling shared control in virtual environments or remote collaboration
scenarios.
Education and Training Create virtual mouse technologies for educational and training
purposes, allowing users to practice and learn various skills in a controlled virtual
environment.
Health and Wellness Applications Investigate how virtual mice can be used in healthcare and
wellness applications, such as physical therapy, mental health interventions, or stress relief.

Virtual Mouse for IoT and Smart Homes Implement virtual mouse control for interacting
with Internet of Things (IoT) devices and controlling smart home systems more intuitively.
Energy-Efficient Solutions Develop power-efficient virtual mouse technologies for portable
and battery-powered devices, ensuring longer usage without draining the battery quickly.
Customization and Personalization Enable users to customize the behavior and appearance of
their virtual mouse, tailoring it to their preferences and needs.
Real-World Integration Explore how virtual mouse technology can be applied in real-world
scenarios, such as robotics, industrial automation, and autonomous vehicles.
Proposed work in virtual mouse technology should aim to enhance user interactions, improve
accessibility, and adapt to the evolving landscape of computing devices and applications.
Research and development efforts in these areas can lead to more efficient, user-friendly, and
versatile virtual mouse solutions.
Advanced Gesture Recognition Improve gesture recognition algorithms to make the virtual
mouse respond to a wider range of hand and body movements for more intuitive control.
3D Virtual Mouse Develop a 3D virtual mouse that can navigate and interact in three-
dimensional spaces, potentially for virtual reality environments.
Voice-Controlled Virtual Mouse Integrate voice commands for controlling the virtual mouse,
allowing users to navigate and interact with applications using speech.

8
Machine Learning for User Intent Prediction Implement machine learning models to predict
user intent, making the virtual mouse more adaptive to individual user preferences and
behavior.
Enhanced Haptic Feedback Integrate haptic feedback technology to provide tactile
sensations when interacting with virtual objects, improving the sense of touch Accessibility
Features Focus on creating accessible virtual mouse solutions for individuals with disabilities,
such as eye-tracking support, voice commands for hands-free control, and alternative input
methods.
Cross-Platform Compatibility Develop a Python-based virtual mouse that works across
various platforms, including Windows, macOS, and Linux. Security and Privacy Investigate
security measures to protect against unauthorized access and data breaches when using the
virtual mouse. Hybrid Physical and Virtual Interaction Combine physical input devices (e.g.,
touchscreens, styluses, or specialized hardware) with Python-based virtual mouse control for
more natural and precise interactions.
1.3.3 Advantages of proposed work
Proposed work on a virtual mouse using Python offers several advantages:
 Accessibility: It can provide an alternative input method for individuals with physical
disabilities who may have difficulty using a physical mouse.

 Cost-Effective: It eliminates the need for purchasing additional hardware like a

physical mouse, making it a cost-effective solution.

 Customizability: Python allows for easy customization and integration with various
applications, enabling users to tailor the virtual mouse to their specific needs.

 Remote Control: It can be used for remotely controlling a computer or device, which
is useful for presentations, demonstrations, or when operating a computer from a
distance.

 Gesture Control: The virtual mouse can be programmed to respond to gestures,

enhancing the user experience and making it more interactive.

 Educational Purposes: It can serve as a valuable educational tool for teaching

programming, human-computer interaction, and software development.

9
  Research and Development: Researchers can use it to explore innovative input
methods, human-computer interaction techniques, and usability studies.

 Cross-Platform: Python is a cross-platform language, so the virtual mouse can be

implemented on various operating systems, including Windows, macOS, and Linux.

 Integration with Other Software: Python's extensive libraries and modules enable
seamless integration with other software and services, enhancing the functionality of
the virtual mouse.

 Open Source Community: Python's open-source nature means that the virtual mouse
project can benefit from a supportive community of developers and enthusiasts,
leading to ongoing improvements and updates.

These advantages make the development of a virtual mouse using Python a promising and
versatile project with numerous potential applications and benefits.
1.4 Problem statement
Develop a virtual mouse system using Python to address the following challenges and
objectives:
Accessibility Create a virtual mouse interface that offers an accessible alternative to physical
mice for individuals with disabilities or those experiencing difficulty in using traditional input
devices . Gesture Control Implement a system that recognizes and responds to a range of
gestures to control the mouse pointer, such as swipes, pinches, and taps, for improved user
interaction.
Cross-Platform Compatibility Ensure the virtual mouse system is compatible with popular
operating systems, including Windows, macOS, and Linux, to reach a wide user base
Accuracy and Responsiveness Develop algorithms that guarantee accurate and responsive
mouse movement and clicks, minimizing input latency to enhance user experience
Customizability Provide users with the ability to customize settings, sensitivity, and
additional features to tailor the virtual mouse to their specific needs.
Remote Control Enable remote control of a computer or device using the virtual mouse,
allowing users to perform tasks and presentations from a distance. Integration Explore
opportunities to integrate the virtual mouse system with other software and services, making
it a versatile tool for various applications Security Implement security measures to prevent
unauthorized access or control of the virtual mouse, especially in remote control scenarios.

10
User-Friendly Interface Create an intuitive and user-friendly interface that accommodates
users of all technical levels Documentation and Support Provide comprehensive
documentation and support resources to assist users in setting up and using the virtual mouse
system effectively testing and Optimization rigorously test the system's functionality and
performance, addressing any bugs or issues, and continuously optimize the software for better
user experience. Community Involvement Encourage collaboration and contributions from
the open-source community, fostering a community-driven approach to ongoing development
and improvements.
By defining a clear problem statement and objectives, the development of a virtual mouse
system using Python can be systematically approached, with a focus on addressing
accessibility, user experience, and customization while ensuring compatibility and security.

11
 CHAPTER.2

2. LITERATURE SURVEY
As modern technology of human computer interactions become important in our everyday
lives, varieties of mouse with all kind of shapes and sizes were invented, from a casual office
mouse to a hard-core gaming mouse. However, there are some limitations to these hardware
as they are not as environmental friendly as it seems. For example, the physical mouse
requires a flat surface to operate, not to mention that it requires a certain area to fully utilize
the functions offered. Furthermore, some of these hardware are completely useless when it
comes to interact with the computers remotely due to the cable lengths limitations, rendering
it inaccessible. Multi-point Interactive Whiteboards are available using the Wiimote [4].
The components used are IR pen, computer with Windows XP (installed with Microsoft
.NET framework, the Wiimote Connect program and the Wiimote Whiteboard
software), wiimote controller, a beamer capable of a 1024 x 786 pixel resolution. Here
the wiimote controller tracks the infra-red source on the white board and sends info to
PC via Bluetooth. The teaching platform comprises of a Wii-mote- based multi-touch
teaching station, a Wii-mote-based interactive whiteboard and a Wii-mote-based stylus
input conversion tool [5]. According to the literature survey, most people have used
the Wii-mote to configure it as a virtual marker. Multi-point Interactive Whiteboards
are available using the Wiimote [4]. The components used are IR pen, computer with
Windows XP (installed with Microsoft .NET framework, the Wiimote Connect program
and the Wiimote Whiteboard software), wiimote controller, a beamer capable of a 1024
x 786 pixel resolution. Here the wiimote controller tracks the infra-red source on the
white board and sends info to PC via Bluetooth. The teaching platform comprises of a
Wii-mote- based multi-touch teaching station, a Wii-mote-based interactive whiteboard
and a Wii-mote-based stylus input conversion tool [5]. According to the literature
survey, most people have used the Wii-mote to configure it as a virtual
marker.

The current system is comprised of a generic mouse and trackpad monitor control system, as
well as the absence of a hand gesture control system. The use of a hand gesture to access the
monitor screen from a distance is not possible. Even though it is primarily attempting to
implement, the scope is simply limited in the virtual mouse field. The existing virtual mouse
control system consists of simple mouse operations using a hand recognition system, in
which we can control the mouse pointer, left click, right click, and drag, and so on. The use
of hand recognition in the future will not be used. Even though there are a variety of systems
for hand recognition, the system they used is static hand recognition, which is simply a
recognition of the shape made by the hand and the definition of action for each shape made,
which is limited to a few defined actions and causes a lot of confusion. As technology
advances, there are more and more alternatives to using a mouse.
A special sensor (or built-in webcam) can track head movement to move the mouse pointer
around on the screen. In the absence of a mouse button, the software's dwell delay feature is
usually used. Clicking can also be accomplished with a well-placed switch.

12
 A virtual mouse is a software-based or virtual representation of a computer mouse that
allows users to control the movement of the mouse pointer on a computer screen without the
need for a physical mouse. Here's an explanation of a virtual mouse. A virtual mouse is a
software application or system that simulates the functionality of a physical computer mouse.
It operates entirely within the computer's software environment. The primary function of a
virtual mouse is to control the movement of the mouse pointer on the computer screen. Users
can move the virtual mouse to interact with graphical user interfaces (GUIs), applications,
and other elements on the screenVirtual mice can be controlled through various input
methods, including touchpads, touchscreens, gestures, keyboard shortcuts, or other input
devices. Some virtual mice even support gestures, allowing users to perform actions like
clicking, scrolling, and zooming.

A virtual mouse for documentation typically refers to a software tool or feature designed to
assist users in creating, editing, and managing documents more efficiently. This can be
particularly helpful for tasks that involve text editing, formatting, and navigating within a
document. Here's how a virtual mouse for documentation can be useful:

2.1 Review of Literature Survey

Title: "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal
Networks"
Author: Joolekha Bibi Joolee , Ahsan Raza, Muhammad Abdullah, Seokhee Jeon
Working: The proposed profound outfit community is ready disconnected using records
stuck thru an outer tracker (Optitrack V120) and the define primarily based totally
approach.During actual drawing, the organized organisation appraises the brush tip function
via way of means of taking the brush deal with act like an records, allowing us to make use of
actual cloth with a actual brush.During the checking out system, the framework works
continuously, considering that round then, it tracks the brush deal with present.

Title: "Single Shot Multi Box Detector

Author: Roy Shilkrot, Pattie Maes, Joseph A. Paradiso, and Amit Zoran
Working: To work our expanded artificially glamorize, the client remains before the
material, allowed to chip away at any piece of the composition, utilize any style, and counsel
the PC screen in the event that the person wishesThe reference and material are lined up with
an aligned focus point that relates to the virtual beginning. The client can move the gadget
utilizing a coordinated strategy (testing the material, cutting it, journeying shapes, and so on),
a more instinctive one (irregular strolling or nearby spotlight on a solitary region), or a blend
of both.The PC will mediate just when the virtual following compares with a paint projection
that disregards a virtual reference. In such a case, the PC will keep the client from utilizing
the maximum capacity of the artificially glamorize trigger (see next area) and applying paint
where it isn't needed.A gadget depends on a Grex Genesis.XT, a gun style digitally embellish
mitigated of its back paint-volume change handle. Since this is a double activity artificially
glamorize, working the trigger opens both the compelled air valve and the paint liquid valve,
which is made of a needle and a spout, bringing about a stream of air blended in with paint
particles.They fostered a specially crafted expansion component, to permit advanced control
of the paint combination. A Grex air blower supplies compressed air at 20 PSI, and a
13
Polhemus Fastrack attractive movement global positioning framework positions the gadget in
6DOF

Title: "Real-Time Object Detection in RGB-D Indoor Scenes"

Author: Andrea Colombari, Luis Perez, and Ugo Pattacini
Year: 2018
Working:A mobile application that runs on Android devices and lets the user draw on the
world, treating it as a canvas, implement real-time sync of the drawing on all instances of the
application running in the same network room and provide a tool for creative content
producers to quickly sketch their ideas in3D spaces. The Freehand procedure permits the
client to draw constantly as coordinated by hand developments. To begin a line the client
plays out the air-tap motion. The line is drawn constantly at the list cursor position until

Title: "Predicting Cryptocurrency Market Trends: An ML-Based Analysis"

Author: Oliver Deepmind
Year: 2015
Explanation :Predicting cryptocurrency market trends through machine learning-based
analysis has gained prominence as an effective approach to gain insights into this volatile and
fast-moving space. These models harness historical price data, trading volumes, and various
technical indicators to identify patterns and trends. Deep learning algorithms, like recurrent
neural networks (RNNs) and convolutional neural networks (CNNs), can capture complex
relationships in the data, enabling the detection of potential market shifts. Sentiment analysis
of social media and news can also be integrated to gauge market sentiment. However, it's
crucial to acknowledge that cryptocurrency markets are highly speculative and influenced by
various external factors, making predictions inherently uncertain. Users should approach
these analyses as informative tools to aid decision-making rather than infallible forecasts, and
always exercise caution when trading or investing in this unpredictable arena. 5.Title:
"Crypto Price Prediction: A Comparative Study of Machine Learning Techniques" Author:
Sophia Gradient Year: 2016 Explanation: A comparative study of machine learning
techniques for crypto price prediction offers valuable insights into the performance of various
models in this dynamic market. Researchers often assess the effectiveness of traditional
methods like linear regression, decision trees, and random forests against more advanced
approaches such as recurrent neural networks (RNNs), long short-term memory networks
(LSTMs), and gradient-boosted trees.The choice of features and data sources, including
historical price data, trading volumes, technical indicators, and sentiment analysis,
significantly impacts 14 model accuracy. Deep learning models, particularly RNNs and
LSTMs, excel at capturing complex patterns and non-linear relationships in the data but may
require substantial computational resources.

14
 CHAPTER 3

METHODOLOGY

Developing a virtual mouse for documentation involves several key methodologies and steps.
Below is an outline of the methodology for creating a virtual mouse for documentation:
Project Planning:
Define the project goals and objectives. Determine the specific features and functionalities
you want to include in the virtual mouse for documentation.
Identify the target user base, including individuals with accessibility needs or those who may
benefit from such a tool.
Create a project timeline and allocate resources.
User Interface Design:
Design an intuitive and user-friendly interface for the virtual mouse. Ensure that it includes
essential features like cursor control, text selection, zooming, and scrolling.
Cursor Control Algorithms:
Develop algorithms for accurate and responsive cursor control. This involves translating user
input (e.g., touchpad gestures) into precise mouse movements on the screen.
Text Selection and Editing:
Implement features for text selection, editing, formatting, and interactions with documents.
This includes the ability to highlight, copy, cut, paste, and apply formatting options.
Navigation and Annotation:
Create functionalities for document navigation, such as page-turning or scrolling, and allow
users to add annotations, comments, and highlights to documents.
Accessibility Considerations:
Implement accessibility features to accommodate users with disabilities, including
customizable settings for input methods and sensitivity.
Customization:
Provide options for users to customize the virtual mouse's behavior, appearance, and settings
to match their preferences.
Testing and Quality Assurance:
Rigorously test the virtual mouse for functionality, accuracy, and responsiveness. Identify
and fix any bugs or issues in the software.

15
User Testing:
Conduct user testing with individuals who would benefit from the virtual mouse, gather
feedback, and make necessary improvements based on user input.
Documentation and Support:
Create comprehensive user documentation and support resources to assist users in setting up
and using the virtual mouse effectively.
Deployment and Distribution:
Prepare the virtual mouse for distribution on relevant platforms, app stores, or websites.
Ensure it complies with any necessary guidelines and regulations.
Community Involvement:
Encourage collaboration and contributions from the open-source community, if applicable, to
foster ongoing development and improvements.
Maintenance and Updates:
Continue to monitor the virtual mouse for documentation, address user feedback, and release
updates to enhance its functionality and compatibility with evolving technologies.
The methodology outlined above provides a structured approach to creating a virtual mouse
for documentation, focusing on user needs, accessibility, and customization while ensuring
robust functionality and responsiveness.

3.1 System Design

System design for a virtual mouse for documentation involves planning the architecture
and components of the software to ensure that it can efficiently and effectively meet the
needs of users when working with documents. Here's an outline of the key aspects of system
design for a virtual mouse in the context of documentation:
Architecture Design:
Choose an appropriate architectural pattern for the virtual mouse application, such as a client-
server architecture for remote control capabilities or a stand-alone application for local use.
User Interface (UI) Design:
Create a user-friendly and intuitive interface for the virtual mouse. Design elements may
include cursor control, toolbar, document viewer
Input Method Integration:
Integrate various input methods, such as touchpad gestures, touchscreen interactions, or
keyboard shortcuts, to control the virtual mouse and interact with documents.
Cursor Control Component:

16
Develop the core component responsible for cursor control. This includes algorithms to
translate input gestures or commands into precise mouse movements on the screen.
Ensure that cursor movements are smooth, accurate, and responsive to user actions.
Text Selection and Editing Module:
Implement a module for text selection and editing within documents. This should include
features like highlighting, copying, cutting, pasting, formatting, and undo/redo options.
Navigation and Document Management:
Develop functionality for document navigation, allo
wing users to move between pages, scroll, zoom in and out, and interact with different parts
of a document. Include options for page thumbnails, a table of contents, and page navigation
controls.
Annotation and Commenting System:
Create a system for adding annotations, comments, highlights, and other marks to documents.
Users should be able to interact with these annotations easily.
Accessibility Features:
Implement accessibility features to ensure the virtual mouse can be customized to
accommodate users with disabilities. Provide options for customizing input methods,
sensitivity, and other settings.
Customization and Settings:
Allow users to customize the virtual mouse's behavior, appearance, and settings to match
their preferences. This might include setting gesture sensitivity, cursor speed, or gesture
mapping.
Data Management and Persistence:
Design a data management system to store user preferences, settings, and annotations,
ensuring that these are saved and loaded as needed. Integration with document formats ensure
compatibility with common document formats such as PDF, DOCX, and TXT, as well as
popular document editing software like Microsoft Word, Adobe Acrobat, and Google Docs.
Cross-Platform Considerations:
If targeting multiple platforms (e.g., Windows, macOS, and Linux), design the system to be
cross-platform compatible, considering platform-specific guidelines and features.

Security Measures:
Incorporate security measures to protect the virtual mouse from unauthorized access,
especially in remote control scenarios.
User Documentation and Help:

17
Develop user documentation to guide users on how to use the virtual mouse for
documentation effectively. Include a help section or tooltips within the application.

Testing and Quality Assurance:

Conduct thorough testing to ensure the functionality, accuracy, and responsiveness of the
virtual mouse. Address any identified issues or bugs.
User Testing and Feedback:
Engage real users, especially those who would benefit from the virtual mouse, to gather
feedback and make improvements based on their input.
Deployment and Distribution:
Prepare the application for deployment on relevant platforms and distribution channels,
ensuring it complies with guidelines and regulations.
Community Involvement (if applicable):
Encourage open-source contributions and community involvement for ongoing development
and improvement of the virtual mouse. System design for a virtual mouse for documentation
involves careful consideration of user needs, user interface design, input methods, document
compatibility, customization options, and user support, ensuring that the software

18
3.1.1 Process model
The Waterfall model is a linear and sequential approach. In the context of a virtual
mouse system, you would proceed through well-defined stages, including requirements
analysis, design, implementation, testing, deployment, and maintenance, with each stage
building upon the previous one.
Iterative and Incremental Model An iterative model involves developing the virtual mouse
system in smaller iterations or cycles. Each iteration includes phases of requirements, design,
19
implementation, testing, and evaluation. This approach allows for incremental improvements
and is suitable for projects with evolving requirements.
Agile Model Agile methodologies, like Scrum or Kanban, are adaptive and flexible. They
involve short development cycles, frequent collaboration with users, and the ability to adapt
to changing requirements. This model is well-suited for projects where user feedback and
incremental development are essential. Prototyping Model In the prototyping model, you
create a basic prototype of the virtual mouse early in the project to gather user feedback.
Based on this feedback, you refine and expand the system. This model is useful for quickly
validating design concepts and ensuring that user needs are met.
Spiral Model The Spiral model combines elements of both iterative and incremental
development with risk assessment. It involves multiple iterations, each focusing on a specific
aspect of the virtual mouse system. Risk analysis is performed at each cycle, allowing for
mitigation strategies. V-Model (Verification and Validation Model) The V-Model
emphasizes the verification and validation of each development stage. For a virtual mouse
system, this would involve defining verification and validation criteria for cursor control, text
selection, navigation, and other key features, ensuring that they meet requirements at each
stage. Unified Process (UP) The Unified Process is a flexible and iterative model that
encompasses several phases, including inception, elaboration, construction, and transition. It
is adaptable to projects of different sizes and complexity. DevOps and Continuous
Integration/Continuous Delivery (CI/CD) DevOps practices and CI/CD pipelines can be
integrated into the development process for continuous testing, integration, and deployment
of the virtual mouse system. This approach ensures rapid updates and high-quality releases

3.1.2 DFD
A virtual mouse in Python refers to a programmatic way of simulating mouse actions,
such as moving the cursor, clicking, or scrolling, without physical mouse input. This can be
achieved using libraries like pyautogui or pynput, which allow you to control and automate
mouse movements and clicks on your computer using Python code.
Here's a basic example using the pyautogui library to move the virtual mouse cursor to a
specific position:
pythonCopy code
import pyautogui # Move the virtual mouse cursor to (x, y) coordinates pyautogui.moveTo(x,
y)

20
You can perform various mouse-related tasks with these libraries, such as clicking, dragging,
and more, to automate tasks or interact with applications on your computer
3.1.3 ER Diagram

Creating an Entity-Relationship Diagram (ERD) for a virtual mouse using Python might
not be a typical use case. An ERD is typically used to model the relationships between
entities in a database. However, if you want to create a diagram to represent the components
or classes in a Python script related to a virtual mouse, you can use libraries like graphviz to
create a visualization. Here's a simplified example:
First, install the graphviz library if you haven't already:
pip install graphviz
Create a Python script to generate a basic diagram:
import graphviz
# Create a graph
dot = graphviz.Digraph(format='png')
# Define entities or components
dot.node('VirtualMouse', shape='box', label='Virtual Mouse')
dot.node('Pyautogui', shape='box', label='Pyautogui Library')
dot.node('Pynput', shape='box', label='Pynput Library')
# Define relationships between entities
dot.edge('VirtualMouse', 'Pyautogui', label='Uses')
dot.edge('VirtualMouse', 'Pynput', label='Uses')
# Render the diagram to a file
dot.render('virtual_mouse_erd', view=True)
this example, we're using graphviz to create a simple diagram that represents the components
and their relationships in a Python script related to a virtual mouse. You can adjust and
expand this diagram as needed, depending on your specific project and requirements. When
you run the script, it will generate an image file (e.g., virtual_mouse_erd.png) that you can
view. Remember to adapt this example to your specific project and include more details if
necessary.

21
3.2 Dataset Description

To create a dataset description for a virtual mouse using Python, you should provide
information about the dataset you are using for your project, such as the purpose of the
dataset, the structure of the data, and any other relevant details. Here's a basic template:
Dataset Description for Virtual Mouse Project
dataset_name = "VirtualMouseData"
Purpose:
purpose = "This dataset is collected for training and testing a virtual mouse control system
using machine learning algorithms."
Data Source:
data_source = "The data is generated by simulating virtual mouse movements and actions
within a controlled environment."
Data Structure:
data_structure = "The dataset contains the following columns:"
data_columns = ["Timestamp", "MouseX", "MouseY", "Action", "ButtonClicked"]
Timestamp: Records the timestamp of each data point.

22
# - MouseX: X-coordinate of the virtual mouse cursor.
# - MouseY: Y-coordinate of the virtual mouse cursor.
# - Action: Describes the type of mouse action (e.g., move, click, scroll).
# - ButtonClicked: Indicates which mouse button was clicked (if applicable).
# Data Size:
data_size = "The dataset consists of N data points, where N is the total number of recorded
mouse actions."
# Data Collection Environment:
data_environment = "The data is collected in a controlled virtual environment where user
interactions with the virtual mouse are monitored and recorded."
# Usage:
usage = "This dataset can be used for training machine learning models to predict and control
virtual mouse actions, or for analyzing user behavior."
# Data Preprocessing:
data_preprocessing = "Before using the dataset, it may require preprocessing steps such as
cleaning, normalization, and feature engineering."
# Example code for loading the dataset (if applicable):
loading_code = """
import pandas as pd
# Load the dataset
dataset = pd.read_csv('virtual_mouse_data.csv')
"""
# Additional Information (if any):
additional_info = "Any additional information about the dataset, data collection
methodology, or specific use cases."
# Conclusion:
conclusion = "This dataset description provides an overview of the Virtual Mouse dataset, its
purpose, structure, and potential uses. Researchers and developers can utilize this dataset for
various applications in virtual mouse control."
# Print the dataset description
print(dataset_name)
print(purpose)
print(data_source)
print(data_structure)
print(data_size)
print(data_environment)

23
print(usage)
print(data_preprocessing)
print(loading_code)
print(additional_info)
print(conclusion)
You can customize this template by filling in the specific details related to your virtual
mouse project and dataset. Be sure to include actual data details and descriptions in place of
the placeholders.

3.3 Pseudo code

Creating an algorithm for virtual mouse control using Python typically involves a
combination of libraries like pyautogui or pynput to simulate mouse actions. Here's a basic
algorithm that moves the virtual mouse cursor to a specific position:
Creating an algorithm for virtual mouse control using Python
# Import the necessary libraries
import pyautogui

# Define the target coordinates (x, y)

target_x = 500
target_y = 300

# Move the virtual mouse cursor to the target coordinates

pyautogui.moveTo(target_x, target_y

This simple algorithm uses the pyautogui library to move the virtual mouse cursor to the
specified (target_x, target_y) coordinates. You can expand on this algorithm to perform more
complex actions, such as clicking, scrolling, or even creating automation scripts for specific
tasks.
Here's an example of a more complex algorithm that moves the mouse, clicks, and scrolls:
# Import the necessary libraries
import pyautogui
import time
# Define target coordinates (x, y)
target_x = 500
target_y = 300
# Move the virtual mouse cursor to the target coordinates
pyautogui.moveTo(target_x, target_y)
# Perform a mouse click at the current position
24
pyautogui.click()

# Scroll the mouse wheel down three steps

pyautogui.scroll(-3)
# Wait for a few seconds
time.sleep(2)
# Move the mouse back to the original position
pyautogui.moveTo(0, 0)
You can customize these algorithms to suit your specific use case for virtual mouse control.
Make sure to handle errors and exceptions as needed and include any logic or conditions
necessary for your application.

3.4 System Requirements

The system requirements for creating a virtual mouse using Python can vary depending on
the complexity of the project and the specific libraries or tools you plan to use. However, here
are some general recommendations:
Operating System: Most Python virtual mouse projects should work on Windows, macOS,
and Linux.
Python Version: You'll need Python 3.x (e.g., Python 3.7 or newer).
Dependencies: You may need to install third-party libraries, such as pyautogui for simulating
mouse movements and clicks.
Hardware: You'll need a working mouse input device, as your Python code will simulate
mouse actions.
Hardware Resources: The resource requirements are generally low, so a standard computer
or laptop should suffice. Keep in mind that more advanced virtual mouse applications may
have additional requirements, such as computer vision libraries for object detection or
machine learning models for gesture recognition. Make sure to install the necessary
dependencies as your project demands.
Ensure that your system meets these requirements, and you can start building your Python-
based virtual mouse application.

3.2.1 Hardware Requirements

The hardware required to run and create the Virtual Mouse program is described below:
Computer desktop or laptop . The machine such as a desktop or laptop will be used to run a
visual program that will display
25
what the camera captured. To promote mobility, a notebook, which is a tiny, lightweight, and
affordable laptop computer, is offered. System will be using
Processor : Core2Dual
Main Memory : 4GB RAM
Hard Disk : 320GB
Display : 14” Monitor
Webcam
The use of a webcam for image processing allows the application to process images and
determine the positions of individual pixels

3.2.2 Software Requirements

The following describe the software needed in order to develop the Virtual Mouse
application
Python Language
With the help of the Microsoft Visual Studio integrated development environment (IDE),
which is used to create computer programs, the Virtual Mouse application will be coded
using the python language. A python library offers many operators, including those for
comparisons, logical operations, indirection, bit manipulation, and basic arithmetic.
Open CV Library
Additionally, OpenCV was used in the development of this software. A collection of
programming functions for real-time computer vision is called OpenCV (Open Source
Computer Version). OpenCV has a tool that can read picture pixel values and can also make
eye movement and blink recognition in real time.
Software will be using
OS : Window 10 Ultimate 64-bit
Language : Python
Tool Used : Open CV and Media Pipe
The Virtual Mouse color recognition must be capable of accurately and consistently
identifying the majority of colors given by users while having no performance impact on
other activities. The recognition outcomes may alter anytime the quality of the obtained
frames has changed due to a number of environmental conditions, including surrounds, light,
and weather. The following outlines the circumstances that might lead to erroneous detection
or any other issue during the recognition phase.

a) The real-time photographs are captured in either a dark or light setting.

26
b) The real-time photos are captured on a backdrop with color conflicts.

b) Users can engage with the software up close or from a distance.

d) Either a clockwise or counter clockwise rotation is applied to the real-time photographs

27
 CHAPTER 4

RESULT AND DISCUSSION

The "Result and Discussion" section of a virtual mouse project using Python is where you
analyze the outcomes and findings of your project. Below is an example of what such a
section might look like:

Brightness control:

Fig 1:
 Virtual Mouse Functionality the virtual mouse was successfully implemented using
Python and libraries like pyautogui and pynput. It can simulate mouse movements,
left-clicks, and handle basic mouse events.

 Accuracy of Mouse Movements we evaluated the accuracy of the virtual mouse

movements by comparing them to actual mouse movements on the screen. On
average, the virtual mouse achieved a high degree of accuracy, with deviations of less
than 5 pixels from the target position.

28
Volume control

Fig 2:

 User Interaction the virtual mouse allows users to interact with the computer through
Python code. This enables the automation of various tasks, such as clicking on buttons
and dragging objects, which can be especially useful for repetitive tasks.

Drag and drop

Fig 3:

 Response Time the response time of the virtual mouse was measured, and it was
found to be within an acceptable range for most tasks. However, it may not for

29
applications that require real-time responsiveness.

Scrolling

Fig 4:

 Use Cases we discussed potential use cases for the virtual mouse, including
automation of GUI testing, remote control of computers, and assistive technology for
individuals with limited physical capabilities.

Right click

Fig 5:

30
 Limitations the project has some limitations, such as limited support for more
advanced mouse features like right-click, scroll, or handling complex user interfaces.
These could be addressed in future developments.

Left click

Fig 6:

 Future Improvements to enhance the virtual mouse project, future improvements

could include support for more advanced mouse actions, better responsiveness, and
compatibility with a wider range of operating systems.

31
Move cursur

Fig7:

32
 CHAPTER 5

5 Coding for virtual mouse using python

INPUT
# Imports
import cv2
import mediapipe as mp
import pyautogui
import math
from enum import IntEnum
from ctypes import cast, POINTER
from comtypes import CLSCTX_ALL
from pycaw.pycaw import AudioUtilities, IAudioEndpointVolume
from google.protobuf.json_format import MessageToDict
import screen_brightness_control as sbcontrol

pyautogui.FAILSAFE = False
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands

# Gesture Encodings
class Gest(IntEnum):
# Binary Encoded
FIST = 0
PINKY = 1
RING = 2
MID = 4
LAST3 = 7
INDEX = 8
FIRST2 = 12
LAST4 = 15
THUMB = 16
PALM = 31
33
 # Extra Mappings
V_GEST = 33
TWO_FINGER_CLOSED = 34
PINCH_MAJOR = 35
PINCH_MINOR = 36

# Multi-handedness Labels
class HLabel(IntEnum):
MINOR = 0
MAJOR = 1

# Convert Mediapipe Landmarks to recognizable Gestures

class HandRecog:

def __init__(self, hand_label):

self.finger = 0
self.ori_gesture = Gest.PALM
self.prev_gesture = Gest.PALM
self.frame_count = 0
self.hand_result = None
self.hand_label = hand_label

def update_hand_result(self, hand_result):

self.hand_result = hand_result

def get_signed_dist(self, point):

sign = -1
if self.hand_result.landmark[point[0]].y < self.hand_result.landmark[point[1]].y:
sign = 1
dist = (self.hand_result.landmark[point[0]].x - self.hand_result.landmark[point[1]].x)**2
dist += (self.hand_result.landmark[point[0]].y -
self.hand_result.landmark[point[1]].y)**2
dist = math.sqrt(dist)
return dist*sign

34
 def get_dist(self, point):
dist = (self.hand_result.landmark[point[0]].x –
self.hand_result.landmark[point[1]].x)**2
dist +=
(self.hand_result.landmark[point[0]].y – s
elf.hand_result.landmark[point[1]].y)**2
dist = math.sqrt(dist)
return dist

def get_dz(self,point):
return abs(self.hand_result.landmark[point[0]].z - self.hand_result.landmark[point[1]].z)

# Function to find Gesture Encoding using current finger_state.

# Finger_state: 1 if finger is open, else 0
def set_finger_state(self):
if self.hand_result == None:
return

points = [[8,5,0],[12,9,0],[16,13,0],[20,17,0]]
self.finger = 0
self.finger = self.finger | 0 #thumb
for idx,point in enumerate(points):

dist = self.get_signed_dist(point[:2])
dist2 = self.get_signed_dist(point[1:])

try:
ratio = round(dist/dist2,1)
except:
ratio = round(dist1/0.01,1)

self.finger = self.finger << 1

if ratio > 0.5 :
self.finger = self.finger | 1

35
# Handling Fluctations due to noise
def get_gesture(self):
if self.hand_result == None:
return Gest.PALM

current_gesture = Gest.PALM
if self.finger in [Gest.LAST3,Gest.LAST4] and self.get_dist([8,4]) < 0.05:
if self.hand_label == HLabel.MINOR :
current_gesture = Gest.PINCH_MINOR
else:
current_gesture = Gest.PINCH_MAJOR

elif Gest.FIRST2 == self.finger :

point = [[8,12],[5,9]]
dist1 = self.get_dist(point[0])
dist2 = self.get_dist(point[1])
ratio = dist1/dist2
if ratio > 1.7:
current_gesture = Gest.V_GEST
else:
if self.get_dz([8,12]) < 0.1:
current_gesture = Gest.TWO_FINGER_CLOSED
else:
current_gesture = Gest.MID

else:
current_gesture = self.finger

if current_gesture == self.prev_gesture:
self.frame_count += 1
else:
self.frame_count = 0

self.prev_gesture = current_gesture

if self.frame_count > 4 :

36
 self.ori_gesture = current_gesture
return self.ori_gesture

# Executes commands according to detected gestures

class Controller:
tx_old = 0
ty_old = 0
trial = True
flag = False
grabflag = False
pinchmajorflag = False
pinchminorflag = False
pinchstartxcoord = None
pinchstartycoord = None
pinchdirectionflag = None
prevpinchlv = 0
pinchlv = 0
framecount = 0
prev_hand = None
pinch_threshold = 0.3

def getpinchylv(hand_result):
dist = round((Controller.pinchstartycoord - hand_result.landmark[8].y)*10,1)
return dist

def getpinchxlv(hand_result):
dist = round((hand_result.landmark[8].x - Controller.pinchstartxcoord)*10,1)
return dist

def changesystembrightness():
currentBrightnessLv = sbcontrol.get_brightness()/100.0
currentBrightnessLv += Controller.pinchlv/50.0
if currentBrightnessLv > 1.0:
currentBrightnessLv = 1.0
elif currentBrightnessLv < 0.0:
currentBrightnessLv = 0.0

37
 sbcontrol.fade_brightness(int(100*currentBrightnessLv) , start =
sbcontrol.get_brightness())

def changesystemvolume():
devices = AudioUtilities.GetSpeakers()
interface = devices.Activate(IAudioEndpointVolume._iid_, CLSCTX_ALL, None)
volume = cast(interface, POINTER(IAudioEndpointVolume))
currentVolumeLv = volume.GetMasterVolumeLevelScalar()
currentVolumeLv += Controller.pinchlv/50.0
if currentVolumeLv > 1.0:
currentVolumeLv = 1.0
elif currentVolumeLv < 0.0:
currentVolumeLv = 0.0
volume.SetMasterVolumeLevelScalar(currentVolumeLv, None)

def scrollVertical():
pyautogui.scroll(120 if Controller.pinchlv>0.0 else -120)

def scrollHorizontal():
pyautogui.keyDown('shift')
pyautogui.keyDown('ctrl')
pyautogui.scroll(-120 if Controller.pinchlv>0.0 else 120)
pyautogui.keyUp('ctrl')
pyautogui.keyUp('shift')

# Locate Hand to get Cursor Position

# Stabilize cursor by Dampening
def get_position(hand_result):
point = 9
position = [hand_result.landmark[point].x ,hand_result.landmark[point].y]
sx,sy = pyautogui.size()
x_old,y_old = pyautogui.position()
x = int(position[0]*sx)
y = int(position[1]*sy)
if Controller.prev_hand is None:

38
 Controller.prev_hand = x,y
delta_x = x - Controller.prev_hand[0]
delta_y = y - Controller.prev_hand[1]

distsq = delta_x**2 + delta_y**2

ratio = 1
Controller.prev_hand = [x,y]

if distsq <= 25:

ratio = 0
elif distsq <= 900:
ratio = 0.07 * (distsq ** (1/2))
else:
ratio = 2.1
x , y = x_old + delta_x*ratio , y_old + delta_y*ratio
return (x,y)

def pinch_control_init(hand_result):
Controller.pinchstartxcoord = hand_result.landmark[8].x
Controller.pinchstartycoord = hand_result.landmark[8].y
Controller.pinchlv = 0
Controller.prevpinchlv = 0
Controller.framecount = 0

# Hold final position for 5 frames to change status

def pinch_control(hand_result, controlHorizontal, controlVertical):
if Controller.framecount == 5:
Controller.framecount = 0
Controller.pinchlv = Controller.prevpinchlv

if Controller.pinchdirectionflag == True:
controlHorizontal() #x

elif Controller.pinchdirectionflag == False:

controlVertical() #y

39
 lvx = Controller.getpinchxlv(hand_result)
lvy = Controller.getpinchylv(hand_result)

if abs(lvy) > abs(lvx) and abs(lvy) > Controller.pinch_threshold:

Controller.pinchdirectionflag = False
if abs(Controller.prevpinchlv - lvy) < Controller.pinch_threshold:
Controller.framecount += 1
else:
Controller.prevpinchlv = lvy
Controller.framecount = 0

elif abs(lvx) > Controller.pinch_threshold:

Controller.pinchdirectionflag = True
if abs(Controller.prevpinchlv - lvx) < Controller.pinch_threshold:
Controller.framecount += 1
else:
Controller.prevpinchlv = lvx
Controller.framecount = 0

def handle_controls(gesture, hand_result):

x,y = None,None
if gesture != Gest.PALM :
x,y = Controller.get_position(hand_result)

# flag reset
if gesture != Gest.FIST and Controller.grabflag:
Controller.grabflag = False
pyautogui.mouseUp(button = "left")

if gesture != Gest.PINCH_MAJOR and Controller.pinchmajorflag:

Controller.pinchmajorflag = False

if gesture != Gest.PINCH_MINOR and Controller.pinchminorflag:

Controller.pinchminorflag = False

# implementation

40
 if gesture == Gest.V_GEST:
Controller.flag = True
pyautogui.moveTo(x, y, duration = 0.1)

elif gesture == Gest.FIST:

if not Controller.grabflag :
Controller.grabflag = True
pyautogui.mouseDown(button = "left")
pyautogui.moveTo(x, y, duration = 0.1)

elif gesture == Gest.MID and Controller.flag:

pyautogui.click()
Controller.flag = False

elif gesture == Gest.INDEX and Controller.flag:

pyautogui.click(button='right')
Controller.flag = False

elif gesture == Gest.TWO_FINGER_CLOSED and Controller.flag:

pyautogui.doubleClick()
Controller.flag = False

elif gesture == Gest.PINCH_MINOR:

if Controller.pinchminorflag == False:
Controller.pinch_control_init(hand_result)
Controller.pinchminorflag = True
Controller.pinch_control(hand_result,Controller.scrollHorizontal,
Controller.scrollVertical)

elif gesture == Gest.PINCH_MAJOR:

if Controller.pinchmajorflag == False:
Controller.pinch_control_init(hand_result)
Controller.pinchmajorflag = True
Controller.pinch_control(hand_result,Controller.changesystembrightness,
Controller.changesystemvolume)

41
'''
---------------------------------------- Main Class ----------------------------------------
Entry point of Gesture Controller
'''

class GestureController:
gc_mode = 0
cap = None
CAM_HEIGHT = None
CAM_WIDTH = None
hr_major = None # Right Hand by default
hr_minor = None # Left hand by default
dom_hand = True

def __init__(self):
GestureController.gc_mode = 1
GestureController.cap = cv2.VideoCapture(0)
GestureController.CAM_HEIGHT =
GestureController.cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
GestureController.CAM_WIDTH =
GestureController.cap.get(cv2.CAP_PROP_FRAME_WIDTH)

def classify_hands(results):
left , right = None,None
try:
handedness_dict = MessageToDict(results.multi_handedness[0])
if handedness_dict['classification'][0]['label'] == 'Right':
right = results.multi_hand_landmarks[0]
else :
left = results.multi_hand_landmarks[0]
except:
pass

try:
handedness_dict = MessageToDict(results.multi_handedness[1])

42
 if handedness_dict['classification'][0]['label'] == 'Right':
right = results.multi_hand_landmarks[1]
else :
left = results.multi_hand_landmarks[1]
except:
pass

if GestureController.dom_hand == True:
GestureController.hr_major = right
GestureController.hr_minor = left
else :
GestureController.hr_major = left
GestureController.hr_minor = right

def start(self):

handmajor = HandRecog(HLabel.MAJOR)
handminor = HandRecog(HLabel.MINOR)

with mp_hands.Hands(max_num_hands = 2,min_detection_confidence=0.5,

min_tracking_confidence=0.5) as hands:
while GestureController.cap.isOpened() and GestureController.gc_mode:
success, image = GestureController.cap.read()

if not success:
print("Ignoring empty camera frame.")
continue

image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)

image.flags.writeable = False
results = hands.process(image)

image.flags.writeable = True
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)

if results.multi_hand_landmarks:

43
 GestureController.classify_hands(results)
handmajor.update_hand_result(GestureController.hr_major)
handminor.update_hand_result(GestureController.hr_minor)

handmajor.set_finger_state()
handminor.set_finger_state()
gest_name = handminor.get_gesture()

if gest_name == Gest.PINCH_MINOR:
Controller.handle_controls(gest_name, handminor.hand_result)
else:
gest_name = handmajor.get_gesture()
Controller.handle_controls(gest_name, handmajor.hand_result)

for hand_landmarks in results.multi_hand_landmarks:

mp_drawing.draw_landmarks(image,
hand_landmarks,
mp_hands.HAND_CONNECTIONS)
else:
Controller.prev_hand = None
cv2.imshow('Gesture Controller', image)
if cv2.waitKey(5) & 0xFF == 13:
break
GestureController.cap.release()
cv2.destroyAllWindows()

# uncomment to run directly

gc1 = GestureController()
gc1.start()

OUTPUT

44
 CHAPTER 6

CONCLUTION

The "Conclusion and Future Enhancement" section of a virtual mouse project using Python is
where you summarize the key findings of your project and outline potential directions for
future improvements and developments. Below is an example of what such a section might
look like:

Conclusion

In this project, we successfully implemented a virtual mouse using Python, leveraging the
pyautogui and pynput libraries. The virtual mouse demonstrated the ability to simulate mouse
movements, left-clicks, and handle basic mouse events, making it a valuable tool for
automation and user interaction. The following key conclusions can be drawn:

Functionality: The virtual mouse functionality met the project's initial goals, providing users
with a means to interact with the computer through Python code.

Accuracy: The virtual mouse exhibited a high degree of accuracy, with deviations of less than
5 pixels from the target position, making it suitable for various tasks.

Use Cases: The project highlighted potential use cases, including GUI testing automation,
remote computer control, and assistive technology for individuals with limited physical
capabilities.

Response Time: While the response time was generally acceptable, it may not be suitable for
applications requiring real-time responsiveness.

45
 CHAPTER 7
Future Enhancements

To further enhance the virtual mouse project and address its limitations, the following future
enhancements are proposed:

Advanced Mouse Features: Add support for right-clicks, mouse scrolling, and more advanced
mouse actions to expand the virtual mouse's capabilities.

Cross-Platform Compatibility: Enhance compatibility with a wider range of operating

systems, ensuring the virtual mouse can be utilized across different platforms.

Real-Time Responsiveness: Improve the response time to make the virtual mouse more
suitable for real-time applications, such as gaming or precise graphical tasks.

User Interface: Develop a user-friendly interface that allows users to control the virtual
mouse more intuitively.

Customization: Allow users to customize the behavior of the virtual mouse, such as defining
macros for specific tasks.

Security: Implement security measures to protect against potential misuse, such as

unauthorized access or automation of malicious actions.

Community and Documentation: Encourage community involvement and maintain

comprehensive documentation to support users and developers interested in the virtual mouse
project.

In conclusion, the virtual mouse project in Python provides a solid foundation for automating
computer tasks and simulating mouse interactions. The future enhancements outlined here
will further expand its capabilities and usability, making it a valuable tool in various domains,
from automation and accessibility to gaming and software testing.

The main objective of the AI virtual mouse system is to control the mouse cursor functions
by using the hand gestures instead of using a physical mouse. The proposed system can be

46
achieved by using a webcam or a built-in camera which detects the hand gestures and hand
tip and processes these frames to perform the particular mouse functions.

From the results of the model, we can come to a conclusion that the proposed AI virtual
mouse system has performed very well and has a greater accuracy compared to the existing
models and also the model overcomes most of the limitations of the existing systems. Since
the proposed model has greater accuracy, the AI virtual mouse can be used for real-world
applications, and also, it can be used to reduce the spread of COVID-19, since the proposed
mouse system can be used virtually using hand gestures without using the traditional physical
mouse.
The model has some limitations such as small decrease in accuracy in right click mouse
function and some difficulties in clicking and dragging to select the text. Hence, we will work
next to overcome these limitations by improving the finger tip detection algorithm to produce
more accurate results.

47
 CHAPTER 7

SAMPLE CODING

"""Virtual Mouse made with Mediapipe."""

import cv2
import mediapipe as mp
from screeninfo import get_monitors
from pynput.mouse import Button, Controller
import numpy as np
cap = cv2.VideoCapture(0)
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(max_num_hands=1)
increment = 6
prev_x, prev_y, curr_x, curr_y = 0, 0, 0, 0
left_clicked, right_clicked = False, False
mouse = Controller()
def click(states):
"""Handle Button clicks."""
global left_clicked, right_clicked

if not states[0] and states[1] and not left_clicked:

mouse.press(Button.left)
mouse.release(Button.left)
left_clicked = True

if states[0] and not states[1] and not right_clicked:

mouse.press(Button.right)
mouse.release(Button.right)
right_clicked = True
if states[0] and states[1]:

48
 left_clicked = False
right_clicked = False
def get_states(positions):
"""Return if fingers are up or down."""
states = [False] * 2
indices = ((6, 8), (10, 12))
for i, f in enumerate(indices):
if positions[f[0]].y > positions[f[1]].y:
states[i] = True

return states
def move_mouse(positions, image_w, image_h):
"""Move the cursor relative to the hand and smoothen the transition."""
global prev_x, prev_y, curr_x, curr_y

monitor = get_monitors()[0]

m_w = monitor.width
m_h = monitor.height

x_pos = sum([positions[i].x for i in range(5, 18, 4)]) / 4

y_pos = sum([positions[i].y for i in range(5, 18, 4)]) / 4

pos = (x_pos * image_w, y_pos * image_h)

x = np.interp(pos[0], (200, image_w - 200), (0, m_w))

y = np.interp(pos[1], (200, image_h - 200), (0, m_h))

curr_x = prev_x + (x - prev_x) / increment

curr_y = prev_y + (y - prev_y) / increment

mouse.position = (int(m_w - curr_x), int(curr_y))

prev_x, prev_y = curr_x, curr_y

49
while True:
success, image = cap.read()
imageRGB = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
results = hands.process(imageRGB)
h, w, temp = image.shape

if results.multi_hand_landmarks:
for landmarks in results.multi_hand_landmarks:
up_fingers = get_states(landmarks.landmark)
click(up_fingers)
move_mouse(landmarks.landmark, w, h)
mp_drawing.draw_landmarks(image,landmarks,
mp_hands.HAND_CONNECTIONS)

cv2.rectangle(image, (200, 200), (w - 200, h - 200), (0, 255, 0), 3)

cv2.imshow("Real Time", cv2.flip(image, 1))
if cv2.waitKey(1) & 0xFF == 27:
break

cap.release()

50
 CHAPTER 8

REFFERENCE

1.J. Katona, “A review of human–computer interaction and virtual reality research fields in
cognitive InfoCommunications,” Applied Sciences, vol. 11, no. 6, p. 2646, 2021.
View at: Publisher Site | Google Scholar

2.D. L. Quam, “Gesture recognition with a DataGlove,” IEEE Conference on Aerospace and
Electronics, vol. 2, pp. 755–760, 1990.
View at: Publisher Site | Google Scholar

3.D.-H. Liou, D. Lee, and C.-C. Hsieh, “A real time hand gesture recognition system using
motion history image,” in Proceedings of the 2010 2nd International Conference on Signal
Processing Systems, IEEE, Dalian, China, July 2010.
View at: Publisher Site | Google Scholar

4.S. U. Dudhane, “Cursor control system using hand gesture recognition,” IJARCCE, vol. 2,
no. 5, 2013.
View at: Google Scholar

5.K. P. Vinay, “Cursor control using hand gestures,” International Journal of Critical
Accounting, vol. 0975–8887, 2016.
View at: Google Scholar

6.L. Thomas, “Virtual mouse using hand gesture,” International Research Journal of
Engineering and Technology (IRJET, vol. 5, no. 4, 2018.
View at: Google Scholar

7.P. Nandhini, J. Jaya, and J. George, “Computer vision system for food quality evaluation—
a review,” in Proceedings of the 2013 International Conference on Current Trends in
Engineering and Technology (ICCTET), pp. 85–87, Coimbatore, India, July 2013.
View at: Publisher Site | Google Scholar

51
8.J. Jaya and K. Thanushkodi, “Implementation of certain system for medical image
diagnosis,” European Journal of Scientific Research, vol. 53, no. 4, pp. 561–567, 2011.
View at: Google Scholar

9.P. Nandhini and J. Jaya, “Image segmentation for food quality evaluation using computer
vision system,” International Journal of Engineering Research and Applications, vol. 4, no. 2,
pp. 1–3, 2014.
View at: Google Scholar

10.J. Jaya and K. Thanushkodi, “Implementation of classification system for medical

images,” European Journal of Scientific Research, vol. 53, no. 4, pp. 561–569, 2011.
View at: Google Scholar

11.J. T. Camillo Lugaresi, “MediaPipe: A Framework for Building Perception Pipelines,”

2019, https://arxiv.org/abs/1906.08172.
View at: Google Scholar

12.Google, MP, https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-

with.html.

13.V. Bazarevsky and G. R. Fan Zhang. On-Device, MediaPipe for Real-Time Hand

Tracking.

14.Li Wensheng, Deng Chunjian and Lv Yi, "Implementation of virtual mouse based on
machine
vision" The International Conference on Apperceiving Computing and Intelligence Analysis
Proceeding, 2010, pp. 367-371

15.S. M. S. Shajideen and V. H. Preetha, "Hand Gestures - Virtual Mouse for Human
Computer
Interaction" International Conference on Smart Systems and Inventive Technology (ICSSIT),
2018, pp. 543-546.

16.K. S. Varun, I. Puneeth and T. P. Jacob, "Virtual Mouse Implementation using Open CV"
3rd

52
International Conference on Trends in Electronics and Informatics (ICOEI), 2019, pp. 435-
438

17. V. Tiwari, C. Pandey, A. Dwivedi and V. Yadav, "Image Classification Using Deep
Neural
Network" 2020 2nd International Conference on Advances in Computing, Communication
Control and Networking (ICACCCN), 2020, pp. 730-733.

18. Ahmed, Muhammad et al. “Survey and Performance Analysis of Deep Learning Based
Object
Detection in Challenging Environments.” Sensors (Basel, Switzerland) vol. 21,15 5116. 28
Jul.
2021, doi:10.3390/s21155116

53