Virtual Marker Using Python

A Mini Project Report Submitted in partial fulfillment of the requirement for the award of
the degree of

BACHELOR OF TECHNOLOGY
In
ELECTRONICS & COMMUNICATIONS ENGINEERING
By
SOMAROUTHU SAI VENKATA MANI KRISHNA 20B95A0420

TENTU ANU SARNYA 19B91A04M3

DONGA SATYA PAVAN KARTHIK 19B91A04K3

SANDRALA MINISHA 19B91A04J9

Under the esteemed guidance of

K. N. V. SATYA NARAYANA
M.tech,

DEPARTMENT OF ELECTRONICS AND COMMUNICATIONS ENGINEERING

S.R.K.R. ENGINEERING COLLEGE(AUTONOMOUS)
(Affiliated to JNTU, KAKINADA)
(Recognized by A.I.C.T.E., Accredited by N.B.A., & Accredited by N.A.A.C. with ‘A’ Grade, New Delhi)

CHINNAMIRAM,
BHIMAVARAM-534204
(2019-2023)
 S.R.K.R. ENGINEERING COLLEGE(AUTONOMOUS)
(Affiliated JNTU, KAKINADA)

(Recognized by A.I.C.T.E. Accredited by N.B.A., Accredited by NAAC. with ‘A’ Grade NEW DELHI)

CHINNAMIRAM, BHIMAVARAM - 534204

DEPARTMENT OF ELECTRONICS AND COMMUNICATIONS ENGINEERING

CERTIFICATE
This is to certify that the project work entitled
VIRTUAL MARKER USING PYTHON
Is the bonafide work of

Mr/Miss SOMAROUTHU SAI VENKATA MANI KRISHNA (20B95A0420), TENTU ANU

SARANYA(19B91A04M3), DONGA SATYA PAVAN KARTHIK(19B91A04K3),

SANDRALA MINISHA (19B91A04J9)submitted the partial fulfillment of the requirement for

the award of the degree of BACHELOR OF TECHNOLOGY in ELECTRONICS AND

COMMUNICATIONS ENGINEERING during the year 2019-2023.

Guide: Head Of the Department:

K. N. V. SATYA NARAYANA Dr. N. UDAYA KUMAR
M.Tech, M.Tech,Ph.D,M.I.S.T.E,S.M.I.E.E.E,F.I.E.T.E,F.I.
Asst. Professor, Department of E.C.E Professor and Head Of the E.C.E
Department
 ACKNOWLEDGMENTS

Our most sincere and grateful acknowledgment to our alma mater SAGI RAMA
KRISHNAM RAJU ENGINEERING COLLEGE for allowing us to fulfill our aspirations
and for the successful completion of the project.
We are grateful to our principal Dr. M. JAGAPATHI RAJU, for providing us
with the necessary facilities to carry out our project.
We convey our sincere thanks to Dr. N. UDAYA KUMAR, Head of the
Department of Electronics and Communication Engineering, for his kind cooperation in the
successful completion of the project work.
We express our sincere thanks to our esteemed guide K.N.V.SATYA
NARAYANA, Asst. Professor, Department of Electronics and Communication Engineering,
for giving valuable and timely suggestions for the project work, constant encouragement, and
support in times of trouble throughout the project work.
We extend our sense of gratitude to all our teaching and non-teaching staff and all
our friends, who indirectly helped us in this endeavor.
-Project Associates

SOMAROUTHU.SAI VENKATA MANI KRISHNA (20B95A0420)

TENTU.ANU SARANYA (19B91A04M3)

DONGA.SATYA PAVAN KARTHIK (19B91A04K3)

SANDRALA. MINISHA (19B91A04J9)

iii
 LIST OF FIGURES

Figure Name Page No

Figure 1 Workflow of the system 4
Figure 2 Proposed Architecture 4
Figure 3 Image Extraction 5
Figure 4 Background Abstraction 5
Figure 5 Edge enhancement 5
Figure 6 X,Y Coordinates 6
Figure 7 Shape of the resultant image 7
Figure 8 Red color image 9
Figure 9 Edge enhancement 9
Figure 10 Bounding box with coordinates 10
Figure 11 resultant marking of writing 10

iv
 LIST OF TABLES

TABLE NO PAGE NO

Table 1 Alphabets accuracy 7

v
 Abstract
Writing in air has been one of the most fascinating and challenging research areas in
field of image processing and pattern recognition in the recent years. It contributes
immensely to the advancement of an automation process and can improve the interface
between man and machine in numerous applications. Several research works have been
focusing on new techniques and methods that would reduce the processing time while
providing higher recognition accuracy.
Object tracking is considered as an important task within the field of Computer
Vision. The invention of faster computers, availability of inexpensive and good quality video
cameras and demands of automated video analysis has given popularity to object tracking
techniques. Generally, video analysis procedure has three majors
steps: firstly, detecting of the object, secondly tracking its movement from frame to frame
and lastly analyzing the behavior of that object. For object tracking, four different issues are
taken into account; selection of suitable object representation, feature selection for tracking,
object detection and object tracking. In real world, Object tracking algorithms are the
primarily part of different applications such as: automatic surveillance, video indexing and
vehicle navigation etc.
The project takes advantage of this gap and focuses on developing a motion-to-text
converter that can potentially serve as software for intelligent wearable devices for writing
from the air. This project is a reporter of occasional gestures. It will use computer vision to
trace the path of the finger. The generated text can also be used for various purposes, such as
sending messages, emails, etc. It will be a powerful means of communication for the deaf. It
is an effective communication method that reduces mobile and laptop usage by eliminating
the need to write.

Keywords - Air Writing, Character Recognition, Object Detection, Real-Time Gesture

Control System, Smart Wearables, Computer Vision.

vi
 CONTENTS
TOPIC NAME PAGE
Title page i
Certificate ii
Acknowledgement iii
List of Figures iv
List of Tables v
Abstract vi
Contents vii
1. INTRODUCTION 1-2
2. METHODOLOGY 3-7
2.1 Object Localization 5-6
2.2 Plot (x,y) coordinates 6
2.3 Blynk application and Arduino IDE Preparation and Running 6-7

3. RESULTS AND DISCUSSION 8-10

4. CONCLUSION 11-12
4.1 Conclusion 12
REFERENCES 13
APPENDEX 14

vii
 CHAPTER I
INTRODUCTION

1
1. INTRODUCTION
In the era of digital world, traditional art of writing is being replaced by digital art. Digital art
refers to forms of expression and transmission of art form with digital form. Relying on
modern science and technology is the distinctive characteristics of the digital manifestation.
Traditional art refers to the art form which is created before the digital art. From the recipient
to analyses, it can simply be divided into visual art, audio art, audio-visual art and audio-
visual imaginary art, which includes literature, painting, sculpture, architecture, music, dance,
drama and other works of art. Digital art and traditional art are interrelated and
interdependent. Social development is not a people’s will, but the needs of human life are the
main driving force anyway. The same situation happens in art. In the present circumstances,
digital art and traditional art are inclusive of the symbiotic state, so we need to systematically
understand the basic knowledge of the form between digital art and traditional art. The
traditional way includes pen and paper, chalk and board method of writing. The essential aim
of digital art is of building hand gesture recognition system to write digitally. Digital art
includes many ways of writing like by using keyboard, touch- screen surface, digital pen,
stylus, using electronic hand gloves, etc. But in this system, we are using hand gesture
recognition with the use of machine learning algorithm by using python programming, which
creates natural interaction between man and machine. With the advancement in technology,
the need of development of natural ‘human – computer interaction (HCI) systems to replace
traditional systems is increasing rapidly.

2
CHAPTER 2
METHODOLOGY

3
2. METHODOLOGY
This system needs a dataset for the Fingertip Detection Model. The Fingertip Model's
primary purpose is used to record the motion, i.e., the air character.

Figure 1. workflow of the System

Air writing can be merely achieved using a stylus or air-pens that have a unique colour . The
system, though, makes use of fingertip. We believe people should be able to write in the air
without the pain of carrying a stylus. We have used Deep Learning algorithms to detect
fingertip in every frame, generating a list of coordinates.

Figure 2. Proposed Architecture

4
 2.1 Object localization
Extracted image from video sequence is shown in figure 3. After the extraction the object is
localized by the following:

Figure 3. Image extraction

Extract color image from reference: In this proposed method we are basically tracking
the motion of the index finger which is colored red; we do not have a reference image so
every previous image is the reference to next image. Now take the difference of the images
and extract the color and object movement. Figure 4 shows the background abstracted image.

Figure 4. Background abstraction

Edge enhancement (EE): The edge enhancement technique makes the object localization
algorithm robust to noise, varying lighting conditions, obscuration and object fading even in
the low-contrast imagery. Figure 5 shows the edge enhancement image. The edge-
enhancement process consists of four operations:

Figure 5. Edge enhancement

5
Normalization:
We have found out in our experiments that the dynamic range of the edge image, E, is often
too narrow towards darker side as compared to the available pixel-value range [0, 255],
especially in low contrast imagery. Conventionally, the edge image is converted into a
binary image using a predefined threshold; however, this approach does not work well in a
template matching application, because the rich content of the gray-level edge-features of
the object is lost in the process of binarization

2.2 Plot X, Y coordinates

After extracting the properties of object from still video frame, we track the object
throughout the video frame and get its x, y coordinates as shown in figure 6. After extraction
of x, y coordinates we represent these coordinates in form of image which is further take in
to consideration. Here, we get the mirror images, so, we just shuffle the x coordinates rather
than inverting the image while y coordinates remain same Figure 7 shows the plotted
image.

Fig. 6: (X, Y) (207,186) coordinates

2.3 OCR Analysis

OCR which is abbreviation of Optical Character Recognition is a designed tool in image
processing used to read and recognize the character by reading an image. The purpose of
using OCR in this is, to read the resultant character after tracking the object. The OCR in
this work is defines as follows:

6
Load Template
A template is basically a data set used to compare the resultant image and find the relation
with a text. The data set basically contains the test images that can possible be drawn while
drawing a text in air.

Convert test image in binary

To process an image, it is necessary to convert an image in to binary because a binary
image contains the value of pixel in 0 or 1 which are easy to deal with so. After this we
label, labeled image correspond to different regions. The set of elements of labeled equal to
1 corresponds to region 1; Elements Labeled equal to 2 corresponds to region 2 and so on.

Find connected line segment and Correlation

The purpose of finding connected line segment is to read the shape of resultant image.
Figure 7 presents the resultant image. This is done by reading labeled image line wise and
store the result in an array. The test image which is shown below in figure is show the
working of connected line segment.

Fig. 7: Read the shape of resultant image

1 A 96 14 N 94
2 B 90 15 O 83
3 C 89 16 P 86
4 D 81 17 Q 85
5 E 98 18 R 90
6 F 100 19 S 96
7 G 88 20 T 98
8 H 98 21 U 83
9 I 100 22 V 81
10 J 100 23 W 96
11 K 92 24 X 92
12 L 93 25 Y 94
13 M 96 26 Z 98
Table 1:Alphabets accuracy in percentage

7
 CHAPTER 3
RESULTS AND DISCUSSIONS

8
 3. RESULTS AND DISCUSSION
3.1 Result

For experimentation, we use laptop having windows Operating System. We use its webcam
for capturing video and images. We text out proposed method for both indoor and outdoor
locations and found that it works well.

The system proposed in this method is track the colored finger in a video file by capturing
video from any video capturing device. Then it read video file and process video frame wise
by reading images; after reading frame wise images from video file, the designed algorithm
extracts the red color from frames.

Fig. 8: Red color image

Then apply edge enhancement to detect the edge of colored finger the enhanced edge is
shown in figure 9.

Fig. 9: edge enhancement

9
After edge enhancement the designed system extract the (X, Y) coordinates of colored
image by applying bounding box on the extract image, the bounding box on colored image
is shown in figure 10.

Fig. 10: Bounding Box with X, Y coordinates

Then this image is given to optical character recognition (OCR) module for recognition.
After loading complete template for alphabetical and numeric characters, it is compared
with the resultant image to find the maximum comparison and result is displayed in a text
file showing resultant character as shown in figure 11.

Fig. 11: Resultant marking of writing using virtual marker

10
CHAPTER 4
CONCLUSION

11
 4. CONCLUSION
4.1 CONCLUSION:
This paper presents a video based pointing method which allows writing of English text in air
using webcam. Proposed method has two main tasks: first it tracks the colored fingertip in the
video frames and then apply English OCR over plotted images in order to recognize the
written characters. Moreover, proposed method provides a natural human-system interaction
in such way that it does not require Keypad, Pen or Glove etc. for character input. It just
requires a mobile camera and red color for reorganization a fingertip. For the experiments, we
have developed an application using OpenCV with python language. The proposed method
gains the average accuracy of 92.083% in order to recognize the accurate alphabets. The
overall writing delay gained by the proposed method was 50 ms per character. Moreover,
proposed methodology can be applicable for all disconnected languages but having one
serious issue that it is color sensitive in such a way that existence of any red color in the
background before starting the analysis can lead to false results.

12
 5. REFERENCES

[1.]Alper Yilmaz, Omar Javed, Mubarak Shah, “Object Tracking: A Survey”, ACM
Computer Survey. Vol. 38, Issue. 4, Article 13, Pp. 1-45, 2006

[2.]Erik B. Sudderth, Michael I. Mandel, William T. Freeman, Alan S. Willsky, “Visual

Hand Tracking Using Nonparametric Belief Propagation”, Mit Laboratory For Information
& Decision Systems Technical Report P-2603, Presented at IEEE CVPR Workshop On
Generative Model Based Vision, Pp. 1-9, 2004
[3.]Robert Y. Wang, Jovan Popovi´c, “Real-Time Hand-Tracking with a Color Glove”,
2008 [4.]T. A. C. Bragatto, G. I. S. Ruas, M. V. Lamar, “Real-time Video Based Finger
Spelling Recognition System Using Low Computational Complexity Artificial Neural
Networks”, IEEE ITS, pp. 393-397, 2006
[5.]Yusuke Araga, Makoto Shirabayashi, Keishi Kaida, Hiroomi Hikawa, “Real Time
Gesture Recognition System Using Posture Classifier and Jordan Recurrent Neural
Network”, IEEE World Congress on Computational Intelligence, Brisbane, Australia, 2012
[6.]Ruiduo Yang, Sudeep Sarkar, “Coupled grouping and matching for sign and gesture
recognition”, Computer Vision and Image Understanding, Elsevier, 2008

[7.]http://learnrnd.com/news.php?id=How_Printing_Batteries_Made,_Its_Structure_Usage
_and_Futur e_Applications
[8.]H.M. Cooper, “Sign Language Recognition: Generalising to More Complex Corpora”,
PhD Thesis, Centre for Vision, Speech and Signal Processing Faculty of Engineering and
Physical Sciences, University of Surrey, UK, 2012
[9.] Robert Wang, Sylvain Paris, Jovan Popovi´c, “Practical Color-Based Motion
Capture”, Eurographics/ ACM SIGGRAPH Symposium on Computer Animation, 2011
[10.] Kim, Sung Deuk, Jaeyoun Yi, Hyun Mun Kim, and Jong Beom Ra. "A deblocking
filter with two separate modes in block-based video coding." Circuits and Systems for
Video Technology, IEEE Transactions on 9, no. 1 (1999): 156-160

13
 APPENDEX

CODE - https://github.com/rohit13000/miniproject.git

14