459-Article Text-2622-1-10-20230409
459-Article Text-2622-1-10-20230409
459-Article Text-2622-1-10-20230409
2 (2023)
Abstract— The technique of human interaction with computers has developed very rapidly. Body movement is the easiest and
most expressive way and hand movements are flexible. We can use gestures as a simple identification command. This research
discusses the application of hand gesture reading as a remote control for learning media and analyzes its quality of service
using Wireshark software. This system uses a Raspberry Pi as a computing center. Raspberry Pi reads hand gestures using
the Convex Hull algorithm, which can read the number of fingers raised on the hand by taking the outermost point in the
contour scan of the hand. Each gesture in the form of the number of fingers was allocated to the keyboard commands used to
select answers on the learning media. On the learning media side, a quiz system was created that uses keyboard commands to
select the answers. Then the Raspberry Pi is connected to the laptop using a third-party application called VNC. Based on
QoS measurement results, the throughput result is 62 kbps, which is included in the very good category. Packet Loss of 0%,
which is also included in the very good category. The delay of 52.2 ms, which is included in the very good category. Thus, the
overall quality of the network can be categorized as very good.
Keywords— Convex Hull, Hand Gesture Detection, Learning Media, OpenCV, Quality of Service.
I. INTRODUCTION the mouse pointer was smoothed using the Kalman filter method
and hand movements were identified using the color detection
The interaction between people and computers is always method [2].
changing as technology advances. At first it still used a keyboard An android-based system for hand gesture recognition was
and mouse, but now it can already use touch screen technology. developed by M. Anshary. To recognize hand motions, the
Now that interactions are more natural and users may system use the convex hull method and convexity defects. The
communicate with one another through body language, speech, background color and light intensity in this system are adjustable
facial expressions, and eye contact, the next revolution is in order to determine the background color and light intensity that
beginning to take shape. work best for identifying hands [3].
The computer is able to easily understand the user's A. R. Adnan used PoseNet and the K Nearest Neighbor
intentions with the support of advanced sensors and algorithms. algorithm to construct a system for identifying human arm
This may increase the system's activity and interaction. Users and movements. This motion identification is utilized to regulate the
computers still interact using mouse in their communication. flame of two switches connected over the IoT network [4].
Given that it is simpler to recognize hand gestures, hand gesture Another example is the Convolutional Neural Network (CNN)-
recognition can be utilized to facilitate interaction rather than based hand gesture detection system developed by A. Adi. This
using a mouse. system is implemented in wheeled robots, where there are 12
There are various situations that occur during lectures that hand gestures to control the 12 maneuvers of wheeled robots [5].
require lecturers to stay in the lecturer's room while teaching A finger tracking device was also developed by Alan Tompunu
students in class. Lecturers can present material via Google Meet using a Raspberry Pi 3. OpenCV and Python were used in this
or Zoom and quizzes may be given by lecturers at certain points. research. An accuracy rate of 75% and an average video
There are a number of mobile applications that can be used as the processing time of 9.2 seconds were the result of this study [6].
quiz system itself, however the system necessitates that students In this project, a hand gesture identification system was
bring their own smartphones, and sometimes there are devices developed for learning media in the form of a quiz program.
that are incompatible with the program. Another issue is the Convex hull [7] and convexity defect methods [8] are used in this
inconsistent and sometimes missing internet connectivity identification system together with the Python [9] programming
between students. language and OpenCV plugins [10].
A presentation control system was developed by M.H. For accuracy smoothing and enhancement, the system
Khoirul using hand gesture recognition. The Support Vector employs an external webcam camera. Hand gestures taken by the
Machine Classification technique is used by the system to classify webcam camera will be converted into RGB color [11] types so
skin tones according to background color. The camera for this that they can be processed by OpenCV. After that, the
experiment was a Logitech C270 webcam, and the data background color changes to green. After that, the background
processing system was an Intel NUC5i7RYH. There are only five color changes to green. From these results, the algorithms for
keyboard commands available for the system [1]. convex hull and convexity defect are applied. By seeking out the
A hand gesture identification system that D.R.M. Harika fingertip point with the least amount of skin tone, the algorithm
developed can be used to control the mouse pointer. In this work,
* Coresponding author
E-ISSN: 2654-6531 P- ISSN: 2407-0807 103
Journal of Telecommunication Network (Jurnal Jaringan Telekomunikasi) Vol. 13, No.2 (2023)
locates the fingertips' points. Students' answers to the questions right area, then the presentation will proceed to the next question,
displayed are identified by the hand movements they make. while if there is one finger detected in the lower right area, then
The factors examined in this study were website system the presentation will proceed to the previous question. Fig. 2 and
success, accuracy, hand gesture recognition process time Fig. 3 shows how the system work.
[12][13], and Quality of Service (QoS) [14]. The performance of
transmission system could be seen by Quality of Service
measurement, including delay, throughput, packet loss and jitter
[14][15]. This measurement utilizes a wireshark software.
II. METHOD
A. Block Diagram System
Fig. 1 depicts a system overview that describes the hardware
process of the system.
Webcam
Raspberry Presentation /
Logitech Laptop
Pi 3B Test Application
C270
Figure 3. Webcam capture display design
Room 1 Proyektor Room 2
C. Flowchart Convex Hull Algorithm
The convex hull of any point P is denoted by the notation
CH(P). In simple terms, we can suppose that point P is a
collection of nails stuck in a tree or wall and the convex hull is
Router Wifi the smallest area like rubber made in a circle where every point P
Figure 1. Block Diagram System is inside the rubber area. Convex Hull is formed from lines
Hand gestures are performed by users who are in Room 1 connecting the points P which are in the outermost area. In the
and the webcam captures the movements and hand gestures implementation, the convex hull has several methods that can be
demonstrated by the user. The program is a hand gesture used, namely Jarvis Wrap. In simple terms the Jarvis Wrap
detection system and is created through the Thonny IDE. In the method can be explained in the following steps:
program, there is a convex hull algorithm, and this program is • Take one of the points that is located on the outermost, for
tasked with computing the input video data into a collection of
example, which has the highest or lowest x coordinate, and
dots that form the hand. From these dots, the gesture and the
number of fingers of the hand that will be used for the output it could also be the one with the highest or lowest y-
process will be determined. The output of the number of fingers coordinate.
indicates the answer choice chosen by the user. The Raspberry Pi • Find the nearest line that makes the largest angle when
is a platform where video input data is converted into output data drawn by the line from the first point
in the form of answer choices chosen by the user. • The second step is done continuously until it returns to the
B. System Planning starting point.
To accomplish the convex hull algorithm, theory convexity
defects is needed. The use of the convex hull approach is thought
to be less exact since one finger might occasionally be identified
at both ends. To identify it accurately, we need an assistance
algorithm. As a result, the convexity defect method is employed
in this work. The convex hull technique is utilized to detect
knuckles, which makes determining the number of elevated
fingers more particular and exact. These convexity defects are the
deepest concavities or deviations between two points convex hull.
This theory calculates the angle generated between 2 fingers.
Angles are calculated using the cosine Equation 1.
𝑐 = √𝑎2 + 𝑏 2 − 2𝑎𝑏. cos(𝛼) (1)
where a is length of finger 1, b is length of finger 2, c is distance
between finger 1 and finger 2, d is the angle of finger 1 and finger
Figure 2. User's position from behind 2. Thus, to calculate 𝛼 use the following Equation 2.
𝑎2 +𝑏 2 −𝑐 2 180
In order to ensure that the convex hull only selected up the 𝛼 = 𝑎𝑐𝑜𝑠 ( )𝑥 (2)
2𝑎𝑏 𝜋
hand part, the camera capture was divided by a bulkhead. This
If the angle 𝛼 is less than 90 degrees, then 𝛼 is a convexity
sealing is carried out because the convex hull method can only
differentiate between the outermost ends of the set of dots, defect. The number of convexity defects is the number of fingers
making it unable to distinguish between hands and faces at this raised minus 1. Figure 4 depicts the flowchart of Convex Hull
time. Inside the area, there is also another seal on the upper right Algorithm process.
and lower right sides. If there is one finger detected in the upper
Start A B
A
Hand Look for the Look for the
point to the point to the Start Number of
edge line yes Choosing the first answer
right of it left of it fingers = 1
coordina
te data Users perform hand
gestures No
Searched for If connected make If connected make
Webcam captures hand Number of Choosing the second
the far left the largest angle the largest angle yes
gestures fingers = 2 answer
point of the
hand No
Video changed to BGR
Connect, and Connect, and mode then to gray Number of
use the point yes Choosing the third answer
use the point fingers = 3
A
as a reference as a reference Video thresholding
point point process No
A
the value of J will be reset, but if the value of C is equal to the
number of questions, then the process will continue to value
calculation. Finally, value is calculated with the formula
time - =1 display C order
question, answer, time Value=N/number of questions*100 and presented on display.
A pop-up page will appear, and the title, class, and quiz id
input fields will be filled in automatically according to the
selected quiz, as presented in Fig. 12.
Figure 12. Pop Up page to set the number of students to take the quiz
gesture identification. The table III shows the result of the E. Delay testing for hand gesture recognition
recognition This hand gesture recognition accuracy test is conducted
TABLE III at 3 various distances (30 cm, 50cm, and 80cm) and 3 various
EXAMPLE RESULT OF HAND GESTURE RECOGNITION
light intensity (<50 lux, 50-100 lux, and >100lux). The delay
Image Description
calculation will be calculated by time that raspberry pi gets the
gesture recognition, minus by time that raspberry pi gets the
frame from webcam. The Table V depicts the results of
1 finger up calculating the delay.
TABLE V
DELAY TEST RESULT
Average Delay per Average Delay
Light
No. Distance Light Intensity per Light
Intensity
and Distance Intensity
2 finger up 30 0.0273
1 <50 50 0.0293 0.0280
80 0.0275
30 0.003
2 50-100 50 0.0036 0.0032
80 0.003
30 0.0033
3 >100 50 0.003 0.004
3 finger up 80 0.0056
The packet loss test results are 0%, which means there are no
failed packets. The value of 0% packet loss is included in the very
good category (<3%), according to TIPHON.
0% which is also included in the very good category. Delay of RGB Images," in IEEE Transactions on Industrial
52.2 ms which is included in the very good category. Thus, the Informatics, vol. 18, no. 9, pp. 5992-6002, Sept. 2022, doi:
overall quality of the network can be categorized as very good. 10.1109/TII.2021.3134016.
[14] P. E. Mas’udia, C. A. Pratama, D. Purwati, Y. Ratnawati,
REFERENCES
M. Sarosa, and N. Hidayati, “Rancang Bangun dan
[1] M.H.Khoirul, “Sistem Pengontrol Presentasi Analisis QoS pada Sistem Informasi Penjualan Obat
Menggunakan Pengenalan Gestur Tangan Berbasis Fitur dengan Layanan Antar-Jemput Berbasis Android,”
pada Contour Dengan Metode Klasifikasi Support Vector Techno.Com, vol. 21, no. 3, pp. 633–643, 2022, doi:
Machine,” Jurnal Pengembangan Teknologi Informasi 10.33633/tc.v21i3.6209.
dan Ilmu Komputer, vol. 4, no. 4, pp. 1083-1089, 2020.
[15] D. Priadi, “Pengukuran Quality of Service (QoS) Pada
[2] D. R. M. Harika, “Rancang Bangun Pengontrol Presentasi Aplikasi File Sharing dengan Metode Client Server
Berbasis Slide dengan Teknik Analisis Gerakan Jari dan Berbasis Android”, Jartel, vol. 6, no. 1, pp. 39-49, May
Tangan,” JOIN ( Jurnal Online Informatika ), vol. 1, no. 2, 2018.
2016.
[3] M. Anshary, “Prototype Program Hand Gesture Recognize
Using the Convex Hull Method and Convexity Defect on
Android,” JOIN ( Jurnal Online Informatika ), vol. 5, no.
2, pp. 205 - 2011, 2020.
[4] A.R.Adnan, “Klasifikasi Gestur Lengan Manusia
Menggunakan Metode KNN Untuk Kendali Stop Kontak
Pintar Berbasis Internet of Things,” e-Proceeding of
Engieering Telkom University, vol. 8, no. 1, pp. 9-16,
2021.
[5] I. C. H.A. Adi, “Sistem Pengenal Isyarat Tangan Untuk
Mengendalikan,” Indonesian Journal of Electronics and
Instrumentation Systems (IJEIS), vol. 9, no. 2, pp. 193-202,
2019.
[6] S. Qin, "Real-time Hand Gesture Recognition from Depth
Images," J Sign Process Syst, 2018.
[7] A. Tompunu, "FINGER TRACKING AND
RECOGNITION USING OPENCV RASPBERRY PI 3,"
Proceeding Forum in Research, Science, and Technology
(FIRST), 2017.
[8] Wilkinson, "A Raspberry Pi-based camera system and
image processing procedure for low cost and long-term
monitoring of forest canopy dynamics.," Methods Ecol,
vol. 12, pp. 1316-1322, 2021.
[9] J. Minichino, "Learning OpenCV 3 Computer Vision with
Python Second Edition," in Preface, Brimingham, UK,
Packt Publishing, 2015, p. vii.
[10] K. &. Atul, "The AI Learner," 9 November 2020. [Online].
Available: https://theailearner.com/2020/11/09/convexity-
defects-opencv/. [Accessed 12 Agustus 2022].
[11] Bhowmik, Interactive Display : Natural Human - Interface
Technologies, John Wiley & Sons, 2015.
[12] A. Skuric, V. Padois, N. Rezzoug and D. Daney, "On-Line
Feasible Wrench Polytope Evaluation Based on Human
Musculoskeletal Models: An Iterative Convex Hull
Method," in IEEE Robotics and Automation Letters, vol.
7, no. 2, pp. 5206-5213, April 2022, doi:
10.1109/LRA.2022.3155374.
[13] S. -J. Horng, D. -T. Vu, T. -V. Nguyen, W. Zhou and C. -
T. Lin, "Recognizing Palm Vein in Smartphones Using