See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/343961443

Design And Implementation Of A Wireless Gesture Controlled Robotic Arm With

Vision

Article · February 2020

CITATIONS READS

0 168

3 authors:

Raed Abdulla Ahmed Gadalla

Asia Pacific University of Technology and Innovation Asia Pacific University of Technology and Innovation
64 PUBLICATIONS   180 CITATIONS    5 PUBLICATIONS   0 CITATIONS   

SEE PROFILE SEE PROFILE

Arun Seeralan Balakrishnan

Asia Pacific University of Technology and Innovation
9 PUBLICATIONS   1 CITATION   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

RFID based system for IoT View project

Mechatronics Engineering View project

All content following this page was uploaded by Raed Abdulla on 29 August 2020.

The user has requested enhancement of the downloaded file.

 Journal of Applied Technology and Innovation (e -ISSN: 2600-7304) vol. 4, no. 1, (2020) 24

Design And Implementation Of A Wireless Gesture

Controlled Robotic Arm With Vision
Ahmed Ayman Gadalla Raed Abdulla Arun Seeralan
School of Engineering School of Engineering School of Engineering
Asia Pacific University of Technology Asia Pacific University of Technology Asia Pacific University of Technology
and Innovation (APU) and Innovation (APU) and Innovation (APU)
Kuala Lumpur, Malaysia Kuala Lumpur, Malaysia Kuala Lumpur, Malaysia
[email protected] [email protected] [email protected]

Abstract The fundamental aim of the project is to develop a and interfaced to the Micro Controller Unit (MCU), which
robotic arm that can be handled by a normal layman to complete will transmit the signal to the robotic harvester that carries the
the specific task according to the needs of the customers. In this robotic arm and a receiver to get the data of the sensors from
proposed method, the system was designed and implemented, a the MCU by using a Bluetooth module. The arm contains four
prototype was developed to demonstrate the proposed system. The servo motors that have a maximum travel of 199.5 degrees,
performance of the developed proposed system is evaluated by and it's used to move the robotic arm as like the human hand's
testing the accuracy of the gesture response, testing the range of joints with a total 4 Degrees of Freedom (DoF). The entire
the radio communication device, accuracy test in multiple ranges,
system can be improved later on by making an automated
resistance of the flex sensor in different angels, Recognition Rate
system and test it to cut the grape and mango trees. The system
and Recognition Time. It is observed that the implemented system
has more movement freedom with the navigation platform and a
was designed and made for the agricultural uses, where it uses
day and night vision compared with the existing products. On the a cutter at the end of the robotic arm. [6] made a research on
existing systems, most of them lacks on the ability to navigate implementation of a wireless gesture controlled robotic Arm.
around the operating space, meanwhile the implemented system The aim of the research was to make the computer understands
contains a gesture-controlled navigation platform. Also, most of the human body language. The Image processing was used as
the existing systems lack on providing a monitoring feature to a development for the traditional wired glove or the gesture
monitor the process of the robot from a distance where they only recognition where those two methods only recognize a
designed the system to be operated within the range of vision while specified movement and react based on this movement. The
the implemented system offers the feature of operating it remotely. system consists of two parts, transmitter and receiver. The
transmitter part contains a PC, RF transmitter and a webcam
Keywords robotic arm, gesture-controlled navigation and all the process is done in this part, and for the receiver it
platform, servo motor, flex sensor, Arduino Nano. contains a RF receiver, microcontroller and motors to move
the arm. The user or the operator has to stay within the
I. INTRODUCTION detecting range of the camera to get the system going in the
In today's reality, in all segments, the majority of the work right way, and in case of not detecting the hand or if the hand
is finished by robots or automated arm having diverse number got out of range the robotic arm might perform a wrong
of level of opportunities (DOF's) according to the prerequisite command. [7] proposed a research about accelerometer based
[1]. Robots are increasingly being integrated into industries to gesture controlled robot with robotic arm. Accelerometer was
replace humans especially to perform the hazardous tasks [2]. used along with Arduino microcontroller, RF transmitter and
A robotic hand arm system in which a robot hand is attached receiver, motors to achieve the movement of the robot. A fire
to a multiple-degree-freedom arm are considered useful in sensor was added to the system to detect if there is a smoke or
performing complicated tasks in various environments [3]. a fire in case if a disaster happened in the industry and it will
These days, the robotic system has turned into a fundamental start buzzing if it got triggered. The system can be improved
component in our everyday life. It works in a wide scope of to fit multiple tasks as for what the job needs. The system
regions. The pattern of actualizing the robotic system has been doesn't have a base to allow it to navigate around the
embraced by numerous industries and countries. Worldwide, workplace. The system designed in a way where while moving
the robotic system is utilized to deal with explicit tasks that the arm a switch needs to be used to switch to the gripper and
are extreme and risky to people. Humans interact in the move it.
physical world by the means of the five senses. However, [8] proposed a high level robot programming using body
gestures have been an important means of communication in and hand gestures, in the research a visual sensor was used to
the physical world from ancient times, even before the detect the human motion, where it allows the movement of the
invention of any language [4]. robot in a different directions. A microcontroller was used
The researchers [5] presented an overview on hand gesture with the robot along with a decoder to translate the movement
based wireless robotic arm control for agricultural of the human, and the system used to do that is Robot
applications, their technology used a glove as the Human Operating System (ROS). Microsoft Kinect is the sensor used
Control Interface (HCI). The sensors are placed in the glove to implement the hand and body gesture. The robot speed is
 Journal of Applied Technology and Innovation (e -ISSN: 2600-7304) vol. 4, no. 1, (2020) 25
20%, while programming, and the response time for the When the data is received by the receiver it will be
sensors was estimated around 43ms, while the response time processed to differentiate which motor it was meant to move,
from the robot after the communication is measured in 115ms. if the data was coming from the first accelerometer then it will
The system is user friendly and easy to use for untrained be sent to the servo motor drive to move the robotic arm, if the
individuals. The system can be used in as a in different robot signal was from the flex sensors it will go through the servo
stages, which makes this method a universal way for robot motor drive then it will move the gripper at the end of the
programming. robotic arm and if the data was coming from the second
accelerometer then the data will be given to the DC motor
Nowadays, most robotic systems in the field are controlled drive to move the navigation platform. Fig 1. and 2. shows the
by usual inputs ways. This design will seem difficult for those overall block diagram of the two parts of the system.
who feel inconvenient to use the robotic system proficiently
and expertly with a controller or has a low understanding of
the robotic system. Training and essential information are Servo 1 (Arm rotation)
required by the users to control the robotic system. In this way, Servo 2 (Arm
the structure of controlling the robotic system with hand Robotic arm and
forward/backward)
motions will turn into a way to solve the current issue as it is Servo 3 (Arm up/down)
gripper motor Servo 4 (Gripper rotation)
progressively convenient and easy to use compared with the drive Servo 5 (Gripper
usual input ways. Users could control the robotic system to open/close)
move as per their own ideal motion by utilizing hand gesture
movements. Other than that, this new strategy of controlling Arduino Nano
the robotic system by utilizing hand gesture movement RF
receiver slave
additionally ready to enable users to work remotely at those
dangerous activities or handle substantial objects. That can DC motor 1
deal with the risky activities, for example, handle poisonous Navigation DC motor 2
substances or bomb transfer, along with dealing with heavy platform motor DC motor 3
objects or robot controlled tasks in the assembling industry. drive for the DC motor 4
This new intelligent strategy among human and machine give wheels
a more convenient strategy to control the robotic system in
many areas, it gives more exact movement of the robotic Fig. 2. Robotic Arm, and Navigation Platform Block Diagram
system rather than utilizing the controller. The fundamental
aim of the project is to develop a robotic arm that can be III. CONSTRUCTIONAL DETAILS
handled by a normal layman to complete the specific task The wireless gesture controlled robotic arm has been
according to the needs of the customers. planned to be constructed in three different phases. Phase one
The wireless gesture controlled robotic arm has been planned
to be constructed in three different phases. is the hand gloves
II. BLOCK DIAGRAM AND OPERATION OF THE PROPOSED and the sensors connections and detection testing for the hand
SYSTEM movement, phase two is the robotic arm structure and the
servo motors connection and testing the reaction to the hand
The overall block diagram shows the schematic diagram
gesture and its movement speed and phase three is the
of how the entire project is developed. Fig 1. shows that when
navigation platform structure and the DC motors connection
a movement is sensed by any of the four sensors placed on the
and test its reaction to the left hand gesture and its movement
hand glove, it will be converted into data and get processed by
speed.
the microcontroller where each sensor is meant to move a
specific part, after that, the microcontroller will transmit the For the structure of the project starting from phase one
processed data to the second microcontroller to perform the there is a wearable gloves where the Arduino nano
movement via the RF transmitter and receiver. microcontroller and the two accelerometers and the flex
sensors will be placed on, for phase two, the robotic arm is
made out of an aluminum 2020 for its durability, roughness
Accelerometer 1 for the and its ability to handle weights, starting from the bottom of
robotic arm movement the robotic arm there is a circular base with the diameter of 12
process CM supported with four screws that connects it with the
navigation platform from the bottom and another four screws
Flex sensor 1 for the from the top which is connected to the rotating bracket which
gripper rotation process
is rotating 180 degree using a MG995 servo motor and it's
Arduino RF connected to the circular base where the rest of the arm parts
Nano transmitte is placed, there is a servo motor to perform the front and back
Flex sensor 2 for the master r
gripper open/close movements which is connected with two brackets by screws
process and a bearing with another servo to perform the up and down
movements and it's connected to another servo with a bracket
Accelerometer 2 for the and screws to perform the rotation of the gripper and it's
platform movement connected to the gripper with a joint and screws then the
process gripper contains a servo motor to perform the opening and
closing of the gripper, and the height of all the arm parts joined
together is 37 CM.
Fig. 1. Hand gloves block diagram
 Journal of Applied Technology and Innovation (e -ISSN: 2600-7304) vol. 4, no. 1, (2020) 26

Fig. 3. The Gloves Schematic Diagram

For phase three which is the navigation platform, the base

of it is made of acrylic sheet for its durability and the ease of
forming, the navigation platform contains two bases, the
bottom one contains four DC motors that rotates 360 degrees
for the wheels rotation and the motors driver and six screws to
join it with the second base on the top of it to carry the robotic
arm and the Arduino nano microcontroller, and the
dimensions are 5 CM height, 15 CM width and 25 CM length.
The height of the whole system joined is 42 CM.

Fig. 4. The Robotic Arm and Navigation Platform Schematic Diagram

IV. IMPLEMETATION AND FLOWCHART OF THE

SYSTEM
The working process of the entire project is segregated
into three different parts, the movement detection process and
the action taking process. The detection process provides a
detailed explanation regarding the hand movement detection
data and transmitting it to the second part to perform the
action. The action taking process provides a detailed
explanation starting from receiving the processed data to
taking this data to perform an action on a specific motor. The
third part is explaining the monitoring process for the robotic
arm. Fig. 5. System flowchart
 Journal of Applied Technology and Innovation (e -ISSN: 2600-7304) vol. 4, no. 1, (2020) 27
V. SYSTEM TESTING getting 10 beeps every 10 second then the signal will be a
A. Accuracy of Gesture Response Test 100% received. The test is done in different ranges to check
how far the communication signal can reach, and it was done
This test is conducted to determine the accuracy of the indoor and outdoor.
accelerometer and the flex sensor on detecting the gesture of
the hand and the response from the robotic arm. The setup was
to run the system and place the accelerometer and the flex
sensor on the hand and make some gesture with the hand in Chart Title
different angles and check the accuracy of the response and if
120%
it did what it meant to do. Rotating the hand should result in
rotating in the base of the robotic arm, turning it to the to the 100%
right will move it up and to the left will move it down, forward 80%
turning should move it forward and turning backwards should 60%
move it backwards and it should be turning as the same degree 40%
as the hand is turning. Bending the flex sensor 90º should close
the gripper and bending it les than 90º should move it halfway 20%
to closing. 0%
Indoor Outdoor

2m 5m 10 m 15 m 30 m 50 m
TABLE I. RESULTS FOR TEST 1
80 m 160 m 300 m 500 m 600 m
Test Sensor Gesture Expected Accuracy
number response
1 Accelerometer Rotate Rotate 30º 65% Fig. 6. Testing the Range of the Radio Communication Graph
30º
2 Accelerometer Forward Forward 30º 80% The range test results of the radio communication module
30º are shown in Fig 6, the test showed a good result in different
3 Accelerometer Backward Backward 30º 80% ranges where the signal was being received in a wide range.
30º
The results of the indoor test of 50 m didn't show any readings,
4 Accelerometer Up 30º Up 30º 80%
and that depends on the walls in between the transmitter and
5 Accelerometer Down 30º Down 30º 80% receiver, the more the walls get in between the weaker the
signal will get. Moreover, a 50 m test was done indoor but in
6 Flex sensor Bend 90º Bend 90º 90% a less walls condition in between the transmitter and the
receiver and it showed a good result where the signal was
7 Flex sensor Bend 60º Bend 60º 90% being received as it should.
Therefore, the indoor test showed that no specific distance
can be given as a maximum distance indoors since it depends
The analysis shown in Table I. shows the accuracy on what stands in between the transmitter and receiver, and for
percentage after testing the gesture of the hand and the
the outdoor test it gave a good results until the range of 1 km
response of it in the robotic arm multiple times. The response
and then it started to lose some data, which makes the
of the gestures forward, backward, up and down were showing
an accuracy of 80% because of the feedback system which maximum distance for the outdoor a 1 km.
keeps calculating the angle of the motor. Moreover, it is
considered as a high accuracy level since it came from the C. Resistance of Flex Sensor
accelerometer, and comparing it to the rotation results that In this test the resistance of the 2.2 inches flex sensor will
came from the gyro meter built inside the accelerometer, the be tested in a different bending angles, since the motion of the
gap can be seen where the results of the rotational gesture finger might be able to yield a continuous range of voltages,
came out to be 60% and what caused this drop in the accuracy and due to that, the test will be performed to insure linearity
percentage for the rotation is that the gyro sensor drifts a little and adequate sensitivity, and sensitivity is important since it
bit from its origin point after some time which caused it losing
has a direct effect on the accuracy. The flex sensor will be
in its accuracy. Furthermore, the flex sensor showed a high
accuracy in responding to the gestures, but it can be affected fixed under the knuckle of the index finger of the right hand
by touching it with hands which might affect its readings. and it was held in place using a tape. The test involves testing
the resistance of the sensor while the finger is held at different
angles. The resistance was tested with the finger pointing up,
B. Testing the Range of the Radio Communication relaxed, straight and in a closed fist. The resistance and the
In this test the range of the radio communication between angle will be measured using the Arduino IDE software.
the controlling glove and a beeping device on the other The voltage of the Arduino is fixed at 4.98 V and the
microcontroller and it will be tested indoor and outdoor to sensor is c
check how far it can reach as the maximum range. The NRF connected to a ground. The resistance of the sensor in a
radio communication device is placed on the hand glove to straight position was measured using a multimeter and it came
transmit the data from the microcontroller to the receiver NRF
which is connected to the other microcontroller for the is made to get the reading of the sensor and converted to a
beeping device to keep beeping every one second as long as digital value and then calculate the voltage to find the
there is a signal being received from the transmitter and resistance and the angle after that.
 Journal of Applied Technology and Innovation (e -ISSN: 2600-7304) vol. 4, no. 1, (2020) 28
150 no walls in between the transmitter and receiver it showed a
Resistance in different angles high accuracy as it can be seen in the result of the 120 m test.
100

50 TABLE II. RESULTS TABLE FOR MULTIPLE RANGES TEST

Test Sensor Range Gesture Expected Accuracy
0 number response
1 Flex 5m Bend 90º Rotate 90º 100%
sensor
Resistance in different angles 2 Flex 15 m Bend 60º Rotate 60º 100%
sensor
Fig. 7. Resistanse in Different Angles Graph 3 Flex 30 m Bend 90º Rotate 90º 100%
sensor
4 Flex 50 m Bend 60º Rotate 60º 80%
Fig 7. shows the results of the test which calculates the sensor
resistance of the flex sensor in a different angles. The Arduino 5 Flex 100 m Bend 90º Rotate 90º 100%
analog to digital conversion can read a voltage from 0 to 5 sensor
volts with a 10-bit resolution, which means 1024 different 6 Flex 120 m Bend 60º Rotate 60º 100%
values can be resolved. The main function is to read the value sensor
of the analog to digital conversion to use it to get the voltage
by getting the digital value and multiply it by the voltage of
the power supply which is 5 volts and divide it by 1024. After VI. HARDWARE RESULTS AND SIMULATION
getting the voltage, the resistance is then found using the After constructing the wireless gesture controlled robotic
following formula: arm, the result came out to be as was required to serve the aim
and objectives. The prototype contains two main part, the first
one is the hand glove which is where the sensors are placed to
detect the gesture from the operator and then transmit it to the
The voltage divider formula converts the voltage to the second part which is the robotic arm with the navigation
resistance of the flex sensor to find the value of R1, after that, platform where it receives the data and operates on moving the
the value of R1 can be used to estimates the sensors bending motors and perform the action required by the operator.
angle by using the map function which will take the linear and
bent resistance values and map them to 0º and 90º, then it will
take the R1 value and estimate the bend angle based on the
two calibration end points values. Furthermore, the test results
showed that, the sensitivity of the sensor is high which will
help on using it in the project to perform the task of controlling
the gripper.

D. Accuracy Test in Multiple Ranges

This test is done to show the accuracy of the gesture and
the response of the system wirelessly in multiple ranges as the
system should be operated from distance. The setup for this
test is to use the hand glove with the sensor, Radio
communication device and the microcontroller as the
transmitter device and the robotic arm with the motor, radio
communication device and the microcontroller as the receiver.
The robotic arm will be placed in multiple ranges and the Fig. 8. Hand Glove with NRF24l01 and Accelerometer
accuracy of responding to the hand gesture will be tested 10
times with respect to the distance at each range. This test will
use the flex sensor gesture which should open and close the
gripper when it gets bended at 90º and bending it in any degree
between 0º and 90º will be closing it halfway.
Table II. shows the results of the response accuracy from
a range test. The response were following the gesture from
multiple ranges with a high accuracy, but for the distance over
50 m indoor it showed a little bit of delay in holding the same
angle which is caused by the angle feedback system and with
the range it became a little bit slower in holding the angle.
Moreover, the system showed a high accuracy with working
in a distance, but this accuracy when tested indoor it showed
this delay on the angle feedback, but when tested outdoor with Fig. 9. Hand glove with Arduino Nano and Flex Sensors
 Journal of Applied Technology and Innovation (e -ISSN: 2600-7304) vol. 4, no. 1, (2020) 29
Figs 8. and 9. shows the hand glove which is made from receiver. After the data is received by the radio
leather and cotton to be comfortable for the operator to wear communication receiver, it will be processed by the
and will make it easier to stick the sensors to it. Fig 8. shows microcontroller to analyze it and Fig the right motor to move.
the MPU6050 accelerometer sensor placed on top of the glove Fig 10. shows the robotic arm and the navigation platform
to make it more comfortable and to detect the gesture of the separated and Fig 11. shows them while attached together.
hand better, aside the accelerometer there is the NRF24L01 The robotic arm parts are made out of aluminum 2020 and
radio communication device to perform the wireless contains a base where it connects it with the navigation
communication between the two parts of the prototype. Both platform from the bottom, and from the top it contains a
devices are fixed to a mini breadboard and attached to the hand rotating base attached to a MG995 servo motor to perform the
glove with a double sided tape. Fig 9. shows the bottom side rotation movement of the robotic arm, and the rotation base is
of the hand glove where the Arduino microcontroller is attached to another circular base where another servo is placed
attached to a mini breadboard with the sensors connected to it and attached to the joint of the arm to perform the front and
to perform the process of the detection and convert it from an back movements of the arm, and in the middle of the arm there
analog values to digital values which will be processed and is a joint with a servo attached to it to perform the up and down
then sent to the second part of the prototype by the radio movements, at the end of the joint there is another servo
communication device, it also can be seen on Fig 9. at the attached to the gripper to perform the rotation of the gripper
bottom of the index and the middle fingers are the two flex when the middle finger flex sensor is bent, and the gripper is
sensor which were placed there to detect the amount of attached to a servo which is used to open and close the gripper
bending of the fingers which will be used to control the gripper if the flex sensor in the index finger is bent. The navigation
on the second part of the prototype, and both sensors along platform is made of acrylic sheet and it contains four DC
with the microcontroller are fixed to the hand glove with a motors connected to four wheels to perform the movement of
double sided tape. it in four directions forward, backward, right and left and it's
attached to the body of the platform by screws and at the first
body of the platform there is the DC and servo motor drives
aside to the radio receiver then it is connected to the top base
by a long screws. At the top base there is the microcontroller
placed under the robotic arm and attached to the base with
screws.
VII. CONCLUSION
The theory behind the project was to implement a way to
ease controlling a robotic arm to replace the traditional way
and to design a wireless gesture controlled robotic arm to
prevent workers from involving into dangerous tasks. The
design was made by utilizing a low-cost sensor which made
the project save much more than invested on. With the
procedure or the working principle which helped to
understand the process of how the system worked which was
implemented after analyzing what the researchers were
lacking to provide to the system to make it more sustainable
Fig. 10. Robotic Arm and the Navigation Platform
for its purpose of preventing workers from involving into
dangerous tasks and to ease controlling the robotic arm. The
system uses a recyclable material, and it provides the ease of
controlling the robotic arm even with a lack of the technical
knowledge to use it which makes it user friendly. Along with
that, the system being cost efficient considering the features it
provides and the low cost of it.
VIII. LIMITATIONS
The major limitation of the project is the coverage range
of the radio device used to communicate between the hand
glove with the sensor and the master microcontroller and the
robotic arm and the navigation platform and the slave
microcontroller. Based on the range testing of the device and
its response, it turned out that the operator might face some
difficulties operating the system indoor in case if there walls
in between the robotic arm and the operator, but in outdoor it
can't reach farther than 1 km. Another part of the system which
also considered is a limitation is that, the system has to be
restarted once in a while regarding a limitation in the
Fig. 11. The Robotic Arm with the Navigation Platform Attached accelerometer module where the gyro sensor drifts a little bit
after a while from operating the system which will make it
Figs 10. and 11. shows the second part of the prototype give less response to the operator or response in a different
which contains the robotic arm and the navigation platform way than desired. Another limitation in the system is that, the
with the microcontroller, motors drive and the NRF24L01 monitoring camera always has to have a WIFI connection in
View publication stats

Journal of Applied Technology and Innovation (e -ISSN: 2600-7304) vol. 4, no. 1, (2020) 30
term of it to stream the process made by the operator through
a live streaming video. Beside always having a WIFI
connection, also a poor connection might cause some delay in
the video streaming which might cause to perform a wrong
movement in the robotic arm.
REFERENCES
[1] B.S.Suryavanshi, S.R. Pati, Design and Development of Arduino
Based Gesture Controlled Robotic ARM International Journal of
Advanced Scientific Research & Development (IJASRD), vol. 4, pp.
202-209, 2016.
[2] J. PRADEEP, P. V. PAUL, Design and implementation of gesture
controlled robotic arm for industrial applications International
Journal of Advanced Scientific Research & Development (IJASRD),
vol. 03, pp.202-209, 2016.
[3] B. Fang, D. Guo, F. Sun, H. Liu, Y. Wu, A robotic hand-arm
teleoperation system using human arm/hand with a novel data
glove 2015 IEEE International Conference on Robotics and
Biomimetics (ROBIO). IEEE, pp. 2483-2488, 2015.
[4] P. Chanda, P. K. Mukherjee, S. Modak, A. Nath, Gesture Controlled
Robot using Arduino and Android. International Journal, vol. 6, pp.
227-234, 2016.
[5] R. K. Megalingam, S. Bandhyopadhyay, G. V. Vivek, M. J. Rahi,
Hand gesture based wireless robotic arm control for agricultural
applications IOP Conference Series: Materials Science and
Engineering. Vol. 225. No. 1. IOP Publishing, 2017.
[6] S. A. Khajone, S. W. Mohod, V. M. Harne, Implementation of a
wireless gesture controlled robotic arm International Journal of
Innovative Research in Computer and Communication
Engineering, Vol. 3, pp. 377-379, 2015.
[7] C. S. Budheliya, R. K. Solanki, H. D. Acharya, P. P. Thanki,
Accelerometer based gesture controlled robot with robotic arm Int.
J. Innov. Res. Sci. Technol 3, Vol. 3, pp.92-97, 2017.
[8] P. Tsarouchia, A. Athanasatosa, S. Makrisa, X. Chatzigeorgioua, G.
Chryssolourisa, High level robot programming using body and hand
gestures Procedia CIRP, pp.1-5, 2016.