Sign Language Converter

Abstract: - People with hearing and speaking disabilities face problems in communicating
with other hearing people. Sign language is a visual language that is used by deaf and
dumb people as their mother tongue. It uses body movements (usually hand movements)
for the purpose of communication. But the sign language is not so easy to be understood
by abled persons. They do require a translator for translation. A translator is a person
who knows both local and sign language. He will listen to the abled person and translate
it into sign language for the specially abled person. And then translate the responses from
specially abled people from sign language to the local language. The application Sign
Language Converter will act as a translator for 2-way communication.
speaking disability that totals over 360 million
people across the globe. The majority of these
people live in countries with comparatively low
incomes. Sign Language is an independent
1. Introduction
language which is different from spoken/written
language, the basic difference is it has limited
Sign language is used by people who have
vocabulary compared to written/spoken. Sign
speaking or listening disabilities so that they can
language is not the same in every country,
exchange information between other people and
different sign languages are used in different
within their own community. This can be used on
countries or regions. There are two separate
a wide scale to make most of the public
languages ASL and BSL among which ASL is
understand what a person is trying to convey to
the most widely used signed language. This
the world through sign language. This will
paper describes a technology in which real time
prevent disabled people from hiring other people
videos are analyzed and are used for hand
who can speak and understand sign language just
movement detection and recognition, thus
to be their communicator. Sign Language
helping them to convey what they want to
Converter is a machine learning program that
explain or tell, in the form of transcripts and
detects and recognizes audio signals received to
converts it into audio. The system developed
text using speech to text API (python modules or
identifies sign language (non verbal
Google API) and then making a visual
communication) done by using some hand
presentation (video) of the converted sign
gestures and the machine is trained to recognize
language which requires machine learning as a
some daily frequently used gestures and convert
part. Sign gestures can be majorly classified into
them into verbal communication so that the other
two types, static and dynamic. The static gesture
people who are not familiar with this non-verbal
is simpler than dynamic gesture recognition, in a
communication can be understood easily which
static only a single image is recognized at a time
will ultimately prevent them to hire a person to
while on the other hand dynamic gesture is a
communicate for them and save their money. We
moving gesture represented by various images.
focus on mainly these two points: - Persons who
Various signs performed by the hand and convert
are disabled to speak or hear converse in sign
it into text/speech.
language and have a problem in communicating
2. Problem Formulation what they actually mean. A mediator is needed
just for the communication between two people
According to the World Health Organization if one is disabled and the other is not.
(WHO), around 5% of the world population
belongs to the people with the hearing and 3. Proposed Solution
 comprises input and output pairs which are used
The current system that we are proposing would to train the model and determine a hidden
work on processing of live video streams rather pattern. This hidden pattern is then used for
than processing of a still image. It actually recognition of specific patterns when working
identifies the signs on a real time basis which are data is provided in future. Supervised learning is
in front of a machine or host that has a webcam in which you have input variable
and tells us about the communications done in (a) and an output variable
transcripts form to read. The current system is (b) and you apply an algorithm to learn the
developed in python and works on machine mapping function from the input to the
learning algorithm that uses supervised learning output. Y = f (A)
as its basis. It uses a specified Dataset which has The aim is to approximate the mapping function
information (Q&A) about a finite number of so well that when you have new input data (a)
sign videos provided, using this information only you can predict the output variables (b) for that
the algorithm recognizes the nonverbal data
communication done in front of the machine. It Supervised learning is called so because the
also has the capability to recognize all types of method of algorithm learning from the training
characters from A to Z or numbers 0 to 9. dataset can be thought of as a trainer supervising
the learning process. We know the accurate
results; the algorithm iteratively makes
4. Technologies used in the project predictions on the training data and is corrected
by the trainer. Learning stops when the algorithm
a) Machine Learning achieves an agreeable level of performance.

Artificial intelligence (AI) is a subarea of c) Deep Neural Networks

computer science that emphasizes the creation of
automated machines that work and reacts like Neural networks is a set of algorithms which is
used to recognize the relationship between the
humans. Machine learning system is a branch of
underlying set of data in the way that a human
artificial intelligence based on an idea that a
brain works which constantly tries to recognize
system can produce general hypothesis by
patterns and categorize and classify information.
learning
It is an excellent tool for finding patterns which
from data provided, identify patterns and make
are complex as well as numerous for
decisions with minimal human intervention.
programmers to teach the machine to recognize.
Machine learning is important because as
It usually involves a large number of processors
models are exposed to new data, they are able
that are operating in parallel and are arranged in
to create a predictive model capable of inferring
tiers. Hidden layer is in between the two input
annotations for future data. They learn from
and output layers. Deep neural network is
SIGN LANGUAGE ORATOR previous
similar to deep learning, with a certain level of
computations to produce reliable, repeatable
complexity and has more than two layers. The
decisions and results without needing multiple
different layers of such a system could be seen
manual edits to the program.
as a nested hierarchy of related concepts or
decision trees. Deep neural network systems
b) Supervised Learning
need large quantities of data in order to be
trained as the systems learn from exposure to
Supervised Learning is a computational task of
huge numbers of data points. Google Brain
automated data (the training data set) to produce
learning to recognize cats after being shown
general hypotheses. The training data set
over ten million images can be seen as an early
example of this. convey to the world through sign language. This
will prevent disabled people from hiring other
d) TensorFlow people who can speak and understand sign
language just to be their communicator.
TensorFlow is a free and open-source software
library for dataflow and differentiable
programming across a range of tasks. It is a 6. Project Objective
symbolic math library, and is also used for
machine The main objectives achieved in the project are
learning applications such as neural networks. It real-time translation, elimination of an interpreter
is used for both research and production at between sign language and common speech. It
Google. It is a standard expectation in the does not require additional hardware and is easy
industry to have experience in TensorFlow to to incorporate in any OS.
work
in machine learning. TensorFlow was developed
by the Google Brain team for internal Google
use. It was released under the Apache 2.0 open 7. Hardware Requirements
source license on November 9, 2015.
The hardware requirements of this project
e) OpenCV are: -
a) Processor: Pentium, AMD or Higher Version
OpenCV (Open Source Computer Vision b) Operating System: Windows XP/ Windows 7/
Library) is a library developed by Intel of Linux
programming functions and is released under c) RAM: 2GB recommended
BSD license hence it’s free for both academic d) Hardware Devices: Keyboard with mouse
and commercial use. It is a powerful library e) Hard disk: 10 GB or more
designed to work on real time applications with f) Display: Standard Output Display
a strong real-time efficiency. It is written in C++ g) Voice Input: Microphone (preferred)
and its primary interface is based on C++ this
makes OpenCV portable to almost any
commercial system includes Python, JAVA, 8. Software Requirements
MATLAB/OCTAVE interfaces which are
supported by Windows, Mac OS, iOS, Linux, The software requirements of this project are
FreeBSD, OpenBSD and Android as it was as follows: -
designed to be a cross-platform, the library can a) Android Studio
take advantage on multi-level processing thus b) MySQL Server
makes it easy for businesses to utilize and c) Python IDLE
modify the code.
5. Project Scope
9. Conclusion
The scope of the project is that it can be used
to help disables people. It can also be used
This is to conclude that the project that we
by abled people to understand sign language
undertook will work upon with a sincere effort.
and help people to communicate through it. It
Most of the requirements will be fulfilled up to
is used on a wide scale to make most of the the mark and the requirements which will be
public understand what a person is trying to
remaining will be completed with a short
extension. This project would definitely satisfy
all the needs and help to the community of deaf
people and people who are disabled to speak.

10.Limitations
Although the project is capable of doing the
conversion of sign language to common speech
and vice-versa, it does face some limitations.
The first one is the need of internet availability
and the second one is the requirement of android
smartphones. Also, it is a bit of a
time-consuming task as the project requires
some time for the conversion. Also, some
amount of memory is utilized for loading
images and videos.

11.Future Enhancements

This project can be extended to achieve greater

efficiency. In future, it can be made available on
iOS platform too. Also, the current GUI can be
improved.