Arabic sign language recognition an image-based approach

ISSN (Print) : 2320 – 3765 ISSN (Online): 2278 – 8875 International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering ( An I SO 3 2 9 7 : 2 0 0 7 Ce rt ifie d Org a n iz a t ion ) Vol. 4 , I ssue 4 , Apr il 2 0 1 5 Linguistic Sign Language Recognition Though Image Based Approach K.Sangeetha1, K.R.Renugadevi2 Assistant Professor, HOD in Dept. of ECE, Krishnasamy College of Engineering and Technology, Cuddalore, Tamilnadu, India1 PG Student [EST], Dept. of EEE, Krishnasamy College of Engineering and Technology, Cuddalore, Tamilnadu, India 2 ABSTRACT: Sign language recognition is a very challenging research area. In this proposed system, a well trained computer system is used to recognize static hand gestures representing linguistic words. The main aim of the paper is conversion of linguistic sign languages into text and speech form. Recognized sign are also translated into Tamil and Hindi languages. It contains three processes of work. First process is pre-processing, in which the obtained images are processed through the steps like segmentation, resize, and gray conversion. Second process is region-based analysis which exploits both boundary and interior pixels of an object. Solidity, perimeter, convex hull, area, major axis length, minor axis length, eccentricity, orientation are some of the shape descriptors used as features in this process. The features derived are used to train the binary classifier first; secondly the testing images are given for classification. Knn classifiers are used for classification which provides a good result with less computation time for larger datasets. Since, the system handled a binary classifier it performed a one-versus-all kind of classification. PCNN (Pulse Coupled Neural Network) is used for pattern recognition. Third process is the hand gesture recognition. KEYWORDS: Sign language recognition, contour based technique, image based, K-nn classifier, PCNN. I. INTRODUCTION Hand Gesture recognition is a language technology with the goal of interpreting human gestures via mathematical algorithms. Sign Language is a non verbal method of communication in which gestures are made using hands. Gestures are an integral part of our day to day communication and some expressions are conveyed by gestures only. Gestures can originate from any bodily motion or state but commonly originate from the face or hand. Current focuses in the field include emotion recognition from the face and hand gesture recognition. Many approaches have been made using cameras and computer vision algorithms to interpret sign language. English is the standard language that is used all over the world for computer keyboards, so our sample sign language is based on English vocabulary. There are 26 alphabets in the English vocabulary. Each alphabet is assigned a unique gesture using hands. The gesture may be single hand gesture or double hand gesture. In this paper we have used single handed gestures only.The Gesture recognition interface acts as a communication channel between humans and machines. The human machine interaction is similar to human interaction, in which, the valuable information are communicated using the human organs like hand gesture, head movement, face expression, voice communication and overall body posture. The design of a gesture recognition system should be based on common hardware support such as webcams or mobile integrated cameras, to be applicable to current PCs, mobile devices, Digital Cameras, etc. While designing such systems, certain parameters have to be included, so that the system will be able to operate under complex or non uniform background, i.e., different light intensity and noisy environment, etc. but here web cam is preferred. However, the identification and recognition of posture, gait and human behaviours is also the subject of gesture recognition techniques. Gesture recognition can be seen as a way for computers to begin to understand human body language, thus building a richer bridge between machines and humans than primitive text user interfaces or even GUIs (graphical user interfaces), which still limit the majority of input to keyboard and mouse. Copyright to IJAREEIE 10.15662/ijareeie.2015.0404106 2498 ISSN (Print) : 2320 – 3765 ISSN (Online): 2278 – 8875 International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering ( An I SO 3 2 9 7 : 2 0 0 7 Ce rt ifie d Org a n iz a t ion ) Vol. 4 , I ssue 4 , Apr il 2 0 1 5 Figure 1 Sign Languages Gesture recognition enables humans to communicate with the machine and interact naturally without any mechanical devices. Using the concept of gesture recognition, it is possible to point a finger at the computer screen so that the cursor will move accordingly. This could potentially make conventional input devices such as mouse, keyboards and even touch-screens redundant. Gesture recognition can be conducted with techniques from computer vision and image processing. More general human pose and movements are taken by cameras (webcam) connected to a computer. II. RELATED WORK [1] Quantitative experimental results show that the system performs efficiently. Less rate configuration for each character. Also will enhance the algorithms needed for sign language recognitions such as neural network.[2] Hand gesture recognition is finding its application for non-verbal communication between human and computer, general fit person and physically challenged people, 3D gaming, virtual reality etc.[3] According to supervised classification pixels that satisfies a certain criteria is assigned to that class of pixel. In this case the extraction is based on pixels. Sub pixel classifier is a kind of supervised classification. [4] have also showed a method, where they detected the center position of hand then they used line detection system to detect the finger print position. Thus they can detect a hand. . According to me it is not necessary that we can have an image of hand with the finger print side. III. GENERAL TERMINOLOGY In hand gesture recognition technology, a camera reads the movements of the human body and communicates the data to a computer that uses the gestures as input to control devices or application. The technology also has the potential to change the way users interact with computers by eliminating input devices such as joysticks, mice and keyboards and allowing the unencumbered body to give signals to the computer through gestures such as finger pointing. An important field is recognizing patterns, particularly visual and sound patterns. It is central to optical character recognition (OCR), voice recognition, and handwriting recognition. Gesture is a form of non-verbal communication using various body parts, mostly hand and face. Gesture is the oldest method of communication in human. Primitive men used to communicate the information of food/ prey for hunting, source of water, information about their enemy, request for help etc. within themselves through gestures. Still gestures are used widely for different applications on different domains. This includes human-robot interaction, sign language recognition, interactive games, vision-based augmented reality etc. Another major application of gestures is found in the aviation industry for placing the aircraft in the defined bay after landing, for making the passengers aware about the safety features by the airhostess. For communication by the people at a visible, but not audible distance (surveyors) and by the physically challenged people (mainly the deaf and dumb) gesture is the only method. IV. PROPOSED SYSTEM In proposed system converting the Linguistic Sign Language into text form and also translating its meaning in two or more languages, thus it overcomes the difficulties in existing system. Copyright to IJAREEIE 10.15662/ijareeie.2015.0404106 2499 ISSN (Print) : 2320 – 3765 ISSN (Online): 2278 – 8875 International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering ( An I SO 3 2 9 7 : 2 0 0 7 Ce rt ifie d Org a n iz a t ion ) Vol. 4 , I ssue 4 , Apr il 2 0 1 5 Proposed System Block Diagram . Figure 2 Hand gesture recognition systems N images to be trained Image Acquisition A collection of 40 static hand images are taken by using USB connected camera or a webcam as shown in figure 2. Since an image based approach has been performed, a certain limitations are followed such as black background and fixed distance between the signer and the background. The resolution of grabbed image is too large (for clarity) to process so that the image is cropped to reduce the size and so finally got an image. Pre-processing This is very much required task to be done in sign language recognition system. It is applied to images before extracting features from hand images. Segmentation Convert a gray scale to binary image so that clear view between the background and gesture. Comparison process is followed in real time process. The images of the hand are captured from the webcam, preprocessed, features extracted, classified and compared with well trained system. A very good segmentation is needed to select adequate threshold of gray level to extract hand from background .i.e. there is no part of hand should have background and background also shouldn’t have any part of hand. TECHNIQUES Shape based technique extract the pixels of the hand gesture that is within a shape, are taken into account to obtain the shape representation. Contour based technique is used to extract the number of fingers and the orientation, elongation of the hand. Contour extraction is the process of retrieving the location of pixel value of the edges of the detected objects from an image. For a binary image, a contour is a region (or blob) of white pixels. All blobs formed due to noise are small, thus the large contour is for further processing specifying it is contour of hand which will be the Region of Interest (ROI). Copyright to IJAREEIE 10.15662/ijareeie.2015.0404106 2500 ISSN (Print) : 2320 – 3765 ISSN (Online): 2278 – 8875 International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering ( An I SO 3 2 9 7 : 2 0 0 7 Ce rt ifie d Org a n iz a t ion ) Vol. 4 , I ssue 4 , Apr il 2 0 1 5 Figure 3 Region of Interest (ROI) Feature extraction In this work shape descriptors have been handled which are used when a region based analysis of the object is performed. In region based techniques, all the pixels within a shape are taken into account to obtain the shape representation. Common region based methods i.e ROI in figure 3, use moment descriptors to describe the shape. Because moments combine information across an entire object rather than providing information just at a single boundary point, they capture some of the global properties missing from many pure contour-based representations: overall orientation, elongation, etc. Solidity, Perimeter, Convex area, Major axis length, Minor axis length, Eccentricity, Orientation are some of the shape descriptors used as features in this work. These shape descriptors are more robust to noise and distortions. Regionbased analysis is invariant to translation, rotation and scale. 1) Solidity: Scalar specifying the proportion of the pixels in the convex hull that are also in the region. In simple terms density in mass per unit volume. But in two dimensional image objects this can be defined as the ratio between the area and convex area of the same object: 2) Perimeter: Perimeter is an important feature of an object. Contour based features which ignore the interior of a shape depend on finding the perimeter or boundary points of the object 3) Convex Area: Convex Image is a binary Image (logical) that specifies the convex hull, with all pixels within the hull filled in (i.e., set to on). 4) Eccentricity: Eccentricity is the ratio between the lengths of the short axis to the long axis as defined in the following equation. = h / h 5) Major Axis Length: Scalar specifying the length (in pixels) of the major axis of the ellipse that has the same normalized second central moments as the region. This property is supported only for 2-D input label matrices. 6) Minor Axis Length: Scalar specifying the length (in pixels) of the minor axis of the ellipse that has the same normalized second central moments as the region. This property is supported only for 2-D input label matrices. 7) Orientation: Scalar specifying the angle (in degrees ranging from -90 to 90 degrees) between the x-axis and the major axis of the ellipse that has the same second moments as the region. This property is supported only for 2-D input label matrices. Copyright to IJAREEIE 10.15662/ijareeie.2015.0404106 2501 ISSN (Print) : 2320 – 3765 ISSN (Online): 2278 – 8875 International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering ( An I SO 3 2 9 7 : 2 0 0 7 Ce rt ifie d Org a n iz a t ion ) Vol. 4 , I ssue 4 , Apr il 2 0 1 5 V. WORKING OF THE ALGORITHM A. Classification In pattern recognition, (k-NN) is a method for classifying objects based on closest training examples in the feature space. K-nn selection, another important problem we should take into account is how to choose a suitable K for this algorithm. K-NN is a type of instance-based learning. The k-NN algorithm is amongst the simplest of all machine learning algorithms: an object is classified by a majority vote of its neighbours, with the object being assigned to the class most common amongst its k nearest neighbours. If k = 1, then the object is simply assigned to the class of its nearest neighbour. K is small and positive integer. Larger values of k reduce the effect of noise on the classification. Choosing an appropriate K is essential to make the classification more successful. Distance is a key word in this algorithm, each object in the space is represented by position vectors in a multidimensional feature space. It is usual to use the Euclidean distance to calculate distance between two vector positions in the multidimensional space. B. PCNN Pulse-coupled networks or pulse-coupled neural networks (PCNNs) are neural models proposed by modelling cat visual cortex and developed for the high performance image processing. A PCNN is a two-dimensional neural network. Each neuron in the network corresponds to one pixel in an input image, receiving its corresponding pixel’s colour information (e.g. intensity) as an external stimulus. Each neuron also connects with its neighbouring neurons, receiving local stimuli from them. The external and local stimuli are combined in an internal activation system, which accumulates the stimuli until it exceeds a dynamic threshold, resulting in a pulse output. Through iterative computation, PCNN neurons produce temporal series of pulse outputs. The temporal series of pulse outputs contain information of input images and can be utilized for various image processing applications, such as image segmentation and feature generation. Compared with conventional image processing means, PCNNs have several significant merits, including robustness against noise, independence of geometric variations in input patterns, capability of bridging minor intensity variations in input patterns. Our classifier is basically a multilayer perceptron (MLP). The neural architecture consists of one input layer, one hidden layer and one output neuron. The input layer contains a number of inputs equal to the samples in the imaginary part of the DFT signal. Then, a hidden layer has an extension of about 10 to 20% of the input layer. Because of the specific tasks used to test the system, the output layer contained only one neuron (target detection). An output value of 1 is equivalent to target detection whereas a value of 0 means no target detection. A standard back propagation algorithm is used for supervised training. VI. RESULTS AND DISCUSSION The hand gesture recognition is done using MATLAB. Geometrical features like Solidity, perimeter, convex hull, area, major axis length, minor axis length, eccentricity, orientation are have been extracted by the shape based technique. Hand gesture recognized in real time process using webcam and translated in two languages. 1.Getting real time image from webcam Copyright to IJAREEIE 10.15662/ijareeie.2015.0404106 2502 ISSN (Print) : 2320 – 3765 ISSN (Online): 2278 – 8875 International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering ( An I SO 3 2 9 7 : 2 0 0 7 Ce rt ifie d Org a n iz a t ion ) Vol. 4 , I ssue 4 , Apr il 2 0 1 5 2. Capturing the image from the web camera in real time. 3. The image is converted and loaded for the sign conversion and the sign is detected in English, Tamil and Hindi. 4. Similarly the process is done for the continuous word recognition. 4. Histogram Plot for a gesture Copyright to IJAREEIE 10.15662/ijareeie.2015.0404106 2503 ISSN (Print) : 2320 – 3765 ISSN (Online): 2278 – 8875 International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering ( An I SO 3 2 9 7 : 2 0 0 7 Ce rt ifie d Org a n iz a t ion ) Vol. 4 , I ssue 4 , Apr il 2 0 1 5 REFERENCES [1] MAGengyu, Linguini, “Operational Gesture Segmentation and Recognition”, Tsinghu Science and Technology, April 2003, 8(2): 163173 [2] Kasprzak, W., Wilkowski, A. and Czapnik, K. Hand Gesture Recognition based on Free-Form Contours and Probabilistic Inference. Int. J. of Applied Math. and Comp. Sc. 22(2) (2012), 437- 448 [3] Laura Dopier, Angelo M. Sabatini, Senior Member, IEEE, and Paolo Dario, Fellow, “A Survey of Glove-Based System sands Their Applications”IEEE Transaction on system man and cybernetics-PART C: APPLICATIONS AND REVIEWS, VOL. 38, NO. 4, JULY 2008 Aliaa A. A.Youssif, AmalElsayedAboutabl, HebaHamdyAli“Arabic Sign Language (ArSL) Recognition System Using HMM” (IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 2, No. 11, 2011 J. Liu and Y. H. Yang, “Multiresolution Color Image Segmentation”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 16, No. 7, pp. 689-700, 1994. Praveenkumar SHavalagi, ShruthiUrfNivedita, “The Amazing digital glove gives voice to voiceless”, International Journal of Advances in Engineering &Technology,Vol. 6, Issue 1, pp. 471-480, Mar. 2013 Jalab, H. A., Static Hand Gesture Recognition for Human Computer Interaction. Information Technology J. 11(09) (2012), 1265 - 1271 [4] [5] [6] [7] Copyright to IJAREEIE 10.15662/ijareeie.2015.0404106 2504