Smart Touch Phones Blind Assistant System

American Journal of Systems and Software, 2014, Vol. 2, No. 3, 72-80 Available online at http://pubs.sciepub.com/ajss/2/3/4 © Science and Education Publishing DOI:10.12691/ajss-2-3-4 Smart Touch Phones Blind Assistant System Hussam Elbehiery, M. S. Abdel-Wahab Faculty of Information Technology, Misr University for Science and Technology (MUST), Cairo, Egypt *Corresponding author: [email protected] Received March 12, 2014; Revised June 27, 2014; Accepted July 03, 2014 Abstract The introduced system including the hardware and software actually allows phones to recognize and audibly identify places, orientation and give the ability to route in the close places almost instantly. Smartphones will become an incredible and depending on the cost of the software potentially affordable aid for blind or impaired individuals. In the past decade there has been a huge leap in technology and technological advancements but unfortunately most of these technological advancements failed to consider an important part of the community which is people with visual disabilities that are also known as visually impaired individuals. Still there are many aspects of the digital world that shall be customized by them. In this work we introduce a mobile application that will allow a visually impaired individual to use a touch screen smartphone without any human assistance, In addition to converting this smartphone into a tool used by the visually impaired individual allowing him to walk freely in the outdoor environment, go from one place to another and overcome several obstacles that he might face while being outside. Keywords: blind assistant, smart phones, GPS, RFID Cite This Article: Hussam Elbehiery, and M. S. Abdel-Wahab, “Smart Touch Phones Blind Assistant System.” American Journal of Systems and Software, vol. 2, no. 3 (2014): 72-80. doi: 10.12691/ajss-2-3-4. 1. Introduction Cell phones have become great tools for providing security and enhancing the lives with better communication. Unfortunately, it is impossible for a blind person to use cell phones. Luckily there are some devices in the market that can eliminate some of the barriers. [4,6] Many new applications are developed to make the technology usable for everyone, even if the user is blind or having some sort of visual impairment. Blind people will soon be able to write a text message without seeing the screen. This facility will start the new era of smart phones created for the blind. Today, there is a number of software available in the market using which blind people can operate different devices and applications [5,9]. These applications especially designed for blind people, who have to purchase an expensive keyboard to be able to type with no-button Smartphones. Smartphones applications are getting more useful for people with disabilities. Using this new application, users can even scan labels in grocery or retail stores using their smartphone camera. Once the item is scanned, the phone will not only tell users if the price is right, but also tell what the item is and read the information provided on the nutrition label [6,7]. If a smartphone has a GPS locator, it can provide directions to the blind or visually impaired person. All smartphones with this imbedded software can use these applications, but the phone must have a built-in camera. The most popular phones on which this assistive technology app can be used are the iPhone and Google Android OS phones. These phones including the iPhone can be purchased at discounted rates in the retailers [1,2]. 2. Visually Foundations Impaired Society and Total blindness is the inability to tell light from dark, or the total inability to see. Visual impairment or low vision is a severe reduction in vision that cannot be corrected with standard glasses or contact lenses and reduces a person's ability to function at certain or all tasks. Legal blindness (which is actually a severe visual impairment) refers to a best-corrected central vision of 20/200 or worse in the better eye or a visual acuity of better than 20/200 but with a visual field no greater than 20° e.g., side vision that is so reduced that it appears as if the person is looking through a tunnel [8]. Right now there are nearly 2 million people in Egypt who have significant sight loss. [9] The fact is there are a lot of people who suffer from visual impairment not just in Egypt but around the universe and most of these people need assistance in accessing the digital world that we are now in. Along the years a lot of people have established foundations in order to provide assistance to the visually impaired some of them provide moral and financial support while such as foundations like Visionary others provide other types of support like a foundation called Vision Rehabilitation Services (VRS) is dedicated to assisting individuals who are blind or visually impaired so they may function independently in all of their environments [8]. 73 American Journal of Systems and Software There are many Egyptian foundations working in the field of the visually impairing like: 1. The Development Association for Empowering Special Needs (DAESN) under Partner donors: ORASCOM Construction Industries (OCI), MITSUBISHI Egypt, and The Egyptian Ministry of Communications [26]. 2. SAWIRIS Foundation for Social Development supported by: ORASCOM Telecom Holding Company (OTHC), Integration Program for Persons with Social Needs (IPPSN) [27]. 3. ALNOOR MAGRABI Foundation [28]. 3. Basic Idea and Goals This work is for the basic concept in enabling visually impaired individual to use a touch screen smart phone with no human assistance and be able to use many of its features like calling, GPS for the service of selflocalization and sending emergency message to an already programed number. In addition, to enable the visually impaired to have the ability to use the smart phone to cross the street and change the traffic light signal. So, the main goals will be to: Enhance the graphical user interface of the Smart Phone, the touch methodology, the GPS to pronounce the user’s location and to guide the visually impaired to a known as well as unknown location, to eliminate the visual options, and changing the traffic light signal using smartphone. 4. System Methodology Enable launching application: Since the application is designed for visually impaired individuals, the process of launching the application should be done as easy as possible and using the few buttons that can be sensed by the user. Therefore, the menu button is used to launch the application. Enhance the graphical user interface of the Smart Phone: Since the application is designed for the individuals who find some difficulties in using the touch screen phone, so some changes in the interface of the mobile phone expressed by three major buttons that occupy the entire screen and each button lead to specific application. Enhancing touch methodology: the standard form in touch screen phones is that the first click on the icon provides access to the chosen application but here in the introduced application there will be a change for that methodology by using the first touch of a certain button to initiate an audio sound telling the user where he is standing and second touch will launch application. Enhancing the interface of navigation system using GPS: GPS is known for being useful to many of its users but this is mostly the case when using it in case of driving but not so much when trying to use it in the state of walking as it does not contain names of a lot of streets and mostly side streets also to be used by a visually challenged individual its merely an fictional idea as it depends mostly on vision and also in case of audible commands. There is the problem of providing multiple choices that the user has to see to choose from. Another problem is that when using GPS at the first launch the application, which shows the user destination without pronouncing it. This problem was not a function in primary suggestion, so in attempting to solve all these encountered problems, there was the following plan which includes separate programs: [4] Enabling the GPS to pronounce the user’s location: The regular GPS shows the user his location only without speaking. We attempt to solve this problem by entering coordinates into Google maps and using separate software to pronounce the location. Guide the visually impaired to a known location: Also using Google maps in order to get accurate directions. Using Test-To-Speech software (TTS) to pronounce these directions to the visually impaired. Using software enabling taking the English directions from Google maps and pronounces them to the visually impaired in Arabic [3]. Guide the visually impaired to an unknown location: GPS is only functional to a visually impaired individual meaning that it depends mostly on a person’s ability to view the map. So, to solve this problem in case of an already programmed location while still working on new locations that was not already programmed in. Eliminating the function of providing several visual options; since the user who will always use the GPS state of walking only. A filter of a certain block radius could be used by calculating the distance that an average person can walk and so eliminating all the possibilities that does not fit those criteria as it is un sensible. That to provide the user with a result that will require by him to walk for over an hour or two hours continuously, and so by using this filter the program automatically eliminates the results that does not fit such criteria and only provide the user with the most sensible options [4]. Establish a connection between the Smartphone and traffic light; this has been done for achieving two objectives the connection in both those cases will be achieved by putting an RFID reader on the traffic light signal and an RFID tag will be attached to the Smartphone [10]. The first objective arises from the fact that we are handling a case of a walking pedestrians across the street we have to consider the possibility of crossing the street and so we have to establish a method to enable the application to let the user know that he came within the range of a traffic light signal and whether or not it’s safe to cross using audible commands/instructions and in case it’s not safe to cross how long does he have to wait and when exactly to cross and as we mentioned earlier this will all be achieved by using the RFID tag and reader and establishing a connection between both the mobile phone and the traffic light system. Changing the traffic light Signal using Smart Phone; which is the second objective that we were aiming to achieve, as we mentioned earlier there is a method by which pedestrians can switch the traffic light signal using a button found in the pole holding the signal but the problem is that this advantage is by far impossible to be used by a visually impaired individual as it would be impossible for him/her to detect the presence of a traffic light signal let alone know that it contains a button and to press it on his own and so we were aiming to enable him American Journal of Systems and Software to dispose the use of the button and switch the traffic light signal using the RFID Tag that is read by the RFID Reader placed in the traffic light signal. Establish a calling application; as its well known the working with a touch screen phone making the calling process for a visually challenged individual very difficult and so in our application we enhance the typical call methodology: The application will open two buttons the first will be for calling an already saved number by voice command. The second button will be a dialer application that speaks the numbers while dialing. Establish a button that sends an emergency message; In case the user is lost or tired or in case of any other emergency then by clicking on the emergency message button: The application will get the current location of the visually impaired individual and send it to an already saved emergency contact number. 74 6.1. Blind Assistant Use Case Diagram A use case diagram at its simplest is a graphical representation of a user's interaction with the system and depicting the specifications of a use case. A use case diagram can portray the different types of users of a system and the diagram is typically used in conjunction with the textual use case and will often be accompanied by other types of diagrams as well. In this use case we have five actors that influence the application and are influenced by it: the visually impaired individual, the GPS application, TTS application, Google database and the mobile’s database as shown in Figure 1 [16]. The use case contains actors and also main functions that are performed by the application on such actors meaning it demonstrates how the application affects the actors and what they are capable to do with it for example we will find that using our application the visually impaired individual will be able to know his location and set a destination and listen to voice instructions on how to reach that destination. 5. RFID Technique Radio-frequency identification (RFID) is the use of a wireless non-contact system that uses radio-frequency electromagnetic fields to transfer data from a tag attached to an object, for the purposes of automatic identification and tracking. Some tags require no battery and are powered and read at short ranges via magnetic fields (electromagnetic induction). Others use a local power source and emit radio waves (electromagnetic radiation at radio frequencies). The tag contains electronically stored information, which may be read from up to several meters away. Unlike a bar code, the tag does not need to be within line of sight of the reader and may be embedded in the tracked object [10]. RFID tags are used in many industries. An RFID tag attached to an automobile during production can be used to track its progress through the assembly line. Pharmaceuticals can be tracked through warehouses. Livestock and pets may have tags injected, allowing positive identification of the animal. RFID systems typically come in three configurations. One is a Passive Reader Active Tag (PRAT) system that has a passive reader which only receives radio signals from active tags (battery operated, transmit only). The reception range of a PRAT system reader can be adjusted from 1-2,000 feet [10]. Thereby allowing for great flexibility in applications such as asset protection and supervision. Another configuration is an Active Reader Passive Tag (ARPT) system that has an active reader, which transmits interrogator signals and also receives authentication replies from passive tags. Finally, there is the Active Reader Active Tag (ARAT) system in which active tags are awoken with an interrogator signal from the active reader. A variation of this system could also use a Battery Assisted Passive (BAP) tag which acts like a passive tag but has a small battery to power the tag's return reporting signal [11]. 6. System Design & Implementation Figure 1. Blind Assistant System Use Case Diagram 6.2. Blind Assistant Sequence Diagram A sequence diagram in a Unified Modeling Language (UML) is a kind of interaction diagram that shows how processes operate with one another and in what order. It is a construct of a Message Sequence Chart. A sequence diagram shows object interactions arranged in time sequence. It depicts the objects and classes involved in the scenario and the sequence of messages exchanged between the objects needed to carry out the functionality of the scenario. Sequence Diagrams typically are associated with use case realizations in the logical view of the system under 75 American Journal of Systems and Software development. Sequence diagrams are sometimes called event diagrams, event scenarios, or timing diagrams. Figure 2, shows the Sequence Diagram of the emergency message function. It operates first the user touch the button once and the a voice message informs him that by clicking twice on the button he will launch the emergency message application and then if the user clicks a double click the application will take his location from the GPS module system in co-ordinates and convert it to a specific address and send this address to an emergency contact number that already save in the mobile’s database.[17] Figure 2. Blind Assistant System (Messaging) Sequence Diagram Figure 3 demonstrates how the Pin out function operates first the user touch the button once and the a voice message informs him that by clicking twice on the button he will launch the pin out application after launching this application the user a voice will inform the user of his exact location and the will have to choose whether the user wants to go to an already saved place in the database or to a new location after choosing using the same touch methodology that was demonstrated earlier and in either cases he will have voice instructions directing him on the easiest route to reach the destination which he had chosen earlier [18]. Also this sequence diagram illustrates how the application convert the text instructions provided by GPS with voice instructions so that the visually impaired would benefit from them this will be achieved using Google maps and a text to speech (TTS) software [19]. American Journal of Systems and Software 76 Figure 3. Blind Assistant System (Pin-out) Sequence Diagram Figure 4 illustrates the function that is also provided by the pin out which is letting the user know where the nearest traffic light is and whether or not it’s safe to cross the street this is achieved by using an RFID reader on the traffic light and an RFID tag on the mobile phone. Also this connection enables the visually impaired individual to switch the traffic light signal in case of an emergency or a difficult situation. This of course does not occur randomly but the traffic light signal provides the drivers with a few seconds notice before switching the signal and then informs the visually impaired individual that it is safe to cross the street. 77 American Journal of Systems and Software As we know the user of this mobile phone is a visually impaired individual and so it will be difficult for him to use the touch screen phone to make a telephone call and so using the same enhanced touch methodology we enable the user to choose whether he wants to choose an already saved number to call it or dial a new number in case of an already saved number the user will dial it using voice commands or dial a new number using a dialer that we already enhanced its touch methodology to enable the user to dial an unsaved phone number.[20] 6.3. Blind Assistant Implementation After launching the application we faced two other dilemmas; The first is the need to enhance the touch methodology to become compatible with the needs of a visually challenged individual meaning so a single touch on an icon announces the place while a double touch on an icon launches the feature and also announces to the user what feature or action took place as shown in Figure 6. The second dilemma was how to help the user in navigation by dividing the screen into not more than three buttons reducing the effort needed by the user to reach the needed button (feature) [21]. Figure 4. Blind Assistant System (Traffic Light) Sequence Diagram This next diagram shown in Figure 5 illustrates the call function provided by the application. Figure 6. Blind Assistant System GUI After launching the application the user will find three features: 1. Pin Out: Which will instantly launch a voice notification telling the user his exact location and asking him about his desired destination? 2. Calling: Which opens an interface with two buttons one for each feature which indicated in Figure 7. Figure 5. Blind Assistant System (Calling) Sequence Diagram Figure 7. Blind Assistant Calling Menu American Journal of Systems and Software So that the user would be able to either call a new number that is not saved in his phone memory (see Figure 8). 78 7. Blind Assistant System Functionality Steps We started off by constructing each one of these applications individually and then tying them all together in a major application we first started with the Pin out feature. 7.1. Pin Out Figure 8. Blind Assistant Dial UP GUI Or the blind man can search his contacts for the number he wants to dial (see Figure 9). Figure 9. Blind Assistant Contacts Menu 3. Emergency SMS: Enables the user to send an emergency message by clicking a button to be able to get help in case he is lost (see Figure 100). As anyone who has ever used the navigation systems based on GPS would know that not all the streets are mentioned especially if it is not a main street, side streets and alleys does not have the same significance in case of labeling them on GPS mainly because people who use either these features use it in the state of driving and so they won’t be in a dire need to know the names of all the side roads. But when providing the same service for a walking individual who is actually visually challenged then the name of the streets do count and so we decided to discard GPS and use Google maps instead in terms of figuring out where the user is and how to reach his destination. Another problem with GPS is its interface is designed to benefit only the visually abled community but it cannot be used by someone with a vision disability. And so instead we decided to use GPS and Google maps together and add a few enhancement of our own to match the needs of the user we are targeting. And so we started by creating the pin out application so that when launched a voice notification informs the user where he is and then asks him to set the destination of where he wants to go and by using the internet the application gets directions to how to get to his location and the voice states upon the user the proper steps he should make to reach such destination. The steps are narrated one by one using voice commands. Also in case the user reaches a traffic light signal we connected bot h the traffic light system and our application using a RFID tag which is attached to the mobile phone and a RFID reader that is attached to the traffic light system when the user enters the range of the signal of the RFID reader a buzzer that is attached to the traffic light signal is switched on and the user is notified when it is safe to cross the street and he will also be able to use the RFID tag to change the light of the traffic light signal to be able to cross the street [22]. 7.2. Calling Figure 10. Blind Assistant Emergency SMS Menu The second feature we worked on was a calling application that enables the user to call an either already registered number or a new number in case of an already registered number we decided to load all the mobile contacts to the application and have them read on the user using our voice notifications the user can either go forward to next contact or go backwards and when he has found the contact he wishes to call he can press the call button all of this is achieved through our enhanced touch and interface methodologies that we have explained earlier [23]. In case of wanting to dial a new number the user will press the second button which initiates a call dialer that we have enhanced to fit the needs of a visually impaired 79 American Journal of Systems and Software individual by increasing the size of the button, enhancing the touch methodology and enabling voice notifications [24]. 7.3. Emergency SMS The third feature was intended to help the visually impaired individual in case of an emergency so by pressing on send message button the application sends the users exact location to an already programed number the user is also able to change that number in case he wants to send the message to another number. In case the user’s location is unknown a message will still be sent to the emergency contact number but with the word unknown in its content [25]. Acknowledgements I would like to express my thanks to Misr University for Science and Technology – 6th October City - Cairo, Egypt for its affordable efforts in publishing this research paper also for providing necessary tools and kits and continuous support during the tests. References [1] [2] [3] 8. Hardware and Software Tools [4] 8.1. Hardware Tools In this research the hardware tools that have been used are: • Mobile Smart phone. • The traffic light signal. • Traffic control circuit. • RFID Reader. • RFID Tags. • Colored LEDs. • Buzzer. 8.2. Software tools In this research the software tools that have been used are: [10,12,13,14,15,25] • Eclipse, for java mobile applications. • Android SDK, Software Development Kit. • JDK, Java Development Kit. • Phone on Map and GPS Tracker, Global Positioning System Android applications [5] [6] [7] [8] [9] [10] [11] 9. Conclusion The introduced application is allow a visually impaired individual to use a touch screen smartphone without any human assistance, also converting the smartphone into a tool used by the visually impaired individual allowing him to walk freely in the outdoor environment. By using this application the visually impaired could walk in the streets and reach their desired destination with no human assistance using the navigation software based GPS. The user interface for the introduced application depends on the easy way to access to functions through little buttons and simple menus for the visually impaired supported by voice commands. The main outline here is what’s better than being on the verge of technological advancement is being able to share such advancement with others so that everyone could benefit from it and also you never know who might add something new to this advancement and push an entire generation forward and so in order to be on top of the world you have to have some assistance from everyone around you as no one can achieve his goals without taking a little help and support from others regardless of their nature, health state or culture. [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] Boyina. S. Rao, “Indoor navigation system for visually impaired person using GPS,” April, 2001. Jack M. Loomis, “Navigation System for the Blind: Auditory Display Modes and Guidance,” June, 2007. Andreas Hub J. D, “Design and Development of an Indoor Navigation and Object Identification System for the Blind,” July, 2011. A. Sakhare, “Smart stick for Blind: Obstacle Detection, Artificial Vision and Real-time assistance via GPS,” 2nd National Conference on Information and Communication Technology (NCICT), January, 2012. Shamsi, M. A., Sharjah, and Jeedella J., “Blind assistant navigation system, Biomedical Engineering,” (MECBME), 2011 1st Middle East Conference 21-24 Feb. 2011. Enas Elbarbary, Hussam Elbehiery, Abdelhalim Zekry, and Adel Elhennawy, “Portable Blind Assistant,” ICENCO 2009 5th International Computer Engineering Conference Cairo University, December 27-28, 2009 Cairo, Egypt. Cole DA, Viola MV, Yow, L, Humayun, MS, Weiland, JD, Greenberg R and Greenbaum E, "The U.S. Department of Energy’s Artificial Sight Program", 10th Annual Conference of the International FES Society, Department of Energy Medical Sciences Division’s Program on Artificial Sight, Montreal, Canada, July 2005. Diane M. Szaflarski, "How We See: The First Steps of Human Vision", Medical Report, Access Excellence Classic Collection, National Health Museum, http://www.accessexcellence.org/AE/AEC/CC/vision_background .php, January 2002. Medical Report, Teaching Hospital of Harvard Medical School, Massachusetts Eye and Ear Infirmary, Boston, "Why Do People Become Blind?", http://www.meei.harvard.edu/patient/blindness.php, 24th November 2008. V. Daniel Hunt, Albert Puglia, and Mike Puglia, “RFID: A Guide to Radio Frequency Identification,” Wiley-Blackwell publishing, ISBN-10: 0470107642, April 2007. BS Choi., “Mobile robot localization scheme based on RFID and sonar fusion system,” Industrial Electronics, IOSR Journal of Computer Engineering (IOSR-JCE), ISIE 2009. Nikolaos bourbakis, “Sensing Surrounding 3-D Space for Navigation of the Blind,” IEEE Engineering in Medicine and Biology Magazine, Feb. 2008. John C. Russ, "The Image Processing Handbook", Third Edition, CRC Press, CRC Press LLC, IT Knowledge, Earth Web, Inc., 7th January 1998. Rafael C. Gonzalez, Richard E. Woods, "Digital Image Processing", Second Edition, Prentice-Hall, Inc., New Jersey, U.S.A., 2002. Mark S. Nixon, Alberto S. Aguado, "Feature Extraction and Image Processing", First Edition, British Library Cataloguing in Publication Data, UK, 2002. Kathy Sierra and Bert Bates, “Head First Java; covers Java 5.0,” A Brain Friendly Guide, O’REILLY Publication, 2nd Edition, ISBN10: 0596009208, February 9, 2005. Bruce Eckel, “Thinking in Java,” President, Mind View, Inc., Prentice Hall, 4th Edition, USA, ISBN-10: 0131872486, February 2006. Joshua Bloch, “Effective Java,” Addison-Wisely Publications, 2nd Edition, USA, ISBN-10: 0321356683, May 2008. The Java Tutorials from Sun the gold standard for learning Java, and it’s free, http://java.sun.com/docs/books/tutorial/java/index.html, Java Ranch Big Moose Saloon web site http://saloon.javaranch.com/cgibin/ubb/ultimatebb.cgi?category=1 The Java Developers Almanac 1.4 http://www.exampledepot.com. American Journal of Systems and Software [23] Robert Liguori and Patricia Liguori, “Java Pocket Guide,” O'Reilly Media, Inc., 2nd Edition, ISBN-13: 978-0-596-51419-8, March 2008. [24] Paul Deitel, Harvey Deitel and Abbey Deitel, “Android How to Program,” Prentice Hall, ISBN-10: 0-13-299054-7, USA, February 08, 2012. [25] James Steele and Nelson To, “The Android Developer's Cookbook: Building Applications with the Android SDK: Building Applications with the Android SDK,” Addison-Wisely [26] [27] [28] [29] 80 Publications, 1st Edition, USA, ISBN-10: 0321741234, October 2010. Neil Smyth, “Android 4.2 App Development Essentials,” Payload Media, ISBN-10: B00DPY991O, USA, June 2013. The Development Association for Empowering Special Needs (DAESN), www.daesn.org, www.daesnegypt.com. Sawiris Foundation for Social Development http://www.sawirisfoundation.org/. Alnoor Magrabi Foundation, http://magrabi.org/