Mobile Application for the Blind and Their Family

TEM Journal. Volume 10, Issue 3, Pages 1039‐1044, ISSN 2217‐8309, DOI: 10.18421/TEM103-05, August 2021. Mobile Application for the Blind and Their Family Harco Leslie Hendric Spits Warnars 1, Nicholas Nicholas 2, Muhammad Raihan 2, Arief Ramadhan 1, Teddy Mantoro 3, Wan Adilah Wan Adnan 4 1 Computer Science Department, BINUS Graduate Program – Doctor of Computer Science, Bina Nusantara University, Jakarta, 11480, Indonesia 2 Information Systems Department, School of Information Systems, Bina Nusantara University, Jakarta, 11480, Indonesia 3 Faculty of Engineering and Technology, Sampoerna University, Jakarta, Indonesia 12780 4 Faculty of Computer and Mathematical Sciences, Universiti Teknologi Mara (UiTM), Selangor, 40450, Malaysia Abstract – Many people want to use gadgets, but they cannot due to some reasons. One of the reasons is the person who is visually impaired and some children are challenged by birth. To overcome this problem, we have designed a mobile application to make communication easier between the blind and their family or their friends. In this project, we want to utilize some of the technology that already exists to develop a mobile application that can be used by the blind to communicate easily to their family. Keywords – Mobile application, Information Systems, Mobile application for disabled family. 1. Introduction Along with the days, humans at this point are required to be ready to adapt to the event of technical knowledge within the world. 2. Previous and Current Research DOI: 10.18421/TEM103-05 https://doi.org/10.18421/TEM103-05 Corresponding author: Harco Leslie Hendric Spits Warnars, Computer Science Department, BINUS Graduate Program – Doctor of Computer Science, Bina Nusantara University, Jakarta, Indonesia 11480. Email: [email protected] Received: Revised: Accepted: Published: 23 March 2021. 06 July 2021. 12 July 2021. 27 August 2021. © 2021 Harco Leslie Hendric Spits Warnars et al; published by UIKTEN. This work is licensed under the Creative Commons Attribution‐NonCommercial‐NoDerivs 4.0 License. The article is published www.temjournal.com with There are numerous blind people in the world. This is often becoming one of the everyday issues where blind people find it difficult to induce information from the web and communicate via the web. [1] says that in Indonesia, many blind people still do not get access to education which is often caused by the shortage of technology that Indonesia currently has. To support this problem, we would make a special discussion forum for people with visual defects. The technology that we apply is technology-supported mobile applications, so that blind people can get information quickly and may communicate with fellow blind people. The appliance technology that we use to assist people with visual defects in making words is Google Talkback. Google Talkback is an application that creates it easy for blind sufferers to interact with their smartphones. Google Talkback uses sound and vibration to assist blind people so that they know what is on their smartphone screen. Open TEM Journal – Volume 10 / Number 3 / 2021. Access at Before this research was made, there are several similar studies about a mobile application that can be used for visually impaired people. Ranjan et al. [2] designed an android-based application for people with visual defects. It is regularly used in teaching on a large scale for teaching blind people efficiently. The application used a text-to-speech module (TTS) to get input and provide users the output. It also makes use of a touch display and vibration motors for consumer interaction. Another Android-based application [3] is made for deaf and blind communications, which was later named Voisee. Compared to [2], this app is meant to communicate between 2 disabled individuals, either the deaf or the blind. It is developed using Eclipse IDE. Two Java libraries are used to make this application, such as tts.speech.SpeechToText and 1039 TEM Journal. Volume 10, Issue 3, Pages 1039‐1044, ISSN 2217‐8309, DOI: 10.18421/TEM103‐05, August 2021. tts.speech.RecognizerIntent. Both of these libraries allow forming a custom voice command to use the app. The fundamental features in this app are to create, reply, send, and forward messages. Replying to messages can be done by typing the messages or using a voice command with Speech-to-text (STT) engine. Reading messages can be done using Text-toSpeech (TTS) engine. The app can be controlled using both TTS and STT engines for the blind. Related to [3], there is research that focused on how the blind can access graphical-based content on Facebook. [4] said blind people are very interested in participating in some of the activities on Facebook like posting photos or giving feedback. Recently, Facebook has made a forward leap in beating this issue by utilizing a programmed image caption tool that provides a list of various objects and schemes in the photo. [5] said the increasing popularity of visual content on social media is very important for blind people to interact with the content. Right now, technology offers a blind tool that is used to identify and understand visual content, such as image recognition, graphic touches, and crowd systems. [6] said that user satisfaction is one of the crucial things in studying the use of technology and information technology is evaluated through customer satisfaction to measure the success rate of the system. [7] reports a usability study of Open Art mobile application conducted at the Gallery of the Twentieth Century Polish Art of National Museum in Kraków, Poland from 13 to 16 December 2016. This study observes thirteen people, which includes: one blind, five partly argus-eyed, three deaf, and hearing difficulties yet as four argus-eyed / hearing participants. The results show that the app was well received with a positive influence on the museum visiting experience for the blind. There is a wearable technology made by Choudhary et al. [8] which is a translation glove that translates the Braille alphabet. This technology can help deaf-blind people to communicate and interact with each other. The glove allows users to send messages easily through the use of capacitive touch sensors. The wearer can receive and recognize incoming messages with a help of feedback from vibration motors on the glove. It opens communication opportunities for information exchange that were not available to deaf-blind individuals. In [9], Jawale et al. developed an aid for blind people based on ultrasonic navigation. This method was made to help blind people to be able to walk easily in urban areas. It had been developed using the ultrasonic sensor, GPS, Bluetooth technology that was bridged through Arduino. There are two systems named detection mode and fixed mode. This system uses Bluetooth to send instructions or information 1040 about the objects that are around it. The instructions sent by Bluetooth are within a variety of sounds. This method is meant to make the blind ready to walk safely in urban areas. Still talking about aid for blind people, [10] created ultrasonic smart glasses for the cheap blind. The glasses’ main function is obstacle detection using ultrasonic sensors. Also, this device can alert the user in a form of sound using the buzzer. The device is running on an Arduino NANO microcontroller that runs the code to take the distance between the user and obstacles from the sensor. There is a different invention with similar functions on [11]. Yahaya et al. made an Ultrasonic Walking Stick using a few similar components from [10], which is the Arduino UNO board with an ATmega328 microcontroller. The main functions of [11] are obstacle detection and pit detection. Awad et al. developed an android mobile application to assist blind people. [12] has many useful features like light detection, color detection, object detection, and banknote recognition. [12] developed using a different platform with [3] although both apps can assist blind people. That is Android Studio IDE to write the source code, PHP and MySQL database, OpenCV for android for image recognition and manipulation, CraftAR library for the on-device image recognition, and lastly CNNdroid to learn useful feature representation directly from image data using the trained Convolutional Neural Networks (CNN) which is popular for deep learning architecture that for running. There is a mobile application to assist blind people by identifying objects with a help of Augmented Reality. [13] was created using Unity 3D and Vuforia SDK. The app runs on an Android smartphone that encompasses an operation to spot the article and provides info within the variety of sound which can be used by those who cannot distinguish objects with identical form but completely different functions, which then can be referred to as blind people. To access this application, the Google Assistant is used by saying ‘Hey Google’ and ‘open blind reader’. The app will be directly opened after google assistant recognized the voice. Users can directly use the camera to scan the required object and provides the data within the type of sound. The research and development do not stop here, but there is also a mobile phone for the blind [14] that is based on the GSM technology-based smartphone. The phone uses a braille keyboard that helps the blind to grasp numbers and letters in choosing keys. The system uses an ATMEG16A microcontroller to receive the message. The message is received by the GSM module, and then the microcontroller converts it into the braille language. TEM Journal – Volume 10 / Number 3 / 2021. TEM Journal. Volume 10, Issue 3, Pages 1039‐1044, ISSN 2217‐8309, DOI: 10.18421/TEM103‐05, August 2021. The Learning Management System (LMS) is a kind of online learning platform to access learning schedules, assessments, learning history, accessing teaching materials, and some online learning features, such as online discussions [15]. Forums, chat, and video conferences which is useful to support the learning process so that users understand certain material. The user form is the core of the discussion forum [16]. Without user discussion, nothing will happen, there are no questions and there is no answer. User participation is the most important thing in web-based discussion forums. Several studies say that by studying the behavior of forum users they can understand what are things that can attract user attention and what motivates them to use web-based forums. The application that we propose is a discussion forum that later can be used for the blind and, it is a tool to communicate between the sighted (normal) people and the blind. There is a mobile game application for the blind [17] that contains questions on a touch-screen device. The study was originally made to collect information about interaction difficulties faced by the blind and to analyze possible solutions. The solution given to this study is a mobile game designed with eight exercises. Those exercises have many kinds of questions. Other research made an in-context Q&A application [18] to bridge the gap for the blind using smartphones. The authors have completed many workshops involving 42 blind participants. Hint Me! the software proposed in [18] is a human-powered service that allows a blind user to get in-app help through posing questions or searching formerly answered questions on a shared knowledge-base. [19] developed a mobile game for interactive learning. The purpose of this development is to assist blind children to enhance their interest and enthusiasm for gaining knowledge of and enlarge their understanding of study material. The software design consists of one entry page and 7 menus, which are select, adventure, battle, training, ranking, guide, and settings. All the instructions were delivered in a form of audio or text-to-speech. Similarly, Bouraoui et al. made [20] to help Tunisian blind children which includes simple interfaces to teach blind students Braille alphabets, both in Arabic and French. There are numerous features of this application, like learning activity, exploring activity, checking out user’s understanding, and memory-like game. The app was tested in a Tunisian specialized school with 8 students aged between 6 and 8 years old. In a study conducted by Bülbül et al. [21], they developed a smartphone application to assist visually impaired students to study physics. With this application, teachers who do not have laboratories that are equipped with various tools at their school TEM Journal – Volume 10 / Number 3 / 2021. could have a chance to experiment. This study found some apps in the Google play store for physics, but no application can fully or partially support the blind. There are adaptations for usage using current applications and sensors in a smartphone. However, blind learners need some tactile, audio support, or help from a sighted friend. Two of the foremost popular tools for blind learners are just employing a smartphone or using other tools like a selfie stick. 3. Proposed Idea The system was developed using an Android application and was designed using the UML notation. We created the Use-case Diagram which represents user activities in the system and Class Diagram that represents the Database structure which includes tables and the relations between them in a database. Both of these diagrams are used as a figure of how the application system works. We also created a user interface mockup using Axure rp 9. Figure 1. Use Case Diagram of the Proposed System We have designed a use case diagram that aims to find out what access the user has in the figure, described in Figure 1. From the picture we can see that the user can create a new account as a user, log in to the app both as a user or ophthalmologist, access the discussion forum which includes creating a thread, commenting on it, and delete the post. Then, the user can use a chat feature to communicate with the expert, who is mainly an ophthalmologist about their health problems, and later users can rate the ophthalmologist after done chatting. Lastly, the user can update their profile and access information that is provided by the application developer. Besides, we have made class diagrams as seen in Figure 2 that explains how the system runs and what 1041 TEM Journal. Volume 10, Issue 3, Pages 1039‐1044, ISSN 2217‐8309, DOI: 10.18421/TEM103‐05, August 2021. rights are obtained by each class for our application where there are seven classes such as user, chat, forumdiscussion, comments, ophthalmologist, rating, and chatdetail. Users have access to access chat with the expert (doctor) and the detailed chatting will be saved in class chatdetail. The rating feature is optional which can be used later by the user to rate the expert (doctor) with whom he chatted and will be saved in class rating. Moreover, Users can create forum threads and send one or more comments on every thread that other users made. In Figure 3a, it is seen that this page is used to log into account of our application. This page is shown when we opened the app for the first time. There are two buttons, one for login but the user has to input the registered email address and password. The other button is redirecting the user to the sign-up page. After the user clicks the sign-up button on the Login page, Figure 3b appears. In this section, the user is asked to fill in personal data in the form of account data and user personal data. (a) (b) Figure 4. (a) Home Page, (b) Discussion forum page Figure 2. Class Diagram of the Proposed System As mentioned before, we made a user interface design using the Axure rp 9 app. We have made a UI design based on the activity that can be done while using this app, which is shown in the pictures below: (a) (b) Figure 3. (a) Login Page, (b) Sign-up page 1042 In Figure 4a, there is a home menu in which the user can receive news or information relating to the world as well as important messages from the admin and staff working on the application. On the next page, which is a discussion forum as shown in Figure 4b, the user will choose what categories the user wants to dig up for information or in what section the user wants to participate in speaking in the forum. (a) (b) Figure 5. (a) Chat Page, (b) Rating page TEM Journal – Volume 10 / Number 3 / 2021. TEM Journal. Volume 10, Issue 3, Pages 1039‐1044, ISSN 2217‐8309, DOI: 10.18421/TEM103‐05, August 2021. Figure 5a shows the chat page in which the user can have a consultation with an expert or with other users. If the user has done talking with an expert, the user will be asked to rate the expert that is shown in Figure 5b, but this is optional or not mandatory. If this project succeeds in getting enthusiasm from our users we will develop an e-commerce website for blind sufferers so that those blind sufferers can sell and that their productivity increases. This is good because this project can improve the economy, especially in the Indonesian region. Moreover, some Artificial Intelligence (AI) technology will be applied such as implementation Sentiment Analysis, where communication in the forum will be mined and recognize as positive, neutral, and negative. Other AI technology such as chatbot or automatic dialogue systems will be applied to deal with users' questions regarding any issue in the systems or blind people that automatically will be answered by the systems. Whenever there are new questions where the systems can not answer then the systems will accommodate the admin to label with the correct answer. Acknowledgments (a) (b) Figure 6. (a) Update Profile Page, (b) Change Password page Figure 6 shows the Update Profile page. On this page, the user can update their profile, which includes their name, email, username, phone number, date of birth, address, and lastly password. For passwords, there is a separate view because, in order to change a password, the user has to input their current password, new password and repeat the new password to confirm. When the user has finished updating their profile, the user can click the confirm and save changes button. 4. Conclusion This project that we are doing concludes that we hope that this project can be carried out to help blind people to communicate with each other and share information, especially communicating with their immediate family. So that it can build a community in the hope of helping fellow blind people to be able to interact with the world and besides that, it makes it easier for families who have visually impaired family members to understand blind family members by sharing their experiences. others and use it as a lesson in understanding visually impaired family members. In order to make this project a success we as a team had to choose a technology that fits the current environment and for its future development, besides that culture becomes the most difficult issue in this project because everyone has different habits. TEM Journal – Volume 10 / Number 3 / 2021. This work is supported by the Research and Technology Transfer Office, Bina Nusantara University as a part of Bina Nusantara University’s International Research Grant contract number: No.017/VR.RTT/III/2021 contract date: 22 March 2021. References [1]. Ball, H. A., McWhirter, L., Ballard, C., Bhome, R., Blackburn, D. J., Edwards, M. J., ... & Carson, A. J. (2020). Functional cognitive disorder: dementia's blind spot. Brain: a journal of neurology, 143(10), 2895-2903. [2]. Ranjan, A., & Navamani, T. M. (2019). Androidbased blind learning application. Ambient Communications and Computer Systems. AISC, 904, 247-255. [3]. Landicho, J. A. (2016). Voisee Communicator: An Android Mobile Application for Hearing-impaired and Blind Communications. Int. J. Interact. Mob. Technol., 10(4), 26-31. [4]. MacLeod, H., Bennett, C. L., Morris, M. R., & Cutrell, E. (2017, May). Understanding blind people's experiences with computer-generated captions of social media images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 5988-5999). [5]. Voykinska, V., Azenkot, S., Wu, S., & Leshed, G. (2016, February). How blind people interact with visual content on social networking services. In Proceedings of the 19th acm conference on computer-supported cooperative work & social computing (pp. 1584-1595). [6]. Isaac, O., Abdullah, Z., Ramayah, T., & Mutahar, A. M. (2017). Internet usage, user satisfaction, tasktechnology fit, and performance impact among public sector employees in Yemen. Technology, 34(3), 210241. 1043 TEM Journal. Volume 10, Issue 3, Pages 1039‐1044, ISSN 2217‐8309, DOI: 10.18421/TEM103‐05, August 2021. [7]. Jankowska, A., Szarkowska, A., Krejtz, K., Fidyka, A., Kowalski, J., & Wichrowski, M. (2017). Smartphone app as a museum guide. Testing the Open Art application with blind, deaf, and sighted users. Rivista internazionale di tecnica della traduzione, 19, 113-130. [8]. Choudhary, T., Kulkarni, S., & Reddy, P. (2015, January). A Braille-based mobile communication and translation glove for deaf-blind people. In 2015 international conference on pervasive computing (ICPC) (pp. 1-4). IEEE. [9]. Jawale, R. V., Kadam, M. V., Gaikawad, R. S., & Kondaka, L. S. (2017, September). Ultrasonic navigation based blind aid for the visually impaired. In 2017 IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI) (pp. 923-928). IEEE. [10]. Agarwal, R., Ladha, N., Agarwal, M., Majee, K. K., Das, A., Kumar, S., ... & Saha, H. N. (2017, October). Low cost ultrasonic smart glasses for blind. In 2017 8th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) (pp. 210-213). IEEE. [11]. Yahaya, S. A., Jilantikiri, L. J., Oyinloye, G. S., Zaccheus, E. J., Ajiboye, J. O., & Akande, K. A. (2019). Development of Obstacle and Pit-Detecting Ultrasonic Walking Stick for the Blind. FUOYE Journal of Engineering and Technology, 4(2). [12]. Awad, M., El Haddad, J., Khneisser, E., Mahmoud, T., Yaacoub, E., & Malli, M. (2018, April). Intelligent eye: A mobile application for assisting blind people. In 2018 IEEE Middle East and North Africa Communications Conference (MENACOMM) (pp. 16). IEEE. [13]. Mambu, J. Y., Anderson, E., Wahyudi, A., Keyeh, G., & Dajoh, B. (2019, August). Blind Reader: An Object Identification Mobile-based Application for the Blind using Augmented Reality Detection. In 2019 1st International Conference on Cybernetics and Intelligent System (ICORIS) (Vol. 1, pp. 138141). IEEE. 1044 [14]. Men, M. A., Devhare, M. M., & Mithare, M. S. (2018). Cell Phone for Visually Impaired people. Cell, 5(03), 14-20. [15]. Murad, D. F., Heryadi, Y., amad Isa, S. M., Budiharto, W., & Wijanarko, B. D. (2018, November). Text mining analysis in the log discussion forum for online learning recommendation systems. In 2018 International Seminar on Research of Information Technology and Intelligent Systems (ISRITI) (pp. 421-426). IEEE. [16]. Hoogeveen, D., Wang, L., Baldwin, T., & Verspoor, K. M. (2018). Web forum retrieval and text analytics: A survey. Foundations and Trends in Information Retrieval, 12(1), 1-163. [17]. Leporini, B., & Palmucci, E. (2018, July). Accessible Question Types on a Touch-Screen Device: The Case of a Mobile Game App for Blind People. In International Conference on Computers Helping People with Special Needs (pp. 262-269). Springer, Cham. [18]. Rodrigues, A., Montague, K., Nicolau, H., Guerreiro, J., & Guerreiro, T. (2017, October). In-context Q&A to support blind people using smartphones. In Proceedings of the 19th international ACM SIGACCESS conference on computers and accessibility (pp. 32-36). [19]. Sari, A. C., Fadillah, A. M., Jonathan, J., & Prabowo, M. R. D. (2019). Interactive gamification learning media application for blind children using android smartphone in Indonesia. Procedia Computer Science, 157, 589-595. [20]. Bouraoui, A., & Soufi, M. (2018, July). Br’Eye: An Android mobile application to teach Arabic and French Braille alphabets to blind children in Tunisia. In International Conference on Computers Helping People with Special Needs (pp. 357-364). Springer, Cham. [21]. Bülbül, M. Ş., Yiğit, N., & Garip, B. (2016, April). Adapting smart phone applications about physics education to blind students. In Journal of Physics: Conference Series (Vol. 707, No. 1, p. 012039). IOP Publishing. TEM Journal – Volume 10 / Number 3 / 2021.