See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/254005127

MOSOCO: A Mobile Assistive Tool to Support Children with Autism Practicing

Social Skills in Real-Life Situations

Article · May 2012

DOI: 10.1145/2207676.2208649

CITATIONS READS

85 2,216

8 authors, including:

Lizbeth Escobedo David H. Nguyen

Ensenada Center for Scientific Research and Higher Education Accenture
19 PUBLICATIONS   252 CITATIONS    33 PUBLICATIONS   1,258 CITATIONS   

SEE PROFILE SEE PROFILE

Louanne E. Boyd Alejandro Rangel

University of California, Irvine Ensenada Center for Scientific Research and Higher Education
20 PUBLICATIONS   317 CITATIONS    2 PUBLICATIONS   110 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Go Go Games View project

Zody's clock catastrophe View project

All content following this page was uploaded by David H. Nguyen on 13 June 2014.

The user has requested enhancement of the downloaded file.

 MOSOCO: A Mobile Assistive Tool to Support Children
with Autism Practicing Social Skills in Real-Life Situations
Lizbeth Escobedo1, David H. Nguyen2, LouAnne Boyd3, Sen H. Hirano4, Alejandro Rangel1,
Daniel García-Rosas1, Monica Tentori1, Gillian R. Hayes4
1 2 3 2
Computer Science Nokia Research Division of Special Department of
School, UABC Center Education, NOC Informatics, University
{lescobedo, arangel, david.h.nguyen SELPA of California, Irvine
jose.daniel.garcia.rosas, @nokia.com [email protected] {gillianrh, shirano}
mtentori} @ics.uci.edu
@uabc.edu.mx
ABSTRACT Most of the available social skills tools use visual supports
MOSOCO is a mobile assistive application that uses (i.e., ―those things we see that enhance the communication
augmented reality and the visual supports of a validated process‖ [19]. See Figure 1) and social stories (i.e., ―picture
curriculum, the Social Compass, to help children with or written description of social situations‖ [13, 32]). Taken
autism practice social skills in real-life situations. In this together, these tools go beyond the teaching of basic social
paper, we present the results of a seven-week deployment skills and tackle such broad social issues as how situational
study of MOSOCO in a public school in Southern contexts evolve in ongoing social interactions [13].
California with both students with autism and neurotypical
The Social Compass curriculum is a behavioral and
students. The results of our study demonstrate that educational curriculum identified as the best social
MOSOCO facilitates practicing and learning social skills, curriculum for social skills training. It uses social stories
increases both quantity and quality of social interactions, and paper-based visual supports for teaching children with
reduces social and behavioral missteps, and enables the autism social skills [6]. The minimal social skills required
integration of children with autism in social groups of for a ―basic and complete social interaction‖ are addressed
neurotypical children. The findings from this study reveal by six core lessons from the Social Compass curriculum:
emergent practices of the uses of mobile assistive eye contact, space and proximity, start interaction, asking
technologies in real-life situations. questions, sharing interests, and finishing interaction [33].
Author Keywords Each Social Compass lesson is composed of eight steps and
Assistive technology, autism, child-computer interaction, is conducted in groups of both students with autism and
social skills, augmented reality, mobile applications neurotypical (NT) students. First, (1) the teacher introduces
ACM Classification Keywords the social skill to be learned and (2) its related visual
K.3.1 [Computers and Education]: Computer Uses in support. Then, (3) the teacher reads a social story that helps
Education; K.4.2 [Computers and Society]: Social Issues- students connect the visual support, the social skill and the
Assistive technologies for persons with disabilities social situation. Later, (4) the teacher demonstrates how the
skill is used in social contexts, and (5) students rehearse the
INTRODUCTION AND MOTIVATION
skill in pairs. Next, (6) students role-play to practice the
Autism is characterized by social skills impairments [1].
skill in four scenarios and (7) self-rate their performances.
Children with autism exhibit problems when initiating and
Finally, (8) students are encouraged to practice the newly
terminating interactions, learning the interests of others, and
learned social skill in different real-life social situations. A
joining social groups [12]. Direct instructions for skill
central aspect of the Social Compass training—as with
acquisition are an effective way to teach them social skills,
helping to cope with these social challenges [35].

Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are Figure 1. The visual supports of the Social Compass’
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
basic social skills lessons. From left to right: eye
or republish, to post on servers or to redistribute to lists, requires prior contact, space and proximity, start an interaction,
specific permission and/or a fee. asking questions, sharing interests, and finish an
CHI‟12, May 5-10, 2012, Austin, Texas, USA. interaction.
Copyright 2012 ACM 978-1-4503-1015-4/12/05...$10.00.
other social skills training [12, 23, 32, 35]—is enabling
students to use newly learned skills in daily social contexts,
supporting maintenance and generalization in real life.
Although the Social Compass curriculum has been
successfully used during social skills activities in
classrooms [6], there are still open questions as to how
students might generalize their newly learned social skills
to situations outside the classroom. However, there is
currently limited support for this kind of mobile, dynamic
instruction. To offset this challenge, we explored the use of
mobile and augmented reality technologies to provide
―anytime and anywhere‖ support for students practicing Figure 2. The MOSOCO system
social skills in their daily lives.
available assistive technologies. One of the main challenges
In this paper, we first describe the design and in this context is mobility. Although mobile systems have
implementation of the Mobile Social Compass been theorized as an effective educational method enabling
(MOSOCO), a mobile assistive application that extends the ―anytime and anywhere‖ social skills practicing and
Social Compass training and supports real-life social learning [8, 28], mobile assistive technologies that support
situations [33]. Then, we present the results from a socialization and social skills practice for children with
deployment of MOSOCO. Finally, we close discussing with autism in this context are scarce. Some mobile applications
directions for future work. for encouraging and supporting socialization of NT children
have been proposed (e.g., Camelot [34], Mobile Stories
RELATED WORK
[8]), others have explored other domains, (e.g., Personal
Interactive technologies have the potential to provide a
Travel Assistant [7] ), but none of these solutions have been
wide variety of support to children with autism. For
used to support the socialization of children with autism.
example, interactive visual supports [15] help children with
autism manage their schedules [17], remediate their speech These related efforts open new research questions around
and language disabilities (e.g., [5, 27]) and provide how mobile assistive technologies for encouraging
feedback on their pronunciation (e.g., [10, 14]). socialization and learning and practicing of social skills
improve the quality of interactions of students with autism
Other projects have focused more directly on the
and their NT peers. Additionally, the lack of exploration of
development and practicing of social skills. Both games
tools for both children with and without autism opens
(e.g., SIDES [29]) and multitouch displays (e.g., Tablets
questions around whether these tools might foster
[21]) have been developed to engage children with autism
integrated social groups.
in social activities and enable novel forms of social
expression. These entertainment-based assistive In light of the related literature, this work has three
technologies also help children with autism learn how to contributions. First, although many other researchers have
work with others, be patient and understand emotions [29]. explored the use of assistive technologies for social skills
These research projects have made the compelling practice in classrooms (e.g., [9, 16, 20]), we examine the
argument that children with autism are more willing to usability and usefulness of socially assistive technologies in
initiate play and to interact appropriately when using real life situations outside classrooms. Second, we present a
entertainment-based assistive technologies. However, there mobile assistive tool that could be useful to other
is limited evidence of a sustained improvement in their researchers in designing and evaluating mobile assistive
social skills following the use of these tools. technologies for use in real-life situations. Third, the results
of our deployment study reveal how new social practices
To cope with this challenge of sustainability of skill
mediated by MOSOCO can improve the quality of
development, the StoryTable project focused on the use of
interactions and social skills in generalized settings.
storytelling to encourage socialization and enable practicing
of appropriate social skills [11]. This research has shown THE MOSOCO SYSTEM
that support during social engagements increased MOSOCO is a mobile augmented reality application
appropriate expression of emotions and responses to peer inspired by previous research on the Social Compass
questions by children with autism [11]. However, most of curriculum [33] (see Figure 2). In our previous work, we
these assistive technologies are often available only for use focused to analyze interactions among students, which led
inside classrooms or during group therapies. Children with to design insights. Those insights, along with the Social
autism still struggle when generalizing what they learned to Compass curriculum, inspired the design and development
real-life situations outside the classroom [3]. of MOSOCO. MOSOCO facilitates practicing social skills
in real-life situations, guiding students through the six
Supporting real-life social situations poses new research fundamental social skills of the Social Compass curriculum.
questions and strains the design and use of currently
MOSOCO provides children with autism with interactive
 Figure 3: Students using MOSOCO during recess (left) two students making eye contact (right) Learning and
tagging PIP’s interests
features to encourage them to make eye contact, maintain and don’ts (e.g., “don‟t stare”). These suggestions mimic
appropriate spatial boundaries, reply to conversation verbal support available from teachers during the Social
initiators, share interests with partners, disengage Compass classroom activities and can be used to practice
appropriately at the end of an interaction, and identify the skills (Figure 3). MOSOCO monitors skill completion
potential communication partners. MOSOCO runs on an to mimic teacher support further. For example, the system
Android smartphone wirelessly connected to a server. detects eye contact through a vision-based algorithm for eye
MOSOCO uses the smartphone’s camera to augment a real- detection. If MOSOCO infers that eye contact was not
life social situation with visual support—mimicking the properly achieved, it shows a warning notifying that a
Social Compass curriculum that augments social stories social misstep has occurred alongside a picture and text-
with paper-based visual supports. based advice suggesting how to recover from it (e.g., ―look
MOSOCO was designed with both individual and group at your partner’s eyes‖). Some common social missteps
features in mind, augmenting each individual student’s observed in our data include students using an inappropriate
social skills, needs, and capabilities, and enabling tone of voice (e.g., shouting), avoiding eye contact or
synchronized group interactions and shared experiences. staring, invading personal space, and interrupting or
Individual features include: a unique identifier for each breaking an interaction (e.g., students with autism usually
student using the system (typically a name), visualization of break an interaction without an explicit farewell). Once
student progress in the form of stars that can be earned, an MOSOCO detects that both students have made eye
interaction self-report form, and social cues, i.e., what a contact, it emits a sound and highlights the next step.
user should do (“DOs”) and should not do (“DONTs”) Students repeat this process to follow each of the rest of the
when interacting with others. Collaborative and group IVS steps until it is time to ―end a conversation‖ (step 6).
features include: the Interaction Visual Schedule (IVS) During a successful sustained interaction, students perform
(i.e., a six-step visual schedule that provides step-by-step ―verbal or physical social activities‖ (Figure 3). Verbal
guidance through the six basic social skills learned during interactions include activities like asking questions and
the Social Compass training), a Potential Interaction learning and marking interests from the Social Compass
Partner (PIP) suggestion (i.e., a student who might like to curriculum. Physical interactions include traditional
interact with the student using the device), and a roster of activities performed during recess, such as playing sports or
connected students (i.e., a listing of student profiles board games.
including photographs and personal interests).
After each interaction, students complete self-report forms
When they first start MOSOCO, students are instructed to to grade themselves on the six practiced social skills. They
find a suggested PIP. MOSOCO suggests the PIP by select ―thumbs up‖ or ―thumb down‖ on the interface,
showing the partner’s name and photograph (see Figure 4). matching the simple self-grading students do during
When the system detects that the user is in close proximity classroom social skills activities. If no social missteps were
to the suggested PIP, both students are automatically detected directly by the system, students earn stars based on
―paired up,‖ indicating that they have started an interaction. the number of times both partners grade the interaction with
At this point, their Interaction Visual Schedules (IVS) are a ―thumbs up,‖ encouraging students to help each other
synchronized. Students may also manually pair themselves master the different skills. Finally, after the interaction and
with another student. self-evaluation, MOSOCO calculates a new PIP suggestion
Once MOSOCO’s IVS is activated, the first visual support, for both students giving them the opportunity to interact
eye contact, is colored. MOSOCO suggests skill- with other students.
appropriate do’s (e.g., “smile if somebody looks at you”)
By receiving this information through a personal device, area was surrounded with six HD video cameras, thus
each student gets direct assistance and reinforcement for recording the children’s interactions from six different
practicing their social skills, similar to the kind of human angles. Researchers observed the students during both the
prompting and reinforcement they might receive during Social Compass training and breaks, including lunch and
classrooms activities. Meanwhile, every activity is logged recess for a total time of observation of just under 11 hours,
in the background, enabling automatic report generation for (pre-deployment: 4 hr. 3 min., deployment: 4 hr. 37 min.,
teachers to measure student progress. post-deployment: 2 hr. 12 min.).
METHODS Weekly interviews were recorded and conducted with the
MOSOCO was deployed in a public school in Southern participants across each study phase. Interviews were face-
California during regularly scheduled lunch and recess to-face and semi-structured, and students were divided in
breaks. All students enrolled in the Social Compass training three groups (i.e., g1 = 5 NT students, g2 = 4 NT students,
were between the ages of 8 and 11 (m=10.08, sd=0.90). g3 = 3 students with autism). Interviews generally lasted
From this group, we selected three students with autism between 15 minutes to half an hour, in keeping with the
who demonstrated minimal social skills but otherwise short blocks of time available to talk with school students.
demonstrated age-appropriate functioning; for each of
During pre-deployment, students were asked about daily
them, we invited three NT students to serve as their PIPs (n
routines and general communication practices. During
= 12 total). This ratio of NT students mimics the school
deployment, students were asked to discuss how the use of
population. The evaluation included three stages: pre-
the system went during that particular week, comment on
deployment (3 weeks), deployment (3 weeks) and post-
how MOSOCO impacted their interactions and social
deployment (1 week).
groups, and encouraged to tell stories and discuss what they
During pre-deployment, students participated in the found interesting, surprising, or different that week. At the
standard Social Compass curriculum in the classroom, end of the study, the teaching staff and students completed
which included direct instruction from trained specialists surveys about their experiences with MOSOCO and took
and paper-based tools. Researchers then installed part in a 60-minute group interview. Unlike in the previous
MOSOCO and gave a training session on the system to the interviews in which the students were broken into small
students and teachers. During the deployment phase, the groups, in this final interview, all students met together.
students used MOSOCO during the Social Compass class Recorded interviews and videos were transcribed.
and during their breaks. Finally, during post-deployment, Analysis
the students participated in no lessons, and the system was All field notes, interview and videos transcripts were
not available to them. inspected together using a mixed-methods approach.
To meet the regulations of the school and the university Researchers first analyzed the data for evidence of whether
research ethics board, students enrolled in the study were MOSOCO supported the needs of students. Researchers
confined to a private playground area during recess and used open coding [31] and multi-phased affinity analysis
lunch breaks (approximately 25 minutes per day). This area [24] to uncover emergent themes from the interview data, in
(150x50 ft) was separated from the rest of the playground particular in relation to uses of the system. We then
by cones but incorporated common landscape elements developed a coding scheme focused on student social
(e.g., a tree, a wall, and benches) and traditional school experiences. This coding scheme was cross-analyzed during
playground items (e.g., balls, ropes, board games). discussions among the research team to validate and refine
its categories and properties.
Data Collection
At the beginning of each session, researchers set up video A quantitative analysis was conducted to estimate the time
cameras and sectioned off the playground for the study students spent in different interactions and social groups as
using bright orange cones. The sectioned-off playground well as the number of social missteps committed. Using our
coding scheme, the research team coded the recorded
videos for interactions, behaviours, and social missteps.
Inter-observer reliability1 was acceptable (r=.912). For each
participant under each condition, we calculated:
 total and descriptive statistics (i.e., average and standard
deviation) of interactions, social missteps, and topic of
conversations
 average time in a sustained interaction, social misstep
and topic conversation

1
Refers to the degree of agreement between two data sets, scored
Figure 4: MOSOCO suggesting a PIP independently by two different observers.
Finally, a single t-test was used to compare students with During the three weeks of intervention, all of the students
autism social missteps and behaviours, and an independent mastered the features of MOSOCO and the smartphone
sample t-test was conducted to compare the number of itself. This allowed them to explore uses for both the
physical, verbal and total of interactions using and not using smartphone and MOSOCO, sometimes circumventing the
MOSOCO, of both students with autism and NT students. prescribed lessons. For example, at the end of the third
The data presented a normal distribution. week, students started to ―skip‖ practicing of some of the
more basic skills in MOSOCO, shortening interactions to
RESULTS
New social practices mediated by MOSOCO emerged earn more stars while competing with other students.
during our deployment. In particular, MOSOCO had an “….I pressed ok ok ok and then I got so much stars, and
impact on the practicing and learning of social skills and I was like wow!” (s5, NT student)
changed the makeup and character of student peer groups.
The students provided suggestions for encouraging greater
General Use and Adoption adherence to the lessons during interviews:
Overall, students positively received MOSOCO, finding it
useful, fun, and helpful. Students showed excitement while “To really go with the person suggested by
using the system, and learned how to use it with minimal [MOSOCO], it would be interesting to have in
training in less than an hour. At the beginning of each [MOSOCO] more sophisticated functions, like face
session, they typically ran to the area where the phones detection gear.” (s4, NT student)
were stored to start using them. “Find games that are like appropriate—that are mixed
“It‟s really cool… I was very comfortable… I‟m always with the social compass—you have to do eye contact.
waiting for this part of the school day.” (s10, student Oh, and then you get a point or something like that?
with autism) Make it more game like.” (s7, NT student)

“It was good, really good, and really awesome.” (s11, Another, particularly remarkable interaction that
student with autism) exemplifies how students mastered the smartphone
occurred during the third week of use. Even though all of
Students were more interested in using MOSOCO and other the phone’s buttons were deactivated, Student 5 (NT)
smartphone applications than the typical playground items closed MOSOCO’s application and launched other
available, calling out features they liked. applications. He then taught this skill to other students who
“I liked that you can earn stars. [That] was fun.” (s1, began occasionally to listen to music, select different
NT student) ringtones, take pictures, and play games with the GPS.

“…I‟m more interested in the phone and [MOSOCO], “Well I found this app thing that said; GPS Test. So I
because I can learn a lot of different things.” (s2, NT hit it so I could test it. It‟s cool; it also tells you your
student) speed.” (s6, NT student)

The primary motivation for using MOSOCO was to learn “I just spent time with myself listening to ringtones
and to practice social skills, with all students reporting that sitting down.” (s3, NT student)
they learned the Social Compass lessons. When discussing These results demonstrate that assistive technologies are
what they learned specifically, students referenced the six important and useful outside classrooms but that classroom
fundamental social skills (e.g., ―space and proximity,‖ management strategies will continue to be needed to
―ending a conversation,‖ and ―eye contact‖). support students in staying focused on their tasks rather
“I got to learn, because I didn‟t know about how far than playing with the technologies in other ways.
[away from another person] you are supposed to be.” Impact on Student Behavior and Performance
(s4, NT student) During the deployment phase, the smartphone and
“… I‟ve learn how to end [a conversation properly].” MOSOCO evolved from a novel artifact and a ―toyish‖
(s12, student with autism) application to a true assistive support and means for social
skills learning and practicing outside the classroom.
“[MOSOCO] helped me how to make eye contact and Overall, the use of MOSOCO positively impacted student
ask questions.” (s10, student with autism) interactions with each other. Students learned the basic
steps of a ―proper‖ interaction, how to avoid and identify
Both sets of students expressed that they would like to use
missteps, and how to help others interact.
MOSOCO for longer integrated in their everyday lives,
rather than only in the restricted areas of the study. Social Skills Practice and Learning
Generalizing skills beyond classroom learning is a
“I‟ll want to do it again. Yeah I think it would be
challenge for any student, but particularly so for students
helpful for kids that are still learning [the Social with autism. Thus, the ability to practice social skills
Compass curriculum].” (s4, NT student)
outside the classroom was a key feature of the MOSOCO Improving the quality of interactions
system. Both NT students and those with autism repeatedly
expressed that MOSOCO helped them to improve the
“It really does help me practice the Social Compass quality of their interactions.
lessons.” (s11, student with autism)
“Talking certainly to a person in the way that they
“[It helps me] to do the social things, eye contact, taught us: how to talk „formally.‟ ” (s7, NT student)
proximity, and all those things, so I can practice them.”
(s6, NT student) “It helps me like to socialize better.” (s10, student with
autism)
In addition, students noted that the system provides them
confidence when interacting with others, reassuring them of The time students with autism were involved in social
what they should or should not be doing. missteps was 56% lower on average using MOSOCO (no-
support: 2 hours 2 min 13 sec; using-MOSOCO: 54 min 38
“I forgot something; I‟ll look into [MOSOCO‟s social sec; t=3.42, p=.003). This reduction can be attributed to
cues].” (s12, student with autism) greater student awareness and improved ability to identify
“It usually gives me hints.” (s5, NT student) social missteps using the prompts provided by MOSOCO.
Additionally, the system augmented understanding of the
“The eye contact and all the other tools and the do‟s usage of specific social skills given particular situations.
and don‟ts—I found those useful for any of the
conversations I start.” (s8, NT student) “Like sometimes before Social Compass, when I was
doing eye contact, I looked away, but after the lessons I
MOSOCO can help students learn by repetition, in keeping know eyes should go on.” (s5, NT student)
with many other therapies for children with autism (e.g.,
discrete trial training [30] and rote learning [22]). “[MOSOCO] helped reduce the social missteps.” (s10,
student with autism)
“It helps me practice and practice.” (s10, student with
autism) Having social skill awareness with MOSOCO, the total
time a student continually remained engaged in a social
“To reenact” (s11, student with autism) misstep reduced 76.52% (no-support: 3 min 33 sec; using-
After so many repetitions, students were able to quickly MOSOCO: 50 sec; t=3.75, p=.002). In addition to reducing
recall the goal of the lesson and how to use the social skill missteps for students with autism, MOSOCO supported NT
in a given particular situation. students in learning how to be more understanding and
permissive if such missteps were presented. NT students
“[MOSOCO] helps you, because if you forgot to put developed new strategies for handling the social missteps
that into your brain, it then remembers you [sic].” (s10, instead of mocking or teasing the students with autism as
student with autism) they had done before.
Students with autism and NT students differed somewhat, Of course, for young children, encountering this kind of
however, in their motivation for continued use of the difference can still be challenging, as exemplified by the
system once the novelty wore off. While students with student below who describes being understanding most of
autism continued describing wanting to practice newly the time but resorting to antisocial behaviors at other times
learned social skills, NT students adopted ―mentoring‖ to handle social missteps.
behavior. They described feeling responsible for assisting
the practicing of social skills with the students with autism “I ask [a student with autism] what he wants to tell me
and helping them to improve their interactions. and then we just get along, but sometimes I just ignore
him because he does that [referring to social missteps]
“I like to help those shy children [the students with way too many times” (s5, NT student)
autism] to have a fun time. It‟s awesome.” (s4, NT
student) As with social missteps, we observed that other behavioral
issues (e.g., rocking, jumping or making noises),
―Yeah! I practice with [a student with autism] eye traditionally demonstrated by students with autism when
contact and all those things.” (s5, NT student) interacting with others were substantially reduced during
Most of the interactions we observed with NT students the deployment of MOSOCO (no-support: 50; using-
involved hints and explanations about a particular social MOSOCO: 1). This large difference warrants further
skill. When children with autism would get ―stuck‖ in a investigation with a larger sample. These behaviors may
step, NT students would help them to ―stay on task.‖ This have been limited by the interest in and engagement with
mimics the physical prompting given by teachers during the smartphone and MOSOCO.
classroom lessons.
 Figure 5. Percentage of students’ number of interactions: Children with autism (left) and NT children (right)
Students with autism struggle to make eye contact when MOSOCO also helped students improve awareness of space
interacting with others [26], a problem one might imagine and proximity when interacting with others. Beyond just
could be exacerbated by concentration on a mobile device. understanding the anatomy of a ―proper‖ interaction,
However, all students agreed that with MOSOCO they students learned about the implications of not maintaining
never had problems making eye contact during an socially acceptable distances.
interaction.
Students also learned that one of the main goals of social
“It tells me to find somebody, and then I found a interactions is to know the personality of your interaction
person and then I hear the buzz sound and it tells me to partners (i.e., interests, likes, and dislikes). They began to
do eye sight, I mean eye communicate. Yes you have to recognize the need for multiple conversation topics and to
communicate with somebody‟s eyes I guess.” (s4, NT develop new relationships through these conversations.
student)
“[MOSOCO] guided us through a „proper‟
Similarly, ―how to end a conversation‖ was another basic conversation and taught us how to input emotions and
social skill that students agreed MOSOCO helped them to feelings.” (s1, NT student)
improve. Before the use of MOSOCO, students said that is
These findings indicate that MOSOCO helped both students
very common to end up being alone because their partner
with autism and their NT peers to enhance the quality of
walked away without saying anything.
interactions by encouraging them to follow basic steps. The
“I‟ve been improving on how to like end conversations, reminders then solidified these directions during
usually I‟m like „bye‟ then walk away.” (s5, NT student) interactions.
“First you would say „hi‟ and then walk away without Student interactions and social groups
ending the conversation. Now you know how to end the In this work, we were interested in two types of
conversation…” (s10, student with autism) interactions. First, given that MOSOCO is largely about
helping students with autism communicate with peers, we
Students made references to their interactions before using were interested in investigating how their one-on-one
MOSOCO, highlighting that this application serve as a interactions can be assisted or improved by mobile assistive
milestone for improving their social behavior. technologies. Second, we were interested in how
“I‟ve been like starting and ending conversations communication patterns amongst students influence social
differently. We had less way less blanks [using groups through interactions mediated by MOSOCO.
MOSOCO].” (s1, NT student) Student-Student
Eleven of the twelve students expressed that MOSOCO Overall, the number of interactions between students
helped them to understand that an interaction has an increased 56.73% when using MOSOCO (no-support: 386
―interactive behavior,‖ involving at least three of the basic interactions; using-MOSOCO: 605 interactions) (Figure 5).
social skills. Surprisingly, we observed less time spent not interacting
“In the beginning, you have to make eye contact, and with others for students with autism during use of
the next step is proximity I think, and then asking MOSOCO (no-support: 27; using-MOSOCO: 7), NT
questions and then asking interests, and then is students spent more time alone (no-support: 14; using-
cleaning up the conversation—like how to end it, like MOSOCO: 136) (Figure 5). We hypothesize that this result
say “bye” or something.” (s2, NT student) could be related to the interest of NT students in interacting
with the smartphone, like other youth their age [2]. The
“…you have to make good eye contact, study your large differences in these initial comparisons warrant
things to make sure you made the right move, ending further detailed exploration. In contrast, students with
the conversation instead of just walking away.” (s10, autism, who are typically considered at a greater risk for
student with autism)
fixating on a tool, appeared to embrace the application as a What could be happening here is that MOSOCO is
support that seamlessly mediated their interactions. providing students a ―ticket to talk‖. MOSOCO provides
the ―ticket‖ for them so that it is easier to start
Although in our observations, NT students interacted
conversations and talk with each other.
physically with other students less while using MOSOCO
(no-support: 145; MOSOCO: 13), students with autism “We see each other in the face and we just play.” (s4,
interacted physically more with MOSOCO (no-support: 50; NT student)
MOSOCO: 60). Likewise, students using MOSOCO
After the removal of MOSOCO during post deployment,
displayed substantially, but not significantly, more verbal
NT children integrated children with autism into their
interactions than without (no-support: 160; using-
playgroups, and the students with autism joined without
MOSOCO: 389). These preliminary results warrant further
resistance. This kind of social integration was not seen
detailed exploration with a larger sample to understand the
during our baseline data collection nor reported by any staff
potential trends we see in these non-significant
members prior to that. Moreover, this change did not go
comparisons. However, these initial results imply that
unnoticed by the students with autism who told us during
MOSOCO may be better at supporting conversations than
interviews that now they can join groups more easily.
physical games, which is in keeping with the goals of social
skills training in schools. Students’ highlighted that after “All you do is just go and have eye contact and they
the use of MOSOCO they got to know their partners better, know you‟re playing with them and they know they‟re
knowing their interests and dislikes. Additionally, we playing with you.” (s10, student with autism)
observed conversations that were qualitatively more
intimate during the deployment than in the pre-deployment MOSOCO provided new strategies for the creation and
condition, an effect that could be in part due to the greater transformation of social groups. Students universally
reported that before using MOSOCO they had clearly
familiarity the students had with one another having been in
class together for weeks. defined social groups of friends with a sort of social
protocol dictated by the social group’s leader for admitting
With MOSOCO we observed that students with autism new members. These social protocols, typical of school
learn new and traditional strategies for starting an circles, tend to demand from an individual an attempt to be
interaction, even though it has been reported that children integrated into the group.
with autism find it difficult to properly start a conversation
“She has to agree and I have to agree and she has to
[18], read others body language [4] or join a group of
children playing [25]. Students expressed that MOSOCO agree too.” (s9, NT student)
helped them to understand body language and recognize The complexity of these interactions combined with limited
invitations for starting an interaction. social skills means that most children with autism tend not
“[MOSOCO] helped me play with others.” (s12, to be part of a social group and instead spend recess by
student with autism) themselves. However, after using MOSOCO, we observed
that students with autism learned new strategies for creating
In our observations, students with autism spent more time and entering a predefined social group.
continuously engaged when using MOSOCO (no-support: 6
“Just walk up and stand in line. Then they play.” (s10,
min 8 sec; using-MOSOCO: 8 min 49 sec). This result
warrants further investigation with a larger sample size as student with autism)
well as a more detailed observation of the types of This kind of breaking down of the barriers of strict social
engagement and the potential ramifications of use of groups provides some indication that even in schools with
MOSOCO on those activities. no students with autism, such tools might be useful.
Group Interactions Additionally, making group integration a more
straightforward process allowed students with autism to
Social skills used during one-on-one interactions also bled enjoy topics of conversation and the company of classmates
over into groups. In particular, students exhibited new rather than just the execution of an activity (e.g., a game).
strategies for joining a group of children already playing.
“I just face them and then I say hello and then they
Before the deployment of MOSOCO, the strategies were
fairly ad hoc and tended to encourage socialization with start talking to me and then I talk so we start a big
known classmates: conversation.” (s7, NT student)
CONCLUSION AND FUTURE WORK
“I just look around for one of my friends.” (s1, NT
Appropriately designed assistive technologies can play a
student)
crucial role in improving social learning and social skill
While using MOSOCO, however, the students appeared practice in real-life social situations for students with
more motivated to interact with everyone. This motivation autism and their NT peers. A key contribution of this work
enabled students to flow more easily between groups or to are the results of the deployment of MOSOCO in order to
enter large groups without explicitly requesting to do so. understand the design space and the user experience of this
mobile assistive tool in real-life social situations. The There are some limitations to this work. First, the design of
exploration of 12 students’ use and experience with MOSOCO is based on the application of one particular
MOSOCO throughout its deployment provides a set of classroom-based social skills training. Use of this tool in
emergent practices of the use of assistive technologies in schools in which the Social Compass curriculum is not
real situations complementing classroom activities. currently the norm would broaden our understanding of its
general applicability. Likewise, open questions remain in
Our results demonstrate that MOSOCO is easy to use and
exploring other assistive technologies based on a different
helps students practice their social skills. In our evaluation,
social skills training techniques in this context. Finally, as
use of this tool reduced students with autism social missteps
in any deployment study, perfectly controlled conditions are
56% and behavioral issues 98%, increases the number of
not possible, and some effects may be related to the
interactions and their quality 52% of both students with
sustained engagement of the students in the Social Compass
autism and NT students. We found out that instead of the
curriculum over the course of the study rather than to tool
smartphones pulling children out of interactions, students
use alone. Also our study is limited by the small number of
were motivated towards social interactions and used the
participants. With the current sample, it’s not possible to
system without additional prompting by teachers. This
derive findings that can extend across larger populations.
behavior occurred because both students with autism and
We want to clarify that our effort centered on acquiring a
NT students were interested in sharing what they
vast and detailed understanding of the practices of a few
discovered, practicing the newly learned social skills and
individuals across the multitude of experiences of a shared
earning the rewards designed into MOSOCO. By freeing
playground. Therefore, the significance of our results lies
students from teachers’ prompts, MOSOCO also allowed
on the qualitative nature of the inquiry and analysis. We
students to discover other functionalities of the smartphone
leave open for future work the study of the impact of
that played a positive role in improving student’s
MOSOCO with a bigger sample or in another context.
interactions. A deeper and bigger study is needed to explore
this sort of assisted independence and its relevance to long- ACKNOWLEDGMENTS
term MOSOCO use. Additionally, our results indicate that This work was supported by grants from NSF CAREER
using mobile assistive technologies in real-life social #0846063, UC Mexus, PROMEP #9044, and CONACYT
situations have an impact on children’s social groups. But #10256. We thank the STAR group at UCI for reviewing
open questions remain as to how social group dynamics are early drafts of this paper. And importantly, we thank the
currently changing due the use of an assistive technology participants and their school for helping with this study.
that mediates topics of conversation, with which potential REFERENCES
users more likely choose to interact, and the creation and 1. American Psychiatric Association (1994). Diagnostic and
transformation of their existing social groups—an analysis statistical manual of mental disorders (4th ed.)
in this direction is currently in progress.
2. Berg, S., Taylor, A. and Harper, R. Mobile phones for the
Future work should also focus on use of tools such as these next generation: Device designs for teenagers. In Proc.
outside of schools with interaction partners who have CHI 2003, ACM Press (2003), 433-440.
limited knowledge of the tools and lessons. Additionally, 3. Betz, A., Higbee, T. and Pollarda, J. Promoting
the experience of teachers, other school staff, and parents in generalization of mands for information used by young
setting up and using such tools for students should be children with autism. Research in autism spectrum
explored with new tools built to support these activities if disorders, (2009), 4(3).
necessary. 4. Blake, R., Turner, L., Smoski, M., Pozdol, S. and Stone,
The deployment and evaluation of MOSOCO exposed W. Visual recognition of biological motion is impaired in
tensions around its use outside classrooms and contributes children with autism. Psychological science, 2003, 14(2).
important implications for the design of assistive 5. Bowker, S. A,. Autism, AAC, and Proloquo2go. In
technologies in the future, especially those destined for use perspectives on augmentative and alternative
in real-life situations and in mobile contexts. Our results communication, (2009), 137-145.
indicate that students were highly interested in earning stars 6. Boyd, L., Mcreynolds, C. and Chanin, K. The social
so assistive technologies should incorporate the potential compass curriculum: a story-based intervention package for
for game-like interactions to earn points and rewards. Also, social skills. NOC SELPA, editor 2010, Fullerton, CA,
we noted that social cues play an important role for helping USA, 190.
students ―stay on topic‖ when interacting with peers. 7. Carmien, S., Dawe, M., Fischer, G., Gorman, A., Kintsch,
However, social cues were sometimes triggered at A. and Sullivan, J. F. Socio-Technical Environments
inappropriate times, as the application was highly unaware Supporting People with Cognitive Disabilities Using Public
of the topic of conversation. Context-aware tools with voice transportation. In ToCHI 2005, 12(2), 233-262.
recognition capabilities will enable the application to
8. Fails, J., Druin, A. and Guha, M. Mobile collaboration:
recognize the conversation context and react accordingly.
collaboratively reading and creating children’s stories on
mobile devices. In IDC 2010, 20-29.
 9. Farr, W., Yuill, N. and Raffle, H. Collaborative benefits of disorders. Personal and ubiquitous computing, 2011, 1-
a tangible interface for autistic children. In Proc. CHI 12.
2009. 22. Koegel, R.L. and Kern, L. Pivotal response treatments for
10. Fell, H., Cress, C., MacAuslan, J., and Ferrier, L. autism: communication, social, and academic
visiBabble for reinforcement of early vocalization. In development. (2006) Brookes publishing company.
Proc. of the 6th international ACM SIGACCESS on 23. Krantz, P.J. and Mcclannahan, L.E. Social interaction
Computers and accessibility, ACM Press, 2003, 161-168. skills for children with autism: a scriptading procedure for
11. Gal, E., Bauminger, W., Pianesi, F., Stock, O., Zancanaro, beginning readers. Journal of applied behavior analysis,
M., and Weiss, P. Enhancing social communication of 1998, 31,191–202.
children with high-functioning autism through a co- 24. Kuniavsky, M. Observing the user experience: a
located interface. AI Soc. 24, 1 (2009), 75-84. practitioner’s guide to user research. (2003) Morgan
12. Gonzalez-Lopez, A. and Kamps, D. M. Social skills kaufmann.
training to increase social interactions between children 25. Libby, S., Powell, S., Messer, D. and Jordan, R.
with autism and their typical peers. Focus on autism and Spontaneous play in children with autism: a reappraisal.
other developmental disabilities, 1997, 12, 2-14. Journal of autism and developmental disorders, 1998,
13. Gray, C., Social stories and comic strip conversation with 28(2), 487-497.
childs with asperger syndrome and high functioning 26. Mcarthur, D. and Adamson, L.B. Joint attention in
autism. In Asperger syndrome or highg-functioning preverbal children: autism and developmental language
autism. Schopler et. Al. (eds.) Plenum press, NY, 1998, disorder. Journal of autism and developmental disorders,
167-198. 1996, 26(3), 481-496.
14. Hailpern, J., Karahalios, K., Dethorne, L., and Halle, J. 27. Monibi, M. and Hayes, G. R. Mocotos: Mobile
Vocsyl: Visualizing syllable production for children with communications tools for children with special needs. In
ASD and speech delays. In Proc. of the 12th international IDC 2008, ACM Press, 121-124.
ACM SIGACCESS conference on Computers and
accessibility 2010, ACM Press, 297-298. 28. Patten, B., Sánchez, I.A. and Tangney, B. Designing
collaborative, constructionist and contextual applications
15. Hayes, G.R., Hirano, S., Marcu, G., Monibim M., for handheld devices. Computers & education, 2006, 6(3).
Nguyen, D.H., and Yeganyan, M. Interactive visual
supports for children with autism. Personal and 29. Piper, A.M., O’Brien, E., Morris, M. and Winograd, T.
ubiquitous computing, 2010, 14(7). Sides: a cooperative tabletop computer game for social
skills development. In Anniversary conference on CSCW
16. Hendrix , K., L., van Herk, R., Verhaegh, J., and 2006, ACM Press, 1-10.
Markopoulos, P. Increasing children’s social competence
through games, an exploratory study. In IDC 2009. 30. Smith, T. Discrete trial training in the treatment of autism.
Focus on autism and other developmental disabilities,
17. Hirano, S.H., Yeganyan, M. T., Marcu, G., Nguyen, D. 2001, 15(2), 86-92.
H., Boyd, L., and Hayes, G. R. Vsked: evaluation of a
system to support classroom activities for children with 31. Strauss, A. and Corbin, J. Basics of qualitative research:
autism. In Proc. CHI 2010, ACM Press, 1633-1642. techniques and procedures for developing grounded
theory. Ed. Sage. 1998, thousand Oaks, CA.
18. Hobson, R.P. and Lee, A. Hello and goodbye: a study of
social engagement in autism. Journal of autism and 32. Swaggart, B.L., Gagnon, E., Jones-Bock, S., Earles, T.L.,
developmental disorders, 1998, 2(1): 117-127. Quinn, C., Smith-Myles, B. and Simpson, R.L. Using
social stories to teach social and behavioral skills to
19. Hodgdon, L.A. Visual strategies for improving visual children with autism. Focus on autism and other
communication. Practical support for school and home. developmental disabilities, 1995, 10, 1-6.
Quirk roberts publishing, 1999.
33. Tentori, M. and Hayes, G.R. Designing for interaction
20. Hourcade, J.P., Bullock-Rest, N.E. and Hansen, T.E. immediacy to enhance social skills of children with
Improving the quality of communication and social autism. In ubicomp 2010, ACM Press, 51-60.
interactions for children with autism spectrum disorders
through multi-touch tablet applications. In clinical aac 34. Verhaegh, J. Soute,I., Kessels, A. and Markopoulos, P. On
research conference 2010. the design of camelot, an outdoor game for children. In
IDC 2006, ACM Press, 9-16.
21. Hourcade, J.P., Bullock-Rest, N.E. and Hansen, T.E.
Multitouch tablet applications and activities to enhance 35. Wise, M.J. and Harris, S.L. Teaching social skills to
the social skills of children with autism spectrum people with autism. Behavior modification, 2001, 25(5),
785-802.

View publication stats