Developing and Validating a Synthetic Teammate

Form Approved OMB No. 0704-0188 REPORT DOCUMENTATION PAGE Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 222024302 Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY) 31-01-2010 2. REPORT TYPE Final Performance Report 4. TITLE AND SUBTITLE 3. DATES COVERED (From - To) 12/01/2008 - 11/30/2009 5a. CONTRACT NUMBER DEVELOPING & VALIDATING A SYNTHETIC TEAMMATE Sb. GRANT NUMBER N000140910201 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) Christopher W. Myers & Nancy J. Cooke 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER Cognitive Engineering Research Institute 5810 S. Sossaman Mesa, AZ 85212 9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) Dr. Paul Bello Office of Naval Research 875 N. Randolph St. Suite 425 Arlington, Va 22203 10. SPONSOR/MONITOR'S ACRONYM(S) ONR 11. SPONSOR/MONITOR'S REPORT NUMBER(S) 12. DISTRIBUTION / AVAILABILITY STATEMENT Approved for public release; distribution is unlimited 13. SUPPLEMENTARY NOTES 14. ABSTRACT The research was focused on the development of a synthetic teammate over the course of a year. The scientific and technical objectives were to develop and integrate computational accounts of macrocognitive processes identified as necessary for working as part of a team (e.g., language, task behavior, situation assessment/awareness, etc.). Each of the proposed project milestones have been achieved: 1) we integrated language comprehension, agent-environment interaction, and language generation components into a single system that behaves as a synthetic teammate; 2) we developed a situation component and integrated it with the synthetic teammate; 3) we continued to refine the synthetic teammate through test-develop-retest iterations; however, instead of adding humans as teammates we developed agents that were low in cognitive fidelity using systems of finite state machines that acted as the photographer and navigator. 15. SUBJECT TERMS Cognitive model, vehicle situation assessment, language comprehension and generation, uninhabited air 16. SECURITY CLASSIFICATION OF: a. REPORT b. ABSTRACT 17. LIMITATION OF ABSTRACT c. THIS PAGE UL 18. NUMBER OF PAGES 19a. NAME OF RESPONSIBLE PERSON Christopher Myers 19b. TELEPHONE NUMBER (include area code; 480-988-6561 x687 Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39 18 20100222584 Developing & Validating a Synthetic Teammate PI: Dr. Christopher W. Myers* 711th Human Performance Wing / RHAC Air Force Research Laboratory 6030 South Kent Street Mesa, AZ 85212 Co-PI: Dr. Nancy J. Cooke Cognitive Engineering Research Institute 5810 South Sossaman Road, Suite 106 Mesa, AZ 85212 ONR Award: N000140910201 Report Date: January 31, 2010 Duration of Effort: December 1, 2008 - November 30, 2009 *Dr. Myers was employed at the Cognitive Engineering Research Institute throughout the duration of the award. Scientific and Technical Objectives The research was focused on the development of a synthetic teammate over the course of a year. The scientific and technical objectives were to develop and integrate computational accounts of macrocognitive processes identified as necessary for working as part of a team (e.g., language, task behavior, situation assessment/awareness, etc.). Each of the proposed project milestones have been achieved: 1) we integrated language comprehension, agent-environment interaction, and language generation components into a single system that behaves as a synthetic teammate; 2) we developed a situation component and integrated it with the synthetic teammate; 3) we continued to refine the synthetic teammate through test-develop-retest iterations; however, instead of adding humans as teammates we developed agents that were low in cognitive fidelity using systems of finite state machines that acted as the photographer and navigator. The following sections provide more specifics regarding our proposed scientific and technical objectives. Background For the two years prior to the current ONR award, a team of scientists at the Cognitive Engineering Research Institute (CERI, an independent not-for-profit research institute in Mesa, AZ) and the Performance and Learning Models team (PALM) at the Air Force Research Laboratory (AFRL) have worked on the development, integration, and validation of a synthetic teammate (Myers et al., under review). The synthetic teammate will interact with human teammates in real-time to accomplish a reconnaissance task within an Uninhabited Air Vehicle synthetic task environment (UAV-STE). The UAV-STE is a team-based task that involves three interdependent team members, each with a different role. The team members are the Data Exploitation Mission Planning and Communications operator (DEMPC, the navigator) who is responsible for producing a dynamic flight plan, including speed and altitude restrictions, an Air Vehicle Operator (AVO, the pilot) who controls flight systems, and a Payload Operator (PLO, the photographer) who monitors sensor equipment and photographs ground targets. The team members' common goal is to photograph ground targets and this requires interaction between team members. A single UAV-STE mission consists of 11-12 targets and lasts a maximum of 40 minutes; each team performs five 40-minute missions. The synthetic teammate will act as the AVO in the UAV-STE, and is being developed using the ACT-R computational cognitive architecture (Anderson, 2007). ACT-R has been under continuous development for several decades and is now capable of accurately reproducing human microcognitive processes (e.g., memory retrieval, skill acquisition, etc.) Without detailing ACT-R, cognition revolves around the interaction between a central procedural system and several peripheral modules. There are modules for vision, motor capabilities, memory, one for storing the model's intentions for completing the task (i.e., the control state), and a module for storing the mental representation of the task at hand (problem state, see Figure 1). (For more detail on ACT-R, see Anderson, 2007.) We have chosen ACT-R for two important reasons. First, there is an abundance of ACT-R expertise between PALM and CERI. Second, and more importantly, ACT-R provides a good foundation to investigate how macrocognitive processes (e.g., meta- cognition, situation assessment/awareness, etc.) affect microcognitive processes, and vice versa (Cooke & Myers, 2008). Because ACT-R provides good quantitative predictions of human performance across many microcognitive processes, using them as a foundation for developing macrocognitive processes will help to uncover how micro and macro processes interact within complex task environments. Task Environment Figure 1. The modules of the ACT-R 6.0 computational cognitive architecture. Adapted from Anderson (2007) Synthetic teammate development has been, and will continue to be, managed through a divide-and-conquer strategy across a set of components, combined with a synthesis strategy for component integration. To support synthesis and cognitive plausibility of the three major components, they are all being developed using the ACT-R architecture. The major components include: 1) language comprehension, 2) language generation and dialog management, and 3) task behavior. Each of these components will interact through a central situation component (see Figure 2). Language Generation & Dialog Management Visual Information (text chat, flight parameters, etc.) Language Comprehension Situation Component Manual Output (typing, mousedicks, etc.) Task Behavior Figure 2. Functional components of the synthetic teammate Language Comprehension The language comprehension system has been under development since 2002 (Ball, Heiberg & Silber, 2007). It is based on a linguistic theory of the grammatical encoding of referential and relational meaning (Ball, 2007a) combined with a theory of language processing based on the activation, selection, and integration of constructions corresponding to a wide range of linguistic input (Ball, 2007b). The component incrementally processes input in real-time, constructing a linguistic representation that encodes referential and relational meaning. Language Generation and Dialog Modeling The language generation and dialog component was developed over the course of 18 months, beginning in November 2006. The component does an adequate job of matching human behavior using variabilized "utterance templates" in concert with a strict constraint hierarchy based on principles of optimality theory (Prince & Smolensky, 1993/2004). The constraint hierarchy is implemented using ACT-R's spreading activation mechanism. Situational constraints activate utterance templates that are instantiated as chunks within ACT-R's declarative memory, and the template with the greatest activation is selected for language generation. Despite the constraint hierarchy, noise in ACT-R's spreading activation mechanism allows for occasional hierarchy reversals. Furthermore, the constraint hierarchy is adaptive, resulting in variation in the constraint hierarchy over time as a function of communication experience with different teammates. Task Behavior Component The task behavior component has been under development since summer 2007. The task model was developed as a flat goal-subgoal hierarchy based on task environment constraints. Task goals are stored as chunks in declarative memory, and are retrieved when appropriate. For example, a declarative memory chunk representing a task-related activity (i.e, a goal) is retrieved from memory as a function of the model's situation. As the situation changes, different goal chunks are retrieved from memory, producing flexible and robust goal selection and execution. Situation Component The situation component was implemented as one ACT-R declarative memory chunk that was stored in the synthetic teammate's problem state buffer, and was used in a similar capacity to the "blackboard" in blackboard architectures used in artificial intelligence research. The chunk contained 49 slots for storing information associated with task environment and language generation states. As the synthetic teammate operated as the AVO in the UAV reconnaissance task, information within the situation component chunk was updated to reflect the state of the task environment and communications between teammates. The information held in the situation component is used to retrieve and execute specific goals, leading to situated cognitive control within each functional component of the synthetic teammate (see Figure 2). Information held in the situation component implementation did not decay, making situation information potentially eternally available to the synthetic teammate, going well above human situation assessment/awareness capabi 1 ities. Integrated Components The task behavior and the language generation and dialog components were combined into a single, integrated cognitive system (Gray, 2007) capable of behaving in the UAVSTE and sharing task-related information with teammates as a function of a representation stored in the situation component prior to the current ONR award. A key to this initial integration was the introduction of implicit task switching between the two components. Background Summary There have been significant advances made in developing a cognitively plausible model that that can interact with teammates using the ACT-R architecture. First, language comprehension capabilities have been steadily advancing over the past two years. Second, language generation capabilities have advanced to the point that the synthetic teammate is capable of generating utterances based on a primitive situation representation. Third, knowledge and rules enabling task-relevant behavior have been integrated with the language generation component, producing a composite component. Although much progress was made, more needed to occur to ultimately reach the goal of developing a synthetic teammate that can interact with humans in real-time and provide insight into how macrocognitive and microcognitive process interact. The following section covers the statement of work for the current ONR award. Statement of Work The following sections detail the planned improvements to the situation component, deliverables, and milestones associated with the current ONR award. Planned Improvements to the Situation Component The situation component is important for representing complex agent states resulting from agent-task-team interactions. Inputs to the situation component will include task environment states (e.g., the UAV's current altitude, speed, course, etc.) combined with communication from teammates (e.g., desired UAV airspeeds and altitudes, upcoming targets, etc.), and background knowledge (e.g. information about the other teammates). Development of the situation component will contribute some of the groundwork necessary for developing high-level, or macrocognitive, processes (e.g., reasoning over complex scenarios, simulating others beliefs, predicting future states, etc.) within a computational cognitive architecture that operates at a relatively low-level of cognitive and perceptual/motor behavior (i.e., ACT-R 'operates' at 50 ms cycles, Anderson, 2007). Although the blackboard implementation has worked well for developing and integrating the task behavior and language generation and dialog components, the situation component must be improved to reflect human capabilities. Furthermore, improvement to the situation component will facilitate the integration of the language comprehension component and provide a sufficient foundation for other macrocognitive processes (i.e, reasoning, predicting future states, etc.) within the ACT-R architecture. The situation component will combine relevant information gleaned from linguistic and task environment inputs, supporting language generation and continuing task behavior. Different from the blackboard implementation, the planned implementation of the situation component will be primarily propositional in nature—information and relationships between information described in the linguistic input and derived from the task environment and background knowledge will be encoded in a propositional manner. Because the synthetic teammate is modeled using ACT-R, propositional representations will ultimately be implemented using ACT-R's chunk-based representations. Integrating the Language Comprehension Component. A key commitment of the language comprehension component is a general capability to handle basic grammatical constructions of English, enabling use of the component across different applications and models. Issues like lexical and structural ambiguity, ungrammatical inputs, and variability in input forms cannot be ignored. Despite the advanced state of the component, the requirements for adequate language comprehension over an open-ended range of input are significant, and much remains to be done. Another key commitment is the position that adherence to well-established constraints on human language processing may actually facilitate the goal of building a functional language comprehension component (Ball, 2006). For example, the common use of a separate part-of-speech tagging routine whose output feeds a separate parsing process is eschewed in favor of an integrated approach that is capable of operating incrementally in real-time. Moving the situation component to a propositional format will facilitate integrating the language comprehension component with the task behavior/language generation system since the linguistic representations derived from the language comprehension component contain much of the information needed to generate propositional representations. Complete component integration will result in a combined computational cognitive model that is likely to exceed most existing cognitive models in size. Questions of efficiency and complexity will need to be addressed to ensure that the synthetic teammate is capable of functioning in real-time without exceeding resource capacities within the timeframe for a mission in the UAV-STE (~40 minutes). Continued Development with Humans In-the-Loop. The development of the synthetic teammate will be in vain without validation, and we view validation as more than producing a working agent. It is possible that the synthetic teammate's interactions are unnatural, disruptive, or unfaithful to those of humans performing the same task even though it adequately pilots the UAV in the STE. Consequently, we plan on beginning human-in-the-loop validation procedures as soon as practically possible. Looking to previous research on synthetic teammates for direction, it is clear that validation is rarely accomplished, or goes beyond anecdotal comments from human participants. We are planning a hierarchical validation procedure: • Key-press level - comparing model data to human data within a goal, such as setting airspeed, altitude, course, current waypoint, sending a message, etc. • Goal-selection level - comparing goal execution sequences to humans (general order of goals executed to reach waypoints. This level of analysis will help validate the situation component, as the component has a direct influence on which goal is selected and when. • Mission level - comparing synthetic teammate mission performance to human performance across a set of UAV-STE missions. • Team level - comparing team performance between teams that have an incorporated synthetic teammate and all-human teams. Validation at the key-press and goal-selection levels began with official start of the proposed research (Myers, 2009). Mission level and team level validation procedures will begin on once the synthetic teammate is fully capable of interacting with human teammates. Deliverables Prototype demonstration* 8/01 /2009 Final technical report 1/31/2010 Publication and presentation 1 /31 /2010 *We recognized that a fully functional and valid synthetic teammate was likely unattainable within a year; however we nonetheless planned to have a partially functional prototype to showcase the accomplishments of our year-long effort. Also in the interest of risk mitigation this effort has advanced the state-of-the-art of cognitive modeling and synthetic teammates in the following ways (Ball, Myers, Heiberg, et al., submitted): • Improved understanding for managing and coordinating the development of large scale computational cognitive models • Improved understanding of how to develop macrocognitive processes in a cognitive architecture situated around microcognitive processes. • Advantages and disadvantages of propositional representations within the situation component • Advantages and disadvantages of using a situation component as a large scale model's form of cognitive control • Advantages and disadvantages associated with modeling approaches for each of the components • Improved understanding about solutions to software and hardware issues associated with developing large scale models Project Schedule and Milestones Milestone: Date: December 1,2008 Project kick-off March 1, 2009 Situation component specification finalized June 1,2009 Begin integrating language comprehension component with task behavior/language generation and dialog management system. August 1,2009 Prototype demonstration September 1,2i November 30, 2009 in model validation with teammat Project end Summary of Scientific and Technical Objectives, Background, & Approach The synthetic teammate is being developed using the ACT-R computational cognitive architecture (Anderson, 2007). The synthetic teammate will interact with two humans in real-time to accomplish a UAV reconnaissance task within a synthetic task environment. The UAV-STE is a dynamic command and control task which requires three teammates (photographer, navigator, and pilot) to interact with each other to attain the common goal of photographing as many reconnaissance targets as possible over a the course of multiple 40-minute missions. The synthetic teammate plays the role of the pilot in the UAV-STE. Four macrocognitive components have been identified for the synthetic teammate to adequately interact with its environment and teammates: language comprehension, language generation/dialog management, situation assessment, and agent-environment interaction. As part of a larger collaboration with scientists at AFRL and Michael Matessa of Alion, each of the components have been developed and integrated, facilitated through the use of the ACT-R cognitive architecture. The language generation/dialog management component was successfully integrated with the agent-environment interaction component prior to the beginning of the current ONR award. The current ONR award provided support for the successful development and integration of the situation assessment and language comprehension components with the agentenvironment interaction and language generation components. The refinement of each synthetic teammate macrocognitive component, as well as the interactions among them, have benefited from development iteration with teammates in the loop. Failures of the model working with teammates have provided fodder for further development of the synthetic teammate's components. Eventually, the synthetic teammate will provide insight into how individual teammates affect team processes, and how team processes affect individuals' cognitive processes. Concise Accomplishments (Accomplishments resulting directly from the current ONR award are italicized) • Major expansion of the linguistic coverage of the language comprehension component (language comprehension; Dr. Jerry Ball of AFRL) • Addition of an externalized, persistent Declarative Memory (DM) system to ACT-R 6 (persistent DM: Drs. Scott Douglass and Jerry Ball of AFRL) • Significant improvement in the handling of variability in the linguistic input (linguistic input variability: Mrs. Mary Freiman of L3 Communications) • Modification of the ACT-R cognitive architecture to improve the word recognition subcomponent (improved word recognition; Mary Freiman of L3 Communications) • Situation assessment component specified in sufficient detail to support processing of scripted communication and is poisedfor expansion beyond scripted communication (situation assessment) • Situation assessment component fully integrated with synthetic teammate model (full integration) • Specification (Dr. Scott Douglass) and development of low cognitive fidelity teammates in-the-loop development and testing of synthetic teammate (lowfidelity agents) • Keypress level model validation effort demonstrated an excellent fit between model and human data (key-press level validation) • Demonstration of synthetic teammate after integration of language comprehension, agent-environment interaction, language generation, and situation components (demonstration) 10 • Data analysis of human teams from an experiment that manipulated communication mode (text-chat vs. audio) that serves as baseline data for comparisons with the synthetic teammate incorporated in teams (further analyses) Expanded Accomplishments Language comprehension. A key commitment of the project was for the synthetic teammate to be capable of handling a broad range of linguistic inputs. To support this, the language comprehension component has been significantly expanded over the last year. In particular, the language comprehension component is now capable of handling a broad range of linguistic constructions including declarative, interrogative and imperative sentences of various types, sentences including a broad range of verbal constructions including intransitive, transitive, and ditransitive verbs, and verbs which take a clausal or locative complement. Besides expanding the general linguistic coverage, the language comprehension component has been expanded to handle the specific constructions that occur in the corpus collected in past experiments. These include existential there constructions (e.g. "there are no restrictions"), predicate nominal constructions (e.g. "the next waypoint is a target"), and contractions of subject and auxiliary (e.g. "It's a target"). Efforts are currently underway to develop mechanisms for automatically expanding the lexical coverage of the language comprehension component. In particular, we are using the British National Corpus and WordNet as resources for expanding the coverage of the system. To take advantage of these resources, the lexical entries they contain must be mapped into the lexical ontology used by the language comprehension component. We are very near to having a capability to do this. Persistent DM. To meet the large declarative memory (DM) requirements of the synthetic teammate, we have developed an external DM storage and retrieval capability in ACT-R 6 based on PostgreSQL, a powerful, open source object-relational database management system (DBMS). This "persistent-DM" module outsources the storage of ACT-R DM elements, or chunks, to an industrial-strength external DBMS while leaving ACT-R's DM retrieval calculus untouched. The persistent-DM module modifies the behavior of ACT-R's declarative module by: (1) introducing seven control parameters; (2) providing programmatic support for managing the interaction between ACT-R and the PostgreSQL DBMS; (3) extending the retrieval process; and (4) modifying the comparison of chunk slots. The addition of the persistent-DM DBMS does not interfere with ACT-R predictions associated with the retrieval and storage of declarative knowledge, but instead outsources the declarative system from Lisp to improve computational efficiency. Linguistic input variability and improved word recognition. Whereas previous experiments were based on audio communications, the most recent experiment (funded by AFOSR in the precursor to the current ONR award) involved text messaging and we now have a corpus of text messages that reveals tremendous variability in the form of the linguistic input (i.e., misspellings, abbreviations, etc.). The word recognition subcomponent of the language comprehension component has been significantly modified to handle this variability. In addition, the perceptual encoding mechanism of the ACT-R cognitive architecture was modified to support the perception of a perceptual II span that is not limited to space delimited words as in ACT-R's default mechanism. This modification improves the performance of the word recognition component at the same time that it improves cognitive fidelity. Situation assessment. The situation assessment component of the synthetic teammate represents the current situation as informed by the linguistic input, the task environment, the discourse context, and background knowledge. Further, the situation component constitutes the primary meaning representation of the system, although the linguistic representations that are mapped into the situation component also encode important aspects of meaning. Consequently, the situation component grounds the meaning of referring expressions in the linguistic input in the objects and situations from the task environment, discourse context and background knowledge, all of which can be stored in the situation component. We have developed a situation component that handles propositional and discourse content, and designed the component to eventually handle spatial and imaginal content within the ACT-R cognitive architecture. The result was a new ACT-R module that contains eight buffers. The buffers are used for maintaining declarative memory chunks that represent the perceived situation of the synthetic teammate. These chunks are retrieved from memory and used to process received communications, reason, and make decisions. Full integration. The initial integration of the four cognitive components of the synthetic teammate was grounded in the ability of the integrated synthetic teammate to handle the communications required to fly thru the first two waypoints of a UAV reconnaissance mission. Because the amount of communications required to achieve this is limited, we refer to these as scripted communication. For the initial integration, the focus was on getting the components to work together. Once this was achieved, the focus shifted to expanding the capability of the synthetic teammate to process a broader range of communications and behaviors over the course of a 40-minute reconnaissance mission involving more than 10 waypoints. The language comprehension component is currently capable of generating the situation representations corresponding to communications from the scripted communications. The agent-environment interaction component is capable of responding to changes in the situation assessment component, such as changing airspeed, altitude, waypoints, et cetera. The language generation/dialog management component is capable of generating the synthetic teammate's text-based communications for responding to comprehended communications and/or providing updates to teammates on the performance of the UAV. Low-fidelity agents. In order to test the synthetic teammate's macrocognitive processes when interacting with teammates, we developed low-cognitive-fidelity photographer and navigator agents using systems of finite state automata. The formalism for developing the agents was developed by Dr. Scott Douglass at AFRL as an existence proof for developing models using a hybrid of text and a graphical modeling language (e.g., General Modeling Environment, GME) that can be automatically transformed into an executable model. The models of the photographer and navigator were developed by the PI of the grant and interact with the synthetic teammate through the instant messaging 12 system. The models can be scaled-up in a simple manner to produce communications based on increased environmental complexity and were developed to eliminate the need and cost of having humans play the role of the synthetic teammate's navigator and photographer. Key-press level validation. An effort to validate the agent-environment interaction component of the synthetic teammate at the key press and strategy level of analysis was completed. The results demonstrated that the model adequately approximated the performance of human operators performing for necessary setting tasks for flying the UAV from one location to another: setting the airspeed, course, altitude, and new location. Consequently, the model is a good approximation to human performance at the key press and setting task strategy levels of analysis. The next validation effort of the agent-environment interaction component is to compare the synthetic teammate's strategies for choosing which setting task is performed, and the environment state/context when it was selected (Myers, 2009). Demonstration. A demonstration was performed for Dr. Paul Bello, the project's manager at the Office of Naval Research. A video version of the demonstration provided to Dr. Bello is included as supplemental information with this report. Further analyses. Recent data analyses have revealed that there is no statistical difference in team performance between teams that communicate using audio and teams that communicate with a text-based communication system. However, evidence has emerged that teams using text-based communications coordinate in patterns that differ from communication patterns of teams that use audio communications. Specifically, there is an inherent lag in communication reception when using text-based communications that does not occur in audio-based communications. The average lag time for the text-based communications was 10.52 seconds. A 3 (teammate) x 4 (mission) mixed Analysis of Variance (ANOVA) revealed that communication reception lag was a function of mission and teammate, where the lag decreased as mission number increased, but did so differentially based on teammate role (navigator, pilot, photographer, see Figure 1). 13 Pilot Navigator Photographer 25 -r 12 3 4 Mission Figure 1. The teammate x mission interaction effect on communication lag time. Team coordination (K) was computed using a ratio of times associated with key components of information sharing, such as information associated with upcoming targets (i.e., information; Iw), altitude and airspeed negotiation between the photographer and the pilot (negotiation; Nw), and feedback from the photographer that a good picture was taken (feedback; Fw): K-N The formula for calculating coordination score (K) has been used in previous experiments to determine differences in coordination dynamics between treatment groups in team experiments (Cooke, Gorman, Pedersen, et al., 2007). To determine if there was a difference in coordination score between audio-comm and text-comm teams, a 2 (communication mode) x 4 (mission) mixed ANOVA was conducted on coordination scores. There was a significant main effect for which text-comm had a significantly lower coordination score than audio-comm (p = 0.042). This is not to say that the audio comm condition coordinated "better" but only to say that the two communication conditions coordinated differently. Further, a measure that reveals the stability of team coordination dynamics, the Hurst exponent, was also analyzed to determine if there was a coordination stability difference between communication groups. An independent samples Mest on the average Hurst exponents across teams revealed that text-comm teams were, on average, coordinated in a more stable fashion (M= 0.9527, SD = 0.0131) than voice-comm teams (M= 0.8988, SD = 0.061), f(15) = 2.287, p = 0.037. 14 Major Problems/Issues There are no major problems other than the sheer amount of work necessary to get to the point of bringing human teammates into the development loop. Mentioned above, the integration of the situation assessment and language comprehension components with the rest of the synthetic teammate has proven to be a challenge. The challenge results from making changes to the components in order to accommodate integration while at the same time maintaining the efficacy of each individual component. Although this is not a "major problem" it should be noted that integration is also not trivial. A further issue is synthetic teammate/CERTT UAV-STE integration. This type of integration issue is a regular occurrence when integrating any modeling formalism with a dynamic synthetic task environment. Stuart Rodgers and Dr. Steven Shope have developed a mechanism for providing environment information to the synthetic teammate, which is currently under refinement to improve system response times. Technology Transfer • • • Interactions with Dr. Greg Trafton and Dr. Raj Ratwani of the Naval Research Laboratory regarding approaches to embodied models of situation awareness/assessment. Interactions with Dr. Wink Bennett, Craig Eidman and Dr. Gary Boyle at the Air Force Research Laboratory on training applications of synthetic wingmen in the Joint Terminal Attack Controller Dome at the Air Force Research Laboratory in Mesa, AZ. Interactions with the US Air Force Scientific Advisory Board to provide input to their study on the application of cognitive modeling to Virtual Training Technologies. References Anderson, J. R. (2007). How Can the Human Mind Occur in the Physical Universe? Oxford: Oxford University Press. Ball, J. (2006). Can NLP Systems be a Cognitive Black Box? Papers from the AAAI Spring Symposium, Technical Report SS-06-02, 1-6. Menlo Park, CA: AAAI Press Ball, J. (2007a). A Bi-Polar Theory of Nominal and Clause Structure and Function. Annual Review of Cognitive Linguistics, 27-54. Amsterdam: John Benjamins Ball, J. (2007b). Construction-Driven Language Processing. Proceedings of the 2nd European Cognitive Science Conference, 122-121. Edited by S. Vosniadou, D. Kayser & A. Protopapas. NY: LEA. Ball, J., Heiberg, A. & Silber, R. (2007). Toward a Large-Scale Model of Language Comprehension in ACT-R 6. Proceedings of the 8th International Conference on 15 Cognitive Modeling, 173-179. Ed by R. Lewis, T. Polk & J. Laird. NY: Psychology Press. Cooke, N. J., Gorman, J. C, Pedersen, H. K., Winner, J. L., Duran, J., Taylor, A., Amazeen, P. G, & Andrews, D. (2007). Acquisition and retention of team coordination in command-and-control. Technical Report for AFOSR Grant FA9550-04-1-0234 and AFRL Award No. FA8650-04-6442. Cooke, N. J., & Myers, C. W. (2008). Agent Based Approaches to Macrocognition: An ACT-R Model of a Synthetic Teammate. Gray, W. D. (Ed.). (2007). Integrated Models of Cognitive Systems. Oxford: OUP. Prince, A. & Smolensky, P. (1993/2004). Optimality Theory: Constraint Interaction in Generative Grammar. Oxford: Blackwell. Foreign Collaborations and Supported Foreign Nationals None. Productivity Myers, C. W., Best, B., Cooke, N. J., Lewis, M., McNeese, M., Zachary, W. (2009). Synthetic Agents as Full-fledged Teammates. Panel discussion at the 53rd Annual Meeting of the Human Factors and Ergonomics Society, San Antonio, TX. Ball, J. (2008). A Naturalistic, Functional Approach to Modeling Language Comprehension. Papers from the AAAI Fall 2008 Symposium, Naturally Inspired Artificial Intelligence. Menlo Park, CA: AAAI Press. Ball, J., Myers, C. W., Heiberg, A., Cooke, N. J., Matessa, M., & Freiman, M. (2009). The Synthetic Teammate Project. Proceedings of the 18f Annual Conference on Behavior Representation in Modeling and Simulation. Sundance, UT. Ball, J., Myers, C. W., Heiberg, A., Cooke, N. J., Matessa, M., & Freiman, M., Rodgers, S. (submitted). The Synthetic Teammate Project. Computational and Mathematical Organization Theory Journal. Freiman, M. & Ball, J. (2008). Computational cognitive modeling of reading comprehension at the word level. Proceedings of the 38? Western Conference on Linguistics. Myers, C. W., Cooke, N. J., Ball, J. T., Heiberg, A., Gluck, K. A., & Robinson, F. E. (submitted). Collaborating with Synthetic Teammates. In W. Bennett (Ed.), Collaboration in Complex Task Environments. Myers, C. W. (2009). An Account of Model Inspiration, Integration, & Sub-task validation. Paper presented at the 9th International Conference on Cognitive Modeling, Manchester, UK.