Search

We describe a lifelong learner modeling project that focuses on the use of written and behavioral data to detect patterns of learning over time. Related work in essay analysis and machine learning is discussed. Although primarily focused on isolated learning experiences, we argue there is promise for scaling these techniques up to the lifelong learner modeling problem., Presented at the Proceedings of the User Modeling, Adaptation, and Personalization (UMAP) Workshop on Lifelong Learner Modeling (54-61, June 2009). U.S. Government or Federal Rights License

Accurately estimating the person's head position and orientation is an important task for a wide range of applications such as driver awareness and human-robot interaction. Over the past two decades, many approaches have been suggested to solve this problem, each with its own advantages and disadvantages. In this paper, we present a probabilistic framework called Monocular Adaptive View-based Appearance Model (MAVAM) which integrates the advantages from two of these approaches: (1) the relative precision and user-independence of differential registration, and (2) the robustness and bounded drift of keyframe tracking. In our experiments, we show how the MAVAM model can be used to estimate head position and orientation in real-time using a simple monocular camera. Our experiments on two previously published datasets show that the MAVAM framework can accurately track for a long period of time (more than 2 minutes) with an average accuracy of 3.9 degrees and 1.2 inches with an inertial sensor and a 3D magnetic sensor., See also ADM002187. Presented at the Army Science Conference (26th) held in Orlando, Florida on 1-4 December 2008. Published in the Proceedings of the Army Science Conference (26th), December 2008. Sponsored in part by the Naval Research Laboratory (NRL) under grant NRL-55-05-03. The original document contains color images.

We describe a text classification approach based on statistical language modeling. We show how this approach can be used for several natural language processing tasks in a virtual human system. Specifically, we show it can be applied to language understanding, language generation, and character response selection tasks. We illustrate these applications with some experimental results., See also ADM002187. Presented at the Army Science Conference (26th) held in Orlando, Florida on 1-4 December 2008. Published in the Proceedings of the Army Science Conference (26th), December 2008.

The infusion of commercial game technology into U.S. Army training, simulation, and instructional domains has resulted in more immersive and engaging experiences for Soldiers to hone their skills. However, the influx of such technology comes at a significant cost, specifically in the creation of virtual environments in which these skills are simulated and practiced. Today's typical commercial triple-A game title cost upwards of $40- $60M and four to six years to develop, much of which is spent on producing the digital assets used to populate the scene (models, animations, etc). Additionally, this content is often suited for a custom type of rendering technology, and often cannot be reused without significant manual modification. Unfortunately, the Army has neither the financial or personnel resources available to create such highly immersive, reusable virtual content, nor the time to invest when current operations call for training or simulation data in a matter of hours, not months or years. In this paper, we discuss a research initiative aimed at significantly reducing the time and cost for converting, optimizing, and enhancing existing geospatial data for today's virtual environments. The goal is a completely automated process for ingesting existing military terrain data and outputting a technology agnostic representation in less than 24 hours., See also ADM002187. Proceedings of the Army Science Conference (26th) Held in Orlando, Florida on 1-4 December 2008, The original document contains color images.

We demonstrate three classes of off-topic responses which allow a virtual question-answering character to handle cases where it does not understand the user's input: ask for clarification, indicate misunderstanding, and move on with the conversation. While falling short of full dialogue management, a combination of such responses together with prompts to change the topic can improve overall dialogue coherence., See also ADM002187. Presented at the Army Science Conference (26th) held in Orlando, FL on 1-4 December 2008. The original document contains color images.

Post Traumatic Stress Disorder (PTSD) is reported to be caused by traumatic events that are outside the range of usual human experience including (but not limited to) military combat, violent personal assault, being kidnapped or taken hostage and terrorist attacks. Initial data suggests that at least 1 out of 5 Iraq War veterans are exhibiting symptoms of depression, anxiety and PTSD. Virtual Reality (VR) delivered exposure therapy for PTSD has been previously used with reports of positive outcomes. The current paper will present the rationale and description of a VR PTSD therapy application (Virtual Iraq) and present initial findings from its use with active duty service members. Virtual Iraq consists of a series of customizable virtual scenarios designed to represent relevant Middle Eastern VR contexts for exposure therapy, including a city and desert road convoy environment. User-centered design feedback needed to iteratively evolve the system was gathered from returning Iraq War veterans in the USA and from a system deployed in Iraq and tested by an Army Combat Stress Control Team. Results from an open clinical trial using Virtual Iraq at the Naval Medical Center-San Diego with 20 treatment completers indicate that 16 no longer met PTSD diagnostic criteria at post-treatment, with only one not maintaining treatment gains at 3 month follow-up., See also ADM002187. Proceedings of the Army Science Conference (26th) Held in Orlando, Florida on 1-4 December 2008, The original document contains color images. Sponsored in U.S. Army Research Development and Engineering Command.

In this paper we contrast three architectures for natural language questioning characters. We contrast the relative costs and benefits of each approach in building characters for tactical questioning. The first architecture works purely at the textual, using cross-language information retrieval techniques to learn the best output for any input from a training set of linked questions and answers. The second architecture adds a global emotional model and computes a compliance model, which can result in different outputs for different levels, given the same inputs. The third architecture works at a semantic level and allows authoring of different policies for response for different kinds of information. We describe these architectures and their strengths and weaknesses with respect to expressive capacity, performance, and authoring demands., See also ADM002187. Published in the Proceedings of the Army Science Conference (26th) Held in Or.lando, Florida on 1-4 December 2008.

Increasingly, the military has requirements for teaching cultural awareness, which demands flexible representations of cultural knowledge. The Culturally-Affected Behavior project seeks to define a language for encoding ethnographic data in order to capture cultural knowledge and use that knowledge to affect human behavior models. Having anthropologists encode ethnographic data will validate the language and will result in a library of culture models for immersive training.

We describe some key issues involved in building an intelligent tutoring system for the ill-defined domain of interpersonal and intercultural skill acquisition. We discuss the consideration of mixed-result actions (actions with pros and cons), categories of actions (e.g., required steps vs. rules of thumb), the role of narrative, and reflective tutoring, among other topics. We present these ideas in the context our work on an intelligent tutor for ELECT BiLAT, a game-based system to teach cultural awareness and negotiation skills for bilateral engagements. The tutor provides guidance in two forms: (1) as a coach that gives hints and feedback during an engagement with a virtual character, and (2) during an after-action review to help the learner reflect on their choices. Learner activities are mapped to learning objectives, which include whether the actions represent positive or negative evidence of learning. These underlie an expert model, student model, and models of coaching and reflective tutoring that support the learner. We describe several other cultural and interpersonal training systems that situate learners in goal-based social contexts that include interaction with virtual characters and automated guidance. Finally, our future work includes evaluations of learning, expansion of the coach and reflective tutoring strategies, and integration of deeper knowledge-based resources that capture more nuanced cultural aspects of interaction., Presented at the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) held in Orlando, FL on 26-29 Nov 2007. Published in the Proceedings of Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), Paper 7417, 2007. The original document contains color images.

Case method instruction has been universally endorsed as an effective teaching approach, yet little empirical research provides evidence of this claim. This report describes research that investigated the importance of the discussion component of case method instruction to learning. In addition to collecting data from 182 United States Military Academy cadets, this investigation extends past research on case method instruction by incorporating the perspectives of instructors. Findings indicated that students produced better quality answers and were better able to diagnose leadership problems in the scenario after participating in discussion, even though the time allotted for discussion was relatively short. Self-reports of instructors provided corroborating evidence that the instructional approach was useful for accomplishing learning objectives and that class discussion quality was high. Additional results point to the importance of having an instructor's manual available for discussing case studies. Results also suggest that the particular case study used, "Power Hungry," generalizes from an audience of junior officers and NCOs to a cadet audience., Prepared in cooperation with Kansas State University, Manhattan, KS.

We present an evaluation of a spoken dialogue system that engages in dialogues with soldiers training in an immersive Call for Fire (CFF) simulation. We briefly describe aspects of the Joint Fires and Effects Trainer System, and the Radiobot-CFF dialogue system, which can engage in voice communications with a trainee in call for fire dialogues. An experiment is described to judge performance of the Radiobot CFF system compared with human radio operators. Results show that while the current version of the system is not quite at human-performance levels, it is already viable for training interaction and as an operator-controller aid., The original document contains color images.

Large corpora of parsed sentences with semantic role labels (e.g. PropBank) provide training data for use in the creation of high-performance automatic semantic role labeling systems. Despite the size of these corpora, individual verbs (or rolesets) often have only a handful of instances in these corpora, and only a fraction of English verbs have even a single annotation. In this paper, we describe an approach for dealing with this sparse data problem, enabling accurate semantic role labeling for novel verbs (rolesets) with only a single training example. Our approach involves the identification of syntactically similar verbs found in Prop-Bank, the alignment of arguments in their corresponding rolesets, and the use of their corresponding annotations in Prop-Bank as surrogate training data.

We argue that metacognition is a critical component in the development of intercultural competence by highlighting the importance of supporting a learner's self-assessment, self-monitoring, predictive, planning, and reflection skills. We also survey several modern immersive cultural learning environments and discuss the role intelligent tutoring and experience management techniques can play to support these metacognitive demands. Techniques for adapting the behaviors of virtual humans to promote cultural learning are discussed, as well as explicit approaches to feedback. We conclude with several suggestions for future research, including the use of existing intercultural development metrics for evaluating learning in immersive environments and to conduct more studies of the use of implicit and explicit feedback to guide learning and establish optimal conditions for acquiring intercultural competence.

Post Traumatic Stress Disorder (PTSD) is reported to be caused by traumatic events that are outside the range of usual human experience, including (but not limited to) military combat, violent personal assault, being kidnapped or taken hostage, and terrorist attacks. Initial data suggest that at least 1 out of 6 Iraq War veterans are exhibiting symptoms of depression, anxiety, and PTSD. Virtual Reality (VR) delivered exposure therapy for PTSD has been used with reports of positive outcomes. The aim of the current paper is to present the rationale, technical specifications, application features, and user-centered design process for the development of a Virtual Iraq PTSD VR therapy application. The VR treatment environment is being created via the recycling of virtual graphic assets that were initially built for the U.S. Army-funded combat tactical simulation scenario and commercially successful X-Box game, Full Spectrum Warrior, in addition to other available and newly created assets. Thus far we have created a series of customizable virtual scenarios designed to represent relevant contexts for exposure therapy to be conducted in VR, including a city and desert road convoy environment. User-centered design feedback needed to iteratively evolve the system was gathered from returning Iraq War veterans in the USA and from a system in Iraq tested by an Army Combat Stress Control Team. Clinical trials are currently underway at Camp Pendleton and at the San Diego Naval Medical Center. Other sites are preparing to use the application for a variety of PTSD and VR research purposes., Published in the Proceedings of the 6th International Conference on Disability, Virtual Reality, and Associated Technologies, p113-122, 2006 (held in Esbjerg, Denmark, on 18-20 Sep 2006). Sponsored in part by the Telemedicine and Advanced Technology Research Center (TATRC) and the U.S. Army Research, Development, and Engineering Command (RDECOM).

AXL.Net is a prototype web-based immersive technology solution that supports case method teaching for U.S. Army leader development. The AXL.Net system addresses three challenges: (1) designing a pedagogically-sound research prototype for leader development, (2) integrating research technologies with the best of Web 2.0 innovations to enhance case method teaching, and (3) providing an easy to use system. Initial evaluations show that the prototype application and framework is effective for leader development., See also ADM002075. Presented at the Army Science Conference (25th) held in Orlando, FL, on 27-30 Nov 2006. Prepared in collaboration with Army Research Institute. Sponsored in part by U.S. Army RDECOM. Document includes briefing charts in addition to text. The original document contains color images.

OUTLINE: Virtual Reality Call for Fire Training * The Radiobot-CFF System * Evaluation Method * Evaluation Results * Next Steps., See also ADM002075. Presented at the Army Science Conference (25th) held in Orlando, FL, on 27-30 Nov 2006. The original document contains color images.

Many image-based modeling and rendering techniques involve photographing a scene from an array of different viewpoints. Usually, this is achieved by moving the camera or the subject to successive positions, or by photographing the scene with an array of cameras. In this work, we present a system of mirrors to simulate the appearance of camera movement around a scene while the physical camera remains stationary. The system thus is amenable to capturing dynamic events avoiding the need to construct and calibrate an array of cameras. We demonstrate the system with a high speed video of a dynamic scene. We show smooth camera motion rotating 360 degrees around the scene. We discuss the optical performance of our system and compare with alternate setups., See also ADM002075., The original document contains color images. Presented at the Army Science Conference (25th) held in Orlando, FL on 27-30 November 2006.

We present a fully image-based approach for capturing and modeling real human locomotion under varying illumination and viewpoint that overviews the techniques and results presented by [Einarsson et al, 2006]. An actor performs repeatable locomotive actions (walking/running) on a rotating treadmill while being filmed from a vertical array of 3 high-speed cameras under controlled rapidly changing lighting conditions. The known rotation of the treadmill, repeatability of the actor's motion, timing of the lighting pattern and capture rate of the cameras are all carefully synchronized so that the actor is imaged in (approximately) the same position in the locomotion at the same point in the lighting pattern but having rotated a known amount due to the known turntable motion. This allows us to effectively multiply the number of cameras from 3 1 in azimuth to 3 36. Small perturbations in the actor's repeating cyclic position are corrected for using optical flow, and optical flow is also used to align images temporally. This leads to a flowed reflectance field data structure. Datasets are compressed using image compression. Image-based relighting and a combination of view morphing and light field rendering implemented on the GPU allow us to render the subject under novel viewpoint and illumination. To composite the person into a scene we derive an alpha matte from retro-reflective material and a back-lit diffuse backdrop, and implement a voxel-based visual hull process to compute how the person should cast shadows on the ground plane. We demonstrate realistic composites of real subjects into real and virtual environments applicable to the area of training simulation., Presented at the Army Science Conference (25th), "Transformational Army Science and Technology - Charting the future of S&T for the Soldier," held in Orlando, Florida on 27-30 Nov 2006. Published in proceedings of the same. See also ADM002075. The original document contains color images.

The U.S. Army is currently interested in developing state-of-the-art training methods that leverage technology based on established and emerging immersive mixed and virtual environment systems employing both head mounted and spatially immersive display technologies. A primary motivation for utilizing mixed and virtual environment systems is that they create cost-effective simulations of relevant military challenges while training a variety of skills and processes. However, in a sometimes single-minded effort to directly advance training technology, the area of performance testing and assessment has often been overlooked. There are a number of compelling reasons to address the assessment component concurrently with efforts to build better training tools. It is nearly a given in modern psychology that, before one can begin to focus on training or enhancement of any behavior, one needs to be able to measure and understand the performance that needs to be trained. This requires assessment tools that have demonstrated reliability and validity for measuring criterion performance both pre and post training implementation. Without good assessment metrics, the measurement of training effectiveness is compromised, and as well, the ability to investigate and understand the components of effective performance that are needed to drive the evolution of a training system is not possible. To address this need, we have initiated development of a comprehensive, standardized, norm-based VR cognitive performance assessment test (VRCPAT) battery., Presented at the RTO Human Factors and Medicine Panel (HFM) Workshop held in United States Military Academy, West Point, NY on 13-15 Jun 2006 and published in proceedings of the same as paper 26. See also ADM002024, Virtual Media for Military Applications (Media virtuels pour les applications militaires) (CD-ROM), RTO-MP-HFM-136, AC/323(HFM-136)TP/68. The original document contains color images.

Real time computer graphics are limited in that they can only be displayed on projection screens and monitors. Monitors and projection screens cannot be used in live fire training or scenarios in which the displays could be physically damaged by trainees. To address this issue, we have developed projection systems using computer vision based color correction and image processing to project onto non-ideal surfaces such as painted walls, cinder blocks, and concrete floors. These projector-camera systems effectively paint the real world with digital light. Any surface can become an interactive projection screen allowing unprepared spaces to be transformed into an immersive environment. Virtual bullet holes, charring, and cracks can be added to real doors, walls, tables, chairs, cabinets, and windows. Distortion correction algorithms allow positioning of projection devices out of the field of view of trainees and their weapons. This paper describes our motivation and approach for implementing projector-camera systems for use within the FlatWorld wide area mixed reality system., The original document contains color images.

ELECT BiLAT is a prototype game-based simulation for Soldiers to practice conducting bilateral engagements in a cultural context. The prototype provides students with the experience of preparing for a meeting including familiarization with the cultural context, gathering intelligence, conducting a meeting and negotiating when possible, and following up on meeting agreements as appropriate. The ELECT BiLAT architecture is based on a commercial game engine that is integrated with research technologies to enable the use of virtual human characters, scenario customization, as well as coaching, feedback and tutoring. Because the prototype application is intended to be a learning environment, pedagogy has been central throughout development. The project followed a five-phase process: (1) analyze the training domain; (2) develop a story board prototype; (3) implement a computer version of the training prototype; (4) refine training objectives and link their conditions and standards to game activities; and (5) develop training support content for students, instructors, and training developers. The goal is an authorable game-based environment that uses the pedagogy of guided discovery for training Soldiers in the conduct of bilateral engagements within a specific cultural context., Presented at the Army Science Conference (25th) held in Orlando, FL on 27-30 Nov 2006. Published in the Proceedings of the Army Science Conference (25th), 2006.

Although the representation of physical environments and behaviors will continue to play an important role in simulation-based training, an emerging challenge is the representation of virtual humans with rich mental models (e.g., including emotions, trust) that interact through conversational as well as physical behaviors. The motivation for such simulations is training soft skills such as leadership, cultural awareness, and negotiation, where the majority of actions are conversational, and the problem solving involves consideration of the emotions, attitudes, and desires of others. The educational power of such simulations can be enhanced by the integration of an intelligent tutoring system to support learners understanding of the effect of their actions on virtual humans and how they might improve their performance. In this paper, we discuss our efforts to build such virtual humans, along with an accompanying intelligent tutor, for the domain of negotiation and cultural awareness., Published in Simulation, v82 n11, p685-701, Nov 2006.

This paper focuses not on the detection and correction of specific errors in the interaction between machines and humans, but rather cases of massive deviation from the user's conversational expectations and desires. This can be the result of too many or too unusual errors, but also from dialogue strategies designed to minimize error, which make the interaction unnatural in other ways. We study causes of irritation such as over-fragmentation, over-clarity, overcoordination, over-directedness, and repetitiveness of verbal action, syntax, and intonation. Human reactions to these irritating features typically appear in the following order: tiredness, tolerance, anger, confusion, irony, humor, exhaustion, uncertainty, lack of desire to communicate. The studied features of human expressions of irritation in nonface- to-face interaction are: intonation, emphatic speech, elliptic speech, speed of speech, extra-linguistic signs, speed of verbal action, and overlap., The original document contains color images.

As artificial intelligence (AI) systems and behavior models in military simulations become increasingly complex, it has been difficult for users to understand the activities of computer-controlled entities. Prototype explanation systems have been added to simulators, but designers have not heeded the lessons learned from work in explaining expert system behavior. These new explanation systems are not modular and not portable; they are tied to a particular AI system. In this paper, we present a modular and generic architecture for explaining the behavior of simulated entities. We describe its application to the Virtual Humans, a simulation designed to teach soft skills such as negotiation and cultural awareness., The original document contains color images.

The University of Southern California and its collaborative research partner, Fakespace Labs, are participating in a number of research programs to invent and implement new forms of display technologies for immersive and semi-immersive applications. This paper briefly describes three of these technologies and highlights a few emerging results from those efforts. The first system is a rear projected 300 degree field of view cylindrical display. It is driven by 11 projectors with geometry correction and edge blending hardware. A full scale prototype will be completed in March 2006. The second system is a 14 screen projected panoramic room environment used as an advanced teaching and meeting space. It can be driven by a cluster of personal computers or low-cost DVD players, or driven by a single personal computer. The third is a prototype stereoscopic head mounted display that can be worn in a fashion similar to standard dust protection goggles. It provides a field of view in excess of 150 degrees., The original document contains color images. Presented at the IEEE Emerging Displays Workshop, held in Alexandria, VA in March 2006.

Effective face-to-face conversations are highly interactive. Participants respond to each other, engaging in nonconscious behavioral mimicry and backchanneling feedback. Such behaviors produce a subjective sense of rapport and are correlated with effective communication, greater liking and trust, and greater influence between participants. Creating rapport requires a tight sense-act loop that has been traditionally lacking in embodied conversational agents. Here we describe a system, based on psycholinguistic theory, designed to create a sense of rapport between a human speaker and virtual human listener. We provide empirical evidence that it increases speaker fluency and engagement., Prepared in cooperation with the Ecole Speciale Militaire de St-Cyr, the University of Twente and the Massachusetts Institute of Technology. Presented at the International Conference of Virtual Agents (6th), IVA'06, held in Marina del Rey, CA on 21-23 Aug 2006. Published in proceedings of the same; ISBN 3540375937. The original document contains color images.

Simulation is a common feature in computer entertainment. However, in computer games, simulation and story are often kept distinct by interleaving periods of interactive play and cut scenes -- short noninteractive scenes that transition from one mission to the next, providing the player with goals and motivation for the next segment of game play. In this mode of alternating between game play and cut scenes, story elements and simulation are kept strictly separate. The authors describe a technique for an "Interactive Narrative" system that more closely integrates simulation and storyline. "Interactive Narrative" is an approach to interactive entertainment that enables the player to make decisions that directly affect the direction and/or outcome of the narrative experience being delivered by the computer system. The technique uses a combination of semi-autonomous character agents and high-level story direction. The storyline is decomposed into directives to character agents to achieve particular world states. Otherwise, character agents are allowed to behave autonomously. When the player's actions create inconsistency between the simulation state and storyline, the storyline is dynamically adapted and repaired to resolve any inconsistencies., Prepared in cooperation with Procedural Arts LLC, Portland, OR. The original document contains color images.

Robust intelligent systems will require a capacity for metacognitive reasoning, where intelligent systems monitor and reflect on their own reasoning processes. A large-scale study of human strategic reasoning indicates that rich representational models of commonsense psychology are available to enable human metacognition. In this paper, we argue that large-scale formalizations of commonsense psychology enable metacognitive reasoning in intelligent systems. We describe our progress toward developing 30 integrated axiomatic theories of commonsense psychology, and discuss the central representational challenges that have arisen in this work to date.

Post-Traumatic Stress Disorder (PTSD) is reported to be caused by traumatic events that are outside the range of usual human experiences, including (but not limited to) military combat, violent personal assault, being kidnapped or taken hostage, and terrorist attacks. Initial data suggests that 1 out of 6 Iraqi War veterans are exhibiting symptoms of depression, anxiety, and PTSD. Virtual Reality (VR) exposure treatment has been used in previous treatments of PTSD patients with reports of positive outcomes. The aim of the current paper is to present the rationale, technical specifications, application features, and user-centered design process for the development of a Virtual Iraq PTSD VR therapy application. The VR treatment environment is being created via the recycling of virtual graphic assets that were initially built for the U.S. Army-funded combat tactical simulation scenario and commercially successful X-Box game, "Full Spectrum Warrior," in addition to other available and newly created assets. Thus far, the authors have created a series of customizable virtual scenarios designed to represent relevant contexts for exposure therapy to be conducted in VR, including a city and a desert road convoy environment. User-Centered tests with the application are currently underway at the Naval Medical Center, San Diego, and within an Army Combat Stress Control Team in Iraq, with clinical trials scheduled to commence in February 2006., Prepared in cooperation with Virtually Better, Inc., Decatur, GA, and the United States Army, 98th Med Det (CSC), Tallil AB, LSA Adder, Iraq. Pub. in the IEEE Virtual Reality 2006 Proceedings, p64-71, 2006.

In this paper, the authors present a computational model of dynamic perceptual attention for virtual humans. The computational models of perceptual attention that they surveyed fell into one of two camps: top-down and bottom-up. Biologically inspired computational models typically focus on the bottom-up aspects of attention, while most virtual humans implement a top-down form of attention. Bottom-up attention models only consider the image information without taking into consideration saliency based on tasks or goals. As a result, the outcome of a purely bottom-up model will not consistently match the behavior of real humans in certain situations. Modeling perceptual attention as a purely top-down process, however, also is not sufficient for implementing a virtual human. A purely top-down model does not take into account the fact that virtual humans need to react to perceptual stimuli vying for attention. Top-down systems typically handle this in an ad hoc manner by encoding special rules to catch certain conditions in the environment. The problem with this approach is that it does not provide a principled way of integrating the ever-present bottom-up perceptual stimuli with top-down control of attention. The model presented here extends the prior model introduced by D. Traum and J. Rickel with perceptual resolution based on psychological theories of human perception. This model allows virtual humans to dynamically interact with objects and other individuals, balancing the demands of goal-directed behavior with those of attending to novel stimuli. The model has been implemented and tested with the MRE Project., The original document contains color images.

In this paper we argue that previous models of cognitive abilities (e.g. memory, analogy) have been constructed to satisfy functional requirements of implicit commonsense psychological theories held by researchers and nonresearchers alike. Rather than working to avoid the influence of commonsense psychology in cognitive modeling research, we propose to capitalize on progress in developing formal theories of commonsense psychology to explicitly define the functional requirements of cognitive models. We present a taxonomy of 16 classes of cognitive models that correspond to the representational areas that have been addressed in large-scale inferential theories of commonsense psychology. We consider the functional requirements that can be derived from inferential theories for one of these classes, the processes involved in human memory. We argue that the breadth coverage of commonsense theories can be used to better evaluate the explanatory scope of cognitive models, as well as facilitate the investigation of larger-scale cognitive systems., Presented at the American Association of Artificial Intelligence Workshop on Modular Construction of Human-Like Intelligence held in Pittsburgh, PA on 10 Jul 2005 and published in the proceedings, AAAI Technical rept. WS-05-08, p56-62, 2005; ISBN 978-1-57735-244-0.

This paper presents scalable solutions for achieving virtual humans who are able to manipulate objects in interactive virtual environments. The scalability trades computational time with the ability to address increasingly difficult constraints. In time-critical environments, arm motions are computed in a few milliseconds using fast analytical Inverse Kinematics. For other types of applications in which collision-free motions are required, a randomized motion planner capable of generating motions of average complexity in approximately a second of computation time is employed. The steps required for defining and computing different types of manipulations are described in this paper., The original document contains color images.

The Institute for Creative Technologies was created at the University of Southern California with the goal of bringing together researchers in simulation technology to collaborate with people from the entertainment industry. The idea was that much more compelling simulations could be developed if researchers who understood state-of-the-art simulation technology worked together with writers and directors who knew how to create compelling stories and characters. This paper presents our first major effort to realize that vision, the Mission Rehearsal Exercise Project, which confronts a soldier trainee with the kinds of dilemmas he might reasonably encounter in a peacekeeping operation. The trainee is immersed in a synthetic world and interacts with virtual humans: artificially intelligent and graphically embodied conversational agents that understand and generate natural language, reason about world events and respond appropriately to the trainee's actions or commands. This project is an ambitious exercise in integration, both in the sense of integrating technology with entertainment industry content, but also in that we have also joined a number of component technologies that have not been integrated before. This integration has not only raised new research issues, but it has also suggested some new approaches to difficult problems. In this paper we describe the Mission Rehearsal Exercise system and the insights gained through this large-scale integration., Published in Multimodial Intelligent Information Presentation, edited by O. Stock and M. Zancanaro, Springer, 2005; ISBN 1-4020-3049-5. The original document contains color images.

This paper presents an activity-based framework for empirical discourse analysis of mitigation in public environments such as Swedish and Bulgarian courtroom examinations. Mitigation is defined as a pragmatic, cognitive and linguistic behavior the main purpose of which is reduction of vulnerability. The suggested framework consists of mitigation processes, which involve mitigating argumentation lines, defense moves, and communicative acts. The functions of mitigation are described in terms of the participants' actions and goals separately from politeness strategies. The conclusions and observations address two things: issues related to the pragmatic theory of communication -- especially mitigation -- and issues related to the trial as a social activity. For instance, non-turntaking confirmations by examiners are often followed by volunteered utterances, which in some cases may be examples of rehearsed testimonies. At the same time the witnesses' tendency to volunteer information, even on the behalf of their own credibility, indicates that they also favor pro-party testimonies. Despite the objective judicial role of the prosecutor or judge and/or despite the examiners' accommodating style the verbal behavior of the witnesses exhibits constant anticipation of danger.

While storytelling has long been recognized as an important part of effective knowledge management in organizations, knowledge management technologies have generally not distinguished between stories and other types of discourse. In this paper we describe a new type of technological support for storytelling that involves automatically capturing the stories that people tell to each other in conversations. We describe our first attempt at constructing an automated story extraction system using statistical text classification and a simple voting scheme. We evaluate the performance of this system and demonstrate that useful levels of precision and recall can be obtained when analyzing transcripts of interviews, but that performance on speech recognition data is not above what can be expected by chance. This paper establishes the level of performance that can be obtained using a straightforward approach to story extraction, and outlines ways in which future systems can improve on these results and enable a wide range of knowledge socialization applications., Presented at the International Conference on Knowledge Capture (3rd) (K-CAP'05) held in Banff, Canada on 2-5 Oct 2005. Published in the Proceedings of the International Conference on Knowledge Capture (3rd) (K-CAP'05), 2005.

Human-behavior models (HBMs) and artificial intelligence systems are called on to fill a wide variety of roles in military simulations. Each of the "off the shelf" human behavior models available today focuses on a specific area of human cognition and behavior. While this makes these HBMs very effective in specific roles, none are single-handedly capable of supporting the full range of roles necessary in an urban military scenario involving asymmetric opponents and potentially hostile civilians. The research presented here explores the integration of three separate human behavior models to support three different roles for synthetic participants in a single simulated scenario. The Soar architecture, focusing on knowledge-based, goal-directed behavior, supports a fire team of U.S. Army Rangers. PMFServ, focusing on a physiologically/stress constrained model of decision-making based on emotional utility, supports civilians that may become hostile., Presented at the Conference on Behavior Representation in Modeling and Simulation (13th) held in Arlington, VA on 17-20 May 2004. Published in the Proceedings of the Conference on Behavior Representation in Modeling and Simulation (13th), 2004.

This paper presents a novel panning algorithm called Speaker-Placement Correction Amplitude Panning (SPCAP) which guarantees conservation of loudspeaker power output. The method is appropriate for any speaker arrangement (e.g. ITU 5.1, 10.2, etc.), and scales with the number of speakers. SPCAP works by correcting initial pan values based on speaker placement to achieve constant power output. Because panning occurs over an arbitrary number of speakers (i.e. is not pair-wise), SPCAP provides two significant advantages over discrete panning schemes. First, pan values for current and future surround-sound formats (e.g. 5.1 and 10.2) are guaranteed to conserve power under any lower-resolution setup, making dynamic up/down mixing in non-standard setups feasible. Second, SPCAP provides a framework for producing wide (non point-source) sounds., Presented at the Audio Engineering Society Convention (117th) held in San Francisco, CA on 28-31 October 2004. Published in the Proceedings of the Audio Engineering Society Convention (117th), 2004. The original document contains color images.

This paper presents ARIA (Application Rendering Immersive Audio). This system provides a means for the research community to easily test and integrate algorithms into a multichannel playback/recording system. ARIA uses a host-based architecture, meaning that programs can be developed and debugged in standard C++ without the need for expensive, specialized DSP programming and testing tools. ARIA allows developers to exploit the speed and low cost of modern CPUs, provides cross-platform portability, and simplifies the modification and sharing of codes. This system is designed for real-time playback and processing, thus closing the gap between research testbed and delivery systems., Presented at the Audio Engineering Society Convention (117th) held in San Francisco, CA on 28-31 Oct 2004.

This paper describes the virtual humans developed as part of the Mission Rehearsal Exercise project, a virtual reality based training system. This project is an ambitious exercise in integration, both in the sense of integrating technology with entertainment industry content, but also in that we have joined a number of component technologies that have not been integrated before. This integration has not only raised new research issues, but it has also suggested some new approaches to difficult problems., The original document contains color images.

The virtual world of the Mission Rehearsal Exercise project is three dimensional, but the scenery is projected on a 2D screen. As the user moves around in the virtual theatre, the view on the scenery should change accordingly. The goal of this assignment is to gather experience on camera movement to make the projection on the screen realistic.

This technical report elaborates on the technical details of the EMA model of emotional appraisal and coping. It should be seen as an appendix to the journal article on this topic (Gratch & Marsella, to appear). This article specifies details of how EMA represents and reasons about causal events, beliefs, desires, intentions, actions. EMA relies on a cognitive component that combines techniques form decision-theoretic planning with models of beliefs, desires and intentions. Syntactic features of the resulting representations are then appraised along a number of dimensions, resulting in a set of appraisal frames and emotional responses.

This report describes the addition of an emotion dialogue to the Mission Rehearsal Exercise (MRE) system. The goal of the MRE system is to provide an immersive learning environment for army officer recruits. The user can engage in conversation with several intelligent agents in order to accomplish the goals within a certain scenario. Although these agents did already posses emotions, they were unable to express them verbally. A question - answer dialogue has been implemented to this purpose. The implementation makes use of proposition states for modelling knowledge, keyword scanning for natural language understanding and templates for natural language generation. The system is implemented using Soar and TCL. An agent can understand emotion related questions in four different domains, type, intensity, state, and the combination of responsible-agent and blameworthiness. Some limitations arise due to the techniques used and to the relative short time frame in which the assignment was to be executed. Main issues are that the existing natural language understanding and generation modules could not be fully used, that very little context about the conversation is available and that the emotion states simplify the emotional state of an agent.

In this work, we present a technique for capturing a time-varying human performance in such a way that it can be re-illuminated in postproduction. The key idea is to illuminate the subject with a variety of rapidly changing time-multiplexed basis lighting conditions, and to record these lighting conditions with a fast enough video camera so that several or many different basis lighting conditions are recorded during the span of the final video's desired frame rate. In this poster we present two versions of such a system and propose plans for creating a complete, production-ready device., The original document contains color images.

This paper reviews none available transcription annotation tools, considering in particular the special difficulties arising from transcribing and annotating multi-party, multi-model dialogue. Tools are evaluated as to the ability to support the user's annotation scheme, ability to visualize the form of the data, compatibility with other tools, flexibility of data representation, and general user-friendliness., Presented at the International Conference on Language Resources and Evaluation (4th) (LREC 2004), held in Lisbon, Portugal in May 2004.

This report describes the effort to address the following research objective: "To begin to define, prototype, and demonstrate an interchange standard among Human Behavior Modeling (HEM) -related models in the Department of Defense (DoD), Industry, Academia, and other Government simulations by establishing a Laboratory for the Study of Human Behavior Representation Interchange Standard." With experience, expertise, and technologies of the commercial computer game industry, the academic research community, and DoD simulation developers, the Institute for Creative Technologies discusses their design and implementation for a prototype HBM interface standard and also describes their demonstration of that standard in a game-based simulation environment that combines HBM models from the entertainment industry and academic researchers., Sponsored in part by the Defense Modeling and Simulation Office, Alexandria, VA.

This paper presents a process for capturing spatially and directionally varying illumination from a real-world scene and using this lighting to illuminate computer-generated objects. We use two devices for capturing such illumination. In the first we photograph an array of mirrored spheres in high dynamic range to capture the spatially varying illumination. In the second, we obtain higher resolution data by capturing images with an high dynamic range omnidirectional camera as it traverses across a plane. For both methods we apply the light field technique to extrapolate the incident illumination to a volume. We render computer-generated objects as illuminated by this captured illumination using a custom shader within an existing global illumination rendering system. To demonstrate our technique we capture several spatially-varying lighting environments with spotlights, shadows, and dappled lighting and use them to illuminate synthetic scenes. We also show comparisons to real objects under the same illumination., The original document contains color images. Pub. at Eurographics Symposium on Rendering, p1-10, 2003. Sponsored in part by University of Southern California and Linkoping University.

Think Like a Commander Excellence in Leadership (TLAC-XL) is an application designed for learning leadership skills both from the experiences of others and through a structured dialogue about issues raised in a vignette. The participant watches a movie, interacts with a synthetic mentor and interviews characters in the story. The goal is to enable leaders to learn the human dimensions of leadership, addressing a gap in the training tools currently available to the U.S. Army. The TLAC-XL application employs a number of Artificial Intelligence technologies, including the use of a coordination architecture, a machine learning approach to natural language processing, and an algorithm for the automated animation of rendered human faces., The original document contains color images.

This paper presents an overview of techniques for generating photoreal computer graphics models of real-world places and objects. Our group's early efforts in modeling scenes involved the development of Facade, an interactive photogrammetric modeling system that uses geometric primitives to model the scene, and projective texture mapping to produce the scene appearance properties. Subsequent work has produced techniques to model the incident illumination within scenes, which we have shown to be useful for realistically adding computer-generated objects to image-based models. More recently, our work has focussed on recovering lighting-independent models of scenes and objects, capturing how each point on an object reflects light. Our latest work combines three-dimensional range scans, digital photographs, and incident illumination measurements to produce lighting-independent models of complex objects and environments., The original document contains color images.

Natural language generation is a broad field, given the wide variety of different applications for text generation. Perhaps one of the most challenging of these applications is natural language generation for spoken dialogue systems. In spoken dialogue systems, real-time throughput is required, which constrains the processing to less than a second if the system is to seem natural, especially given other processing of input and output. Thus text generation approaches which involve selecting from among many possible alternatives or involve complex calculations to determine preferences (Langkilde& Knight 1998) is not appropriate. Generation in dialogue is also somewhere in between single-shot sentence generation and generation of extended discourse. On the one hand, single short utterances must be generated because one can not predict a priori exactly how the other dialogue participant(s) will react, and subsequent generation may depend more on the input that is newly provided than any previously available information. On the other hand, dialogues generally have a coherent structure, depending on the goals and overall structure of the task that is being discussed as well as the immediately previous utterance (Grosz & Sidner 1986). Thus text-planning notions are still relevant, even if one can not count on being able to produce paragraph-level or longer utterances as pre-planned due to the interactive nature of dialogue.

We describe a process for compositing a live performance of an actor into a virtual set wherein the actor is consistently illuminated by the virtual environment. The Light Stage used in this work is a two-meter sphere of inward-pointing RGB light emitting diodes focused on the actor, where each light can be set to an arbitrary color and intensity to replicate a real-world or virtual lighting environment. We implement a digital two-camera infrared matting system to composite the actor into the background plate of the environment without affecting the visible-spectrum illumination on the actor. The color response of the system is calibrated to produce correct color renditions of the actor as illuminated by the environment. We demonstrate moving-camera composites of actors into real-world environments and virtual sets such that the actor is properly illuminated by the environment into which they are composited., Presented at SIGGRAPH 2002 held in San Antonio, TX on 21-26 July 2002. Published in the Proceedings of SIGGRAPH, 2002. The original document contains color images.