Search

There is social concern about rapid increase of people with dementia. One of triggers of behavioral psychological symptoms of dementia (BPSD) is inappropriate care by caregivers. The development of adequate training system for professional caregivers is crucial. Previous study showed multimodal comprehensive care methodology Humanitude; which consists with verbal and non-verbal comprehensive structured communication skills, has shown significant improvement of BPSD in long-term care facilities in Japan. Following this result, a quantitative study of dementia care using video analysis was conducted. Informatics researchers created a tool of multimodal behavior analysis system, which supports 1) video annotations, 2) visualization of annotation data, 3) quantitative analysis, 4) create a query of expert’s interpretation. Four professional caregivers enrolled the study. Two caregivers were trained for Humanitude as intervention arm, and others applied conventional care as control arm. Caregivers in both arms provided care to the same residents in a nursing home. While control arm failed all the care due to refusal by the residents, the care to the same residents by intervention arm were calmly accepted. The cares were video recorded and analyzed. The elements performed during the care was eye contact 0.0%, verbal communication 28.4%, and touch 0.2% in control and 12.5%, 54.8%, and 44.5% in intervention arm, respectively. The average of positive attitudes during care was 0 in control and 35 in intervention arm. In conclusion, this quantitative analysis system can be used as objective description of cares and contribute to educations for caregivers to improve the quality of care.

the multimodal communication analysis by a computational approach. With the rapid growth of an aging population, it is becoming extremely important to elucidate communication skills for improving the quality of life of the elderly. Focusing on the dementia care method by HUMANITUDE, we have analyzed dementia care skills to evaluate their effectiveness. We designed three layers for representing skills and named them “Intra-modality,” “Inter-modality” and “Multimodal-interaction.” These results are a part of the findings of empirical analysis of building human relationships.

Many parents need child-rearing support not only through face-to-face interaction but also through the Internet as the child's nursing environment changes. However, current web content for child-rearing causes parents to worry unless there is a proper interpretation supplied. Since many parents worry about interpersonal relationships in parenting, child behavior examples lead to a deep-level understanding of the cause of problems. We have constructed a multimodal speech behavior corpus that includes annotated metadata on child behavior in everyday life. In this paper, we propose a method for creating video content based on this corpus. We designed content by using behavior descriptions as a structure. As a result of creating with our method, we found that the corpus structure was useful in content design and that the video content was effective in child-rearing support.

We developed a multi-camera shooting support system in a compact studio for novice cameramen. We analyzed the problems they faced when shooting video programs. We have suggested that our automated multi-camera controls and the adjustment of camera angles for face detection reduce the burden on novices. Our experiment shows that a novice cameraman alone can carry out the task of shooting a video program.

Management of Behavioral and Psychological Symptoms of Dementia (BPSD) is a key challenge in geriatric dementia care. A multimodal comprehensive care methodology, Humanitude, with eye contact, verbal communication, and touch as its elements, was provided to three geriatric dementia patients for whom conventional nursing care failed in an acute care hospital. Each episode was evaluated by video analysis. All patients had advanced dementia with BPSD. Failure of care was identified by patient’s shouting, screaming, or abrupt movements of limbs. In this case series, conventional care failed for all three patients. Each element of care communication was much shorter than in Humanitude care, which was accepted by the patients. The average of the elements performed during the care was eye contact 0.6%, verbal communication 15.7%, and touch 0.1% in conventional care and 12.5%, 54.8%, and 44.5% in Humanitude care, respectively. The duration of aggressive behavior of each patient during care was 25.0%, 25.4%, and 66.3% in conventional care and 0%, 0%, and 0.3% in Humanitude, respectively. In our case series, conventional care was provided by less eye contact, verbal communication, and touch. The multimodal comprehensive care approach, Humanitude, decreased BPSD and showed success by patients’ acceptance of care.

In order to create air conditioning service scenarios adaptive to elderly people, we have conducted detailed situational analyses focusing on bodily feeling information. The practical experimental environment has introduced our constructed indoor interaction system to understand situations based on bodily feeling and to control air conditioners properly. The experiments consists of 7 sessions participated by 6 elderly people have accumulated 228 times of bodily feeling inputs. The detailed analyses have done utilizing our developed multimodal behavior observation tool equipped with the functions of behavior description from multiple viewpoints and flexible retrieval. The analysis results indicate the possibility of correspondence of the scenes of situational changes in conversation like topic transitions and turn-taking events or those of reading to the changes of bodily feeling. The multimodal bodily feeling analysis has produced valuable findings to enhance situation understanding.

This paper considers the user-centered spontaneous speech dialogue system TOSBURG-II (Taskoriented dialogue system based on speech understanding and response generation), and discusses the design from the viewpoint of media technology and a multimodal interface. The authors have developed element techniques, including spontaneous speech understanding, user-center dialogue control, multimodal response generation and speech response cancelling, all based on the noise-immune word-spotting and key-words. The concept is that “no constraint is imposed on the user.” By integrating these techniques, the realtime speech dialogue system for an unspecified user is developed. The speech dialogue data acquisition/ evaluation system is constructed on the real system. The system can record real speech data as well as the intermediate result of processing in the dialogue system, such as keyword spotting, speech understanding and dialogue processing. The system can also be utilized in the construction of the speech dialogue corpus and the evaluation/improvement of the human factor aspect, in addition to the evaluation of the system performance. As a result of trial use and evaluation experiment for the real system for unspecified users, it is verified that spontaneous speech understanding based on the interruption function by the user and the multimodal response and keywords, is useful in improving the naturalness of the dialogue and robustness.

We have developed a spontaneous speech dialogue system TOSBURG II, employing keyword-based spontaneous speech understanding and multimodal response generation, with adaptive speech response cancellation. Since in multimodal interaction, the user understands the system's response by a visual output before its speech response is completed, the user often interrupts the system's speech response. Therefore, our adaptive speech response cancellation serves to facilitate natural human-computer interaction by allowing the user's interruption. We have also developed an evaluation environment for dialogue data collection and the performance of TOSBURG II. Unlike conventional data collection systems, TOSBURG II collects in this environment not only speech data and the final results of speech understanding but also its intermediate results as dialogue data, to use them for the evaluation and improvement of the system. The results of our dialogue experiments using TOSBURG II prove the effectiveness of adaptive speech response cancellation for natural interaction, confirming that the dialogue data and the evaluation environment will contribute to a further development of spontaneous speech dialogue systems.

This paper describes the speaker-independent spontaneous speech dialog system TOSBURG II, where the user can input the speech interrupting the speech response from the system. In the speech dialog system TOSBURG, which was previously constructed experimentally, spontaneous utterance dialog was realized by a continuous speech understanding based on the key word and the multimodal response. TOSBURG II aims at dialog that is more flexible and interactive. The speech response component superposed on the input speech signal is cancelled by the active control technique, making it possible for the user to interrupt and input during the system speech response. To improve the precision of speech recognition, noise immunity learning, which is useful in detecting the keyword in the noise, is extended so that it can also be applied to the speech input after the speech response cancellation, where the residual component still remains. To evaluate the effectiveness of speech response cancellation and noise immunity learning, an experiment is executed by superposing the speech data (obtained by the utterance of 350 sentences containing 49 key words by five adult males) on the system speech response. The key word detection rate, false alarm rate, and sentence recognition rate are improved from 58.8 percent, 24.1 FA/H/W (false alarms/hour/word), 27.8 percent, to 89.6 percent, 15.4 FA/H/W, 58.4 percent, respectively. A dialog experiment is also conducted using the constructed realtime system, and the usefulness of the proposed system is verified.

Our goal is to represent commonsense knowledge as computational models, which are applied to spoken dialogue systems that realize smart man-machine communications by correctly understanding speakers' intentions and emotions. For this purpose, we have constructed a multimodal speech behavior corpus which includes metadata annotated from various viewpoints in everyday life. This paper describes a methodology of modeling of thinking processes in problem solving based on child development, by analyzing data of attention-catching.

This paper discusses noise immunity learning in speech recognition by word-spotting. In the proposed method, the learning speech data with superposed noise is constructed by adding the noise data to the clean speech data collected beforehand. Then the recognition dictionary for word-spotting is trained to improve noise immunity. In the learning, the recognition by word-spotting is attempted for the artificially synthesized learning data while gradually increasing the ratio of the contained noise. The recognition is executed using the learning word feature vector automatically extracted based on the similarity. The noise immunity is realized by the simulation in the noisy environment and the automatic learning. The recognition and learning use the word as the unit and the multiple similarity method, which can cope with a wide range of pattern deformation. An evaluation experiment is executed using 13-word speech data (plus noise in a railway station). For the case of 96-dimensional word feature vector (eight dimensions for frequency and 12 dimensions for time), and S/N ratio of 10 dB, the recognition rate is improved from 85.5 percent in the word-spotting method without learning to 94.1 percent with learning. This indicates the usefulness of the proposed method.

We are running a parent-child learning project for constructing a multimodal child behavioral corpus. It is based on a learning environment where children and their parents regularly attend both guided lessons and have chances for free play. Among other purposes, we use the gathered data for analyzing how children construct evaluation schemas of learning processes. One important kind of learning is learning which learning methods are useful and which harmful in different situations. We have examined how learning methods, which are different from skills, develop from simple to complex physical and social manipulation. Our research shows hints on how negative knowledge on unsuccessful learning methods is used to encourage cognitive constructs more advanced than binary repetition, resulting in developing complex methods such as learning to reason by analogies and use them to construct new complex hypotheses about the world, and using adults and other children as sources of knowledge. We are also researching how right kind of support and example from adults close to children are essential in endowing and encouraging children with using these abilities.

We have developed a method for multimodal observation of infant development. In order to analyze development of problem solving skills by observing scenes of task achievement or communication with others, we have introduced a method for extracting detailed behavioral features expressed by gestures or eyes. We have realized an environment for recording behavior of the same infants continuously as multi-angle video. The environment has evolved into a practical infrastructure through the following four steps; (1) Establish an infant school and study the camera arrangement. (2) Obtain participants in the school who agree with the project purpose and start to hold regular classes. (3) Begin to construct a multimodal infant behavior corpus with considering observation methods. (4) Practice development process analyses using the corpus. We have constructed a support tool for observing a huge amount of video data which increases with age. The system has contributed to enrich the corpus with annotations from multimodal viewpoints about infant development. With a focus on the demonstrative expression as a fundamental human behavior, we have extracted 240 scenes from the video during 10 months and observed them. The analysis results have revealed interesting findings about the developmental changes in infants' gestures and eyes, and indicated the effectiveness of the proposed observation method.