Search

In order to improve the accessibility of the deaf and hard of hearing (DHH) visitors at museums, real-time captioning using speech recognition is a very convenient communication support tool. Since viewing exhibits and captioning are both visually events, it is necessary to examine whether a DHH person can simultaneously view the exhibits while processing the textual data. This study focuses on the accessibility of guided tours in museum and discusses the problems of using captioning as communication support for DHH visitors. Results of the experiment are summarized as follows. In the scenes of looking at still images in museums, it turns out that different devices are needed for different purposes. In experiments where a moving object was presented, it was found that when the captioning was frequently updated, the user stopped watching the observation target to read the captioning. In other words, the captioning may have prevented stable and continuous observation.

We have conducted a series of experiments on recognizing emotions conveyed on the hearing of musical performances as a means of building a musical performance assistance system with which deaf and hard of hearing people could communicate through musical performance. Until recently, we had assumed that visual information provided simultaneously with musical performance was useful in helping the emotions emitted from music to be recognized, especially if the information conveyed the same emotions as those expressed in the musical performance. Currently we are not sure that this assumption is correct after having gone through experiments we conducted. In this paper, we describe the background of our plan to build a musical performance assistance system and review our past experiments by comparing the emotions recognized by recipients among several types of stimuli. Reviewing the results of our experiments made us unsure that multichannel information that supposedly helps deaf and hard of hearing people to recognize emotions actually works well. Based on a new question we pose about multichannel music information, we describe an experiment that compare the stimuli provided by music types only, music with video sequences, and video sequences only. The results we obtained do not clarify the question that we pose.

We conducted an experiment to determine the abilities of hearing-impaired and normal-hearing people to recognize intended emotions conveyed in four types of stimuli: a drum performance, a drum performance accompanied by a drawing expressing the same intended emotion, and a drum performance accompanied by one of two types of motion pictures. The recognition rate was the highest for a drum performance accompanied by a drawing even though participants in both groups found it difficult to identify the intended emotion because they felt the two stimuli sometimes conveyed different emotions. Visual stimuli were especially effective for performances whose intended emotions were not clear by themselves. The difference in ability to recognize intended emotions between the hearing-impaired and normal-hearing participants was insignificant. The results of this and a series of experiments will enable us to better understand the similarities and differences between how people with different hearing abilities encode and decode emotions in and from sound and visual media. We should then be able to develop a system that will enable hearing-impaired and normal-hearing people to play music together.