Search

Recently, automatic speech recognition (ASR) and visual speech recognition (VSR) have been widely researched owing to the development in deep learning. Most VSR research works focus only on frontal face images. However, assuming real scenes, it is obvious that a VSR system should correctly recognize spoken contents from not only frontal but also diagonal or profile faces. In this paper, we propose a novel VSR method that is applicable to faces taken at any angle. Firstly, view classification is carried out to estimate face angles. Based on the results, feature extraction is then conducted using the best combination of pre-trained feature extraction models. Next, lipreading is carried out using the features. We also developed audio-visual speech recognition (AVSR) using the VSR in addition to conventional ASR. Audio results were obtained from ASR, followed by incorporating audio and visual results in a decision fusion manner. We evaluated our methods using OuluVS2, a multi-angle audio-visual database. We then confirmed that our approach achieved the best performance among conventional VSR schemes in a phrase classification task. In addition, we found that our AVSR results are better than ASR and VSR results.

In developing countries like Timor-Leste, regular road condition monitoring is a significant subject not only for maintaining road quality but also for a national plan of road network construction. The sophisticated equipment for road surface inspection is so expensive that it is difficult to introduce them in developing countries, and the monitoring is usually achieved by manual operation. On the other hand, the utilization of ICT devices such as smartphones has gained much attention in recent years, especially in developing countries because the penetration rate of the smartphone is remarkably increasing even in developing countries. The smartphones equip various high precision sensors, i.e., accelerometers, gyroscopes, GPS, and so on, in the small body in low price. In this project, we are developing an integrated road condition monitoring system that consists of smartphones, dashcams, and a server. There are similar trials in advanced countries but not so many in developing countries. This system assumes to be used in developing countries. The system is very low cost and does not require trained specialists in the field side. The items that are automatically inspected in this system were carefully selected with the local ministry of public works and include paved and unpaved classification, road roughness, road width, detection and size estimation of potholes, bumps, etc., at present. All the inspected items are visualized in Google Maps, Open Street Map, or QGIS with GPS information. The survey results are collected on a server and updated to more accurate values by the repeated surveys. On the analysis, we use several state-of-the-art machine learning and deep learning techniques. In this paper, we summarize related works and introduce this project’s target and framework, which especially focused on the developing countries, and achievements of each of our tasks.

Recently, visual speech recognition (VSR), or namely lipreading, has been widely researched due to development of Deep Learning (DL). The most lipreading researches focus only on frontal face images. However, assuming real scenes, it is obvious that a lipreading system should correctly recognize spoken contents not only from frontal but also side faces. In this paper, we propose a novel lipreading method that is applicable to faces taken at any angles, using Convolutional Neural Networks (CNNs) which is one of key deep-learning techniques. Our method consists of three parts; the view classification part, the feature extraction part and the integration part. We firstly apply angle classification to input faces. Based on the results, secondly we determine the best combination of pre-trained angle-specific feature extraction scheme. Finally, we integrate these features followed by DL-based lipreading. We evaluated our method using the open dataset OuluVs2dataset including multi-angle audiovisual data. We then confirmed our approach has achieved the best performance among conventional and the other DL-based lipreading schemes in the phrase classification task.

In this paper, we propose a denoising method for cine MRI acquired by MoPS. The MoPS-based cine MRI has a high FPS but contains reconstruction noise. DISPEL, a conventional method, performs denoising in the temporal domain. A neural network is further introduced to remove spatial noise. Different from most those methods requiring noisy and clean images, we choose an unsupervised scheme, N2N. We combine these two methods to perform temporal and spatial denoising for cine MRI. Experimental results show that the proposed method is able to remove noise from cine MRIs acquired by MoPS without removing tissue signal.

In recent years, there has been a demand for advanced maintenance in factories. Data collection from factory equipment is being carried out, and the collected sensor data is widely used for statistical analysis in quality control and failure prediction by machine learning. For example, if it is possible to detect an abnormality using vibration data obtained from an equipment, increase in the operation rate of the plant can be expected. In this research, we aim at early detection of equipment failure by finding signs of abnormality from vibration data, using a deep-learning technique, particularly an autoencoder.

Research on road monitoring system has been actively conducted by using both machine learning and deep learning technique. One of our nearest goal in the framework of road condition monitoring system is to segment all road related object and provide a technical report regarding road condition. Our final objective is to develop a community participant-based system for road condition monitoring. As one of our task, in this research, we start with the segmentation of road and pothole. To conduct this task we proposed a semantic segmentation method for road and pothole image segmentation by using one of the famous deep learning technique U-Net. Various condition of road images were used for training and validating the model. The experiment result showed that U-Net model can achieve 97 % of accuracy and 0.86 of mean Intersection Over Union (mIOU).

The objective of our project is to develop an automatic survey system for road condition monitoring using smartphone devices. One of the main tasks of our project is the classification of paved and unpaved roads. Assuming recordings will be archived by using various types of vehicle suspension system and speeds in practice, hence, we use the multiple sensors found in smartphones and state-of-the-art machine learning techniques for signal processing. Despite usually not being paid much attention, the results of the classification are dependent on the feature extraction step. Therefore, we have to carefully choose not only the classification method but also the feature extraction method and their parameters. Simple statistics-based features are most commonly used to extract road surface information from acceleration data. In this study, we evaluated the mel-frequency cepstral coefficient (MFCC) and perceptual linear prediction coefficients (PLP) as a feature extraction step to improve the accuracy for paved and unpaved road classification. Although both MFCC and PLP have been developed in the human speech recognition field, we found that modified MFCC and PLP can be used to improve the commonly used statistical method.

Tracking the wear states of tools on punching machines is necessary to reduce scrap rates. In this paper, we propose a method to estimate wear state of punches using Temporal Convolutional Network Autoencoder (TCN-Autoencoder), one of the deep learning techniques for learning time-series information with convolutional architecture. Approach involves inputting raw time-series information, such as sensor, vibration and audio data, into TCN-Autoencoder, and calculating the reconstruction error between the output and the input data. The reconstruction error is used as “anomaly score” and indicates the distance from the normal state. By training TCN-Autoencoder only with data annotated as “normal” state, the reconstruction error becomes larger when inputting abnormal state data, which corresponds the wear state of the punch. Performance is evaluated on experimental measurement data that spans various wear states of the punch. The results showed our model can estimate anomalies faster than the conventional machine-learning-based anomaly estimation method, while maintaining the high estimation accuracy. This is due to TCN-Autoencoder being able to learn from both frequency and time domain.

In piano learning, it is difficult especially for beginners to judge by themselves whether their musical performances are appropriate in terms of rhythm and melody. Therefore, we have been developing a piano practice support system, which enables piano beginners to conduct independent practice without their instructors. In this paper, we propose the system with the aid of a deep learning technique: Long Short-Term Memory (LSTM). Our system accepts raw piano sounds, extracting performance information. From these information, we evaluate performance. We evaluated the scheme using actual beginners' performances, and found the proposed system achieved better than previous conventional methods. This paper also presents an application employing our methods. Through subjective evaluation experiments for the proposed application, it turns out almost the all beginners found reflection points, and they maintained their motivation for independent practice.

Image processing techniques have been actively used for research on road condition inspection and achieving high detection accuracies. Many studies focus on the detection of cracks and potholes of the road. However, in some least developed countries, there are some distances of roads are still unpaved and it escaped the attention of the researchers. Inspired by penetration and success in applying deep learning technic to computer vision and to any other fields and by the existence of the various type of smartphone devices, we proposed a low - cost method for paved and unpaved road images classification using convolutional neural network (CNN). Our model is trained with 13.186 images and validate with 3.186 images which collected using smartphone device in various conditions of roads such as wet, muddy, dry, dusty and shady conditions and with different types of road surface such as ground, rocks and sands. The experiment using 500 new testing images showed that our model can achieve high Precision (98.0%), Recall (98.4%) and F1 -Score (98.2%) simultaneously.

For developing countries like Timor-Leste, regular road surface monitoring is a major challenge not only for maintaining road quality but also for national plan of road network construction. In Timor-Leste nearly 50% of roads are still unpaved. For this reason, an automated system is required to do a survey of paved and unpaved roads. In this study, we present a new approach for the use of smartphones sensor to classify paved and unpaved roads, and anomaly detection. Although, the most remarkable factor to differentiate paved and unpaved road is based on amplitude of the vertical acceleration, each vehicle has a different type of suspension system. Therefore, we used high-dimensional features and state-of-the-art machine learning techniques to make the system robust for differences of vehicle and also smartphone type. This study divided into two stages such as paved and unpaved road classification and road anomaly detection such as pothole and bump. For paved and unpaved road classification, we tried to use the SVM, HMM and ResNet and compared the performance of these models. Of all comparison, the ResNet was the best choice in this study, because it outperformed the SVM and HMM on the all performance evaluation criteria. Furthermore, the KNN and DTW are applied for anomaly detection on the paved road. The KNN-DTW are also compared to the other machine learning techniques like SVM and classical KNN using same criteria. As a result of the comparison, the KNN-DTW and SVM performed better than classical KNN.

This research proposes a low-cost solution for detecting road potholes image by using convolutional neural network (CNN). Our model is trained entirely on the image which collected from several different places and has variation such as in wet, dry and shady conditions. The experiment using the 500 testing images showed that our model can achieve (99.80 %) of Accuracy, Precision (100%), Recall (99.60%), and F-Measure (99.60%) simultaneously.

Cine Cardiac Magnetic Resonance (Cine-CMR) is one example of dynamic MRI approaches to image organs that exhibit periodic motion. Conventional routine clinical Cine- CMR are typically obtained at 20–35 frames per second (fps) with temporal window sizes of 40–50 milliseconds. We have recently shown the feasibility of significantly increasing this overall frame rate by an acquisition of MRI k-space using a highly optimized radial sampling pattern with respect to both spatial and temporal coverage. In brief, our proposed approach acquires a significantly undersampled radial MRI k-space while encoding spatially and temporally periodic noise characteristics through the undersampled radial MRI acquisition; however, remnant radial streaking noise remain under physiologic imaging conditions. In this research, we propose to further remove these streaking noise, employing a Spatio-Temporal Denoising Auto-Encoder (ST-DAE) based on deep learning. We evaluate performance of our method in addressing such remnant artifact using ST-DAE; PSNR is used to evaluate image quality, and computational time is also discussed.

In recent years, people with swallowing disorder are increasing. Therefore, it is important to evaluate the swallowing function in the early detection and prevention of swallowing disorder. In this study, we used a capsule that generates sound and estimated the timing at which food is sent to the esophagus by sound signal processing and deep learning. By comparing it with the movement of the epiglottis tracked by the image, we performed a noninvasive and quantitative swallowing function evaluation.

In Timor-Leste, the coffee is one of the most important product for the acquisition of foreign currency. However, there are almost no rationalizations, therefore, enhancing the value of the coffee efficiently at the local of production is desired. The final objective of our study is to develop the automatic coffee beans sorting system for the producers of coffee beans in Timor-Leste. As the first step, we developed an image processing system which classifies the images of green coffee beans into each type of defect. We employed deep convolutional neural networks, the state-of-the-art machine learning technique, for the image processing. As the results, we succeeded to sort defect beans from 72.4% to 98.7% of accuracies based on the types of defects.

PURPOSE To introduce a pair of accelerated non-Cartesian acquisition principles that when combined, exploit the periodicity of k-space acquisition, and thereby enable acquisition of high-temporal cine Cardiac Magnetic Resonance (CMR). METHODS The mathematical formulation of a noniterative, undersampled non-Cartesian cine acquisition and reconstruction is presented. First, a low-pass filtering step that exploits streaking artifact redundancy is provided (i.e., Dynamically Interleaved Streak removal in the Power-spectrum Encoded domain with Low-pass filtering [DISPEL]). Next, an effective radial acquisition for the DISPEL approach that exploits the property of prime numbers is described (i.e., Modulo-Prime Spoke [MoPS]). Both DISPEL and MoPS are examined using numerical simulation of a digital heart phantom to show that high-temporal cine-CMR is feasible without removing physiologic motion vs aperiodic interleaving using Golden Angles. The combined high-temporal cine approach is next examined in 11 healthy subjects for a time-volume curve assessment of left ventricular systolic and diastolic performance vs conventional Cartesian cine-CMR reference. RESULTS The DISPEL method was first shown using simulation under different streak cycles to allow separation of undersampled radial streaking artifacts from physiologic motion with a sufficiently frequent streak-cycle interval. Radial interleaving with MoPS is next shown to allow interleaves with pseudo-Golden-Angle variants, and be more compatible with DISPEL against irrational and nonperiodic rotation angles, including the Golden-Angle-derived rotations. In the in vivo data, the proposed method showed no statistical difference in the systolic performance, while diastolic parameters sensitive to the cine's temporal resolution were statistically significant (P < 0.05 vs Cartesian cine). CONCLUSIONS We demonstrate a high-temporal resolution cine-CMR using DISPEL and MoPS, whose streaking artifact was separated from physiologic motion.

In the speech recognition literature, building corpora for Large Vocabulary Continuous Speech Recognition (LVCSR) is quite important. In addition, in order to overcome performance decrease caused by noise, using visual information such as lip images is effective. In this paper, therefore, we focus on collecting speech and lip-image data for audio-visual LVCSR. Audio-visual speech data were obtained from 12 speakers, each who uttered ATR503 phonetically-balanced sentences. These data were recorded in acoustically and visually clean environments. Using the data, we conducted recognition experiments. Mel Frequency Cepstral Coefficients (MFCCs) and eigenlip features were obtained, and multi-stream Hidden Markov Models (HMMs) were built. We compared the performance in clean condition to those in noisy environments. It is found that visual information is able to compensate the performance. In addition, it turns out that we should improve visual speech recognition for high-performance audio-visual LVCSR.

The objective of this study is the presentation of an analytical method to support health consultants, thereby establishing an analytical method that enables them to select subjects for health guidance using health checkup data and to derive a suitable guidance policy for each subject. This paper examines an analysis method that maps a health checkup using Multi-Dimensional Scaling (MDS). MDS mapping of multivariate health checkup data for a health checkup examinee on a two-dimensional plane facilitates comprehension of a subject’s health condition easily as visual information. This study focuses on the efficacy of visualization from the viewpoint of supporting health consultants. The mode of display by MDS facilitates visual confirmation that groups outside of the scope of health guidance and at high risk are shown in a contrastive position. In addition, a medium risk group was plotted into an in-between position. A plot ofmore detailed classification for all inspection items suggests by concurrence an increased risk. Results of this study indicate that its coordinates are effective both in determining a subject’s health condition intuitively and in use as one index of risk formetabolic syndrome. These results are therefore considered useful for formulating health guidance plans such as priority issues.

This paper develops an Audio-Visual Speech Recognition (AVSR) method, by (1) exploring high-performance visual features, (2) applying audio and visual deep bottleneck features to improve AVSR performance, and (3) investigating effectiveness of voice activity detection in a visual modality. In our approach, many kinds of visual features are incorporated, subsequently converted into bottleneck features by deep learning technology. By using proposed features, we successfully achieved 73.66% lipreading accuracy in speaker-independent open condition, and about 90% AVSR accuracy on average in noisy environments. In addition, we extracted speech segments from visual features, resulting 77.80% lipreading accuracy. It is found VAD is useful in both audio and visual modalities, for better lipreading and AVSR.

In service engineering it is important to estimate when and what a worker did, because they include crucial evidences to improve service quality and working environments. For Service Operation Estimation (SOE), acoustic information is one of useful and key modalities; particularly environmental or background sounds include effective cues. This paper focuses on two aspects: (1) extracting powerful and robust acoustic features by using stacked-denoising-autoencoder and bag-of-feature techniques, and (2) investigating a multi-modal SOE scheme by combining the audio features and the other sensor data as well as non-sensor information. We conducted evaluation experiments using multi-modal data recorded in a restaurant. We improved SOE performance in comparison to conventional acoustic features, and effectiveness of our multimodal SOE scheme is also clarified.

This paper proposes a multi-modal speech recognition method using optical-flow analysis for lip images. Optical flow is defined as the distribution of apparent velocities in the movement of brightness patterns in an image. Since the optical flow is computed without extracting the speaker's lip contours and location, robust visual features can be obtained for lip movements. Our method calculates two kinds of visual feature sets in each frame. The first feature set consists of variances of vertical and horizontal components of optical-flow vectors. These are useful for estimating silence/pause periods in noisy conditions since they represent movement of the speaker's mouth. The second feature set consists of maximum and minimum values of integral of the optical flow. These are expected to be more effective than the first set since this feature set has not only silence/pause information but also open/close status of the speaker's mouth. Each of the feature sets is combined with an acoustic feature set in the framework of HMM-based recognition. Triphone HMMs are trained using the combined parameter sets extracted from clean speech data. Noise-corrupted speech recognition experiments have been carried out using audio-visual data from 11 male speakers uttering connected digits. The following improvements of digit accuracy over the audio-only recognition scheme have been achieved when the visual information was used only for silence HMM: 4% at SNR = 5 dB and 13% at SNR = 10 dB using the integral information of optical flow as the visual feature set.

Human behavior sensing and their analysis are great role to improve service quality and education of employees. This paper shows novel frameworks of detection of customer communication and lead time estimation(LTE) by using multi-sensored data, sound data and accounting data in the restaurant. They are useful for management about work environments and problems for employees. Lead time from order to delivery shows the quality of the service for customers. We found sound data of an employee's speech is useful for these techniques by speech ratio smoothing and POS sound detection.

This paper introduces our recent activities for audio-visual speech recognition on mobile devices and data collection in various environments. Audio-visual automatic speech recognition is effective in noisy or real conditions to enhance the robustness of speech recognizer and to improve the recognition accuracy. We have developed an audio-visual speech recognition interface for mobile devices. In order to evaluate the recognizer and investigate issues related to audio-visual processing on mobile computers, we collected speech data and lip images of 16 subjects in eight conditions, where there were various audio noises and visual difficulties. Audio-only speech recognition and visual-only lipreading were then conducted. Through these experiments, we found some issues and future works not only for construction of audio-visual database but also for robust audio-visual speech recognition.

Voice Conversion (VC) is a technique to convert speech data of source speaker into ones of target speaker. VC has been investigated and statistical VC is used for various purposes. Conventional VC uses acoustic features, however, the audio-only VC has suffered from the degradation in noisy or real environments. This paper proposes an AudioVisual VC (AVVC) method using not only audio features but also visual information, i.e. lip images. Eigenlip feature is employed in our scheme as visual feature. We also propose a feature selection approach for audio-visual features. Experiments were conducted to evaluate our AVVC scheme comparing with audio-only VC, using noisy data. The results show that AVVC can improve the performance even in noisy environments, by properly selecting audio and visual parameters. It is also found that visual VC is also successful. Furthermore, it is observed that visual dynamic features are more effective than visual static information.

Lip reading technologies play a great role not only in image pattern recognition e.g. computer vision, but also in audio-visual pattern recognition e.g. bimodal speech recognition. However, it is a problem that the recognition accuracy is still significantly low, compared to that of speech recognition. Another problem lies which the performance degradation occurs in real environments. To improve the performance, in this paper we employ two adaptation schemes: speaker adaptation and environmental adaptation. The speaker adaptation is performed to recognition models so as to prevent the degradation caused by the difference of speakers. The environmental adaptation is also conducted to deal with environmental differences. We tested these adaptation schemes using a real-world audio-visual corpus CENSREC-2-AV, we have built this corpus containing real-world data (speech signals and lip images) recorded in a driving car, in which subjects uttered Japanese connected digits. Experimental results show that the lip reading recognition performance was largely improved by the speaker adaptation, and further recovered by the environmental adaptation.

In this paper, we introduce human behavior sensing and data collection tequniques in a real environment. In order to improve worker's skill and service quality, it is desired to analyse and visualize worker's behavior scientifically. We acquired multi-sensored data and speech data of wait staffs in a restaurant for several months. Our goal is to combine these data, to analyse objectively and finally to improve their behavior and service. The speech data is useful to detect their speech events and spoken contents. To collect speech data in real-environment is a difficult task because of recording environment and load to the subjects. This paper focuses on collection of speech and discusses the issue of its measurement and analysis. We also apply VAD(Voice activity detection) for speech data and calculate their speech ratio. The speech ratio has a certain inclination related to their operations. The change of speech ratio expresses the efficiency of QC(Quality control) activity.

In this paper, we introduce a bimodal speech recognition corpus in real environments. In recent years, speech recognition technology has been used in noisy conditions. Therefore, it becomes necessary to achieve higher recognition accuracy in real environments. As one of the solutions, bimodal speech recognition using audio and non-audio information is getting studied. However, there are few databases which can be used to evaluate the bimodal speech recognition in real environments. In this paper, we introduce CENSREC-2-AV we have been working to built, as a new bimodal speech recognition corpus. CENSREC-2-AV is one of the databases of the CEN-SREC project; we provided a similar corpus CENSREC-1-AV as a database for bimodal speech recognition for additive noises. In these corpora, there are speech data and lip images. Researchers can evaluate a bimodal speech recognition method built using CENSREC-1-AV which consists of clean data, in real environments by using CENSREC-2-AV.

This paper introduces our recent effort to develop a Service-Operation Estimation (SOE) system using speech and multi-sensored data as well as other acquired data. In SOE, it is essential to analyze employees' data in order to increase the productivity in many service industries. Speech processing techniques, such as voice activity detection and keyword spotting recognition, help the analysis and enhance the precision of the results; the beginning and end times of speech region are used to detect work events, and recognized keywords are used to conduct work estimation. In our system all the results are visualized in a 3D model, and it makes employers and employees help their operations.

In this paper, we newly introduce a collection of databases and evaluation tools called CENSREC-4, which is an evaluation framework for distant-talking speech under hands-free conditions. Distant-talking speech recognition is crucial for a handsfree speech interface. Therefore, we measured room impulse responses to investigate reverberant speech recognition in various environments. The data contained in CENSREC-4 are connected digit utterances, as in CENSREC-1. Two subsets are included in the data: basic data sets and extra data sets. The basic data sets are used for the evaluation environment for the room impulse response-convolved speech data. The extra data sets consist of simulated and recorded data. An evaluation framework is only provided for the basic data sets as evaluation tools. The results of evaluation experiments proved that CENSREC-4 is an effective database for evaluating the new dereverberation method because the traditional dereverberation process had difficulty sufficiently improving the recognition performance. Index Terms: Various environments, Impulse response, Convolution, Real recorded data, Evaluation framework

Voice activity detection (VAD) plays an important role in speech processing including speech recognition, speech enhancement, and speech coding in noisy environments. We developed an evaluation framework for VAD in such environments, called corpus and environment for noisy speech recognition 1 concatenated (CENSREC-1-C). This framework consists of noisy continuous digit utterances and evaluation tools for VAD results. By adoptiong two evaluation measures, one for frame-level detection performance and the other for utterance-level detection performance, we provide the evaluation results of a power-based VAD method as a baseline. When using VAD in speech recognizer, the detected speech segments are extended to avoid the loss of speech frames and the pause segments are then absorbed by a pause model. We investigate the balance of an explicit segmentation by VAD and an implicit segmentation by a pause model using an experimental simulation of segment extension and show that a small extension improves speech recognition.

This paper proposes an audio-visual speech recognition method using lip information extracted from side-face images as an attempt to increase noise robustness in mobile environments. Our proposed method assumes that lip images can be captured using a small camera installed in a handset. Two different kinds of lip features, lip-contour geometric features and lip-motion velocity features, are used individually or jointly, in combination with audio features. Phoneme HMMs modeling the audio and visual features are built based on the multistream HMM technique. Experiments conducted using Japanese connected digit speech contaminated with white noise in various SNR conditions show effectiveness of the proposed method. Recognition accuracy is improved by using the visual information in all SNR conditions. These visual features were confirmed to be effective even when the audio HMM was adapted to noise by the MLLR method.

This study presents a novel system which supports nurses in note-taking by providing a digital pen and character-recognition system laying stress on user interface. The system applies characteristics of a digital pen for improving the efficiency of tasks related to nursing records. The system aims at improving the efficiency of nursing activities and reducing the time spent for tasks for nursing records. In our system, first, notes are written on a check sheet using a digital pen along with a voice that is recorded on a voice recorder; the pen and voice data are transferred to a PC. The pen data are then recognized automatically as characters, which can be viewed and manipulated with the application. We conducted an evaluation experiment to improve efficiency and operation of the system, and its interface. The evaluation and test operations used 10 test subjects. Based on the test operation and the evaluation experiment of the system, it is turned out that improvement for urgent situations, enhancement of portability, and further use of character recognition are required.

We propose a new multimodal speech recognition method using optical flow analysis and evaluate its robustness to acoustic and visual noises. Optical flow is defined as the distribution of apparent velocities in the movement of brightness patterns in an image. Since the optical flow is computed without extracting speaker's lip contours and location, robust visual features can be obtained for lip movements. Our method calculates a visual feature set in each frame consisting of maximum and minimum values of the integral of the optical flow. This feature set has not only silence information but also open/close status of the speaker's mouth. The visual feature set is combined with an acoustic feature set in the framework of HMM-based recognition. Triphone HMMs are trained using the combined parameter set extracted from clean speech data. Two multimodal speech recognition experiments have been carried out. First, acoustic white noise was added to speech waveforms, and a speech recognition experiment was conducted using audio-visual data from 11 male speakers uttering connected Japanese digits. The following improvements of relative reduction of digit error rate over the audio-only recognition scheme were achieved when the visual information was incorporated into the silence HMM: 32% at SNR=10dB and 47% at SNR=15dB. Second, real-world data distorted both acoustically and visually was recorded in a driving car from six male speakers and recognized. We achieved approximately 17% and 11% relative error reduction compared with audio-only results on batch and incremental MLLR-based adaptation, respectively.

It is known that different unit cells can have the same computed lines in some cases (Figure). The phenomenon was first called geometrical ambiguity and studied in [1]. In all the high-symmetric cases provided in [1], such unit cells correspond to derivative lattices of each other. Although this is not true in general for 3-dimensional lattices, this has been assumed in methods to search for geometrical ambiguities (e.g., [2]). Thus, a method to obtain all the geometrical ambiguities in very short time for a given unit cell parameters is provided. Because such a method has not been used for powder indexing, it will have impact in the following sense: firstly, it is useful for checking powder indexing solutions promptly. Some powder auto-indexing methods cannot obtain all the geometrical ambiguities. Even for the software including Conograph which can gain all the ambiguities, it is not straightforward to search for them from many indexing solutions using the figures of merit which are sometimes not reliable [3]. Secondly, the new method indicates that powder indexing has only finitely many solutions at least if peak search succeeds in obtaining all (but a few) diffraction peaks with q-values smaller than some calculated value. (Note that infinitely many solutions may exist for lattices of dimension more than 4.) The result seems to provide a foundation of automatic powder crystal structure analysis, because it is possible to obtain all the ambiguities by computation. The introduced method was implemented in the newest version of Conograph, which will be distributed on the web (http://research.kek.jp/people/rtomi/ConographGUI/web_page.html) by IUCr2014. ACKNOWLEDGEMENT: this research was partly supported by a JSPS KAKENHI grant (No.22740077) and by Ibaraki Prefecture (J-PARC-23D06).