Search

Facial expression recognition (FER) is one of the best non-intrusive methods for understanding and tracking mood and mental states. In this study, we propose early and late fusion methods to recognize five facial expressions (angry, happy, neutral, sad, and surprised) using different combinations from a publicly available database (VIRI) with visible, infrared, and multispectral dynamic imaging (MSX) images and the (NVIE) database. A distinctive feature is the use of concatenation and combining techniques to combine ResNet-18 with transfer learning (TL) to create a model that is significantly more accurate than individual models. In the early fusion, we concatenated features from the modalities and classified facial expressions (FEs). In the late fusion, we combined the outputs of the modalities using weighted sums. For this purpose, we used different weighting factors depending on the accuracy of the individual models. The experimental results demonstrated that the proposed model outperformed the previous works by providing an accuracy of 83.33% when we trained the model (1-step training). Through further fine-tuning (3-step training), we obtained an improved performance of 84.44%. We conducted additional experiments by combining them with another modality (MSX) available in the database. By performing experiments with an additional modality (MSX), we obtained improved performance, which confirms that the additional modality combined with existing modalities can help improve the performance of fusion models for facial expression recognition. We also experimented by changing the backbones (Vgg-16, ShuffleNetv2, MobileNetv2, and GhostNet) in addition to ResNet-18 for visible and MSX data. ResNet-18 outperformed the other backbones in facial expression recognition for visible and MSX data.

Abstract The phantom array effect is one of the temporal light artefacts that can decrease performance and increase fatigue. The phantom array effect visibility shows large individual differences; however, the dominant factors that can explain these individual differences remain unclear. We investigated the relationship between saccadic eye movement speed and phantom array visibility at two different angles and four different directions of saccadic eye movement. The peak speed of saccadic eye movement and the phantom array effect visibility were measured at different modulation frequencies of the light source. Our results show that phantom array visibility increased as eye movement speed increased; the phantom array visibility was higher at a wide viewing angle with fast eye movement speed than at a narrow viewing angle. Moreover, when clustered into subgroups according to individual eye movement speed, the mean speed of the saccadic eye movement of each subgroup is related to the variations in the visibility of the phantom array effect of the subgroup. Therefore, saccadic eye movement speed is related to variations in phantom array effect visibility.

Classifying pathological gaits is crucial for identifying impairments in specific areas of the human body. Previous studies have extensively employed machine learning and deep learning (DL) methods, using various wearable (e.g., inertial sensors) and non-wearable (e.g., foot pressure plates and depth cameras) sensors. This study proposes early and late fusion methods through DL to categorize one normal and five abnormal (antalgic, lurch, steppage, stiff-legged, and Trendelenburg) pathological gaits. Initially, single-modal approaches were utilized: first, foot pressure data were augmented for transformer-based models; second, skeleton data were applied to a spatiotemporal graph convolutional network (ST-GCN). Subsequently, a multi-modal approach using early fusion by concatenating features from both the foot pressure and skeleton datasets was introduced. Finally, multi-modal fusions, applying early fusion to the feature vector and late fusion by merging outputs from both modalities with and without varying weights, were evaluated. The foot pressure-based and skeleton-based models achieved 99.04% and 78.24% accuracy, respectively. The proposed multi-modal approach using early fusion achieved 99.86% accuracy, whereas the late fusion method achieved 96.95% accuracy without weights and 99.17% accuracy with different weights. Thus, the proposed multi-modal models using early fusion methods demonstrated state-of-the-art performance on the GIST pathological gait database.

COVID-19 has infected millions of people worldwide over the past few years. The main technique used for COVID-19 detection is reverse transcription, which is expensive, sensitive, and requires medical expertise. X-ray imaging is an alternative and more accessible technique. This study aimed to improve detection accuracy to create a computer-aided diagnostic tool. Combining other artificial intelligence applications techniques with radiological imaging can help detect different diseases. This study proposes a technique for the automatic detection of COVID-19 and other chest-related diseases using digital chest X-ray images of suspected patients by applying transfer learning (TL) algorithms. For this purpose, two balanced datasets, Dataset-1 and Dataset-2, were created by combining four public databases and collecting images from recently published articles. Dataset-1 consisted of 6000 chest X-ray images with 1500 for each class. Dataset-2 consisted of 7200 images with 1200 for each class. To train and test the model, TL with nine pretrained convolutional neural networks (CNNs) was used with augmentation as a preprocessing method. The network was trained to classify using five classifiers: two-class classifier (normal and COVID-19); three-class classifier (normal, COVID-19, and viral pneumonia), four-class classifier (normal, viral pneumonia, COVID-19, and tuberculosis (Tb)), five-class classifier (normal, bacterial pneumonia, COVID-19, Tb, and pneumothorax), and six-class classifier (normal, bacterial pneumonia, COVID-19, viral pneumonia, Tb, and pneumothorax). For two, three, four, five, and six classes, our model achieved a maximum accuracy of 99.83, 98.11, 97.00, 94.66, and 87.29%, respectively.

Walking is an exercise that uses muscles and joints of the human body and is essential for understanding body condition. Analyzing body movements through gait has been studied and applied in human identification, sports science, and medicine. This study investigated a spatiotemporal graph convolutional network model (ST-GCN), using attention techniques applied to pathological-gait classification from the collected skeletal information. The focus of this study was twofold. The first objective was extracting spatiotemporal features from skeletal information presented by joint connections and applying these features to graph convolutional neural networks. The second objective was developing an attention mechanism for spatiotemporal graph convolutional neural networks, to focus on important joints in the current gait. This model establishes a pathological-gait-classification system for diagnosing sarcopenia. Experiments on three datasets, namely NTU RGB+D, pathological gait of GIST, and multimodal-gait symmetry (MMGS), validate that the proposed model outperforms existing models in gait classification.

Facial expression recognition is very useful for effective human–computer interaction, robot interfaces, and emotion-aware smart agent systems. This paper presents a new framework for facial expression recognition by using a hybrid model: a combination of convolutional neural networks (CNNs) and a support vector machine (SVM) classifier using dynamic facial expression data. In order to extract facial motion characteristics, dense facial motion flows and geometry landmark flows of facial expression sequences were used as inputs to the CNN and SVM classifier, respectively. CNN architectures for facial expression recognition from dense facial motion flows were proposed. The optimal weighting combination of the hybrid classifiers provides better facial expression recognition results than individual classifiers. The system has successfully classified seven facial expressions signalling anger, contempt, disgust, fear, happiness, sadness and surprise classes for the CK+ database, and facial expressions of anger, disgust, fear, happiness, sadness and surprise for the BU4D database. The recognition performance of the proposed system is 99.69% for the CK+ database and 94.69% for the BU4D database. The proposed method shows state-of-the-art results for the CK+ database and is proven to be effective for the BU4D database when compared with the previous schemes.

This study presents a human action recognition system from multi‐view image sequences. The authors’ approach to human action recognition is based on an estimation of local motion from multiple camera views. The authors propose a new motion descriptor, called histogram of motion intensity and direction, to capture local motion characteristics of human activity. After image normalisation, they estimate motion flow using dense optical flow. Using regular grids, they extract local flow motion and estimate the dominant angle and the intensity of optical flow. The histogram of the dominant angle and its intensity are used as a descriptor for each sequence. After the identification of head direction, they concatenate descriptors in each view as a single feature vector from multiple‐view sequences. Classification based on the proposed feature vector using support vector machine shows better performance than three‐dimensional optical flow‐based approaches, but with lower computational requirements. The authors evaluated action recognition on the publicly available i3DPost and the Institut de Recherche en Informatique et en Automatique (INRIA) Xmas Motion Acquisition Sequences database. Experimental results show promising state‐of‐the‐art results and validate the advanced performance of the authors’ approach.

Recently LED (Lighting Emitting Diode) lighting sources are applied not only for displays like LED BLU (back light unit) TV but also for general lighting like LED lamps for home and office. The color temperature, or chromaticity, and brightness of LED lighting can be easily controlled. Preferred combinations between illuminance and color temperature of lighting depend on daily living activities (Oi et al., 2007 Symposium on Design of Artificial Environments 214–215). Changes in intensity can be more easily detected than color changes (Almeida et al., 2009 Perception 38 1109–1117). We investigated whether the illumination stimuli of LED lighting can enhance attention and relaxation level by controlling color temperature and illuminance according to activities. EEG signals are used to estimate attention and relaxation levels of human subjects under different lighting conditions. Nine participants with normal eye sight and color vision participated in the experiments with four different activities under different illumination conditions. LED lighting with color temperature 3600 K in 240 lux is used for relaxation activities, and LED lighting with 6600 K in 794 lux is used for the task which requires attention. These lighting conditions are compared with conventional lighting condition with 4600 K in 530 lux. Preliminary experiment results show that low color temperature with low illumination intensity of LED lighting enhances relaxation level and high color temperature with high illuminance improves attention level compared with conventional lighting environment without illuminance and color temperature changes.

Objective: Prior studies on models based on deep learning (DL) and measuring the cardiothoracic ratio (CTR) on chest radiographs have lacked rigorous agreement analyses with radiologists or reader tests. We validated the performance of a commercially available DL-based CTR measurement model with various thoracic pathologies, and performed agreement analyses with thoracic radiologists and reader tests using a probabilistic-based reference. Materials and Methods: This study included 160 posteroanterior view chest radiographs (no lung or pleural abnormalities, pneumothorax, pleural effusion, consolidation, and n = 40 in each category) to externally test a DL-based CTR measurement model. To assess the agreement between the model and experts, intraclass or interclass correlation coefficients (ICCs) were compared between the model and two thoracic radiologists. In the reader tests with a probabilistic-based reference standard (Dawid–Skene consensus), we compared diagnostic measures—including sensitivity and negative predictive value (NPV)—for cardiomegaly between the model and five other radiologists using the non-inferiority test. Results: For the 160 chest radiographs, the model measured a median CTR of 0.521 (interquartile range, 0.446–0.59) and a mean CTR of 0.522 ± 0.095. The ICC between the two thoracic radiologists and between the model and two thoracic radiologists was not significantly different (0.972 versus 0.959, p = 0.192), even across various pathologies (all p-values > 0.05). The model showed non-inferior diagnostic performance, including sensitivity (96.3% versus 97.8%) and NPV (95.6% versus 97.4%) (p < 0.001 in both), compared with the radiologists for all 160 chest radiographs. However, it showed inferior sensitivity in chest radiographs with consolidation (95.5% versus 99.9%; p = 0.082) and NPV in chest radiographs with pleural effusion (92.9% versus 94.6%; p = 0.079) and consolidation (94.1% versus 98.7%; p = 0.173). Conclusion: While the sensitivity and NPV of this model for diagnosing cardiomegaly in chest radiographs with consolidation or pleural effusion were not as high as those of the radiologists, it demonstrated good agreement with the thoracic radiologists in measuring the CTR across various pathologies.