Search

A coupled multimodal emotional feature analysis (CMEFA) method based on broad-deep fusion networks, which divide multimodal emotion recognition into two layers, is proposed. First, facial emotional features and gesture emotional features are extracted using the broad and deep learning fusion network (BDFN). Considering that the bi-modal emotion is not completely independent of each other, canonical correlation analysis (CCA) is used to analyze and extract the correlation between the emotion features, and a coupling network is established for emotion recognition of the extracted bi-modal features. Both simulation and application experiments are completed. According to the simulation experiments completed on the bimodal face and body gesture database (FABO), the recognition rate of the proposed method has increased by 1.15% compared to that of the support vector machine recursive feature elimination (SVMRFE) (without considering the unbalanced contribution of features). Moreover, by using the proposed method, the multimodal recognition rate is 21.22%, 2.65%, 1.61%, 1.54%, and 0.20% higher than those of the fuzzy deep neural network with sparse autoencoder (FDNNSA), ResNet-101 + GFK, C3D + MCB + DBN, the hierarchical classification fusion strategy (HCFS), and cross-channel convolutional neural network (CCCNN), respectively. In addition, preliminary application experiments are carried out on our developed emotional social robot system, where emotional robot recognizes the emotions of eight volunteers based on their facial expressions and body gestures.

Multimodal medical image fusion aims to improve the clinical practicability of medical images by integrating complementary information from multiple medical images. However, in traditional fusion methods, the fusion rules based on prior knowledge or logic usually cannot match the feature representation perfectly, which results in partial information loss. Furthermore, most deep learning-based fusion methods depend on convolutional operations, which only focus on local features and have limited retention of context information. To address the above issues, we propose an end-to-end dynamic residual focal transformer network for multimodal medical image fusion, termed DRFT. The DRFT framework is an end-to-end network with no need to manually design fusion rules. Firstly, the context-gated convolution is introduced to construct the context dynamic extraction module (CDEM) to extract the key semantic information more accurately from multimodal medical images. Then, a new residual transformer fusion module (RTFM) is designed by incorporating the focal transformer into the residual mechanism, which can not only extract the deep semantic features, but also adaptively learn the optimal fusion scheme. Finally, the nest architecture is employed to extract multiscale features. In addition, a new objective function consisting of global detail loss and fusion enhancement loss is designed to enrich the modal information in the fused image. Notably, the proposed network does not require the two-stage training strategy as opposed to the traditional encoder–decoder fusion structure. Extensive experimental results on mainstream datasets show that, compared with the state-of-the-art methods, the proposed DRFT delivers better performance in both qualitative and quantitative evaluation. [ABSTRACT FROM AUTHOR]

As an essential information resource, online reviews play an important role in consumers’ decision-making processes. To solve the product ranking problem through online reviews, two important issues are involved: sentiment analysis (SA) for online reviews and product ranking based on multi-criteria decision-making (MCDM) methods. However, merely a few studies have considered the impact of SA accuracy, which can significantly affect the final decision-making process. This paper proposes a novel data-driven method for ranking products through online reviews based on interval type-2 fuzzy sets (IT2FSs) and SA. In this method, after acquiring online reviews, the explicit and implicit attributes are extracted from the website itself and the latent Dirichlet allocation (LDA) model, respectively. Thereafter, a deep learning model is adopted to identify the five sentiment intensities of online reviews, based on which the SA results are represented as IT2FSs by considering the classification effect. After type-reduction for IT2FSs, the ranking order is obtained based on the exponential TODIM (ExpTODIM) method. Furthermore, a case study on ranking travel products from Trip.com Group through online reviews is provided to illustrate the effectiveness and applicability of the proposed method. [ABSTRACT FROM AUTHOR]

Drones play a pivotal role in various industries of Industry 4.0. For achieving the application of drones in a dynamic environment, finding a clear path for their autonomous flight requires more research. This paper addresses the problem of finding a navigation path for an autonomous drone based on visual scene information. A deep learning-based object detection approach can localize obstacles detected in a scene. Considering this approach, we propose a solution framework that includes masking with a color-based segmentation method to identify an empty area where the drone can fly. The scene is described using segmented regions and localization points. The proposed approach can be used to remotely guide drones in dynamic environments that have poor coverage from global positioning systems. The simulation results show that the proposed framework with object detection and the proposed masking technique support drone navigation in a dynamic environment based only on the visual input from the front field of view. [ABSTRACT FROM AUTHOR]

Recent advances in deep learning have tremendously leveraged the performance of network representation learning (NRL). Multiple deep learning-based NRL models have been proposed recently to effectively handling primitive tasks of information network analysis and mining (INAM) domain, including link prediction (LP). LP is considered as an important one due to its multiple applications in many disciplines. In the recent few years, LP in dynamic networks has attracted a lot of attention from researchers to propose novel algorithms for better capturing both rich structural and evolutional information of complex information networks (INs). However, recent models are mainly concentrated on preserving the sequential representations of a given network over time. They have largely ignored other important structural features, such as: intracommunity which contributes to the creation of links between network nodes. In this article, we propose a novel community-driven dynamic NRL technique upon the RNN+GCN framework, called: ComGCN. Specifically, the ComGCN model is a combination of microscopic (node embedding-based) and mesoscopic (intracommunity-based) dynamic network embedding approach which enable effectively handling the LP problem in context of dynamism. Extensive experiments on real-world dynamic networks demonstrated the effectiveness of the proposed model compared with recent state-of-the-art baselines. [ABSTRACT FROM AUTHOR]

License plate (LP) detection in the wild remains challenging due to the diversity of environmental conditions. Nevertheless, prior solutions have focused on controlled environments, such as when LP images frequently emerge as from an approximately frontal viewpoint and without scene text which might be mistaken for an LP. However, even for state-of-the-art object detectors, their detection performance is not satisfactory for real-world environments, suffering from various types of degradation. To solve these problems, we propose a novel end-to-end framework for robust LP detection, designed for such challenging settings. Our contribution is threefold: (1) A novel information-theoretic learning that takes advantage of a shared encoder, an LP detector and a scene text detector (excluding LP) simultaneously; (2) Localization refinement for generalizing the bounding box regression network to complement ambiguous detection results; (3) a large-scale, comprehensive dataset, LPST-110K, representing real-world unconstrained scenes including scene text annotations. Computational tests show that the proposed model outperforms other state-of-the-art methods on a variety of challenging datasets. [ABSTRACT FROM AUTHOR]

Perception techniques for autonomous driving should be adaptive to various environments. In essential perception modules for traffic line detection, many conditions should be considered, such as a number of traffic lines and computing power of the target system. To address these problems, in this paper, we propose a traffic line detection method called Point Instance Network (PINet); the method is based on the key points estimation and instance segmentation approach. The PINet includes several hourglass models that are trained simultaneously with the same loss function. Therefore, the size of the trained models can be chosen according to the target environment’s computing power. We cast a clustering problem of the predicted key points as an instance segmentation problem; the PINet can be trained regardless of the number of the traffic lines. The PINet achieves competitive accuracy and false positive on CULane and TuSimple datasets, popular public datasets for lane detection. Our code is available at https://github.com/koyeongmin/PINet_new [ABSTRACT FROM AUTHOR]

The two-stage fuzzy fusion based-convolution neural network is proposed for dynamic emotion recognition by using both facial expression and speech modalities, which not only can extract discriminative emotion features which contain spatio-temporal information, but also can effectively fuse facial expression and speech modalities. Moreover, the proposal is able to handle situations where the contributions of each modality data to emotion recognition are very imbalanced. The local binary patterns coming from three orthogonal planes and spectrogram are considered first to extract low-level dynamic emotion, so that the spatio-temporal information of these modalities can be obtained. To reveal more discriminative features, two deep convolution neural networks are constructed to extract high-level emotion semantic features. Moreover, the two stage fuzzy fusion strategy is developed by integrating canonical correlation analysis and fuzzy broad learning system, so as to take into account the correlation and difference between different modal features, as well as handle the ambiguity of emotional state information. The experimental results obtained on benchmark databases show that the accuracies of the proposed method are higher than those of existing methods (such as the hybrid deep model, and the rule-based and machine learning method) on SAVEE, eNTERFACE’05, and AFEW databases. [ABSTRACT FROM AUTHOR]

Magnetic resonance imaging (MRI) is a prevailing method of modal infant brain tissue analysis that precisely segments brain tissue and is vitally important for diagnosis, remediation, and analysis of early brain development. To achieve such segmentation is challenging, particularly for the brain of a six-month-old, owing to several factors: poor image quality; isointense contrast between white and gray matter and the simple incomplete volume consequence of a tiny brain size; and discrepancies in brain tissues, illumination settings, and the vagarious region. This article addresses these challenges with a fuzzy-informed deep learning segmentation network that takes T1- and T2-weighted MRIs as inputs. First, a fuzzy logic layer encodes input to the fuzzy domain. Second, a volumetric fuzzy pooling (VFP) layer models the local fuzziness of the volumetric convolutional maps by applying fuzzification, accumulation, and defuzzification on the adjacency feature map neighborhoods. Third, the VFP layer is employed to design the fuzzy-enabled multiscale feature learning module to enable the extraction of brain features in different receptive fields. Finally, we redesign the Project & Excite module using the VPF layer to enable modeling uncertainty during feature recalibration, and a comprehensive training paradigm is used to learn the ideal parameters of every building block. Extensive experimental comparative studies substantiate the efficiency and accuracy of the proposed model in terms of different evaluation metrics to solve multimodal infant brain segmentation problems on the iSeg-2017 dataset. [ABSTRACT FROM AUTHOR]

The plethora of ways of data representation and their applications to system modeling is inherently associated with dimensionality reduction. In a nutshell, the result of dimensionality reduction should support efficient ways of constructing ensuing models (classifiers and predictors) as well as an interpretation of the data themselves. Furthermore, there should be a suitable measure quantifying the quality of data positioned in the reduced space. We advocate that what makes the reduced data interpretable goes hand in hand with revealing a logic fabric of the data, suppressing redundancy, and finally arriving at a logic description of data. The anticipation is that the reduced data can be described in the form of logic expressions formed over the original highly dimensional data. Evidently, having these above-stated points in mind, the aim of this article is twofold: 1) to develop a logic-oriented data representation with the aid of autoencoders; and 2) to quantify the quality of results of this dimensionality reduction by incorporating a facet of information granularity. In other words, we argue that the result of dimensionality reduction gives rise to information granules whose level of granularity associates with the quality of processing completed by the autoencoder. In light of the recent surge of architectures of deep learning, the study is focused on the construction and analysis of logic-oriented autoencoders. We propose a two-level architecture composed of the logic-oriented processing units organized in two layers of logic processing units. As data representation provided by the autoencoder is not ideal, we augment the original architecture by granular parameters, which give rise to granular logic-oriented autoencoders. A suite of experiments is also reported. [ABSTRACT FROM AUTHOR]

Deep learning is a timely research direction in machine learning, where breakthrough progress has been made in both academe and industries, bringing promising results in speech recognition, computer vision, industrial control and automation, etc. The motivation of deep learning is primarily to establish a model to simulate the neural connection structure of the human brain. While dealing with complex tasks, deep learning adopts a number of transformation stages to deliver the in-depth description and interpretation of the data. Deep learning achieves exceptional power and flexibility by learning to represent the task through a nested hierarchy of layers, with more abstract representations formed successively in terms of less abstract ones. One of the key issues of existing deep learning approaches is that the meaningful representations can be learned only when their hyperparameter settings are properly specified beforehand, and general parameters are learned during the training process. Until now, not much research has been dedicated to automatically set the hyperparameters, and accurately find the globally optimal general parameters. However, this problem can be formulated as optimization problems, including discrete optimization, constrained optimization, large-scale global optimization, and multiobjective optimization, by engaging mechanisms of evolutionary computation. [ABSTRACT FROM AUTHOR]

A fuzzy deep neural network with sparse autoencoder (FDNNSA) is proposed for intention understanding based on human emotions and identification information (i.e., age, gender, and region), in which the fuzzy C-means (FCM) is used to cluster the input data, and deep neural network with sparse autoencoder (DNNSA) is designed for emotional intention understanding in human–robot interaction. It aims to make robots capable of recognizing human emotions and understanding related emotional intention, the FCM is suitable for gathering similar information so that the calculations of dimensionality of DNNSA will be reduced, and the sparse autoencoder of DNNSA can make the neuron of DNNSA sparse to reduce the complexity of the network in such a way human–robot interaction is running smoothly. To validate the proposal, simulation experiments based on benchmark databases such as facial expression database of CK+, and speech emotion corpus of CASIA were completed. The experimental results show that the proposal outperforms the baseline algorithms of Softmax regression (SR), DNNSA, FCM-based SR (FSR), Softplus, Gath Geva-based DNNSA (GDNNSA), and ensemble DNNSA (EDNNSA). Preliminary application experiments are performed in the development of emotional social robot system, where volunteers experience the scenario of “drinking at the bar”. The obtained results indicate that the proposed FDNNSA can promote robot understanding of emotional intention of human. [ABSTRACT FROM AUTHOR]

Processing realized in fuzzy rule-based models is associated with matching individual rules with the existing input data and aggregation of the levels of matching and conclusions of the rules. Rules are processed as individual entities. In this study, we introduce an augmentation of fuzzy models by facilitating interaction among the rules leading to more flexible type membership functions of fuzzy sets forming conditions of the rules (thus resulting in substantially advanced topology of the partition of the input space). Different ways of realizing interaction among the rules are studied. In the sequel, we develop a granular fuzzy model implied by the rule-based model showing how granular parameters of the original rule-based model enhance its quality expressed in terms of coverage of experimental data. The two evaluation criteria of the constructed granular model, namely coverage and specificity are studied. Experimental results are reported for a series of publicly available data. [ABSTRACT FROM AUTHOR]

• Time series Covid-19 prediction problem, a case study in Vietnam. • A dual attention mechanism with RNN-based auto-encoding for handling chaotic time series prediction task. • Extensive experiments with realistic Covid-19 data in Vietnam. • Datasets and implementation source code are available for future studies. For preventing the outbreaks of Covid-19 infection in different countries, many organizations and governments have extensively studied and applied different kinds of quarantine isolation policies, medical treatments as well as organized massive/fast vaccination strategy for over-18 citizens. There are several valuable lessons have been achieved in different countries this Covid-19 battle. These studies have presented the usefulness of prompt actions in testing, isolating confirmed infectious cases from community as well as social resource planning/optimization through data-driven anticipation. In recent times, many studies have demonstrated the effectiveness of short/long-term forecasting in number of new Covid-19 cases in forms of time-series data. These predictions have directly supported to effectively optimize the available healthcare resources as well as imposing suitable policies for slowing down the Covid-19 spreads, especially in high-populated cities/regions/nations. There are several progresses of deep neural architectures, such as recurrent neural network (RNN) have demonstrated significant improvements in analyzing and learning the time-series datasets for conducting better predictions. However, most of recent RNN-based techniques are considered as unable to handle chaotic/non-smooth sequential datasets. The consecutive disturbances and lagged observations from chaotic time-series dataset like as routine Covid-19 confirmed cases have led to the low performance in temporal feature learning process through recent RNN-based models. To meet this challenge, in this paper, we proposed a novel dual attention-based sequential auto-encoding architecture, called as: DAttAE. Our proposed model supports to effectively learn and predict the new Covid-19 cases in forms of chaotic and non-smooth time series dataset. Specifically, the integration between dual self-attention mechanism in a given Bi-LSTM based auto-encoder in our proposed model supports to directly focus the model on a specific time-range sequence in order to achieve better prediction. We evaluated the performance of our proposed DAttAE model by comparing with multiple traditional and state-of-the-art deep learning-based techniques for time-series prediction task upon different real-world datasets. Experimental outputs demonstrated the effectiveness of our proposed attention-based deep neural approach in comparing with state-of-the-art RNN-based architectures for time series based Covid-19 outbreak prediction task. [ABSTRACT FROM AUTHOR]

Near real-time prediction of earthquake-triggered landslides can rapidly forecast the spatial distribution of coseismic landslides just after a great earthquake, and provide effective support for emergency response. However, the prediction of earthquake-triggered landslides has always been a great challenge because of low accuracy and high false alarms. This work proposes a novel fuzzy deep learning (FuDL) model for near real-time earthquake-triggered landslide spatial prediction. Fuzzy learning theory is for the first time employed in earthquake-triggered landslide prediction. The FuDL has high generalization and robustness, effectively improving the accuracy of earthquake-triggered landslide prediction. Eighteen earthquake-triggered landslide inventories worldwide from 2008 to 2022 are employed to conduct ETL prediction. According to the chronological order, 15 earthquake-triggered landslides from 2008 to 2018 are adopted to train the FuDL model, and 3 earthquake-triggered landslides from 2019 to 2022 are utilized for near real-time earthquake-triggered landslide prediction. Furthermore, this work reveals that ground movement, relatively steep and high topography, and strong seismic intensity are critical factors affecting the spatial distribution of earthquake-triggered landslides. In addition, this work conducted a detailed analysis of the distribution patterns of earthquake-triggered landslides on a global scale. • A novel FuDL model is suggested and achieves high accuracy and good generalization. • FuDL model outperforms the state-of-the-art machine learning or deep learning models. • Topography, ground shake, and seismic intensity dominate the distribution of ETLs. [ABSTRACT FROM AUTHOR]