Search

The task of multi-label image recognition is to predict a set of object labels that present in an image. As objects normally co-occur in an image, it is desirable to model label dependencies to improve recognition performance. To capture and explore such important information, we propose Graph Convolutional Networks based models for multi-label recognition, where directed graphs are constructed over classes and information is propagated between classes to learn inter-dependent class-level representations. Following this idea, we design two particular models that approach multi-label classification from different views. In our first model, the prior knowledge about the class dependencies is integrated into classifier learning. Specifically, we propose Classifier-Learning-GCN to map class-level semantic representations (\eg, word embedding) into classifiers that maintain the inter-class topology. In our second model, we decompose the visual representation of an image into a set of label-aware features and propose Prediction-Learning-GCN to encode such features into inter-dependent image-level prediction scores. Furthermore, we also present an effective correlation matrix construction approach to capture inter-class relationships and consequently guide information propagation among classes. Empirical results on generic multi-label recognition demonstrate that the effectiveness of both two proposed models. Moreover, the proposed methods also show advantages in other multi-label related applications.

The performance of a convolutional neural network (CNN) heavily depends on its hyperparameters. However, finding a suitable hyperparameters configuration is difficult, challenging, and computationally expensive due to three issues, which are 1) the mixed-variable problem of different types of hyperparameters; 2) the large-scale search space of finding optimal hyperparameters; and 3) the expensive computational cost for evaluating candidate hyperparameters configuration. Therefore, this article focuses on these three issues and proposes a novel estimation of distribution algorithm (EDA) for efficient hyperparameters optimization, with three major contributions in the algorithm design. First, a hybrid-model EDA is proposed to efficiently deal with the mixed-variable difficulty. The proposed algorithm uses a mixed-variable encoding scheme to encode the mixed-variable hyperparameters and adopts an adaptive hybrid-model learning (AHL) strategy to efficiently optimize the mixed-variables. Second, an orthogonal initialization (OI) strategy is proposed to efficiently deal with the challenge of large-scale search space. Third, a surrogate-assisted multi-level evaluation (SME) method is proposed to reduce the expensive computational cost. Based on the above, the proposed algorithm is named surrogate-assisted hybrid-model EDA (SHEDA). For experimental studies, the proposed SHEDA is verified on widely used classification benchmark problems, and is compared with various state-of-the-art methods. Moreover, a case study on aortic dissection (AD) diagnosis is carried out to evaluate its performance. Experimental results show that the proposed SHEDA is very effective and efficient for hyperparameters optimization, which can find a satisfactory hyperparameters configuration for the CIFAR10, CIFAR100, and AD diagnosis with only 0.58, 0.97, and 1.18 GPU days, respectively.

Facial micro-expression (ME) can disclose genuine and concealed human feelings. It makes MEs extensively useful in real-world applications pertaining to affective computing and psychology. Unfortunately, they are induced by subtle facial movements for a short duration of time, which makes the ME recognition, a highly challenging problem even for human beings. In automatic ME recognition, the well-known features encode either incomplete or redundant information, and there is a lack of sufficient training data. The proposed method, Micro-Expression Recognition by Analysing Spatial and Temporal Characteristics, MERASTC mitigates these issues for improving the ME recognition. It compactly encodes the subtle deformations using action units (AUs), landmarks, gaze, and appearance features of all the video frames while preserving most of the relevant ME information. Furthermore, it improves the efficacy by introducing a novel neutral face normalization for ME and initiating the utilization of gaze features in deep learning based ME recognition. The features are provided to the 2D convolutional neural network that jointly analyses the spatial and temporal behavior for correct ME classification. Experimental results on the publicly available datasets indicate that the proposed method exhibits better performance than the well-known methods.

Weeds are mainly spread by weed seeds being mixed with agricultural and forestry crop seeds, grain, animal hair, and other plant products, and disturb the growing environment of target plants such as crops and wild native plants. The accurate and efficient classification of weed seeds is important for the effective management and control of weeds. However, classification remains mainly dependent on destructive sampling-based manual inspection, which has a high cost and rather low flux. We considered that this problem could be solved using a nondestructive intelligent image recognition method. First, on the basis of the establishment of the image acquisition system for weed seeds, images of single weed seeds were rapidly and completely segmented, and a total of 47 696 samples of 140 species of weed seeds and foreign materials remained. Then, six popular and novel deep Convolutional Neural Network (CNN) models are compared to identify the best method for intelligently identifying 140 species of weed seeds. Of these samples, 33 600 samples are randomly selected as the training dataset for model training, and the remaining 14 096 samples are used as the testing dataset for model testing. AlexNet and GoogLeNet emerged from the quantitative evaluation as the best methods. AlexNet has strong classification accuracy and efficiency (low time consumption), and GoogLeNet has the best classification accuracy. A suitable CNN model for weed seed classification could be selected according to specific identification accuracy requirements and time costs of applications. This research is beneficial for developing a detection system for weed seeds in various applications. The resolution of taxonomic issues and problems associated with the identification of these weed seeds may allow for more effective management and control.

Leaf disease recognition using image processing and deep learning techniques is currently a vibrant research area. Most studies have focused on recognizing diseases from images of whole leaves. This approach limits the resulting models’ ability to estimate leaf disease severity or identify multiple anomalies occurring on the same leaf. Recent studies have demonstrated that classifying leaf diseases based on individual lesions greatly enhances disease recognition accuracy. In those studies, however, the lesions were laboriously cropped by hand. This study proposes a semi-automatic algorithm that facilitates the fast and efficient preparation of datasets of individual lesions and leaf image pixel maps to overcome this problem. These datasets were then used to train and test lesion classifier and semantic segmentation Convolutional Neural Network (CNN) models, respectively. We report that GoogLeNet’s disease recognition accuracy improved by more than 15% when diseases were recognized from lesion images compared to when disease recognition was done using images of whole leaves. A CNN model which performs semantic segmentation of both the leaf and lesions in one pass is also proposed in this paper. The proposed KijaniNet model achieved state-of-the-art segmentation performance in terms of mean Intersection over Union (mIoU) score of 0.8448 and 0.6257 for the leaf and lesion pixel classes, respectively. In terms of mean boundary F1 score, the KijaniNet model attained 0.8241 and 0.7855 for the two pixel classes, respectively. Lastly, a fully automatic algorithm for leaf disease recognition from individual lesions is proposed. The algorithm employs the semantic segmentation network cascaded to a GoogLeNet classifier for lesion-wise disease recognition. The proposed fully automatic algorithm outperforms competing methods in terms of its superior segmentation and classification performance despite being trained on a small dataset.

The capability of Convolutional Neural Networks (CNNs) for sparse representation has significant application to complex tasks like Representation Learning (RL). However, labelled datasets of sufficient size for learning this representation are not easily obtainable. The unsupervised learning capability of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) provide a promising solution to this issue through their capacity to learn representations for novel data samples and classification tasks. In this research, a texture-based latent space disentanglement technique is proposed to enhance learning of representations for novel data samples. A comparison is performed among different VAEs and GANs with the proposed approach for synthesis of new data samples. Two different VAE architectures are considered, a single layer dense VAE and a convolution based VAE, to compare the effectiveness of different architectures for learning of the representations. The GANs are selected based on the distance metric for disjoint distribution divergence estimation of complex representation learning tasks. The proposed texture-based disentanglement has been shown to provide a significant improvement for disentangling the process of representation learning by conditioning the random noise and synthesising texture rich images of fruit and vegetables.

Camouflaged object detection is a challenging task that aims to identify objects having similar texture to the surroundings. This paper presents to amplify the subtle texture difference between camouflaged objects and the background for camouflaged object detection by formulating multiple texture-aware refinement modules to learn the texture-aware features in a deep convolutional neural network. The texture-aware refinement module computes the biased co-variance matrices of feature responses to extract the texture information, adopts an affinity loss to learn a set of parameter maps that help to separate the texture between camouflaged objects and the background, and leverages a boundary-consistency loss to explore the structures of object details. We evaluate our network on the benchmark datasets for camouflaged object detection both qualitatively and quantitatively. Experimental results show that our approach outperforms various state-of-the-art methods by a large margin.

Internet of Things (IoT) has brought changes in many fields by joining physical space with the cyber space. The IoT devices are becoming increasingly complex. With the rapid deployment of cameras, tasks in IoT like visual information are more important, but IoT devices have limited computing resources, including power, computing ability, storage, etc. Some tasks that might be perfectly normal to perform on a computer would be rather challenging on an IoT device. Therefore, how to maintain acceptable performance while minimizing resources is becoming a more consequential part in IoT. In the paper, we aim to solve the problem of object detection and tracking in IoT while minimizing resources. The traditional algorithms need to use convolutional neural network (CNN) to identify different objects in each frame, and then determine the tracking target from identified objects, which typically requires a lot of computing resources. By incorporating spatial attention, and multi-domain network, we proposed a novel algorithm named as Spatial Attention Powered Multi-Domain Network (SA-MDNet). By adding spatial attention mechanism to the original MDNet model, and using multi class cross entropy loss, we are able to distinguish the background and the target in different video sequences effectively and efficiently. This novel algorithm achieves similar performance on the OTB 50/100/2013 datasets compared to several state-of-art models, while uses only a fraction of the memory compared to MDNet.

With the rapid advance of Internet of Things (IoTs), it is difficult for cloud-centric computing to meet the requirements of low latency and ease of use. As an open and distributed system, edge computing integrates computing, networking, storage, and applications. It provides intelligent services on the edge of an IoT. The edge network is comprised of various wireless and wired networks, and the computing and storage resources of edge nodes are limited. These conditions make the edge network expose to a variety of cyber attacks. Additionally, it is difficult for an IoT edge node to support large-scale network data collection and detection for IoT security. Although big data-enabled intrusion detection algorithms can ensure the high accuracy of intrusion detection systems, it is stressful for resource-limited edge nodes to implement those algorithms in IoTs. Motivated by these challenges, we propose an intelligent intrusion detection algorithm implemented by big data mining based on fuzzy rough set, Generative Adversarial Network (GAN), and Convolutional Neural Network (CNN). In our method, we first propose a fuzzy rough set-based algorithm to perform feature selection for big data via IoTs. Then, we take advantage of the efficient feature extraction capabilities of CNN for implementing intrusion detection based on selected features. Furthermore, after combining CNN and GAN, we propose an intelligent algorithm to realize intrusion detection in a variety of scenarios. Finally, the proposed method is compared with existing methods for evaluation. Simulation results show that our method has up to 4% higher accuracy than existing methods.

A largely ignored fact in spectral super-resolution (SSR) is that the subsistent mapping methods neglect the auxiliary prior of camera spectral sensitivity (CSS) and only pay attention to wider or deeper network framework design while ignoring to excavate the spatial and spectral dependencies among intermediate layers, hence constraining representational capability of convolutional neural networks (CNNs). To conquer these drawbacks, we propose a novel deep hybrid 2-D-3-D CNN based on dual second-order attention with CSS prior (HSACS), which can excavate sufficient spatial-spectral context information. Specifically, dual second-order attention embedded in the residual block for more powerful spatial-spectral feature representation and relation learning is composed of a brand new trainable 2-D second-order channel attention (SCA) or 3-D second-order band attention (SBA) and a structure tensor attention (STA). Concretely, the band and channel attention modules are developed to adaptively recalibrate the band-wise and interchannel features via employing second-order band or channel feature statistics for more discriminative representations. Besides, the STA is promoted to rebuild the significant high-frequency spatial details for enough spatial feature extraction. Moreover, the CSS is first employed as a superior prior to avoid its effect of SSR quality, on the strength of which the resolved RGB can be calculated naturally through the super-reconstructed hyperspectral image (HSI); then, the final loss consists of the discrepancies of RGB and the HSI as a finer constraint. Experimental results demonstrate the superiority and progressiveness of the presented approach in terms of quantitative metrics and visual effect over SOTA SSR methods.

Digital security in modern systems very often uses biometric, and increasingly new implementations appear. Such applications can be found everywhere, even when picking up the package from courier, we certify its receipt through our signature on tablet. However, verification of this form is not one of the simplest elements in information processing systems. Given the different sizes, angles, or writing conditions that may affect its stability new methods to evaluate signatures are constantly needed. In this article, we propose the use of spline interpolation and two types of artificial neural networks to verify the identity of a person based on selected local and global features extracted from the image of a signature. Global features are extracted concerning interpolation and graphic processing methods, while local features are verified using Convolutional Neural Networks. Both sets of features are used in the identity verification process. The article presents the model of operation together with experiments, taking into account various parameters of the proposed extraction. We have reached an accuracy of 87.7% on the SVC2004 database.

Gesture Recognition which is an inevitable part of human computer interaction is an ever evolving active research area. This has been in use under varied applications like smart home systems, sign language recognition, augment reality and in device controls. The acquisition of hand gestures are usually through optical sensors, in this work a radar based hand gesture data set is used for classifying the gestures. Micro doppler signatures were used as the input to the model. Radar dataset has fewer data samples compared to optical based data sets. The proposed work uses separable convolutional neural networks model which does depth wise convolution followed by point wise convolution to reduce overfitting effect of the training data. The proposed model was built in such a way that the model is capable of classifying any unseen data without exactly mimicking the training samples. The proposed model has achieved 94.56% as the testing accuracy which is certainly better than the previous work on this Dop Net data set. Moreover the model has also minimized the computational hours of the model using separable convolutions.

The early diagnosis of any disease can be curable with a small amount of human effort. Most people will not be able to detect their illness before it becomes chronic. Breast cancer has been one of the diseases that can be treated until it spreads to all parts of the body when the disease is diagnosed at an earlier stage.Breast cancer is causing an increase in death rates all over the world. The medical practitioner can diagnose diseases incorrectly due to misinterpretation. In both developed and emerging countries, breast cancer is one of the leading causes of death for women. Breast cancer can be detected and classified early on in its progression, allowing patients to get adequate treatment. For detecting and classifying breast cancer in breast cytology videos, deep learning frameworks are suggested. The suggested architecture uses the Convolutional layer and the pooling layer to remove features. The data would be classified using the dense layer.The Convolutional Neural Network and DenseNet are used to extract image features that are then fed into a fully connected layer for the classification of malignant and benign cellsFor image classification of breast cancer, the models are trained, validated, and tested.The distinction is based on CNN and DenseNet's different batch sizes and learning rates. To see how well the proposed CNN and DenseNet programs work, they are compared. In terms of accuracy, the suggested system will outperform CNN and DenseNet in the diagnosis and classification of breast tumors from histological photographs. Hyper parameter tuning is used to measure the efficiency of CNN and DenseNet.

Emotion recognition from EEG signals has attracted much attention in affective computing. Recently, a novel dynamic graph convolutional neural network (DGCNN) model was proposed, which simultaneously optimized the network parameters and a weighted graph G characterizing the strength of functional relation between each pair of two electrodes in the EEG recording equipment. In this paper, we propose a sparse DGCNN model which improves the DGCNN by imposing a sparseness constraint on G. Our work is based on an important observation: the tomography study reveals that different brain regions sampled by EEG electrodes may be related to different functions of the brain and then the functional relations among electrodes are possibly highly localized and sparse. However, introducing sparseness constraint into the graph G makes the loss function of sparse DGCNN non-differentiable at some singular points. To ensure that the training process of sparse DGCNN converges, we apply the forward-backward splitting method. To evaluate the performance of sparse DGCNN, we compare it with four representative recognition methods as well as different features and spectral bands. The results show that (1) sparse DGCNN has consistently better accuracy than representative methods and has a good scalability, and (2) DE, PSD and ASM features on γ bands convey most discriminative emotional information, and fusion of separate features and frequency bands can improve recognition performance.

The recognition a material quality is considered as a process of finding out the constituent material present in an object and it is regarded as a vital part in various applications. Hence, it is considered as a valuable approach for the creation of a system that possesses the ability to achieve recognition of a material. In this paper, we develop a mechanism using convolutional neural networks (CNNs) for material recognition. The CNN model initially trains itself with the features extracted from the image samples. Finally, the classification is carried out with CNN model that learn the classes obtained via CNN of different category of materials. The experimental validation is conducted to test the accuracy of CNN classifiers against various deep learning classifiers. The results on various materials show that the proposed CNN classifier obtains improved recognition accuracy than other methods.

Key semantics can come from everywhere on an image. Semantic alignment is a key part of few-shot learning but still remains challenging. In this paper, we design a Mixer-Based Semantic Spread (MBSS) algorithm that employs a {\it mixer} module to spread the key semantic on the whole image, so that one can directly compare the processed image pairs. We first adopt a convolutional neural network to extract features from both support and query images and separate each of them into multiple Local Descriptor-based Representations (LDRs). The LDRs are then fed into the {\it mixer} for semantic spread, where every LDR attracts complementary information from its peers. In this way, the objective semantic is made spread on the whole image in a data-driven manner. The overall pipeline is supervised by a voting-based loss, guaranteeing a good {\it mixer}. Visualization results validate the feasibility of our {\it mixer}. Comprehensive experiments on three benchmark datasets, miniImageNet, tieredImageNet, and CUB, show that our algorithm achieves the state-of-the-art performance in both $5$ -way $1$ -shot and $5$ -way $5$ -shot settings.

Despite that convolutional neural networks (CNNs) have shown high-quality reconstruction for single image dehazing, recovering natural and realistic dehazed results remains a challenging problem due to semantic confusion in the hazy scene. In this article, we show that it is possible to recover textures faithfully by incorporating semantic prior into dehazing network since objects in haze-free images tend to show certain shapes, textures, and colors. We propose a semantic-aware dehazing network (SDNet) in which the semantic prior is taken as a color constraint for dehazing, benefiting the acquisition of a reasonable scene configuration. In addition, we design a densely connected block to capture global and local information for dehazing and semantic prior estimation. To eliminate the unnatural appearance of some objects, we propose to fuse the features from shallow and deep layers adaptively. Experimental results demonstrate that our proposed model performs favorably against the state-of-the-art single image dehazing approaches.

COVID-19 is one of the biggest pandemics that the world is facing today, and every day, we are coming up with new challenges in this area. Still, much research is already going on to overcome this pandemic, and we also get succeeded to some extent. Diverse sources such as MRI, CT scanning, blood samples, X-ray image, and many more are available to detect COVID-19. Thus, it can be easily said that through image processing, the classification of COVID-19 can be done. In this study, the COVID-19 detection is done by classifying with the use of a type of convolutional neural network termed a detail-oriented capsule network. Chest CT scan imaging for the prediction of COVID-19 and non-COVID-19 are classified in the present paper using a Detailed Oriented capsule network (DOCN). Accuracy, specificity, and sensitivity are parameters used for model evaluation. The proposed model has achieved 98% accuracy, 81% sensitivity, and 98.4% specificity.

In this paper, we introduce a new neonatal facial expression database for pain analysis. This database, called facial expression of neonatal pain (FENP), contains 11000 neonatal facial expression images associated with 106 Chinese neonates from two children's hospitals, i.e., the Children's Hospital Affiliated to Nanjing Medical University and Second Affiliated Hospital Affiliated to Nanjing Medical University in China. The facial expression images cover four categories of facial expressions, i.e., severe pain expression, mild pain expression, crying expression and calmness expression, where each category contains 2750 neonatal facial expression images. Based on this database, we also investigate the pain facial expression recognition problem using several state-of-the-art facial expression features and expression recognition methods, such as Gabor+SVM, LBP+SVM, HOG+SVM, LBP+HOG+SVM, and several Convolutional Neural Network (CNN) methods (including AlexNet, VGGNet, GoogLeNet, ResNet and DenseNet). The experimental results indicate that the proposed neonatal pain facial expression database would be very suitable for the study of both neonatal pain and facial expression recognition. Moreover, the FENP database would be publicly available after signing a license agreement (the users can contact Jingjie Yan (yanjingjie@njupt.edu.cn) or Guanming Lu (lugm@njupt.edu.cn)).

Introduction: Nowadays, deep learning and convolutional neural networks (CNNs) have become widespread tools in many biomedical engineering studies. CNN is an end-to-end tool, which makes the processing procedure integrated, but in some situations, this processing tool requires to be fused with machine learning methods to be more accurate. Methods: In this paper, a hybrid approach based on deep features extracted from wavelet CNNs (WCNNs) weighted layers and multiclass support vector machine (MSVM) was proposed to improve the recognition of emotional states from electroencephalogram (EEG) signals. First, EEG signals were preprocessed and converted to Time-Frequency (T-F) color representation or scalogram using the continuous wavelet transform (CWT) method. Then, scalograms were fed into four popular pre-trained CNNs, AlexNet, ResNet-18, VGG-19, and Inception-v3 to fine-tune them. Then, the best feature layer from each one was used as input to the MSVM method to classify four quarters of the valence-arousal model. Finally, the subject-independent leave-one-subject-out criterion was used to evaluate the proposed method on DEAP and MAHNOB-HCI databases. Results: Results showed that extracting deep features from the earlier convolutional layer of ResNet-18 (Res2a) and classifying using the MSVM increased the average accuracy, precision, and recall by about 20% and 12% for MAHNOB-HCI and DEAP databases, respectively. Also, combining scalograms from four regions of pre-frontal, frontal, parietal, and parietal-occipital and two regions of frontal and parietal achieved the higher average accuracy of 77.47% and 87.45% for MAHNOB-HCI and DEAP databases, respectively. Conclusion: Combining CNN and MSVM increased the recognition of emotion from EEG signals and the results were comparable to state-of-the art studies.

We propose an automatic ternary classification model for Bipolar Disorder (BD) states. As input information, the model uses speech signals from patients' audio-visual recordings of structured interviews. The model classifies the patient's clinical state as Mania, Hypo-Mania, or Remission. We capture Mel-Frequency Cepstral Coefficients (MFCCs) and Geneva Minimalistic Acoustic Parameter Set (GeMAPS) as audio features. We compute linguistic and sentiment features for each subject's transcript. We present a stacked ensemble classifier to classify all fused features after feature selection. A set of three homogeneous Convolutional Neural Networks (CNNs) and a Multi Layer Perceptron (MLP) construct the first-level and second-level of the stacked ensemble classifier respectively. Moreover, we use reinforcement learning to optimize the networks and their hyperparameters. We show that our stacked ensemble framework outperforms existing models on the BD Turkish corpus with a %59.3 Unweighted Average Unit (UAR) on the test set. To the best of our knowledge, this is the highest UAR achieved on this dataset.

Although recent emotion recognition methods (based on facial expression cues) achieve excellent performance in controlled scenarios, the recognition of emotion in the wild remains a challenging problem because of occlusion, large head poses, illumination variations, etc. Recent advances in deep learning show that combining an ensemble of deep learning models can considerably outperform the approach of using only a single deep learning model for challenging recognition problems. This paper presents a novel ensemble deep learning method, deep convolutional neural network (DCNN) ensemble classifier, for improved facial expression recognition (FER) in the wild. Our proposed DCNN ensemble classifier is novel in terms of the following aspects: (1) the process of finding ensemble weights for combining DCNN decision outputs is formulated as a stochastic optimization problem (via simulated annealing) in which the energy to be minimized represents the generalized (test) classification error of the DCNN ensemble classifier and (2) for the creation of DCNN members in an ensemble, we propose the combined use of different types of face representations and bagging [16], which is quite useful in increasing the diversity of the DCNN ensemble. Extensive and comparative experiments on three wild FER datasets, namely FER2013, SFEW2.0, and RAF-DB, show that the proposed DCNN ensemble classifier achieves competitive FER performances when compared with other recently developed methods76.69%, 58.68%, and 87.13% of FER accuracy under the FER2013, SFEW2.0, and RAF-DB evaluation protocols, respectively.

For face-related applications like face recognition and face verification, face detection is vital. Law enforcement agencies extensively use face recognition applications for the detection of known offenders. Security cameras are increasingly deployed everywhere, and these cameras’ captured data can be used for tagging offenders. This is done by offering the face detection system with input images, which are then processed. Later on, attempts will be done to tag the security cameras’ captured data with this processed input data. Effective utilization of deep learning techniques in order to efficiently carry out this task. Computer vision’s various tasks have been taken over by deep Convolutional Neural Networks (CNNs). Of late, the key paradigms for face detection are the region-based CNN (R-CNN) detection models due to their quicker processing speed and progressively better performance. Nevertheless, the huge feature set causes these models’ to be inefficient for real-time face detection. A Grammatical Evolution (GE) Region-Based Fully Conventional Neural Network (R-FCN) for face detection has been proposed in this work in which the GE heuristics are utilized for tuning the hyper-parameters. The WIDER face dataset and Face Detection Dataset and Benchmark (FDDB) dataset were employed for conducting the assessments. In comparison to other advanced paradigms, the proposed technique’s superiority is evident from the experiments conducted.

Graph convolutional neural network (GCN) has effectively boosted the multi-label image recognition task by modeling correlation among labels. In previous methods, label correlation is computed based on statistical information through label diffusion, and therefore the same for all samples. This, however, makes graph inference on labels insufficient to handle huge variations among numerous image instances. In this paper, we propose an instance-aware graph convolutional neural network (IA_GCN) framework for the multi-label classification. As a whole, two fused branches of sub-networks are involved in the framework: a global branch modeling the whole image and a local branch exploring dependencies among regions of interests (ROIs). For both the branches, an image-dependent label correlation matrix (ID_LCM), fusing both the statistical LCM and an individual one of each image instance, is constructed to inject adaptive information of label-awareness into the learned features of the model through graph convolution. Specifically, the individual LCM of each image is obtained by mining the label dependencies based on the predicted label scores of those detected ROIs. In this process, considering the contribution differences of ROIs to multi-label classification, variational inference is introduced to learn adaptive scaling factors for those ROIs by considering their complex distribution. Finally, extensive experiments on MS-COCO and VOC datasets show that our proposed approach outperforms existing state-of-the-art methods.

The estimation of human pose has been an important field of study for the area of computer vision. Due to its significant applications in various important fields such as human computer interaction, action recognition, video surveillance, threat prediction, etc., it has been in the focus of researchers mainly. Dramatic effects of deep learning techniques have resulted in major steps and important developments in estimating human poses. We are proposing a fast and efficient deep learning approach to detecting 2D pose from a monocular image by multiple persons. The method uses a Vector Field (PVF) part affinity, which learns how well the limbs have been connected. The architectural design provides a global pixel-level interpretation of the number of people by enabling a bottom-up approach that maintains high accuracy in real time. The architecture has been designed to use two branches of a convolutional neural network (CNN) with sequential prediction to understand and reference part locations around each other. Successfully, with the help of the MS COCO human pose dataset, we have implemented our approach.

This article presents an intelligent fault diagnosis method for wind turbine (WT) gearbox by using wavelet packet decomposition (WPD) and deep learning. Specifically, the vibration signals from the gearbox are decomposed using WPD and the decomposed signal components are fed into a hierarchical convolutional neural network (CNN) to extract multiscale features adaptively and classify faults effectively. The presented method combines the multiscale characteristic of WPD with the strong classification capacity of CNNs, and it does not need complex manual feature extraction steps as usually adopted in existing results. The presented CNN with multiple characteristic scales based on WPD (WPD-MSCNN) has three advantages: 1) the added WPD layer can legitimately process the nonstationary vibration data to obtain components at multiple characteristic scales adaptively, it takes full advantage of WPD and, thus, enables the CNN to extract multiscale features; 2) the WPD layer directly sends multiscale components to the hierarchical CNN to extract rich fault information effectively, and it avoids the loss of useful information due to hand-crafted feature extraction; and 3) even if the scale changes, the lengths of components remain the same, which shows that the proposed method is robust to scale uncertainties in the vibration signals. Experiments with vibration data from a production wind farm provided by a company using condition monitoring system (CMS) show that the presented WPD-MSCNN method is superior to traditional CNN and multiscale CNN (MSCNN) for fault diagnosis.

Emotional expressivity has always been a simple job for people, but computer programming is much harder to accomplish. Image emotions may be recognised by recent developments in computer vision and machine learning. In this article, we present a new method to detect face emotion. Use neural networks convolutionary (FERC). The FERC is based on a CNN network of two parts: the first portion removed the backdrop of the image, the second part removed the face vector. The expressional vector (EV) is utilised in the FERC model to detect the fve different kinds of regular facial expressions. The double-level CNN is continuous and the weights and exponent values of the final perception layer vary with each iteration. In that it improves accuracy, FERC varies from widely utilised CNN single-level technology. Moreover, EV generation prevents the development of a number of issues before a new background removal process is used (for example distance from the camera).

Affect is often expressed via non-verbal body language such as actions/gestures, which are vital indicators for human behaviors. Recent studies on recognition of fine-grained actions/gestures in monocular images have mainly focused on modeling spatial configuration of body parts representing body pose, human-objects interactions and variations in local appearance. The results show that this is a brittle approach since it relies on accurate body parts/objects detection. In this work, we argue that there exist local discriminative semantic regions, whose "informativeness" can be evaluated by the attention mechanism for inferring fine-grained gestures/actions. To this end, we propose a novel end-to-end \textbf{Regional Attention Network (RAN)}, which is a fully Convolutional Neural Network (CNN) to combine multiple contextual regions through attention mechanism, focusing on parts of the images that are most relevant to a given task. Our regions consist of one or more consecutive cells and are adapted from the strategies used in computing HOG (Histogram of Oriented Gradient) descriptor. The model is extensively evaluated on ten datasets belonging to 3 different scenarios: 1) head pose recognition, 2) drivers state recognition, and 3) human action and facial expression recognition. The proposed approach outperforms the state-of-the-art by a considerable margin in different metrics., Comment: This manuscript is the accepted version of the published paper in IEEE Transaction on Affective Computing

In the last few years, Image Enhancement is a very necessary task in computer vision and to enhance the image. Super Resolution (SR) technique has been used to enhance the resolution of image. Super Resolution is a technique which is used to convert High Resolution (HR) from Low Resolution Image (LR). In this paper, to understand SR problems and to design an efficient and robust model, Convolution neural network (CNN) is used. This research work is focused on proposed a network design for deep convolution neural networks for application of super resolution techniques. To improve the complexity of existing techniques this work is intended towards network designs, different filter size and CNN architecture. The CNN model is most effective model for detection and segmentation in image. This model will improve the efficiency of image reconstruction from LR to HR. The proposed model showed its efficiency not only PET medical images but also on other datasets and acheived advance results as compared to existing works.

Face recognition is one of the useful tasks and can be used for many applications as security systems, it is necessary to find effective and low complexity facial classifier methods. In this paper, we proposed a new one-dimensional CNN deep convolutional neural network (1D-DCNN) classifier was combined with linear discriminative analysis (LDA) techniques to produce a new face recognition methodology. The contribution of this paper is generated of one-dimensional face feature set by LDA from original image database to training of 1D-DCNN classifier, that it contributed in the improvement of facial recognition performance. The model has been tested on MCUT dataset consisted of 3755 images for 276 classes. The results of the implementation of face recognition were accuracy of 100%, precision of 100%, recall of 100%, and F-measure of 100%.

Agricultural productivity plays a key role in determining the economy of a country. Detection of plant leaf disease is a vital task as it greatly affects the agricultural productivity. If proper detection is not done, it may lead to serious damage in the quality and quantity of the agricultural yield. In this research we propose a novel scheme for the detection of plant leaf diseases using deep convolutional neural networks (DCNN). In the proposed framework, initially, the plant leaf images are preprocessed using filtering and enhancement techniques. In our work, image filtering is done using 2D Adaptive Anisotropic Diffusion Filter (2D AADF) for noise removal. Using these de-noised images, enhancement is done using Adaptive Mean Adjustment (AMA) technique. This step helps to intensify the region of interest in the image. Using the enhanced image, segmentation is performed by means of clustering and thresholding. Clustering is done using the Improved Fast Fuzzy C Means Clustering (IFFCMC) Algorithm and image thresholding is performed using the Adaptive Otsu (AO) thresholding algorithm. From the segmented images, features are extracted using grey level co-occurrence matrix (GLCM). Dimensionality reduction of features is performed using principle component analysis (PCA). Finally, classification is done using a novel DCNN architecture. The proposed architecture has four convolutional layers, two fully connected layers and one SoftMax layer. Experimental results show that the proposed framework is effective and achieves best classification results other classifiers.

Healthcare agents, in particular in the oncology field, are currently collecting vast amounts of diverse patient data. In this context, some decision-support systems, mostly based on deep learning techniques, have already been approved for clinical purposes. Despite all the efforts in introducing artificial intelligence methods in the workflow of clinicians, its lack of interpretability - understand how the methods make decisions - still inhibits their dissemination in clinical practice. The aim of this article is to present an easy guide for oncologists explaining how these methods make decisions and illustrating the strategies to explain them. Theoretical concepts were illustrated based on oncological examples and a literature review of research works was performed from PubMed between January 2014 to September 2020, using deep learning techniques, interpretability and oncology as keywords. Overall, more than 60% are related to breast, skin or brain cancers and the majority focused on explaining the importance of tumor characteristics (e.g. dimension, shape) in the predictions. The most used computational methods are multilayer perceptrons and convolutional neural networks. Nevertheless, despite being successfully applied in different cancers scenarios, endowing deep learning techniques with interpretability, while maintaining their performance, continues to be one of the greatest challenges of artificial intelligence.

Agriculture plays a major role in developing countries like India, however the food security still remains a vital issue. Most of the crops get wasted due to lack of storage facility, transportation, and plant diseases. More than 15% of the crops get wasted in India due to diseases and hence it has become one of the major concern to be resolved. There is a need of automatic system that can identify these diseases and help farmers to take appropriate steps to get rid of crop loss. Farmers have followed the conventional method of identifying the plant disease with their naked eyes, and it not possible for all the farmers to identify these diseases the same way. With the advance in Artificial Intelligence, there is a need to incorporate the facilities of the computer vision in the field of agriculture. Deep Learning rich libraries and user as well as developer friendly environment to work with, all these qualities make Deep Learning as the favorable method to get started with this problem. In this paper we have used Deep Learning because of the advantages it offers to work with images especially in image classification to get improvised results. The methodology includes taking leaves of infected crops and label them as per the disease pattern. The images of infected leaves are processed pixel based operations are applied to improve the information from the image. As a next step feature extraction is done followed by image segmentation and at the last classification of crop diseases based on the patterns extracted from the diseased leaves. The CNN (Convolutional Neural Network) is used for the classification of diseases, for the demonstration purpose the public dataset is used consisting of around 87 K images (RGB type images) including healthy as well as diseased leaves.

The security of cyber-physical systems, such as vehicle platoons, is critical to ensuring their proper operation and acceptance to society. In platooning, vehicles follow one another according to an agreed-upon control law that determines vehicle separation. We show that a vehicle within a platoon and under the control of a malicious actor could cause collisions or decrease the efficiency of surrounding vehicles. This paper focuses on detecting, identifying, and mitigating so-called destabilizing attacks that could cause vehicle collisions. Our approach is decentralized and requires only local sensor information for each vehicle to identify the vehicle responsible for the attack and then deploy an appropriate mitigating controller that prevents collisions. We use a Deep Learning approach (Convolutional Neural Network) with various data preprocessing techniques to detect and identify the malicious vehicle. Results indicate that even with decent noise levels in range and relative speed data, we achieve accuracy up to 96.3\%. Also, once the adversarial vehicle is localized, we derive conditions for controller gains using the Routh Hurwitz criterion to mitigate the attack and ensure the stability of the platoon. Realistic simulator CARLA and MATLAB simulation results validate the effectiveness of our proposed approaches.

RGB-D salient object detection aims to detect visually distinctive objects or regions from a pair of the RGB image and the depth image. State-of-the-art RGB-D saliency detectors are mainly based on convolutional neural networks but almost suffer from an intrinsic limitation relying on the labeled data, thus degrading detection accuracy in complex cases. In this work, we present a self-supervised self-ensembling network (S $^3$ Net) for semi-supervised RGB-D salient object detection by leveraging the unlabeled data and exploring a self-supervised learning mechanism. To be specific, we first build a self-guided convolutional neural network (SG-CNN) as a baseline model by developing a series of three-layer cross-model feature fusion (TCF) modules to leverage complementary information among depth and RGB modalities and formulating an auxiliary task that predicts a self-supervised image rotation angle. After that, to further explore the knowledge from unlabeled data, we assign SG-CNN to a student network and a teacher network, and encourage the saliency predictions and self-supervised rotation predictions from these two networks to be consistent on the unlabeled data. Experimental results on seven widely-used benchmark datasets demonstrate that our network quantitatively and qualitatively outperforms the state-of-the-art methods.

Learning discriminative features is of vital importance for automatic Facial Expression Recognition (FER) in the wild. In this paper, we propose a novel Slide-Patch and Whole-Face Attention model with SE blocks (SPWFA-SE), which jointly perceives the discriminative locality characteristics and informative global features of the face for effective FER. Specifically, the well-designed slide patches are proposed to extract local features. Different from the existing methods, our slide patches not only can maintain the information at the edge area of patches, but also do not need to detect facial landmarks. Moreover, to make the model adaptively focus on the distinguishable regions, an attention module is proposed in the patch level to learn the weight of each patch. Furthermore, squeeze-and-excitation blocks are explored in the channel level to learn the weight of each channel. As such, the proposed multi-level feature extraction and attention mechanisms can enhance the representative ability of the learned features. Extensive experiments on five challenging datasets demonstrate that our method can achieve state-of-the-art performance. Cross database experiments on another three databases show the superior generalization performance of our model. Furthermore, complexity analysis results show that our model contains fewer parameters with fast training advantages than other competing models.

Change detection (CD), as one of the central problems in Earth observation, has attracted a lot of research interest over recent decades. Due to the rapid development of satellite sensors in recent years, we have witnessed an enrichment of the CD source data with the availability of very-high-resolution (VHR) multispectral imagery, which provides abundant change clues. However, precisely locating real changed areas still remains a challenge. In this article, we propose an end-to-end superpixel-enhanced CD network (ESCNet) for VHR images, which combines differentiable superpixel segmentation and a deep convolutional neural network (DCNN). Two weight-sharing superpixel sampling networks (SSNs) are tailored for the feature extraction and superpixel segmentation of bitemporal image pairs. A UNet-based Siamese neural network is then employed to mine the different information. The superpixels are then leveraged to reduce the latent noise in the pixel-level feature maps while preserving the edges, where a novel superpixelation module is used to serve this purpose. Furthermore, to compensate for the dependence on the number of superpixels, we propose an innovative adaptive superpixel merging (ASM) module, which has a concise form and is fully differentiable. A pixel-level refinement module making use of the multilevel decoded features is also appended to the end of the framework. Experiments on two public datasets confirmed the superiority of ESCNet compared to the traditional and state-of-the-art (SOTA) deep learning-based CD (DLCD) methods.

To improve the performance of text classification, we propose text augmentation based on attention score (TABAS). We recognized that a criterion for selecting a replacement word rather than a random selection was necessary. Therefore, TABAS utilizes attention scores for text modification, processing only words with the same entity and part-of-speech tags to consider informational aspects. To verify this approach, we used two benchmark tasks. As a result, TABAS can significantly improve performance, both recurrent and convolutional neural networks. Furthermore, we confirm that it provides a practical way to develop deep-learning models by saving costs on making additional datasets.

In the field of reversible data hiding (RDH), how to predict an image and embed a message into the image with smaller distortion are two important aspects. In this paper, we propose a novel and efficient RDH method by innovating an intelligent predictor and an adaptive embedding way. In the prediction stage, we first constructed a convolutional neural network (CNN) based predictor by reasonably dividing an image into four parts. In such a way, each part can be predicted by using the other three parts as the context for the improvement of the prediction performance. Compared with existing predictors, the proposed CNN predictor can use more neighboring pixels for the prediction by exploiting its multi-receptive fields and global optimization capacities. In the embedding stage, we also developed a prediction-error-ordering (PEO) based adaptive embedding strategy, which can better adapt image content and thus efficiently reduce the embedding distortion by elaborately and luminously applying background complexity to select and pair those smaller prediction errors for data hiding. With the proposed CNN prediction and embedding ways, the RDH method presented in this paper provides satisfactory results in improving the visual quality of data hidden images, e.g., the average PSNR value for the Kodak benchmark dataset can reach as high as 63.59 dB with an embedding capacity of 10,000 bits. Extensive experimental results have shown that the RDH method proposed in this paper is superior to those existing state-of-the-art works.

Producing high-resolution urban land cover maps is essential for decision-making and urban management. In this regard, Synthetic Aperture Radar (SAR), especially Polarimetric SAR (PolSAR), has served as a valuable data source to fulfill this task. Direct conversion of low-level features into high-level Land Cover (LC) concept may reduce the final classification accuracy. Therefore, mid-level representation models, such as Bag of Visual Words (BOVW), were employed to resolve the existing semantic gap challenge. In this paper, a Segment-based BOVW (Seg-BOVW) model was developed for urban land cover classification using PolSAR data. To this end, two PolSAR data over San Francisco Bay (SFB) and Flevoland (FL) acquired by RADARSAT-2 were employed to comprehensively evaluate the Seg-BOVW model's performance. First, to exploit the full potential of PolSAR data, 169 low-level features in four categories: (1) original, (2) polarimetric, (3) texture, and (4) decomposition features were extracted. Afterward, a Multi-Objective Genetic Algorithm (MOGA) was implemented to investigate the importance of low-level features for urban land cover mapping. This step resulted in selecting 14 and 6 low-level features, as the high contributing features, for SFB and FL datasets, respectively. The Seg-BOVW model achieved significant overall accuracies of 96.02% and 98.82% for SFB and FL, respectively, indicating the high potential of the proposed method for urban land cover classification. Furthermore, a comparison with other well-known algorithms of Support Vector Machine (SVM), Artificial Neural Networks (ANN), and Convolutional Neural Networks (CNN) was made, suggesting the capability of the Seg-BOVW model to improve the urban land cover classification results. Finally, the Seg-BOVW model was tested with two other PolSAR datasets acquired with different sensors over SFB to examine its applicability with different datasets.

The analysis of large-scale multimodal data has become very popular recently. Image captioning, whose goal is to describe the content of image with natural language automatically, is an essential and challenging task in artificial intelligence. Commonly, most existing image caption methods utilize the mixture of Convolutional Neural Network and Recurrent Neural Network framework. These methods either pay attention to global representation at the image level or only focus on the specific concepts, such as regions and objects. To make the most of characteristics about a given image, in this study, we present a novel model named Multilevel Attention Networks and Policy Reinforcement Learning for image caption generation. Specifically, our model is composed of a multilevel attention network module and a policy reinforcement learning module. In the multilevel attention network, the object-attention network aims to capture global and local details about objects, whereas the region-attention network obtains global and local features about regions. After that, a policy reinforcement learning algorithm is adopted to overcome the exposure bias problem in the training phase and solve the loss-evaluation mismatching problem at the caption generation stage. With the attention network and policy algorithm, our model can automatically generate accurate and natural sentences for any particular image. We carry out extensive experiments on the MSCOCO and Flickr30k data sets, demonstrating that our model is superior to other competitive methods.

Compared with traditional convolutions, grouped convolutional neural networks are promising for both model performance and network parameters. However, existing models with the grouped convolution still have parameter redundancy. In this article, concerning the grouped convolution, we propose a sharing grouped convolution structure to reduce parameters. To efficiently eliminate parameter redundancy and improve model performance, we propose a Bayesian sharing framework to transfer the vanilla grouped convolution to be the sharing structure. Intragroup correlation and intergroup importance are introduced into the prior of the parameters. We handle the Maximum Type II likelihood estimation problem of the intragroup correlation and intergroup importance by a group LASSO-type algorithm. The prior mean of the sharing kernels is iteratively updated. Extensive experiments are conducted to demonstrate that on different grouped convolutional neural networks, the proposed sharing grouped convolution structure with the Bayesian sharing framework can reduce parameters and improve prediction accuracy. The proposed sharing framework can reduce parameters up to 64.17%. For ResNeXt-50 with the sharing grouped convolution on ImageNet dataset, network parameters can be reduced by 96.875% in all grouped convolutional layers, and accuracies are improved to 78.86% and 94.54% for top-1 and top-5, respectively.

Objective Breast cancer and breast tumors have been considered to be the most pervasive form of cancer in medical practice. Breast tumors are life-threatening to women, and their early detection could save lives with the proper treatment. Physical methods for detection of Breast Cancer are time-consuming and often prone to a misdiagnosis at classifying tumors. Recent trends in radiological imaging have significantly improved the efficiency and veracity of breast tumor classification. Artificial intelligence techniques could be used as an automated detection and classification system. Materials and methods In this research, we propose a novel configuration of a Stacking Ensemble with custom Convolutional Neural Network architectures to classify breast tumors from ultrasound images into ‘Normal’, ‘Benign’, and ‘Malignant’ categories. Results After thorough experimentation, our ensemble has performed with an accuracy, f1-score, precision, and recall of 92.15%, 92.21%, 92.26%, 92.17% respectively. Conclusion The presented ensemble leverages three Stacked Feature Extractors coupled with a characteristic meta-learner to provide an overall balanced classification performance, with better accuracy and lower false positives. The architecture works in association with gaussian dropout layers to improve the computation and an alternative pooling scheme to retain essential features.

International audience; Hyperspectral images (HSIs) are of crucial importance in order to better understand features from a large number of spectral channels. Restricted by its inner imaging mechanism, the spatial resolution is often limited for HSIs. To alleviate this issue, in this work, we propose a simple and efficient architecture of deep convolutional neural networks to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). The network is designed to preserve both spatial and spectral information thanks to a new architecture based on: 1) the use of the LR-HSI at the HR-MSI’s scale to get an output with satisfied spectral preservation and 2) the application of the attention and pixelShuffle modules to extract information, aiming to output high-quality spatial details. Finally, a plain mean squared error loss function is used to measure the performance during the training. Extensive experiments demonstrate that the proposed network architecture achieves the best performance (both qualitatively and quantitatively) compared with recent state-of-the-art HSI super-resolution approaches. Moreover, other significant advantages can be pointed out by the use of the proposed approach, such as a better network generalization ability, a limited computational burden, and the robustness with respect to the number of training samples. Please find the source code and pretrained models from https://liangjiandeng.github.io/Projects_Res/HSRnet_2021tnnls.html .

Understanding and predicting types of bugs are of practical importance for developers to improve the testing efficiency and take appropriate steps to address bugs in software releases. However, due to the complex conditions under which faults manifest and the complexity of the classification rules, the automatic classification of Mandelbugs is a difficult task. In this article, we present a deep semantic information-based Mandelbug classification method that combines a semantic model with a deep learning classifier and makes use of both labeled and unlabeled bug reports. By training the bug report semantic model on millions of bug reports, each word in the text of a bug report is represented as a word embedding that preserves the semantic relationship among the words. Then, a convolutional neural network model is designed to capture the high-level features of bug reports to obtain a more accurate classification. Moreover, the effects of the semantic model size and domain on the classification results are investigated, and the quality of word embeddings is evaluated by analyzing several important parameters.

Augmented reality applications use object tracking to estimate the pose of a camera and to superimpose virtual content onto the observed object. Today, a number of tracking systems are available, ready to be used in industrial applications. However, such systems are hard to handle for a service maintenance engineer, due to obscure configuration procedures. In this article, we investigate options towards replacing the manual configuration process with a machine learning approach based on automatically synthesized data. We present an automated process of creating object tracker facilities exclusively from synthetic data. The data is highly enhanced to train a convolutional neural network, while still being able to receive reliable and robust results during real world applications only from simple RGB cameras. Comparison against related work using the LINEMOD dataset showed that we are able to outperform similar approaches. For our intended industrial applications with high accuracy demands, its performance is still lower than common object tracking methods with manual configuration. Yet, it can greatly support those as an add-on during initialization, due to its higher reliability.

Modern convolutional neural network (CNN)-based object detectors focus on feature configuration during training but often ignore feature optimization during inference. In this article, we propose a new feature optimization approach to enhance features and suppress background noise in both the training and inference stages. We introduce a generic inference-aware feature filtering (IFF) module that can be easily combined with existing detectors, resulting in our iffDetector. Unlike conventional open-loop feature calculation approaches without feedback, the proposed IFF module performs the closed-loop feature optimization by leveraging high-level semantics to enhance the convolutional features. By applying the Fourier transform to analyze our detector, we prove that the IFF module acts as a negative feedback that can theoretically guarantee the stability of the feature learning. IFF can be fused with CNN-based object detectors in a plug-and-play manner with little computational cost overhead. Experiments on the PASCAL VOC and MS COCO datasets demonstrate that our iffDetector consistently outperforms state-of-the-art methods with significant margins.

We present a learning-based approach for removing unwanted obstructions, such as window reflections, fence occlusions, or adherent raindrops, from a short sequence of images captured by a moving camera. Our method leverages motion differences between the background and obstructing elements to recover both layers. Specifically, we alternate between estimating dense optical flow fields of the two layers and reconstructing each layer from the flow-warped images via a deep convolutional neural network. This learning-based layer reconstruction module facilitates accommodating potential errors in the flow estimation and brittle assumptions, such as brightness consistency. We show that the proposed approach learned from synthetically generated data performs well to real images. Experimental results on numerous challenging scenarios of reflection and fence removal demonstrate the effectiveness of the proposed method., Project page: https://alex04072000.github.io/SOLD/ Code: https://github.com/alex04072000/SOLD Extension of the CVPR 2020 paper: arXiv:2004.01180