Search

In complex industrial processes, the technique of fault diagnosis has been playing an increasingly considerable role in ensuring the safety of life and property. Unfortunately, the process data of complex industrial processes have the features of high dimension. Feature extraction from high-dimensional data is promising to coping with the fault data with high dimension. Recently, one of manifold learning methods named discriminant locality preserving projection achieves excellent performance in feature extraction. However, the performance of discriminant locality preserving projection (DLPP) is subject to the problem of matrix decomposition in the denominator of the objection function caused by the small sample size (SSS) issue. To overcome this limitation, novel DLPP integrated with Monte Carlo sampling is proposed to enhance the performance of feature extraction through dimensionality reduction. In the proposed MC-DLPP, Monte Carlo sampling is first utilized to generate fault samples for each fault type. With the aid of the virtually generated fault samples, the rank of the matrix in the denominator of the objection function of DLPP increases, thus well addressing the SSS problem. The Softmax classifier is used for fault diagnosis. To test the performance of the improved DLPP-based fault diagnosis, case studies using the Tennessee Eastman process are carried out. Simulation results confirm the presented MC-DLPP achieves superior accuracy in fault diagnosis.

Multi-view classification with limited sample size and data augmentation is a very common machine learning (ML) problem in medicine. With limited data, a triplet network approach for two-stage representation learning has been proposed. However, effective training and verifying the features from the representation network for their suitability in subsequent classifiers are still unsolved problems. Although typical distance-based metrics for the training capture the overall class separability of the features, the performance according to these metrics does not always lead to an optimal classification. Consequently, an exhaustive tuning with all feature-classifier combinations is required to search for the best end result. To overcome this challenge, we developed a novel nearest-neighbor (NN) validation strategy based on the triplet metric. This strategy is supported by a theoretical foundation to provide the best selection of the features with a lower bound of the highest end performance. The proposed strategy is a transparent approach to identify whether to improve the features or the classifier. This avoids the need for repeated tuning. Our evaluations on real-world medical imaging tasks (i.e., radiation therapy delivery error prediction and sarcoma survival prediction) show that our strategy is superior to other common deep representation learning baselines [i.e., autoencoder (AE) and softmax]. The strategy addresses the issue of feature's interpretability which enables more holistic feature creation such that the medical experts can focus on specifying relevant data as opposed to tedious feature engineering.

We propose a simple and effective method to inject word-level information into character-aware neural language models. Unlike previous approaches which usually inject word-level information at the input of a long short-term memory (LSTM) network, we inject it into the softmax function. The resultant model can be seen as a combination of character-aware language model and simple word-level language model. Our injection method can also be used together with previous methods. Through the experiments on 14 typologically diverse languages, we empirically show that our injection method, when used together with the previous methods, works better than the previous methods, including a gating mechanism, averaging, and concatenation of word vectors. We also provide a comprehensive comparison of these injection methods.

We consider a hierarchical structure of a permissioned blockchain with three types of participant: providers, collectors, and governors. Providers forward transactions to collectors; collectors upload received transactions to governors after verifying and labeling them; and governors validate a portion of the labeled transactions they receive, pack valid transactions into a block, and append the block to the ledger. This model has various fields of application including data collection from the Internet-of-Things and second-hand markets. Our main contribution is to propose a reputation-based protocol to help governors evaluate the reliability of collectors. Specifically, given a transaction, each governor runs a softmax-based function to calculate a probability for each collector that sent and labeled this transaction. The probabilities, calculated using collectors' reputations as inputs, represent the likelihood of the lead governor selecting the labeled transaction from collectors to consider for further validation. After the lead governor verifies a transaction, all collectors' reputations are updated in line with the agreement of their labeling and the validity of the transaction as found by the lead governor. We show, both theoretically and empirically, that our protocol can significantly reduce governors' verification workloads while maintaining firm liveness and high incentives.

Agricultural productivity plays a key role in determining the economy of a country. Detection of plant leaf disease is a vital task as it greatly affects the agricultural productivity. If proper detection is not done, it may lead to serious damage in the quality and quantity of the agricultural yield. In this research we propose a novel scheme for the detection of plant leaf diseases using deep convolutional neural networks (DCNN). In the proposed framework, initially, the plant leaf images are preprocessed using filtering and enhancement techniques. In our work, image filtering is done using 2D Adaptive Anisotropic Diffusion Filter (2D AADF) for noise removal. Using these de-noised images, enhancement is done using Adaptive Mean Adjustment (AMA) technique. This step helps to intensify the region of interest in the image. Using the enhanced image, segmentation is performed by means of clustering and thresholding. Clustering is done using the Improved Fast Fuzzy C Means Clustering (IFFCMC) Algorithm and image thresholding is performed using the Adaptive Otsu (AO) thresholding algorithm. From the segmented images, features are extracted using grey level co-occurrence matrix (GLCM). Dimensionality reduction of features is performed using principle component analysis (PCA). Finally, classification is done using a novel DCNN architecture. The proposed architecture has four convolutional layers, two fully connected layers and one SoftMax layer. Experimental results show that the proposed framework is effective and achieves best classification results other classifiers.

This article proposes a robust end-to-end deep learning-induced fault recognition scheme by stacking multiple sparse-denoising autoencoders with a Softmax classifier, called stacked spare-denoising autoencoder (SSDAE)-Softmax, for the fault identification of complex industrial processes (CIPs). Specifically, sparse denoising autoencoder (SDAE) is established by integrating a sparse AE (SAE) with a denoising AE (DAE) for the low-dimensional but intrinsic feature representation of the CIP monitoring data (CIPMD) with possible noise contamination. SSDAE-Softmax is established by stacking multiple SDAEs with a layerwise pretraining procedure, and a Softmax classifier with a global fine-tuning strategy. Furthermore, SSDAE-Softmax hyperparameters are optimized by a relatively new global optimization algorithm, referred to as the state transition algorithm (STA). Benefiting from the deep learning-based feature representation scheme with the STA-based hyperparameter optimization, the underlying intrinsic characteristics of CIPMD can be learned automatically and adaptively for accurate fault identification. A numeric simulation system, the benchmark Tennessee Eastman process (TEP), and a real industrial process, that is, the continuous casting process (CCP) from a top steel plant of China, are used to validate the performance of the proposed method. Experimental results show that the proposed SSDAE-Softmax model can effectively identify various process faults, and has stronger robustness and adaptability against the noise interference in CIPMD for the process monitoring of CIPs.

In this article, we propose a structure-aligned generative adversarial network framework to improve zero-shot learning (ZSL) by mitigating the semantic gap, domain shift, and hubness problem. The proposed framework contains two parts, i.e., a generative adversarial network with a softmax classifier part, and a structure-aligned part. In the first part, the generative adversarial network aims at generating pseudovisual features through the guiding generator and discriminator play the minimax two-player game together. At the same time, the softmax classifier is committed to increasing the interclass distance and reducing intraclass distance. Then, the harmful effect of domain shift and hubness problems can be mitigated. In another part, we introduce a structure-aligned module where the structural consistency between visual space and semantic space is learned. By aligning the structure between visual space and semantic space, the semantic gap between them can be bridged. The performance of classification is improved when the structure-aligned visual-semantic embedding space is transferred to the unseen classes. Our framework reformulates the ZSL as a standard fully supervised classification task using the pseudovisual features of unseen classes. Extensive experiments conducted on five benchmark data sets demonstrate that the proposed framework significantly outperforms state-of-the-art methods in both conventional and generalized settings.

Since electroencephalogram (EEG) signals can truly reflect human emotional state, emotion recognition based on EEG has turned into a critical branch in the field of artificial intelligence. Aiming at the disparity of EEG signals in various emotional states, we propose a new deep learning model named three-dimension convolution attention neural network (3DCANN) for EEG emotion recognition in this paper. The 3DCANN model is composed of spatio-temporal feature extraction module and EEG channel attention weight learning module, which can extract the dynamic relation well among multi-channel EEG signals and the internal spatial relation of multi-channel EEG signals during continuous period time. In this model, the spatio-temporal features are fused with the weights of dual attention learning, and the fused features are input into the softmax classifier for emotion classification. In addition, we utilize SJTU Emotion EEG Dataset (SEED) to appraise the feasibility and effectiveness of the proposed algorithm. Finally, experimental results display that the 3DCANN method has superior performance over the state-of-the-art models in EEG emotion recognition.

With the increasing accumulation of spatial-temporal trajectory data, location-based data mining has recently been extensively studied. A fundamental research topic in this field is learning the embedding vectors of locations through self-supervised pre-training. Pre-trained embedding vectors can utilize the highly available unlabeled trajectory data, and benefit downstream tasks in multiple aspects. However, most existing methods ignore the temporal information hidden in the visited time of locations in trajectories. Considering that human activities are highly regulated by specific periods of a day, temporal information can reflect some intrinsic characteristics of locations, so it is necessary to fuse them into location embedding vectors. In this paper, we propose a Time-Aware Location Embedding (TALE) pre-training method based on the CBOW framework, which is able to incorporate temporal information into the learned embedding vectors of locations. A novel temporal tree structure is designed to extract temporal information during the calculation of Hierarchical Softmax. In order to verify the effectiveness of TALE, we apply the learned embedding vectors into three downstream location-based prediction tasks, i.e., location classification, location visitor flow prediction and user next location prediction. Experiments are conducted on four real-world user trajectory datasets, and the experimental results demonstrate that our TALE model can obviously help downstream tasks gain better performance.

Center loss is widely used as a supervision tool in deep learning method. However, the center loss also has some shortcomings, the most important of which is that it must be combined with softmax loss to run well. In this article, we sum up five shortcomings of center loss and solve all of them by proposing a dual distance center loss (DDCL). Compared with center loss, DDCL can run without the combination of softmax to supervise training the model. In addition, we verify the inconsistency between the proposed DDCL and softmax loss in the feature space. To be specifically, we add the Pearson distance on the basis of the Euclidean distance to the same center, which makes all features of the same class be confined to the intersection of a hypersphere and a hypercone in the feature space, strengthens the intraclass compactness of the center loss, and enhances the generalization ability of center loss. Moreover, by designing a Euclidean distance threshold between all center pairs, we not only strengthen the interclass separability of center loss, but also make the center loss (or DDCL) works well without the combination of softmax loss. We verify the effectiveness of DDCL in four datasets, two of which are widely used in the field of vehicle re-identification named VeRi-776 dataset and VehicleID dataset, and two other datasets are widely used in the field of person re-identification named Market1501 dataset and MSMT17 dataset. The experimental results of the proposed DDCL exceed that of the softmax loss in all the four datasets, indicating that our proposed method not only can run without the combination of softmax, but also has a high accuracy.

Despite the success of stochastic variance-reduced gradient (SVRG) algorithms in solving large-scale problems, their stochastic gradient complexity often scales linearly with data size and is expensive for huge data. Accordingly, we propose a hybrid stochastic-deterministic minibatch proximal gradient~(HSDMPG) algorithm for strongly convex problems with linear prediction structure, e.g.~least squares and logistic/softmax regression. HSDMPG~enjoys improved computational complexity that is data-size-independent for large-scale problems. It iteratively samples an evolving~minibatch of individual losses to estimate the original problem, and efficiently minimizes the sampled smaller-sized subproblems. For strongly convex loss of n components, HSDMPG~attains an $\epsilon$ -optimization-error within $\mathcal{O} \left(\kappa\log^{\zeta+1}\left(\frac{1}{\epsilon}\right)\frac{1}{\epsilon}\bigwedge n\log^{\zeta}\left(\frac{1}{\epsilon}\right)\right)$ stochastic gradient evaluations, where $\kappa$ is condition number, $\zeta=1$ for quadratic loss and $\zeta=2$ for generic loss. For large-scale problems, our complexity outperforms those of SVRG-type algorithms with/without dependence on data size. Particularly, when $\epsilon=\mathcal{O}(1/\sqrt{n})$ which matches the intrinsic excess error of a learning model and is sufficient for generalization, our complexity for quadratic and generic losses is respectively $\mathcal{O} (n^{0.5}\log^{2}(n))$ and $\mathcal{O} (n^{0.5}\log^{3}(n))$ , which for the first time achieves optimal generalization in less than a single pass over data. Besides, we extend HSDMPG~to online strongly convex problems and prove its higher efficiency over the prior algorithms. Numerical results demonstrate the computational advantages of~HSDMPG.

Recently, a popular line of research in face recognition is adopting margins in the well-established softmax loss function to maximize class separability. In this paper, we first introduce an Additive Angular Margin Loss (ArcFace), which not only has a clear geometric interpretation but also significantly enhances the discriminative power. Since ArcFace is susceptible to the massive label noise, we further propose sub-center ArcFace, in which each class contains $K$ sub-centers and training samples only need to be close to any of the $K$ positive sub-centers. Sub-center ArcFace encourages one dominant sub-class that contains the majority of clean faces and non-dominant sub-classes that include hard or noisy faces. Based on this self-propelled isolation, we boost the performance through automatically purifying raw web faces under massive real-world noise. Besides discriminative feature embedding, we also explore the inverse problem, mapping feature vectors to face images. Without training any additional generator or discriminator, the pre-trained ArcFace model can generate identity-preserved face images for both subjects inside and outside the training data only by using the network gradient and Batch Normalization (BN) priors. Extensive experiments demonstrate that ArcFace can enhance the discriminative feature embedding as well as strengthen the generative face synthesis., Comment: ArcFace TPAMI version

The process operating performance assessment is critical for the smelting process of electro-fused magnesium furnaces to improve quality of the magnesia product and pursue optimal comprehensive economic benefit. This paper proposes a new method of multi-source heterogeneous information deep feature fusion (MSHIDFF) to achieve higher accuracy operating performance assessment in the electro-fused magnesium smelting process. Firstly, we utilize convolutional neural network, bidirectional long short-term memory network and stacked auto-encoder to extract deep features from raw image, sound and current of different performance grades. Furthermore, those multi-source deep features are fused and the softmax regression with attention mechanism is employed to train a neural network classifier for the fused deep features of different performance grades. The simulation results show that the proposed MSHIDFF method obtains the superior assessment accuracy.

Nowadays, automatic deception detection has received considerable attention in the machine learning community owing to this research interest to its vast applications in the fields of social media, interviews, law enforcement, and the military. In this study, a novel deep convolution neural network (DCNN) named LieNet is proposed to precisely detect the multi-scale variations of deception automatically. Our approach is a combination of contact and non-contact-based approaches. First, 20 frames from each video are fetched and concatenated to form a single image. Moreover, an audio signal is extracted from video and treated as image input by plotting the signal into two dimensional (2D) plane. Furthermore, 13 channels of EEG signals are plotted into 2D plane and concatenated to generate an image. Second, the LieNet model extracts features from each modality separately. Third, scores are estimated using a softmax classifier for all the modalities. Finally, three scores are combined using score level fusion to obtain a score, which gives support in favor of either deception or truth. The LieNet is validated on the “Bag-of-Lies (BoL)”“, Real-life (RL) trail”, and “Miami University Deception Detection (MU3D)" databases by considering four evaluation indexes viz., accuracy, precision, recall, and F1-score. Experimental outcomes depict that the LieNet defeats an initial work on Set-A and Set-B of the BoL database with average accuracies of 95.91% and 96.04% respectively. The accuracies obtained by the LieNet are 97% and 98% on RL trail and MU3D databases respectively.

Spoofing attacks on the fingerprint scanners become major and serious concern with the growing development and use of biometric technologies. Level 1 and Level 2 features; which are said to be unique and most commonly used features in fingerprint verification systems, are easily get spoofed. Moreover, single feature based specifically designed spoof detection methods are not performed well on different fingerprint scanners and spoofing materials. This paper proposed the fusion of pores perspiration and texture features in static software based approach to identify live and fake fingerprints. The pores perspiration activity is quantified by computing the ridge signal energy and gray level distributions around the detected pores. These pore characteristics are statically determined instead of dynamic measurement. Autoencoder neural network is used to reduce the high dimensional feature vector and learn its low dimensional hidden representation unsupervisedly. The binary classification in two classes: live and spoof is performed by the supervisedly trained softmax classifier. The performance of the classifier is evaluated in terms of Average Classification Error (ACE) and misclassification rates: FerrLive and FerrFake. The experimental results carried out on LivDet 2013 and LivDet 2015 databases show the improvement of the classifier performance in comparison to the state-of-the-art methods.

This paper attempts at improving the accuracy of Human Action Recognition (HAR) by fusion of depth and inertial sensor data. Firstly, we transform the depth data into Sequential Front view Images(SFI) and fine-tune the pre-trained AlexNet on these images. Then, inertial data is converted into Signal Images (SI) and another convolutional neural network (CNN) is trained on these images. Finally, learned features are extracted from both CNN, fused together to make a shared feature layer, and these features are fed to the classifier. We experiment with two classifiers, namely Support Vector Machines (SVM) and softmax classifier and compare their performances. The recognition accuracies of each modality, depth data alone and sensor data alone are also calculated and compared with fusion based accuracies to highlight the fact that fusion of modalities yields better results than individual modalities. Experimental results on UTD-MHAD and Kinect 2D datasets show that proposed method achieves state of the art results when compared to other recently proposed visual-inertial action recognition methods.

Capsule Networks (CapsNets) were proposed to mitigate the shortcomings of Convolutional Neural Networks (CNNs) such as invariance. Even though they have achieved equivariance, they fail to perform on the recognition of complex images and images with varied backgrounds such as CIFAR-10. Real-life images such as those found in plant disease datasets (aside from being complex with varied backgrounds) pose additional challenges such as class imbalance and the availability of a smaller number of annotated datasets. The original CapsNet uses CNNs as feature extractors, SoftMax for normalization, and dynamic routing (DR) to enable active capsules to make predictions resulting in the activation of higher-level capsules. However, CNNs do not serve as superior texture extractors and SoftMax restrains capsules from forming optimal coupling during routing. In this paper, we propose the use of an efficient texture descriptor (Local Binary Pattern -LBP), sigmoid function, and k-means routing respectively in place of CNN, SoftMax, and dynamic routing. We evaluate our model on six publicly available datasets; MNIST, fashion-MNIST, CIFAR-10, tomato, maize, and citrus datasets. Experimental results show that the proposed model generates fewer parameters and performs comparably well with the state-of-the-art multi-lane capsule networks on complex images.

Epilepsy ictal detection based on scalp electroencephalograms (EEGs) has been comprehensively studied in the past decades. But few attentions have been paid to the preictal classification. In this paper, a comprehensive study on epileptic state classification based on deep transfer learning (TL) is presented. The main contributions include: 1) the subband mean amplitude spectrum (MAS) map that characterizes the typical rhythms of brain activities is extracted for EEG representation, 2) 5 representative deep neural networks (DNNs) pre-trained on ImageNet are applied for EEG feature TL, 3) a 7-layer hierarchical neural network (HNN) that consists of 3 fully-connected (Fc) and 3 dropout layers followed by a softmax layer is developed to perform the epileptic state probability learning and classification. Experiments on the benchmark CHB-MIT and iNeuro EEG databases that contain several different types of seizures show that the proposed algorithm achieves the highest overall accuracies of 96.97% and 87.87% on the 5-state epileptic classification, respectively, that outperforms many existing state-of-the-art methods presented in the paper.

In this article, a deep probability model, called the discriminative mixture variational autoencoder (DMVAE), is developed for the feature extraction in semisupervised learning. The DMVAE consists of three parts: 1) the encoding; 2) decoding; and 3) classification modules. In the encoding module, the encoder projects the observation to the latent space, and then the latent representation is fed to the decoding part, which depicts the generative process from the hidden variable to data. In particular, the decoding module in our DMVAE partitions the observed dataset into some clusters via multiple decoders whose number is automatically determined via the Dirichlet process (DP) and learns a probability distribution for each cluster. Compared to the standard variational autoencoder (VAE) describing all data with a single probability function, the DMVAE has the capacity to give a more accurate description for observations, thus improving the characterization ability of the extracted features, especially for the data with complex distribution. Moreover, to obtain a discriminative latent space, the class labels of labeled data are introduced to restrict the feature learning via a softmax classifier, with which the minimum entropy of the predicted labels for the features from unlabeled data can also be guaranteed. Finally, the joint optimization of the marginal likelihood, label, and entropy constraints makes the DMVAE have higher classification confidence for unlabeled data while accurately classifying the labeled data, ultimately leading to better performance. Experiments on several benchmark datasets and the measured radar echo dataset show the advantages of our DMVAE-based semisupervised classification over other related methods.

The requirement of fast fault isolation poses a great challenge to the safe operation of multi-terminal direct current (MTDC) systems. In order to make a better tradeoff between the speed and reliability of the protection scheme, it is imperative to mine more valuable information from fault transient signals. This paper puts forward a data-driven fault detection and classification framework for MTDC systems. Highly comparative time-series analysis (HCTSA) is first adopted to extract extensive features with clear physical interpretations from the original DC fault current waveform, and the most useful features to fault identification are then selected utilizing the greedy forward search. Based on these reduced features, a simple but powerful softmax regression classifier (SRC) is further proposed to calculate the probability of each fault category, so that fault identification can be fast achieved with minor computational burden. Numerical simulations carried out in PSCAD/EMTDC have demonstrated the accuracy of proposed approach under various fault conditions and noise corruptions. In addition, comprehensive comparison studies with the conventional derivative-based method and some typical data-driven methods have been conducted, and it is well-verified that via rationally selecting and fully integrating multi-dimensional features, the proposed approach possesses higher fault classification accuracy than existing methods.

Vibration signals always contain noise and irregularities which makes spectrum analysis difficult to extract high-level features. Recently, graph theory has been applied to spectrum analysis to improve the performance of feature extraction. By converting the raw data into graphs, hidden structural and topological information can be obtained. In this paper, a spatial-temporal graph-based feature extraction, called SuperGraph, for rotating machinery fault diagnosis is proposed. Specifically, graph theory-based spectrum analysis is used to construct the spatial-temporal graph. Then, the Laplacian matrix-based feature vector is extracted from the constructed spatial-temporal graph. By this means, the spatial-temporal graph is converted into the one-dimensional vector for further constructing SuperGraph, where each node of the SuperGraph represents a spatial-temporal graph and the SuperGraph is composed of many local graphs. In the local graph, only the same type of nodes are connected to form a fully connected graph. Thus, the task of graph classification can be transformed into classifying the nodes in the SuperGraph. After graph convolutional network is established for learning and obtaining deep features, the label of nodes is identified from a softmax model. Experiments are conducted on two benchmarking datasets and a practical experimental platform to verify effectiveness of the proposed fau

Most of the deep quantization methods adopt unsupervised approaches, and the quantization process usually occurs in the Euclidean space on top of the deep feature and its approximate value. When this approach is applied to the retrieval tasks, since the internal product space of the retrieval process is different from the Euclidean space of quantization, minimizing the quantization error (QE) does not necessarily lead to a good performance on the maximum inner product search (MIPS). To solve these problems, we treat Softmax classification as vector quantization (VQ) with angular decision boundaries and propose angular deep supervised VQ (ADSVQ) for image retrieval. Our approach can simultaneously learn the discriminative feature representation and the updatable codebook, both lying on a hypersphere. To reduce the QE between centroids and deep features, two regularization terms are proposed as supervision signals to encourage the intra-class compactness and inter-class balance, respectively. ADSVQ explicitly reformulates the asymmetric distance computation in MIPS to transform the image retrieval process into a two-stage classification process. Moreover, we discuss the extension of multiple-label cases from the perspective of quantization with binary classification. Extensive experiments demonstrate that the proposed ADSVQ has excellent performance on four well-known image data sets when compared with the state-of-the-art hashing methods.

Deep learning is a special branch of machine learning using a collage of algorithms to model high-level data motifs. Deep learning resembles the biological communications of systems of brain neurons in the central nervous system (CNS), where synthetic graphs represent the CNS network as nodes/states and connections/edges between them. For instance, in a simple synthetic network consisting of a pair of connected nodes, an output sent by one node is received by the other as an input signal. When more nodes are present in the network, they may be arranged in multiple levels (like a multiscale object) where the ith layer output serves as the input of the next (i + 1)st layer. The signal is manipulated at each layer, sent as a layer output downstream, interpreted as an input to the next, (i + 1)st layer, and so forth. Deep learning relies on multipler layers of nodes and many edges linking the nodes forming input/output (I/O) layered grids representing a multiscale processing network. At each layer, linear and non-linear transformations are converting inputs into outputs.

Chemical-induced disease (CID) relation extraction from biomedical articles plays an important role in disease treatment and drug development. Existing methods are insufficient for capturing complete document level semantic information due to ignoring semantic information of entities in different sentences. In this work, we proposed an effective document-level relation extraction model to automatically extract intra-/inter-sentential CID relations from articles. Firstly, our model employed BERT to generate contextual semantic representations of the title, abstract and shortest dependency paths (SDPs). Secondly, to enhance the semantic representation of the whole document, cross attention with self-attention (named cross2self-attention) between abstract, title and SDPs was proposed to learn the mutual semantic information. Thirdly, to distinguish the importance of the target entity in different sentences, the Gaussian probability distribution was utilized to compute the weights of the co-occurrence sentence and its adjacent entity sentences. More complete semantic information of the target entity is collected from all entities occurring in the document via our presented document-level R-BERT (DocR-BERT). Finally, the related representations were concatenated and fed into the softmax function to extract CIDs. We evaluated the model on the CDR corpus provided by BioCreative V. The proposed model without external resources is superior in performance as compared with other state-of-the-art models (our model achieves 53.5%, 70%, and 63.7% of the F1-score on inter-/intra-sentential and overall CDR dataset). The experimental results indicate that cross2self-attention, the Gaussian probability distribution and DocR-BERT can effectively improve the CID extraction performance. Furthermore, the mutual semantic information learned by the cross self-attention from abstract towards title can significantly influence the extraction performance of document-level biomedical relation extraction tasks.