Search

Condition-based maintenance (CBM) policy can avoid premature or late maintenance and reduce system failures and maintenance costs. Most existing CBM studies cannot solve the dimensional disaster problem in multi-component complex systems. Only some studies consider the constraint of maintenance resources when searching for the optimal maintenance policy, which is hard to apply to practical maintenance. This paper studies the joint optimization of the CBM policy and spare components inventory for the multi-component system in large state and action spaces. We use Markov Decision Process to model it and propose an improved deep reinforcement learning algorithm based on the stochastic policy and actor-critic framework. In this algorithm, factorization decomposes the system action into the linear combination of each component's action. The experimental results show that the algorithm proposed in this paper has better time performance and lower system cost compared with other benchmark algorithms. The training time of the former is only 28.5% and 9.12% of that of PPO and DQN algorithms, and the corresponding system cost is decreased by 17.39% and 27.95%, respectively. At the same time, our algorithm has good scalability and is suitable for solving Markov decision-making problems in large-scale state and action space. • Considering minor and major repair, we model the joint optimization of CBM and spare ordering for large multi-component systems based on MDP. • An improved DRL algorithm is presented to deal with the MDP model in large-scale discrete state and action space. • We validate our DRL algorithm has good time performance and optimal decision-making series solution via comparisons with DQN and PPO algorithm. [ABSTRACT FROM AUTHOR]

Data augmentation strategies have always been important in machine learning techniques and play a unique role in model performance optimization processes. Therefore, in recent years, these techniques have become popular in the artificial intelligence field. In this paper, a new data augmentation strategy is proposed based on the interpretation algorithm of deep convolutional neural networks, i.e., constructing new training samples by deeply exploiting key features extracted from interpretable networks to achieve sample augmentation. Thus, a novel supervised data augmentation approach known as Supervised Data Augmentation–Key Feature Extraction (SDA-KFE) was proposed. By introducing the Neural Network Interpreter-Segmentation Recognition and Interpretation (NNI-SRI) algorithm, an augmentation strategy is proposed that can balance the high accuracy and high robustness of the final model while ensuring a large amount of data augmentation. The advantages of the SDA-KFE algorithm are mainly reflected in the following aspects. First, it is easy to implement. This algorithm is implemented based on the lightweight NNI-SRI algorithm, which lays the foundation for the implementation of SDA-KFE so that it can be easily implemented on convolutional neural networks. Second, this model, which is widely applicable, can be applied to almost any deep convolutional network. Through research and experiments on this proposed algorithm, SDA-KFE can be applied in graphical image binary classification and multiclassification models. Third, SDA-KFE can rapidly construct data samples with diverse variations. Under the premise of determining the classification labels of the generated samples, the distribution of the feature unit composition of the samples can be controlled. Compared with traditional data augmentation methods, SDA-KFE can control the direction of the model performance, i.e., the balance between the pursuit of high accuracy and robust performance of the model. Therefore, the novel supervised augmentation approach proposed in this paper is relevant for optimizing deep convolutional neural networks, solving model overfitting, augmenting data types, etc. The data augmentation algorithm proposed in this paper can be regarded as a useful supplement to traditional data augmentation methods, such as horizontal or vertical image flipping, cropping, color transformation, extension and rotation. [ABSTRACT FROM AUTHOR]

Multimodal emotion recognition, that is, emotion recognition uses machine learning to generate multi-modal features on the basis of videos which has become a research hotspot in the field of artificial intelligence. Traditional multi-modal emotion recognition method only simply connects multiple modalities, and the interactive utilization rate of modal information is low, and it cannot reflect the real emotion under the conflict of modal features well. This article first proves that effective weighting can improve the discrimination between modalities. Therefore, this paper takes into account the importance differences between multiple modalities, and assigns weights to them through the importance attention network. At the same time, considering that there is a certain complementary relationship between the modalities, this paper constructs an attention network with complementary modalities. Finally, the reconstructed features are fused to obtain a multi-modal feature with good interaction. The method proposed in this paper is compared with traditional methods in public datasets. The test results show that our method is accurate in It performs well in both the rate and confusion matrix metrics. [ABSTRACT FROM AUTHOR]

Recent researches show that deep learning model is susceptible to backdoor attacks. Many defenses against backdoor attacks have been proposed. However, existing defense works require high computational overhead or backdoor attack information such as the trigger size. In this paper, we propose a novel backdoor detection method based on intentional adversarial perturbations. The proposed method leverages intentional adversarial perturbation to detect whether an image contains a trigger, which can be applied in both the training stage and the inference stage (sanitize the training set in training stage or detect the backdoor instances in inference stage). Specifically, given an untrusted image, the adversarial perturbation is added to the image intentionally. If the prediction of the model on the perturbed image is consistent with that on the unperturbed image, the input image will be considered as a backdoor instance. Compared with most existing defense works, the proposed method is faster and introduces less computational overhead during backdoor detection process. Moreover, the proposed method maintains the visual quality of the image (as the ℓ 2 norm of the added perturbation is as low as 2.8715, 3.0513 and 2.4362 on Fashion-MNIST, CIFAR-10 and GTSRB datasets, respectively). Experimental results show that, for general backdoor attack, the backdoor detection rate of the proposed defense method is 99.63%, 99.76% and 99.91% on Fashion-MNIST, CIFAR-10 and GTSRB datasets, respectively. For invisible backdoor attack, the backdoor detection rate of the proposed defense method is 99.75% against blended backdoor attack and 98.00% against sample-specific backdoor attack. It is also demonstrated that the proposed method can achieve high defense performance against backdoor attacks under different attack settings (trigger transparency, trigger size and trigger pattern). In addition, the experimental comparison with related work demonstrates that the proposed method has better detection performance and higher detection efficiency. [ABSTRACT FROM AUTHOR]

Usually, manual labeling of large-scale electrocardiograms (ECGs) for deep learning is always expensive, as it requires considerable effort and time from cardiologists. Currently, contrastive learning can utilize unlabeled ECGs to improve deep learning using insufficient labels. However, this field may still lack instructive literature. In this paper, a dense lead contrast (DLC) is proposed for effective contrastive learning on multilead ECGs. It develops contrastive learning between any two leads from different views to explore intralead and interlead invariance. A joint loss combining intralead and interlead contrastive loss guides the DLC pretraining. Moreover, DLC introduces a multibranch network (MBN) for contrastive learning, generates a representation for each lead, and fuses all the leads for a global representation. In the downstream tasks, DLC outperforms the standard contrastive learning paradigm of multilead ECGs by 2.78%∼6.59% in AUROC (linear probe). Using only 10% of the labeled training data, it still outperforms standard contrastive learning by a significant margin in AUROC. Compared with existing methods, DLC shows obvious advantages in all the experiments. Therefore, DLC may be more suitable for the contrastive learning of multilead ECGs. Its good performance based on insufficient labels can alleviate the cardiologists' burden from data labeling. • Dense lead contrast (DLC) conducts an effective self-supervised pre-training. • DLC jointly explores intra- and inter-lead invariance of electrocardiograms (ECGs). • Multiple branch network (MBN) architecture organizes all the leads. • Raw ECGs from a real-world hospital are used to validate algorithm robustness. [ABSTRACT FROM AUTHOR]

• Proposing a new deep learning model for accurately detecting cyber threats to extract a feature representation and learn temporal information from cybersecurity data sequences. • Decreasing the learning rate of the Adam optimizer exponentially to improve its exploitation operator when it cannot improve in the validation loss within several epochs. • Evaluating the proposed model's performance, DeepAK-IoT uses three datasets, two of which are aggregated using heterogeneous data sources, to assess its ability to detect the malicious code that can threaten IoT sources. • Refining DeepAK-IoT provides greater accuracy for the validated datasets, superior to three contemporary models. Our daily lives have been profoundly changed over the past few years owing to the growing presence of the Internet of Things (IoT). Importantly, IoT makes our lives more convenient, simpler, and more efficient; however, gadgets are vulnerable to a wide variety of cyberattacks due to the lack of robust security mechanisms and hardware security support. This paper presents an alternative deep learning model known as DeepAK-IoT to detect cyberattacks against IoT devices. DeepAK-IoT uses three blocks as its foundation: the residual-based-spatial representation (RSR) block, the temporal representation block (TRB), and the detection block (DB). The RSR block uses five residual blocks to extract a feature representation from the output of the preceding layer. The four convolutional layers are connected in parallel with a skip connection within each block to avoid vanishing or exploding gradients. Then, the second block uses the extracted spatial representation to learn a temporal representation to detect cyber threats. The final block decides how to classify the input record. We evaluated the accuracy and generalization ability of DeepAK-IoT using three well-known public datasets: TON-IoT, Edge-IIoTset, and UNSW-NB15. The proposed model was compared to three state-of-the-art deep learning models to demonstrate its effectiveness in detecting cyber threats in IoT systems. According to the experimental results, DeepAK-IoT was found to be a powerful alternative model for managing cyber threats in IoT networks, as it provided 90.57% accuracy for TON IoT, 94.96% for Edge-IIoTset, and 98.41% for UNSW NB15. [ABSTRACT FROM AUTHOR]

The training effect of the neural network model is directly influenced by optimization algorithms. The Barzilai-Borwein(BB) method is used in the stochastic gradient descent (SGD) and other deep learning optimization algorithms because of its outstanding performance in terms of convergence speed. In order to improve the stability of BB step size and the output quality of network model in deep learning, this paper presents two improved optimization algorithms based on the BB method: BBbound and AdaBBbound. BBbound reduces the floating BB step size by generating the upper bound of the step in the current iteration. It avoids the occasional occurrence of long steps; AdaBBbound is an adaptive gradient method based on BB method with dynamic bound. The algorithm has a fast convergence effect in the early stage, then calculates learning rate and smoothly transitions to SGD by setting the better conditions of the BB method. We analyzed the performance of the two algorithms based on the initial conditions and the learning rate change curve. Meanwhile, we tested our algorithms on popular network models such as ResNet and DenseNet. The results showed that the new optimization algorithms have achieved high stability and significantly improved the performance of the network model. [ABSTRACT FROM AUTHOR]

To further enhance the reliability of Machine Learning (ML) systems, considerable efforts have been dedicated to developing privacy protection techniques. Recently, membership privacy has gained increasing attention, with a focus on determining whether a specific data point is present in the confidential training set of an ML model. However, most current attacks only prioritize attack accuracy and fail to extend their range to the evidence that contributes to the member/non-member classification. This limitation greatly reduces the practicality of Membership Inference Attacks (MIA), as real-world data typically includes multiple features, making it challenging to identify which features are involved in the sensitive training set. Therefore, this paper targets one of the fundamental challenges in membership inference attack: measuring the distance between an attack sample and a member sample. Specifically, we propose a novel threat model called Membership Reconstruction Attack (MRA), which aims to reconstruct the exact distribution of the target training set. MRA achieves this by marking each input dimension (e.g., pixels) according to its similarity to the target dataset in feature space. Our attack demonstrates its effectiveness across various settings, including different major datasets (MNIST, CIFAR-10, CIFAR-100) and different model architectures (AlexNet, ResNet, DenseNet, and generative models). Additionally, we evaluate MRA from the defenders' perspective and test several defense approaches against our attack. [ABSTRACT FROM AUTHOR]

This issue of DeepFake technique that may cause great threat to privacy, democracy, and national security has attracted the attention of deep learning researchers. DeepFake detection, therefore, has been a very hot issue in deep learning research. The face landmark feature maps are often used by many DeepFake approaches in generating fake faces. It also provides key information to help to detect manipulated face images. In this paper, we propose a detection approach for manipulated face images. To make full use of face landmark information, we propose a facial geometry prior module (FGPM) to extract facial geometry feature maps. Then the facial geometry feature maps are embedded into upsampling feature maps generated by the CNN-LSTM network. The proposed FGPM exploits facial maps and frequency domain correlation to analyze the discriminative characteristics between manipulated and non-manipulated regions by incorporating the CNN-LSTM network. Finally, a decoder is used to learn the mapping from low-resolution feature maps to pixel-wise to predict manipulation localization. Or a softmax classifier is used to predict real or fake face images. By experimenting on several popular datasets, the proposed detection model has demonstrated the capability of localizing manipulation at the pixel level and with a high prediction on real or fake face images. • Propose a manipulated face detection model which predicts fake face in pix-lever. • Facial geometry feature maps are used in predicting fake face. • Propose a CNN-LSTM network to discriminate manipulated face images. [ABSTRACT FROM AUTHOR]

Recently, 3D deep learning models have been shown to be susceptible to adversarial attacks like their 2D counterparts. Most of the state-of-the-art (SOTA) 3D adversarial attacks perform perturbation to 3D point clouds. To reproduce these attacks in the physical scenario, a generated adversarial 3D point cloud needs to be reconstructed to mesh, which leads to a significant drop in its adversarial effect. In this paper, we propose a strong 3D adversarial attack named Mesh Attack to address this problem by directly performing perturbation on mesh of a 3D object. In order to take advantage of the most effective gradient-based attack, a differentiable sample module that back-propagate the gradient of point cloud to mesh is introduced. To further ensure the adversarial mesh examples without outlier and 3D printable, three mesh losses are adopted. Extensive experiments demonstrate that the proposed scheme outperforms SOTA 3D attacks by a significant margin. We also achieved SOTA performance under various defenses. Our code will be available at: https://github.com/cuge1995/Mesh-Attack. [ABSTRACT FROM AUTHOR]

Current research on machine learning-based intelligent routing focuses on algorithm design and performance optimization. How to deploy it in practice remains a pressing issue. Existing deep learning-based intelligent routing algorithms suffer from a high computational cost, which is hard to be afforded by routers. Considering that the cloud can assist the routers with limited resources to handle complex computation, outsourcing deep learning-based intelligent routing computations to the cloud becomes a feasible solution to support intelligent routing. However, due to the untrustworthiness of the cloud, it is necessary for the routers to verify the truth of outsourced routing results. However, the literature has not yet explored this issue. In this paper, we proposed VeriORouting, a scheme to verify the correctness of outsourced deep learning-based intelligent routing results provided by the cloud. Facing a lazy cloud, VeriORouting allows routers to check the reliability of intelligent routing models by testing model accuracy, and to verify the routing results returned by the cloud without knowing the model by using verification functions generated with multilayer perceptron (MLP) and locality-sensitive hashing (LSH) in advance. We show the robustness of VeriORouting under two attacks raised by the cloud. We evaluate the performance of VeriORouting and compare it with a local intelligent routing scheme. The results show that VeriORouting outperforms the local scheme in terms of computational overhead and storage overhead, especially when the number of routers in the network increases. In terms of communication, VeriORouting reduces communication overhead between routers compared to the local scheme. In addition, we measure the verification performance of VeriORouting under a random attack. VeriORouting achieves a detection success rate of 73% with a false positive rate of 3%, and a detection success rate of 90% with a false positive rate of 25%. • The first scheme to verify deep learning-based intelligent routing outsourced to the cloud. • A mechanism for timely network state information transmission with validation. • A new routing table to reduce redundant executions of verification. • Resistance to two possible attacks from a lazy cloud and advance for practical deployment. [ABSTRACT FROM AUTHOR]

• Local and global self-attention mechanisms are used for character embedding. • CNN with multi-size filters are used to extract character information for NER. • A cross-attention method that fuses character and word embedding for NER is proposed • A modified Mogrifier LSTM is presented to improve the performance of NER. • Proposed methods integrated with a transformer-based model achieve good performance. Clinical and biomedical concept extraction is critical in medical analysis using clinical and biomedical documents from professional literature, EHRs and PHRs. Named entity recognition (NER) accurately marks essential information in the literature based on the characteristics of the target entity, providing a method for extracting clinical and biomedical concepts. The performance of NER is heavily embedding-dependent, so recent studies have proposed the method of generating word embedding from character-level information, which can strengthen the representation ability for word embedding. In this paper, we present a novel neural network model including an attention mechanism network and a convolutional neural network (CNN) to further improve character-level embedding. First, an attention mechanism is applied simultaneously to the local and global character embedding. Then, a CNN with multi-size filters is used to extract more information from the character level, which can capture more meaningful features from words with various lengths. In addition, a cross-attention method is used to leverage the interaction between word embedding and character embedding to generate the final word representation. Finally, we modified Mogrifier LSTM to make it suitable for NER tasks and integrated it into our model. Experimental results show that our method is effective and that the model performs better than the baseline models. We also apply our methods proposed in this paper to the transformer-based model and obtain a 90.36 F1-score on NCBI-Disease. [ABSTRACT FROM AUTHOR]

Designing a Convolutional Neural Networks (CNN) is a complex task and requires expert knowledge to optimize the performance and network architecture. In this paper, a novel data-driven approach is proposed to determine the architecture of CNN models. The proposed Autonomous Convolutional Neural Networks (AutoCNNThe executable code and original numerical results can be downloaded from (https://tinyurl.com/AutoCNN)) algorithm introduces data driven strategies for addition of new convolutional layers, pruning of redundant filters and training cycle optimization. AutoCNN is evaluated using MNIST, MNIST-rot-back-image, Fashion MNIST and the ADHD200 datasets to measure the performance on small datasets with varied feature distributions. The results indicate that AutoCNN optimizes the CNN network architecture and helps maximise the classification performance. The data-driven network determination approach introduced in this paper was found to not only provides competitive performance similar to existing evolutionary computation based network determination algorithms in literature, but was found to be an effective optimization tool to improve the performance of existing CNN architectures. Further, the AutoCNN was found to highly immune to noise in the dataset and has proven to be effective method to transfer knowledge between related datasets. Therefore, the AutoCNN is a highly versatile CNN architecture determination tool that has a wide range of applications in the field of autonomous driving, medical image analysis, image enhancement, camera based security monitoring and image based fault detection. [ABSTRACT FROM AUTHOR]

In the subjective evaluation process, the hesitant fuzzy set (HFS), as a convenient and robust presentation tool, cannot only suitably address the decision makers' (DMs') or experts' hesitant and uncertain issues but also can arise the dimension curse puzzle. Furthermore, the decision-making result is just derived according to the given objective and subjective information, without considering the DM's subjective evaluation and the environment's dynamic influence. Unlike the previous studies, this paper tries to address them from the deep learning viewpoint. To this end, we first define the non-equidimensional HFS (NHFS) and then introduce the equidimensional and classification characters into the NHFS to further develop the equidimensional HFS (EHFS) and the EHFS with the optimal classification result. Then, the equidimensional hesitant fuzzy-generative adversarial network (EHF-GAN) model is proposed to transform the hesitant fuzzy information from the non-equidimensional to the equidimensional form. The generalization and the convergence of the new model are proven to show the models' reasonability. In addition, the double-learning algorithm of the EHF-GAN model is designed, which can fuse the DMs' dynamic judgments and derive the optimal decision-making results. Lastly, this paper applies the proposed model and algorithm to an illustrative example of the new smart city enterprises and then shows their feasibility and effectiveness. [ABSTRACT FROM AUTHOR]

Ambient intelligence (AmI) systems aim to provide users with context-aware assistance services intended to improve the quality of their lives in terms of autonomy, safety, and well-being. Taking the uncertainty and partial observability of these environments into account is of major importance for context recognition and, more specifically, to detect and solve context abnormalities such as those related to the user's behavior or those related to context attribute prediction. In this paper, an ontology-based framework integrating machine learning and probabilistic planning within commonsense reasoning is proposed to recognize the user's context and abnormalities associated with it. The reasoning is performed using event calculus in answer set programming (ECASP); ECASP allows for abductive and temporal reasoning, which results in an eXplainable AI (XAI) approach. A context ontology is proposed to axiomatize the reasoning and introduce the notion of probabilistic fluents into the EC formalism in order to perform probabilistic reasoning. The reasoning incorporates probabilistic planning based on a partially observable Markov decision process (POMDP) to solve knowledge incompleteness. To evaluate the proposed framework, real-life scenarios, based on the Orange4Home and SIMADL public datasets are implemented and discussed. [ABSTRACT FROM AUTHOR]

Differentiable architecture search (DARTS) is an effective neural architecture search algorithm based on gradient descent. However, there are two limitations in DARTS. First, a small proxy search space is exploited due to memory and computational resource constraints. Second, too many simple operations are preferred, which leads to the network deterioration. In this paper, we propose a uniform-space differentiable architecture search, named U-DARTS, to address the above problems. In one hand, the search space is redesigned to enable the search and evaluation of the architectures in the same space, and the new search space couples with a sampling and parameter sharing strategy to reduce resource overheads. This means that various cell structures are explored directly rather than cells with same structure are stacked to compose the network. In another hand, a regularization method, which takes the depth and the complexity of the operations into account, is proposed to prevent network deterioration. Our experiments show that U-DARTS is able to find excellent architectures. Specifically, we achieve an error rate of 2.59% with 3.3M parameters on CIFAR-10. The code is released in https://github.com/Sun-Shiqi/U-DARTS. [ABSTRACT FROM AUTHOR]

Machine learning models, especially deep neural network (DNN) models, have widespread and valuable applications in business activities. Training a deep learning model for commercial use requires plenty of private data, expert knowledge, and computing resources. The huge commercial value of such trained models has attracted the attention of attackers. Attackers can construct a dataset by repeatedly querying the target model for the output of the requested samples and then train a substitute model on this dataset that functions similarly to the target model. In this paper, we propose a defense scheme based on physical unclonable function (PUF) against such black-box model extraction attacks. We deploy a PUF on the user side and the corresponding PUF model on the service provider side to ensure that only legitimate users can obtain the correct model predictions. Our experimental results show that by choosing a suitable fuzzy extractor threshold d , legitimate users can recover more than 99.5% of the prediction results with a little additional computational overhead to the service provider. We perform a model extraction attack in the most favorable case for the attacker, and the prediction accuracy of the obtained substitute model is only about 10%, which demonstrates the effectiveness of our proposed scheme. Compared to existing defenses, our scheme not only effectively prevents black-box model extraction attacks but also ensures that the accuracy of the prediction service for legitimate users is not affected. • Defense based on PUF obfuscation against black-box model extraction attacks. • A PUF and the corresponding PUF model ensure that legitimate users can obtain correct model predictions. • Legitimate users can correctly recover more than 99.5% of the predictions with a little additional computational overhead. • The prediction accuracy of the model extracted by the attacker is only about 10%. [ABSTRACT FROM AUTHOR]

This paper proposes a novel method for efficiently finding the Nash equilibrium in a chance-constrained game (CCG). Conventional numerical solution methods require significant computational time when solving multiple instances of CCG. We introduce CCGnet, a deep learning approach that is capable of efficiently solving multiple instances of CCG in a one-shot manner. CCGnet employs a specialized network structure and training algorithm based on neurodynamic optimization. We demonstrate the strong performance of CCGnet in practice and show that our proposed method outperforms conventional methods. [ABSTRACT FROM AUTHOR]

• Propose a differentiable bi-level optimization network for multi-view clustering. • Construct an interpretable network by stacking multiple differentiable block networks. • Integrate traditional optimization methods with deep learning. • Perform both explicit and implicit optimization simultaneously. Multi-view clustering on traditional optimization methods is derived from different theoretical frameworks, yet it may be inefficient in dealing with complex multi-view data compared to deep models. In contrast, deep multi-view clustering methods for implicit optimization have excellent feature abstraction ability but are inscrutable due to their black-box problem. However, very limited research was devoted to integrating the advantages of the above two types of methods to design an efficient method for multi-view clustering. Focusing on these problems, this paper proposes a differentiable bi-level optimization network (DBO-Net) for multi-view clustering, which is implemented by incorporating the traditional optimization method with deep learning to design an interpretable deep network. To enhance the representation capability, the proposed DBO-Net is constructed by stacking multiple explicit differentiable block networks to learn an interpretable consistent representation. Then all the learned parameters can be implicitly optimized through back-propagation, making the learned representation more suitable for the clustering task. Extensive experimental results validate that the strategy of bi-level optimization can effectively improve clustering performance and the proposed method is superior to the state-of-the-art clustering methods. [ABSTRACT FROM AUTHOR]

In deep learning, auxiliary modules for model training have become increasingly popular, such as Deep Mutual Learning (DML) and Multi-Scale Dense Convolutional Networks (MSDNet), which can maximize the performance of the model without increasing the amount of test computation. Nevertheless, current research does not fully exploit the knowledge between different auxiliary modules. This paper proposes a new model training technique–Siamese Labels Auxiliary (SiLA) Learning, in which the SiLA module is designed to concatenate the outputs of the modules to get auxiliary information. In addition, the knowledge learned from different modules can be fully exploited by auxiliary information. Our experiments show that SiLA Learning effectively improves the performance of standard models and achieves convincing experimental results on image classification tasks without increasing the amount of test computation. Moreover, the experiments show that SiLA Learning can be easily combined with methods such as DML and MSDNet to exploit the knowledge fully and perform best. [ABSTRACT FROM AUTHOR]

The successful application of modern deep neural network models is deeply influenced by the choice of hyperparameters. As one of the most important hyperparameters, the Learning Rate (LR) needs to be fine-tuned. In order to train models that can perform better, finding a suitable LR schedule quickly has become a pressing problem. Current research often relies on manual use of parallel search methods such as grid search, or serial methods such as Bayesian. However, the variety and number of models and datasets result in a very large LR space, and these methods consume significant resources to find only sub-optimal static LR schedules within a given time frame. It was found that in the currently widely used Distributed Data Parallel (DDP) deep learning, where multiple nodes themselves constitute the population, it is possible to use population algorithms to optimise the searched LR schedule dynamically throughout the training process with little additional training time. To maximise the potential of both the searched LR schedule and the population optimisation of the participating multi-nodes, this paper proposes a Leader Population Learning Rate Schedule (LPLRS) for the DDP deep learning environment, which continuously explores for better learning rates in the neighbourhood of the searched LR schedule and guides the subsequent training process. LPLRS achieves higher model classification accuracy on the Cifar 10&100 test datasets on the state-of-the-art Wide Residual Network with Sharpness-Aware Minimization (WRN(SAM)) model compared to the latest SGDR, CLR, and StepLR learning rate schedules. [ABSTRACT FROM AUTHOR]

Graph has currently become an increasingly popular model in many real applications. The efficiency of graph storage is crucial for these applications. Generally speaking, the tuning task of graph storage relies on the database administrators to find the best graph storage. However, DBAs make the tuning decisions by mainly relying on their experiences and intuition. Due to the limitations of the DBAs, the tuning may have an uncertain performance and worse efficiency. In this paper, we observed that an estimator of the graph workload has the potential ability to guarantee the performance of tuning operations. Unfortunately, because of complex characteristics of the graph evaluation task, there exists no mature estimator for graph workload. We formulate the evaluation task of graph workload as a classification task and carefully design the feature engineering process, including graph data features, graph workload features and graph storage features. For this evaluation task, we propose an active auto-estimator (AAE) for the graph workload evaluation by combining the active learning and deep learning. We test the time efficiency and evaluation accuracy of AAE with two open source graph data, LDBC and Freebase. The experimental results show that our estimator could efficiently complete the graph workload evaluation in milliseconds. [ABSTRACT FROM AUTHOR]

• An advanced anomaly detection method via granular computing is proposed; • Granular data descriptors are used to construct anomaly detectors; • A fuzzy rule-based modeling method is applied for anomaly data detection (ADD); • The constructed granular ADD detectors are low-cost and effective; It is meaningful for data-driven studies to detect and remove the anomaly data from a given raw dataset. Considering the existing anomaly data detection (ADD) methods have some drawbacks, e.g. confidence and uncertainty in anomaly description, computation cost of kernel learning and deep learning in ADD, therefore this paper proposes an advanced approach for constructing low-cost and effective ADD detectors. Firstly, making use of the good data description ability of information granules, granular data descriptors are constructed for anomaly and normal data description. Then, based on these data descriptors, reconstruction-based strategy is applied to model anomaly detection. Subsequently, with consideration of anomaly data's uncertainty, fuzzy rules are formed and used to realize the final anomaly data detection. In this study, both synthetic and publicly available datasets are considered in experimental analysis. Numerical results illustrate the proposed method has a superiority ranging from 8.60% to 32.58% to conventional models on ADD performance, verifying the feasibility and effectiveness the proposed methods via granular data descriptors. Moreover, through the discussion on computation complexity, the proposed method is also demonstrated efficient on detecting anomalies. [ABSTRACT FROM AUTHOR]

• A novel lightweight DL technique is presented for fine-tuning lung infection segmentation. • The down-sampling module is innovatively designed to effectively capture lesion features from cross-domain CT images. • A novel contextual enrichment module is introduced to empower the network to capture the combinations of static. • A novel federated periodic client selection protocol is proposed to select the training participants fairly and periodically. The automatic segmentation of COVID-19 pneumonia from a computerized tomography (CT) scan has become a major interest for scholars in developing a powerful diagnostic framework in the Internet of Medical Things (IoMT). Federated deep learning (FDL) is considered a promising approach for efficient and cooperative training from multi-institutional image data. However, the nonindependent and identically distributed (Non-IID) data from health care remain a remarkable challenge, limiting the applicability of FDL in the real world. The variability in features incurred by different scanning protocols, scanners, or acquisition parameters produces the learning drift phenomena during the training, which impairs both the training speed and segmentation performance of the model. This paper proposes a novel FDL approach for reliable and efficient multi-institutional COVID-19 segmentation, called MIC-Net. MIC-Net consists of three main building modules: the down-sampler, context enrichment (CE) module, and up-sampler. The down-sampler was designed to effectively learn both local and global representations from input CT scans by combining the advantages of lightweight convolutional and attention modules. The contextual enrichment (CE) module is introduced to enable the network to capture the contextual representation that can be later exploited to enrich the semantic knowledge of the up-sampler through skip connections. To further tackle the inter-site heterogeneity within the model, the approach uses an adaptive and switchable normalization (ASN) to adaptively choose the best normalization strategy according to the underlying data. A novel federated periodic selection protocol (FED-PCS) is proposed to fairly select the training participants according to their resource state, data quality, and loss of a local model. The results of an experimental evaluation of MIC-Net on three publicly available data sets show its robust performance, with an average dice score of 88.90% and an average surface dice of 87.53%. [ABSTRACT FROM AUTHOR]

• A new supervised fuzzy measure learning algorithm is proposed for combining classifiers. • Different from previous works, any cost function can be optimized, including those well-suited for classification problems. • Three new update policies are proposed for dealing with monotonicity constrains. • The benefit of using appropriate cost functions for classification is experimentally demonstrated. • The new proposal outperforms existing methods in both binary and multi-class problems. Fuzzy measure-based aggregations allow taking interactions among coalitions of the input sources into account. Their main drawback when applying them in real-world problems, such as combining classifier ensembles, is how to define the fuzzy measure that governs the aggregation and specifies the interactions. However, their usage for combining classifiers has shown its advantage. The learning of the fuzzy measure can be done either in a supervised or unsupervised manner. This paper focuses on supervised approaches. Existing supervised approaches are designed to minimize the mean squared error cost function, even for classification problems. We propose a new fuzzy measure learning algorithm for combining classifiers that can optimize any cost function. To do so, advancements from deep learning frameworks are considered such as automatic gradient computation. Therefore, a gradient-based method is presented together with three new update policies that are required to preserve the monotonicity constraints of the fuzzy measures. The usefulness of the proposal and the optimization of cross-entropy cost are shown in an extensive experimental study with 58 datasets corresponding to both binary and multi-class classification problems. In this framework, the proposed method is compared with other state-of-the-art methods for fuzzy measure learning. [ABSTRACT FROM AUTHOR]

Extreme Multi-label Text Classification (XMTC) is a key task of finding the most relevant labels from a large label set for a document. Although some deep learning-based methods have shown great success in XMTC, they still suffer from the following drawbacks. First, although several methods have improved the precision by clustering labels and combining several sub-models to train and predict for one dataset, they were not ideal in terms of computational efficiency. Second, most of those methods need a low dimensional bottleneck layer before the output layer to compress the feature representations to fit the GPU memory, which results in information loss of original features. In this paper, we proposed a novel two-stage X MTC framework with candidate R etrieving and deep R anking (XRR) to address those drawbacks. In the retrieving stage, we designed two retrieval strategies, including an a ligning P oint M utual I nformation (aPMI) method, and a U nified L abel- S emantic E mbedding (ULSE) method, to extract hundreds of candidates from massive labels. In the ranking stage, we presented a deep ranking model using a pre-trained transformer to distinguish the true labels from candidates. Extensive experiments show that XRR outperforms the state-of-the-art methods on five widely used multi-label datasets. [ABSTRACT FROM AUTHOR]

• An advanced anomaly detection method via granular computing is proposed; • Granular data descriptors are used to construct anomaly detectors; • A fuzzy rule-based modeling method is applied for anomaly data detection (ADD); • The constructed granular ADD detectors are low-cost and effective; It is meaningful for data-driven studies to detect and remove the anomaly data from a given raw dataset. Considering the existing anomaly data detection (ADD) methods have some drawbacks, e.g. confidence and uncertainty in anomaly description, computation cost of kernel learning and deep learning in ADD, therefore this paper proposes an advanced approach for constructing low-cost and effective ADD detectors. Firstly, making use of the good data description ability of information granules, granular data descriptors are constructed for anomaly and normal data description. Then, based on these data descriptors, reconstruction-based strategy is applied to model anomaly detection. Subsequently, with consideration of anomaly data's uncertainty, fuzzy rules are formed and used to realize the final anomaly data detection. In this study, both synthetic and publicly available datasets are considered in experimental analysis. Numerical results illustrate the proposed method has a superiority ranging from 8.60% to 32.58% to conventional models on ADD performance, verifying the feasibility and effectiveness the proposed methods via granular data descriptors. Moreover, through the discussion on computation complexity, the proposed method is also demonstrated efficient on detecting anomalies. [ABSTRACT FROM AUTHOR]

Extreme Multi-label Text Classification (XMTC) is a key task of finding the most relevant labels from a large label set for a document. Although some deep learning-based methods have shown great success in XMTC, they still suffer from the following drawbacks. First, although several methods have improved the precision by clustering labels and combining several sub-models to train and predict for one dataset, they were not ideal in terms of computational efficiency. Second, most of those methods need a low dimensional bottleneck layer before the output layer to compress the feature representations to fit the GPU memory, which results in information loss of original features. In this paper, we proposed a novel two-stage X MTC framework with candidate R etrieving and deep R anking (XRR) to address those drawbacks. In the retrieving stage, we designed two retrieval strategies, including an a ligning P oint M utual I nformation (aPMI) method, and a U nified L abel- S emantic E mbedding (ULSE) method, to extract hundreds of candidates from massive labels. In the ranking stage, we presented a deep ranking model using a pre-trained transformer to distinguish the true labels from candidates. Extensive experiments show that XRR outperforms the state-of-the-art methods on five widely used multi-label datasets. [ABSTRACT FROM AUTHOR]

• A new supervised fuzzy measure learning algorithm is proposed for combining classifiers. • Different from previous works, any cost function can be optimized, including those well-suited for classification problems. • Three new update policies are proposed for dealing with monotonicity constrains. • The benefit of using appropriate cost functions for classification is experimentally demonstrated. • The new proposal outperforms existing methods in both binary and multi-class problems. Fuzzy measure-based aggregations allow taking interactions among coalitions of the input sources into account. Their main drawback when applying them in real-world problems, such as combining classifier ensembles, is how to define the fuzzy measure that governs the aggregation and specifies the interactions. However, their usage for combining classifiers has shown its advantage. The learning of the fuzzy measure can be done either in a supervised or unsupervised manner. This paper focuses on supervised approaches. Existing supervised approaches are designed to minimize the mean squared error cost function, even for classification problems. We propose a new fuzzy measure learning algorithm for combining classifiers that can optimize any cost function. To do so, advancements from deep learning frameworks are considered such as automatic gradient computation. Therefore, a gradient-based method is presented together with three new update policies that are required to preserve the monotonicity constraints of the fuzzy measures. The usefulness of the proposal and the optimization of cross-entropy cost are shown in an extensive experimental study with 58 datasets corresponding to both binary and multi-class classification problems. In this framework, the proposed method is compared with other state-of-the-art methods for fuzzy measure learning. [ABSTRACT FROM AUTHOR]

In recent years, matrix completion has attracted a lot of attention. Conventional matrix completions are shallow and ineffective when dealing with complex data structures. Researchers have recently tried to incorporate deep structures into matrix completion; however, considerable challenges still exist. Most matrix completion methods may fail to work effectively in the presence of limited observations. To enhance the generalization of the reconstruction, adversarial methods are proposed that attempt to fool models by providing deceptive input. The aim is to develop an adversarial training algorithm that resists attacks in a deep model, thus at the same time leading to enhancing the generalization. Therefore, in this paper, we propose an elastic adversarial training to design a high-capacity Deep Nonnegative Matrix Factorization (DNMF) model with proper discovery latent structure of the data and enhanced generalization abilities. In other words, the challenges mentioned above are addressed by perturbing the inputs in DNMF with an elastic loss which is intercalated and adapted between Frobenius and ℓ 2 , 1 norms. This model not only dispenses with adversarial DNMF generation but also is robust towards a mixture of multiple attacks to attain improved accuracy. Extensive simulations show that the proposed approach outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Despite the active research on deep learning these days, no existing works attempt to query neural networks for a specific task so that we can quickly obtain a model specialized for the queried task. This paper presents the first study on the problem of querying neural networks, aiming to efficiently find a lightweight model for any on-demand sub-task supported by a large and generic neural network. This problem is well motivated by the fact that such a lightweight model is particularly suitable for deployment in commodity mobile devices with limited computing resources. In this paper, we propose QueryNet , a framework of queriable neural networks, which is based on our class-aware channel pruning technique and the proposed method of merging multiple tiny networks for producing a specialized model for the task. Through the extensive experiments, we show that QueryNet can generate a lightweight neural network for a given task up to 3.25 times faster than learning a specialized model from scratch, and the resulting specialized neural network carries up to 37 times less parameters than a pretrained network, and even shows a slightly better accuracy than that of the pretrained model. [ABSTRACT FROM AUTHOR]

• The deep reinforcement learning is firstly employed to solve source searching problem. • The belief state is maintained by the particle filter. • The feature of belief state is extracted by the DBSCAN algorithm. • Transfer learning is employed to reuse the trained Q-network in heterogeneous tasks. The localization of hazardous sources (e.g. poisonous gas sources) is an important task regarding the security of human society. To find the unknown source in time, various autonomous source searching methods have mushroomed and been employed over the past decade. This paper designs a fresh source searching approach, namely particle clustering-deep Q-network, PC-DQN, which applies the deep reinforcement learning (DRL) techniques as a source searching approach for the first time. Specifically, the search process is formulated as the partially observable Markov decision process, then converted into the Markov decision process based on the belief state (represented by the particle filter). PC-DQN leverages the density-based spatial clustering of applications with noise (DBSCAN) algorithm to extract the feature of belief state, and employ the deep Q-network (DQN) algorithm to find the optimal policy for the source searching task. Through the comparison with two baseline methods (i.e. RANDOM and Entrotaxis algorithm) under various experimental conditions, the viability of our proposed PC-DQN is testified. Results explicitly reveal that the success rate of the PC-DQN maintains at a high level (beyond 99.6%) in all scenarios in this paper, and the mean search step shows evident superiority over baseline methods in most scenarios. Significantly, we also introduce the transfer learning concept to reuse the well-trained Q-network into new scenarios. These findings show important implications of the DRL-based approach as an alternative and more effective source searching approach. [ABSTRACT FROM AUTHOR]

The early and accurate detection of COVID-19 is vital nowadays to avoid the vast and rapid spread of this virus and ease lockdown restrictions. As a result, researchers developed methods to diagnose COVID-19. However, these methods have several limitations. Therefore, presenting new methods is essential to improve the diagnosis of COVID-19. Recently, investigation of the electrocardiogram (ECG) signals becoming an easy way to detect COVID-19 since the ECG process is non-invasive and easy to use. Therefore, we proposed in this paper a novel end-to-end deep learning model (ECG-COVID) based on ECG for COVID-19 detection. We employed several deep Convolutional Neural Networks (CNNs) on a dataset of 1109 ECG images, which is built for screening the perception of COVID-19 and cardiac patients. After that, we selected the most efficient model as our model for evaluation. The proposed model is end-to-end where the input ECG images are fed directly to the model for the final decision without using any additional stages. The proposed method achieved an average accuracy of 98.81%, Precision of 98.8%, Sensitivity of 98.8% and, F1-score of 98.81% for COVID-19 detection. As cases of corona continue to rise and hospitalizations continue again, hospitals may find our study helpful when dealing with these patients who did not get significantly worse. [ABSTRACT FROM AUTHOR]

Accurate and timely crop yield prediction is difficult to achieve due to the nonlinear and dynamic spatial–temporal correlations included during the crop growth process. The latest approaches utilize deep learning techniques to implicitly capture sequential dependencies but fail to consider prior knowledge. In this paper, a novel Knowledge-guided Spatial–Temporal Attention Graph Network (KSTAGE), which incorporates prior knowledge to model dynamic temporal relationships and geospatial topology structures, is proposed. The proposed KSTAGE first utilizes a 3D convolutional neural network (CNN) to embed the initial spectral feature. Following its development, a new Knowledge-guided Temporal Multi-head Attention Algorithm (KTMA), the core of KSTAGE, is applied to automatically generate temporal attention weights from a self-attention mechanism under the guidance of prior knowledge. Furthermore, a new strategy for aligning self-attention scores with the prior distribution is developed. Finally, a location-aware Spatial Attention Graph Network utilizing geospatial knowledge is proposed to aggregate the spatial neighborhood features for the final yield prediction. The experimental results obtained on county-level yield prediction tasks in China and the contiguous United States (CONUS) demonstrate that the proposed KSTAGE achieves obvious improvements over the baselines. [ABSTRACT FROM AUTHOR]

The integration of deep learning (DL) and fuzzy learning (FL) is considered a recently emerging and promising research direction in data embedding. The integrated fuzzy neural learning paradigm within deep learning-based recommendation helped improving the performance of both latent feature representation learning and multiple recommendation problem fine-tuning. In general, feature representation learning is considered an important component of modern deep-learning based recommendation systems (RS). However, many well-known deep learning-based recommendation techniques encounter major challenges, which are mainly related to data uncertainty and the presence of noise in the obtained user and item embeddings. Moreover, most of the deep embedding-based RSs have still relied much on the local proximity relationships. These proximity features are obtained from the direct user-item interactions in order to leverage the recommendation outputs. This limitation has led to several difficulties in effectively extracting the rich latent structural features of users and items. In addition, in previous meta -path-based embedding techniques for the HIN-based recommendation approach, the integration between multi-type path-based representation learning for recommendation often encounters challenges concerning feature uncertainty extraction. To overcome the aforementioned problems, in this paper we propose a novel fuzzy-driven HIN embedding model for recommendation, called FuzzyRec. The proposed FuzzyRec has an integrated hierarchical fused neural fuzzy and deep network-based embedding architecture. It supports effective learning of the representations of users and items, as well as reducing feature ambiguity and noise. Extensive experiments demonstrate the effectiveness of our proposed method and its advantages compared with state-of-the-art baselines for recommendation systems. [ABSTRACT FROM AUTHOR]

As deep learning models become increasingly integral to various 3D applications, concerns about their vulnerability to adversarial attacks grow in tandem. This paper addresses the challenge of enhancing the transferability of 3D adversarial attacks, a critical aspect for evaluating model robustness across diverse scenarios. We propose a novel approach leveraging scale and shear transformations to generate adversarial examples that exhibit improved transferability across multiple 3D models. Our methodology involves carefully integrating scale and shear transformations into the adversarial perturbation generation process with only a marginal increase in computational time. The proposed attack method operates within the Carlini-Wagner (CW) optimization framework. For each iteration, it employs two hyperparameters: p a , determining the probability of transforming the input point cloud, and p s , deciding whether to shear or scale the point cloud. Limited to scaling and shearing transformations, Scale and Shear (SS) attack seamlessly integrates with established attack methods, enhancing flexibility and compatibility in adversarial attacks on 3D models. Extensive experiments show that the SS attack proposed in this paper can be seamlessly combined with the existing state-of-the-art (SOTA) 3D point cloud attack methods to form more powerful attack methods, and the SS attack improves the transferability over 3.6 times compared to the baseline. Moreover, while substantially outperforming the baseline methods, the SS attack achieves SOTA transferability under various defenses. Our code will be available online at: https://github.com/cuge1995/SS-attack. [ABSTRACT FROM AUTHOR]

Human behavior may be one of the most challenging phenomena to model and validate. This paper proposes a method for automatically extracting and compiling evidence on human behavior determinants into a knowledge graph. The method (1) extracts associations of behavior determinants and choice options in relation to study groups and moderators from published studies using Natural Language Processing and Deep Learning, (2) synthesizes the extracted evidence into a knowledge graph, and (3) sub-selects the model components and relationships that are relevant and robust. The method can be used to either (4a) construct a structurally valid simulation model before proceeding with calibration or (4b) to validate the structure of existing simulation models. To demonstrate the feasibility of the method, we discuss an example implementation with mode of transport as behavior choice. We find that including non-frequently studied significant behavior determinants drastically improves the model's explanatory power in comparison to only including frequently studied variables. The paper serves as a proof-of-concept which can be reused, extended or adapted for various purposes. • The structure of behavior models should be validated using existing evidence. • Deep Learning can help automatize behavior evidence extraction and synthesis. • Infrequently studied significant variables improve model performance drastically. [ABSTRACT FROM AUTHOR]

• A novel neural network based robust method is given and two self-expressiveness in the reconstructed domain and the feature domain are constructed without the self-expressive noise estimation. • A new coupled game model to fuse information from different domains is presented. Based on the self-expressive property, we design a game model that is coupled with two deep subspace clustering modules, which can achieve information interaction from two domains. • A three-stage algorithm to train the game model efficiently is designed. We investigate a Game Training Mechanism for our model, which proposes different strategies for pre-training and fine-tuning to improve the convergence and stability of the algorithm. During high-dimensional subspace clustering, data is generally corrupted, which destroys the self-expressive property and results in self-expressive noise, adversely affecting the clustering process. Compared with data noise, self-expressive noise is complex, and its exact estimate is a difficult problem. This paper proposes a Bi-domanial Deep Subspace Clustering model (BD-DSC), which is a coupled game model. BD-DSC leverages two networks to extract the clean data in the reconstructed spatial domain and clean features in the feature domain respectively. The ability of the deep network to clean data and features allows us to avoid estimating self-expressive noise and efficiently construct two self-expressive models in both domains. Furthermore, game theory is used to couple these two self-expressive models in order to achieve efficient information interaction and increase the robustness and stability of the results. By means of co-supervising each other during training, a three-stage algorithm is provided to solve the game model. Extensive experiments have been conducted on four benchmark datasets. Owing to avoiding the self-expressive noise, fusing two modules dynamically and choosing the optimal parameters carefully, our proposed approach is robust to data noise and significantly superior to the state-of-the-art subspace clustering methods with the lowest clustering errors. [ABSTRACT FROM AUTHOR]

Federated learning is a privacy-preserving machine learning paradigm that can train a model with decentralized data. Classical federated learning systems are vulnerable to attacks from malicious clients. Although a number of efforts have been made to improve robustness of federated learning, existing solutions are either too costly or attack-specific. In this paper, we propose the Robust Ternary Gradients Aggregation (RTGA) algorithm, which can efficiently handle different attacks using two novel mechanisms. At the client-side, the ternary quantization mechanism compresses gradients using only two bits to store each coordinate of the gradient vector. We adopt the error-feedback mechanism to compensate for the difference between the compressed and actual gradients. At the server-side, the robust aggregation mechanism is designed to detect malicious clients according to the risk score of each client. The risk score is calculated by solving an optimization problem of the divergence among clients. Furthermore, we provide analysis of the compression and robustness properties of our proposed algorithm. Extensive experiments show that, compared with existing algorithms, RTGA can mitigate different types of attacks while costing much less communications. [ABSTRACT FROM AUTHOR]

• Bi-modal CBIR incorporating image visual features and semantic of text. • DenseNet is to generate image features in the database and the query image input by the user. • BERT is used to generate the text embeddings. • Deep learning based technique is used to obtain the joint representation of image and text modalities. • Proposed model is tested on real world dataset and ablation studies are conducted. In this study, we focus on the task of desired image retrieval from an extensive database. This comes under the purview of image retrieval systems. Users may input an image and a text as a query and expect the system to retrieve an image. The retrieved image should be close to the users' desires expressed in the query. Nowadays, digital media generates more than petabytes of imagery data in one day. Moreover, due to the internet, this massive amount of data is readily available to users. However, extracting the desired images from such a colossal databank is a challenging task. Users always prefer to extract an image that reflects their wishful thinking, and the user may desire to alter their visionary thoughts according to their abstract thoughts. For example, Elizabeth wants to have a laptop that is similar to her friend's laptop. However, she wants to have the same kind of laptop with a built-in GPU and in silver color. So she expects the e-business platform to show a laptop according to her wish. This paper attempts to devise a multi-modal algorithm for such tasks. It takes care of the user's visual and textual query. It has a query image and text as input and retrieves an image similar to the input image but modified according to the text query. This study focuses on a multi-modal image retrieval system that processes both image and text as input queries. Users can input an image and a text query to modify the image or add more information on it. The text reflects the desired modifications in the image. We proposed a bi-modal image retrieval system named " DenseBert 4 Ret " that learns image and text features concurrently. As the name indicates, DenseNet and BERT models are used for image and text features extractions, respectively. It is based on deep learning techniques used for the joint representation of image and text features. We trained the model, which forces the input image to be modified according to the user's textual query. We used deep information learning to train and test our model on three challenging real-world datasets, i.e., MIT States, Fshion200K and FashionIQ. We also show that the proposed model outperforms its predecessor with tuned parameters. [ABSTRACT FROM AUTHOR]

As the training process of deep neural networks involves expensive computational cost, speeding up the convergence is of great importance. Nesterov's accelerated gradient (NAG) is one of the most popular accelerated optimizers in the deep learning community, which often exhibits improved convergence performance over gradient descent (GD) in practice. However, theoretical investigations of NAG mainly focus on the convex setting. Since the optimization landscape of the neural network is non-convex, little is known about the convergence and acceleration of NAG. Nowadays, some works make progress towards understanding the convergence of NAG in training over-parameterized neural networks, where the number of the parameters exceeds that of the training instances. Nonetheless, previous studies are limited to the two-layer neural network, which are far from explaining the remarkable success of NAG in optimizing deep neural networks. In this paper, we investigate the convergence of NAG in training two architectures of deep linear networks: deep fully-connected linear neural networks and deep linear ResNets. Based on the over-parameterization regime, we first analyze the residual dynamics induced by the training trajectory of NAG for a deep fully-connected linear neural network under random Gaussian initialization. Our results show that NAG can converge to the global minimum at a (1 - O (1 / κ)) t rate when the width is near-linear in the depth of the network, where t is the number of iterations and κ > 1 is a constant depending on the condition number of the feature matrix. Compared to the (1 - O (1 / κ)) t rate of GD, NAG achieves an acceleration over GD. For deep linear ResNets, we utilize the same analytical approach and obtain a similar convergence result, while the width requirement is independent of the depth. To the best of our knowledge, these are the first theoretical guarantees for the convergence and acceleration of NAG in training deep neural networks. Numerical results show the acceleration of NAG compared to GD in terms of iterations. In addition, we conduct experiments to evaluate the effect of the depth on the convergence rate of NAG, which validate our derived conditions of the width. We hope our results may shed light on understanding the optimization behavior of NAG for modern deep neural networks. [ABSTRACT FROM AUTHOR]

• We propose a hyper-clustering (hyper-AP clustering) approach to capture the migration trend between individuals and clusters in intelligent transportation systems. • We use the hyper-clustering enhanced spatio-temporal deep learning model such as ST-ResNet and ST-3DNet, and we show that our approach can significantly improve the prediction accuracy of both deep learning methods. • Compared with state-of-the-art methods, the improvement of the prediction performance of our model is 10.9 Bike traffic prediction is essential to operations management in bike-sharing systems (BSS). The spatio-temporal distribution of shared bikes is imbalanced; therefore, it is necessary to predict the number of bikes that are available at each station to facilitate operations. Many researchers have developed deep neural network methods to improve predictive performance. Predicting bike demand from a higher level by clustering mobility trends has been shown to increase prediction accuracy. However, complex iterative optimization problems need to be solved to stabilize the trend iteration, and such solutions are missing in previous works. In this paper, we propose a novel hyper-clustering approach that enhances a spatio-temporal deep neural network for traffic prediction in bike-sharing systems. In particular, our hyper-clustering approach captures mobility trends among individuals and clusters. Experimental results show that the designed hyper-clustering-enhanced spatio-temporal deep neural network model can predict the number of available bikes more accurately. Compared with ten state-of-the-art benchmark methods, our model demonstrates better performance in reducing prediction error by 10.9%–43.1%. [ABSTRACT FROM AUTHOR]

• We propose two algorithms to optimize adversarial perturbation and patch. • We discover that CAEs can quickly decrease the classification accuracy. • We find that two weak attack patterns can be combined to perform stronger attack. • We demonstrate the effectiveness and robustness of the proposed method. Although deep learning has made great progress in many fields, they are still vulnerable to adversarial examples. Many methods for generating adversarial examples have been proposed, which either contain adversarial perturbation or patch. In this paper, we explore the method that creates compound adversarial examples including both perturbation and patch. We show that fusing two weak attack modes can produce more powerful adversarial examples, where the patch covers only 1 % of the pixels at random location in the image, and the perturbation changes only by 2/255 in the original pixel value (scale to 0–1). For both targeted attack and untargeted attack, compound attack can improve the generative efficiency of adversarial examples, and can attain higher attack success rate with fewer iteration steps. The compound adversarial examples successfully attack the models with defensive mechanisms that previously can defend perturbation attack or patch attack. Furthermore, the compound adversarial examples show good transferability on normal trained classifiers and adversarial trained classifiers. Experimental results on a series of widely used classifiers and defense models show that the proposed compound adversarial examples have strong robustness, high effectiveness, and good transferability. [ABSTRACT FROM AUTHOR]

• A semantic-based initialization for Knowledge Graph Embedding models is proposed. • Unseen entities can be represented with the inclusion of ontological information. • The results of baseline KGE models improve in up to 15 • Triple classification accuracy over facts containing unseen entities scales up to 75%. • The proposed initialization is model-agnostic and does not require retraining. Knowledge graphs (KGs) are one of the most common frameworks for knowledge representation. However, they suffer from a severe scalability problem that hinders their usage. KG embedding aims to provide a solution to this issue. Nonetheless, general approaches are incapable of representing and reasoning about information not previously contained in the graph. This paper proposes to leverage semantic and ontological information for a significant benefit of knowledge graph completion, focusing on triple classification. The goal of this task is to determine whether a given fact holds. Furthermore, this paper also considers the classification of facts that include entities that have not been seen during training, denoted out-of-knowledge-base or OOKB entities. An incremental method is presented, composed of six stages. Although the proposal can be applied to any KG embedding model, this work focuses on its application for semantic matching models, such as ComplEx and DistMult. Compared to other approaches, our proposal is model-agnostic, computationally inexpensive, and does not require retraining. The results show that triple classification accuracy scales up to 15% with the proposed approach, as well as accelerating the convergence of the model to its optimal solution. Furthermore, facts containing OOKB entities can be classified with a reasonable accuracy. [ABSTRACT FROM AUTHOR]

One-class classification is a machine learning problem, where training data has only one class. The objective is to determine if the input data is seen class or unseen class. Traditional deep learning algorithms are not suitable for this task since the algorithm can predict only class in training data. In this paper, the one-class classification algorithm using construction error of image transformation network (OCITN) is proposed. In particular, image transformation network (ITN) is introduced as a subtask, which transforms input image into one image, namely goal image. Moreover, the error of ITN, namely construction error (CE), is computed as a distance metric between the goal image and model output. ITN model is trained using only one-class images and is applied for testing images. Since the model is trained with only one-class images, the CE for one-class is small relative to other classes. Thus, one-class classification is made determining CE is large or small. The proposed method is experimented with using MNIST, Fashion MNIST, CIFAR10, CIFAR100, and Cat-vs-Dog datasets. OCITN shows good results where the goal image has high entropy. Additionally, the extension of OCITN, namely OCITNE, is implemented. This method shows the state of the art performance in MNIST (98.0), Fashion MNIST(95.6), and acceptable performance in CIFAR10(78.4). Furthermore, these methods provide high-speed processing, OCITN process 5291 images, and OCITNE 1261 images per second, 137 times and 33 times faster than state of the art. The source code used in this paper can be downloaded from: https://github.com/ToshiHayashi/OCITN. [ABSTRACT FROM AUTHOR]

• Our model combined the adaptive segmentation mask attack with feature space perturbation. • We adopt the binary cross entropy-loss function for semantic image segmentation. • We proposed a target attack method using multi-scale gradients. • Extensive experiments are conducted on four U-Net with two medical datasets. Research shows that deep neural networks are vulnerable to adversarial examples due to the highly linear nature of deep neural networks (DNNs). Therefore, adversarial examples involve security of deep learning. However, there is a lack of research on the impact of adversarial examples on the biomedical segmentation model. Since a large part of medical image problems are segmentation problems, this paper analyzes the impact of adversarial examples on image segmentation models based on deep learning. We propose to fool the biomedical segmentation model and generate target segmentation masks with feature space perturbations and cross-entropy loss function. Different from the traditional gradient-based attack methods, which usually use the gradient of the final loss function, this paper adopts a Multi-scale Attack (MSA) method based on multi-scale gradients. Extensive experiments to attack U-Net on the ISIC skin lesion segmentation challenge dataset and the glaucoma optic disc segmentation dataset have proved that the predicted mask generated by this method has a high intersection over union(IoU) and pixel accuracy with the target mask. Besides, the L 2 and L ∞ distances between the adversarial and clean examples are reduced compared with the state-of-the-art method. [ABSTRACT FROM AUTHOR]

Graph convolution networks (GCNs) have achieved remarkable success in processing non-Euclidean data. GCNs update the feature representations of each sample by aggregating the structure information from K -order (layer) neighborhood samples. Existing GCNs variants rely heavily on the K -th layer semantic information with K -order neighborhood information aggregating. However, semantic features from different convolution layers have distinct sample attributes. The single-layer semantic feature is only a one-sided feature representation. Besides, the semantic features of traditional GCNs will be oversmoothing with multi-layer structure information aggregates. In this paper, to solve the above-mentioned problem, we propose adaptive graph convolutional collaboration networks (AGCCNs) for the semi-supervised classification task. AGCCNs can fully use the different scales of discrimination information contained in the different convolutional layers. Specifically, AGCCNs utilize the attention mechanism to learn the relevance (contribution) coefficient of the deep semantic features from different convolution layers for the task, which aims to effectively discriminant their importance. After multiple optimizations, AGCCNs can adaptively learn the robust deep semantic features via the effective semantic fusion process between multi-layer semantic information. Compared with GCNs that only utilize the K -th layer semantic features, AGCCNs make the learned deep semantic features contain richer and more robust semantic information. What is more, our proposed AGCCNs can aggregate the appropriate K -order neighborhood information for each sample, which can relieve the oversmoothing issue of traditional GCNs and better generalize shallow GCNs to more deep layers. Abundant experimental results on several popular datasets demonstrate the superiority of our proposed AGCCNs compared with traditional GCNs. [ABSTRACT FROM AUTHOR]

Graph mining has been widely studied to analyze real-world graph properties and applied to various applications. In particular, graph subspace clustering performance, defined as partitioning high-dimensional graph data into several clusters by finding minimum weights for the edges, has been consistently improved by exploiting deep learning algorithms with Euclidean features extracted from Euclidean domains (image datasets). Most subspace clustering algorithms tend to extract features from the Euclidean domain to identify graph characteristics and structures, and hence are limited for real-world data applications in non-Euclidean domains. This paper proposes a self-supervised deep geometric subspace clustering algorithm optimized for non-Euclidean high-dimensional graph data by emphasizing spatial features and geometric structures while simultaneously reducing redundant nodes and edges. Quantitative and qualitative experimental results verified the proposed approach is effective for graph clustering compared with previous state-of-the-art algorithms on public datasets. [ABSTRACT FROM AUTHOR]

Multi-view clustering has attracted increasing attention by reason of its ability to leverage the complementarity of multi-view data. Existing multi-view clustering methods have explored nonnegative matrix factorization to decompose a matrix into multiple matrices for feature representations from multi-view data, which are not discriminative enough to deal with the natural data containing complex information. Moreover, most of multi-view clustering methods prioritize the consensus information among multi-view data, leaving a large amount of information redundant and the clustering performance deteriorated. To address these issues, this paper proposes a multi-view clustering framework that adopts a diversity loss for deep matrix factorization and reduces feature redundancy while obtaining more discriminative features. We then bridge the relation between deep auto-encoder and deep matrix factorization to optimize the objective function. This method avoids the challenges in the optimization process. Extensive experiments demonstrate that the proposed method is superior to state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• Contrastive autoencoder performs well for anomaly detection in time series. • Multi-grained contrasting can capture temporal dependency for time series. • Instance contrasting method can learn local invariant characteristics of time series. • Time-domain and frequency-domain data augmentation is effective. With the proliferation of the Internet of Things, a large amount of multivariate time series (MTS) data is being produced daily by industrial systems, corresponding in many cases to life-critical tasks. The recent anomaly detection researches focus on using deep learning methods to construct a normal profile for MTS. However, without proper constraints, these methods cannot capture the dependencies and dynamics of MTS and thus fail to model the normal pattern, resulting in unsatisfactory performance. This paper proposes CAE-AD, a novel contrastive autoencoder for anomaly detection in MTS, by introducing multi-grained contrasting methods to extract normal data pattern. First, to capture the temporal dependency of series, a projection layer is employed and a novel contextual contrasting method is applied to learn the robust temporal representation. Second, the projected series is transformed into two different views by using time-domain and frequency-domain data augmentation. Last, an instance contrasting method is proposed to learn local invariant characteristics. The experimental results show that CAE-AD achieves an F1-score ranging from 0.9119 to 0.9376 on the three public datasets, outperforming the baseline methods. [ABSTRACT FROM AUTHOR]