Search

This paper introduces Elegans-AI models, a class of neural networks that leverage the connectome topology of the Caenorhabditis elegans to design deep and reservoir architectures. Utilizing deep learning models inspired by the connectome, this paper leverages the evolutionary selection process to consolidate the functional arrangement of biological neurons within their networks. The initial goal involves the conversion of natural connectomes into artificial representations. The second objective centers on embedding the complex circuitry topology of artificial connectomes into both deep learning and deep reservoir networks, highlighting their neural-dynamic short-term and long-term memory and learning capabilities. Lastly, our third objective aims to establish structural explainability by examining the heterophilic/homophilic properties within the connectome and their impact on learning capabilities. In our study, the Elegans-AI models demonstrate superior performance compared to similar models that utilize either randomly rewired artificial connectomes or simulated bio-plausible ones. Notably, these Elegans-AI models achieve a top-1 accuracy of 99.99% on both Cifar10 and Cifar100, and 99.84% on MNIST Unsup. They do this with significantly fewer learning parameters, particularly when reservoir configurations of the connectome are used. Our findings indicate a clear connection between bio-plausible network patterns, the small-world characteristic, and learning outcomes, emphasizing the significant role of evolutionary optimization in shaping the topology of artificial neural networks for improved learning performance. [ABSTRACT FROM AUTHOR]

In personal identity recognition systems, detecting a person's age, gender, and language using voice signal characteristics is a crucial issue, especially because of the importance of security considerations. Age, gender, and language classification problems are important in signal processing because they are used to analyze and understand human behavior, interactions, and preferences. This can be especially useful in the fields of human-computer interaction, psychology, and social science research. In this paper, a new system for detecting a speaker's age, gender, and language based on deep learning models is presented. Deep learning models have shown great efficacy in various fields of signal processing. For this paper, a range of deep models were tested, including convolutional neural networks (CNNs), recurrent neural network (RNN), and a fine-tuning ResNet34 architecture. Additionally, techniques such as transfer learning were applied to improve the efficiency of the proposed system. The input voice signals are preprocessed by applying the spectro-temporal transform to obtain additional features that can be fed to the ResNet34 model, which is designed specifically for the task of voice signal processing. The dataset used in this paper was sourced from the Mozilla common voice initiative, which is dedicated to advancing speech recognition and language identification technologies. The performance of the proposed algorithm was evaluated in the presence of Gaussian noise to determine its robustness. The experimental results demonstrated that the proposed algorithm significantly outperformed basic algorithms and other deep neural networks in terms of age and gender recognition from voice signals. [ABSTRACT FROM AUTHOR]

Quantization methods for convolutional neural network models can be broadly categorized into post-training quantization (PTQ) and quantization aware training (QAT). While PTQ offers the advantage of requiring only a small portion of the data for quantization, the resulting quantized model may not be as effective as QAT. To address this limitation, this paper proposes a novel quantization function named Attention Round. Unlike traditional quantization function that map 32 bit floating-point value w to nearby quantization levels, Attention Round allows w to be mapped to all possible quantization levels in the entire quantization space, expanding the quantization optimization space. The possibilities of mapping w to different quantization levels are inversely correlated with the distance between w and the quantization levels, regulated by a Gaussian decay function. Furthermore, to tackle the challenge of mixed precision quantization, this paper introduces a lossy coding length measure to assign quantization precision to different layers of the model, eliminating the need for solving a combinatorial optimization problem. Experimental evaluations on various models demonstrate the effectiveness of the proposed method. Notably, for ResNet18 and MobileNetV2, the PTQ approach achieves comparable quantization performance to QAT while utilizing only 1024 training data and 10 min for the quantization process. • Attention Round quantization function expands the quantization optimization space. • Mixed precision allocation method improves mixed precision quantization efficiency. • Enriched lightweight CNNs contribute to applications in resource-limited scenarios. [ABSTRACT FROM AUTHOR]

U-Net, known for its simple yet efficient architecture, is widely utilized for image processing tasks and is particularly suitable for deployment on neuromorphic chips. This paper introduces the novel concept of Spiking-UNet for image processing, which combines the power of Spiking Neural Networks (SNNs) with the U-Net architecture. To achieve an efficient Spiking-UNet, we face two primary challenges: ensuring high-fidelity information propagation through the network via spikes and formulating an effective training strategy. To address the issue of information loss, we introduce multi-threshold spiking neurons, which improve the efficiency of information transmission within the Spiking-UNet. For the training strategy, we adopt a conversion and fine-tuning pipeline that leverage pre-trained U-Net models. During the conversion process, significant variability in data distribution across different parts is observed when utilizing skip connections. Therefore, we propose a connection-wise normalization method to prevent inaccurate firing rates. Furthermore, we adopt a flow-based training method to fine-tune the converted models, reducing time steps while preserving performance. Experimental results show that, on image segmentation and denoising, our Spiking-UNet achieves comparable performance to its non-spiking counterpart, surpassing existing SNN methods. Compared with the converted Spiking-UNet without fine-tuning, our Spiking-UNet reduces inference time by approximately 90%. This research broadens the application scope of SNNs in image processing and is expected to inspire further exploration in the field of neuromorphic engineering. The code for our Spiking-UNet implementation is available at https://github.com/SNNresearch/Spiking-UNet. [ABSTRACT FROM AUTHOR]

The task of motion prediction is pivotal for autonomous driving systems, providing crucial data to choose a vehicle behavior strategy within its surroundings. Existing motion prediction techniques primarily focus on predicting the future trajectory of each agent in the scene individually, utilizing its past trajectory data. In this paper, we introduce an end-to-end neural network methodology designed to predict the future behaviors of all dynamic objects in the environment. This approach leverages the occupancy map and the scene's motion flow. We are investigating various alternatives for constructing a deep encoder–decoder model called OFMPNet. This model uses a sequence of bird's-eye-view road images, occupancy grid, and prior motion flow as input data. The encoder of the model can incorporate transformer, attention-based, or convolutional units. The decoder considers the use of both convolutional modules and recurrent blocks. Additionally, we propose a novel time-weighted motion flow loss, whose application has shown a substantial decrease in end-point error. Our approach has achieved state-of-the-art results on the Waymo Occupancy and Flow Prediction benchmark, with a Soft IoU of 52.1% and an AUC of 76.75% on Flow-Grounded Occupancy. [ABSTRACT FROM AUTHOR]

Rumors circulating on social media platforms have consistently represented a substantial threat to societal security and stability. Both academia and the industry have dedicated heightened focus to addressing the issue of rumor detection. Recent research has made significant progress in using deep neural networks to model the textual content and propagation structure of rumors. However, these methods model rumor-related features at a coarse-grained level and do not take full advantage of the various contextual information associated with rumors. In this paper, we propose a new model called Hybrid Rumor Detection Model with Co-Attention Mechanisms(CoAHRD), which utilizes the original tweet content, social context information, and user information for rumor detection. First, we use the Fine-Grained Feature Learning(FGFL) algorithm to extract fine-grained features from the tweets. Based on this, we use the Graph Convolutional Network(GCN) to learn the Multi-relation graph propagation structure features of rumors. Next, we combine FGFL features with temporal encoding information to model the temporal structure of rumors. Then, we introduce a User Feature-Based Co-Attention Network(UFCoAN) to learn the tendency of different users to spread rumors. Finally, we fuse the above features through a fully connected layer and perform rumor detection. Extensive experiments on two publicly available datasets, PHEME and TWITTER15, show that our method outperforms the current mainstream methods. In particular, in terms of accuracy, our model improves by 0.9% and 1.6% over the best baseline method on the Twitter15 and PHEME datasets, respectively. • BERTWeet and saliency learning for fine-grained modelling of rumour text. • Using attention mechanisms to learn the temporal structure of rumours. • As a complementary task to rumour detection, segmentation of users into different groups using a user attention network. [ABSTRACT FROM AUTHOR]

Face recognition models and systems based on deep neural networks are vulnerable to adversarial examples. However, existing attacks on face recognition are either impractical or ineffective for black-box impersonation attacks in the physical world. In this paper, we propose EAP, an effective black-box impersonation attack method on face recognition in the physical world. EAP generates adversarial patches that can be printed by mobile and compact printers and attached to the source face to fool face recognition models and systems. To improve the transferability of adversarial patches, our approach incorporates random similarity transformations and image pyramid strategies, increasing input diversity. Furthermore, we introduce a meta-ensemble attack strategy that harnesses multiple pre-trained face models to extract common gradient features. We evaluate the effectiveness of EAP on two face datasets, using 16 state-of-the-art face recognition backbones, 9 heads, and 5 commercial systems. Moreover, we conduct physical experiments to substantiate its practicality. Our results demonstrate that EAP is capable of effectively executing impersonation attacks against state-of-the-art face recognition models and systems in both digital and physical environments. [ABSTRACT FROM AUTHOR]

Although adversarial attacks have revealed weaknesses in Deep Neural Networks (DNNs)-based aerial detectors, they present a new paradigm for concealing vulnerable assets from autonomous detection systems onboard satellites. Among them, adversarial patches were widely applied due to their physical realizability. Nonetheless, most existing adversarial patch-based attack methods are developed to hide objects from detectors to achieve a vanishing attack. This paper proposes a novel Adversarial Patch False Positive Creation Attack (APFP-CA) framework that enables aerial detectors to recognize non-existent objects, thereby realizing a creation attack. Concretely, the APFP-CA models the creation attack as a two-stage optimization problem. The first stage uses a well-designed efficient loss function to increase the objectness score of patches placed anywhere in the scene to achieve untargeted attacks. The second stage involves elaborating two category-dependent loss functions for fulfilling targeted attacks. Experiments conducted on diverse datasets and detectors demonstrate the effectiveness and universality of our method. Transfer attacks across different datasets and models validate the generalizability of the optimized adversarial patches. Finally, we perform several proportionally scaled experiments physically to illustrate that the optimized adversarial patches can successfully deceive aerial detectors in the physical world. The proposed approach offers a novel contribution to adversarial attacks against aerial imagery objectors and holds potential for practical applications in high-security systems. [ABSTRACT FROM AUTHOR]

Neural network compression is a widely used technique when deploying the neural network in energy-constrained and computation-constrained devices. To guarantee the compressed neural network is still usable without too much accuracy loss, the equivalence between the compressed networks and the original ones must be verified. However, the current verification approach can only verify the equivalence of the two neural networks in the same structure which is limited and unseen in real-world scenarios. In this paper, we proposed an equivalence verification method named VeriPrune, which can verify the equivalence of the deep neural networks without the structure limitation. In detail, we proposed an innovative virtual node completing method to solve the problem that node pruning causes structure change and invalidates the existing equivalence verification approaches. To demonstrate the feasibility and efficiency of the proposed approach, we conducted experiments on the public dataset with 49 DNNs and 1272 properties. The results show that the 83.9% properties, the 1067 of the total 1272 properties can be verified. The proposed VeriPrune can be further developed as a CASE tool for the industry settings. [Display omitted] [ABSTRACT FROM AUTHOR]

Recently, deep learning techniques have been widely applied to fault diagnosis due to their outstanding feature extraction abilities. The success of deep-learning-based fault diagnosis methods is highly dependent on the quantity and quality of the training data. In practical scenarios, it is challenging to obtain sufficient high-quality training data for fault diagnosis tasks due to complex environments, which would affect the effectiveness of the deep learning methods. In this paper, a new fault diagnosis method is proposed for motor bearing fault diagnosis under small samples. The Siamese neural networks (SNNs) are employed to extract the fault features. A multi-stage training strategy is proposed to train the SNNs with the aim to prevent the training stagnation problem and handle the small sample problem. A multi-source feature fusion network is developed to make full use of the multi-source sensor data by fusing the extracted fault features for further fault diagnosis. The proposed method is applied to motor bearing fault diagnosis on two real-world datasets. Experimental results demonstrate the effectiveness and feasibility of the introduced method for motor bearing fault diagnosis under small samples. [ABSTRACT FROM AUTHOR]

In this paper, we present an extensive review of the most recent works on Alzheimer's disease (AD) prediction, focusing on Moderate Cognitive Impairment (MCI) conversion prediction. We aimed to identify the most useful brain-magnetic resonance imaging (MRI) biomarkers and deep learning frameworks used for prediction. To achieve this, we analyzed more than 130 studies and reviewed 7 articles. A closer examination revealed that the hippocampus is an important region of interest (ROI) affected early by AD, and many related features help detect the disease in its early stages. However, when considered alone, this ROI is not sufficient to ensure high prediction performance. Therefore, several other brain regions can also provide additional information to improve prediction accuracy. Concerning state-of-the-art deep neural networks, the U-Net represents the most efficient architecture for hippocampus segmentation. The RESU-Net architecture achieved the highest Dice Similarity Coefficient (DSC) value, equal to 94%.For MCI conversion prediction, the best results were obtained by two models identifying significant landmarks from the entire brain for classification. The multi-stream convolutional neural network achieved the best Area Under the Curve (AUC) and specificity of 94.39% and 99.70%, respectively. Finally, a region ensemble model delivered the highest accuracy of 85.90%, highlighting the need for further research to address this challenging problem. [ABSTRACT FROM AUTHOR]

Deep cross-modal clustering has been developing rapidly and attracted considerable attention in recent years. It aims to pursue a consistent subspace from different modalities with deep neural networks and achieves remarkable clustering performance. However, most existing methods do not simultaneously consider the inherently diverse information of each modality and the neighbour geometric structure over cross-modal data, which inevitably degrades the cluster structure revealed by the common subspace. In this paper, we propose a novel method named Deep Cross-Modal Subspace Clustering with Contrastive Neighbour Embedding (DCSC-CNE) to address the above challenge. DCSC-CNE maintains the inherent independence of each modality while concurrently uncovering consistent information across diverse modalities. In addition, we introduce a contrastive neighbour graph in the proposed deep cross-modal subspace clustering framework by performing contrastive learning between positive and negative samples, to highlight the underlying neighbour geometry of the original data and learn discriminative latent (subspace) representations. In this way, DCSC-CNE integrates the consistent-inherent learning and the contrastive neighbour embedding into a unified deep learning framework. Experimental results demonstrate that the proposed method can significantly improve the cross-modal subspace clustering performance compared with state-of-the-art methods on six benchmark datasets. [ABSTRACT FROM AUTHOR]

Neural network pruning is a critical field aimed at reducing the infrastructure costs of neural networks by removing parameters. Traditional methods follow a fixed paradigm including pretraining, pruning, and fine-tuning. Despite the close relationship among these three stages, most pruning methods treat them as independent processes. In this paper, we propose a novel two-stage pruning method, which includes pretraining a network that is instructive for subsequent pruning, and a unified optimization model that integrates pruning and fine-tuning. Specifically, in the first stage, we design a group sparse regularized model for pretraining. This model not only safeguards the network from irreversible damage but also offers valuable insights for the pruning process. In the second stage, we introduce an element-wise sparse regularization into pruning model. This model enables us to pinpoint sparse weights more precisely than pretrained network. It automatically derives effective pruning criteria, and omits the step of fine-tuning. To implement the two-stage process in practice, we utilize stochastic gradient algorithm for the pretraining and design a threshold algorithm for pruning stage. Extensive experiments confirm the competitive performance of our proposed method in terms of both accuracy and memory cost when compared to various benchmarks. Furthermore, ablation experiments validate the effectiveness of the proposed pretraining model's guidance for the pruning process. • We propose SOGP to enhance the connection between pretraining and pruning. • We propose a novel non-convex group sparse regularization GT l 1 . • We construct a unified optimization model to integrate pruning and fine-tuning. • Extensive experiments validate the good pruning and accuracy performance of SOGP. [ABSTRACT FROM AUTHOR]

Recently, various normalization layers have been proposed to stabilize the training of deep neural networks. Among them, group normalization is a generalization of layer normalization and instance normalization by allowing a degree of freedom in the number of groups it uses. However, to determine the optimal number of groups, trial-and-error-based hyperparameter tuning is required, and such experiments are time-consuming. In this study, we discuss a reasonable method for setting the number of groups. First, we find that the number of groups influences the gradient behavior of the group normalization layer. Based on this observation, we derive the ideal number of groups, which calibrates the gradient scale to facilitate gradient descent optimization. This paper is the first to propose an optimal number of groups that is theoretically grounded, architecture-aware, and can provide a proper value in a layer-wise manner for all layers. The proposed method exhibited improved performance over existing methods in numerous neural network architectures, tasks, and datasets. • We propose a method to determine the number of groups of group normalization. • A theoretical analysis of group normalization with activation function is provided. • The proposed method is validated against various practical problems. [ABSTRACT FROM AUTHOR]

Model pruning is a commonly used technique for compressing DNNs and reducing computation requirements to accelerate inference. However, the required granularity of pruning varies across different application scenarios, making it difficult and cumbersome to customize different pruning methods for each hardware or computing platform. Therefore, a unified framework is necessary to accommodate various levels of granularity of pruning. Furthermore, some available methods require additional fine-tuning or model retraining to restore accuracy, which can result in significant time costs. With this motivation, this paper proposes a Multi-Granularity Pruning Framework, namely MGPF, to obtain sparse models of different granularity without fine-tuning the remaining connections. Specifically, a deformable soft mask is introduced as the pruning initiator to achieve different levels of pruning granularity, such as weight pruning, channel pruning, and filter pruning, etc. The model parameters and soft masks are jointly trained, and we just apply L 1 regularization on soft masks for sparsity to ensure that the model can be repaired during training without fine-tuning or retraining. After pruning, the soft masks are absorbed into the model parameters in the form of element product without changing the model structures. Experimental results on three image classification benchmarks CIFAR-10/100 and ImageNet-1K demonstrate the effectiveness of our method for various CNN architectures, datasets, and pruning rates. Particularly, for ResNet-50 on ImageNet-1K, we achieve a higher accuracy under the pruning rate of 98% for unstructured pruning which leads the advanced method by 12%. [ABSTRACT FROM AUTHOR]

Traditional knowledge distillation approaches are typically designed for specific tasks, as they primarily distilling deep features from intermediate layers of a neural network, generally with ingeniously designed knowledge representations, which increase the difficulty of model development and interpretation. In contrast, we empirically show that a soft logits knowledge distillation is enough to significantly narrow the teacher–student model performance gap. In this paper, the S oft H ybrid K nowledge D istillation (SHKD) is proposed to generate representative features from the input and output location of models, and applicable to various teacher and student architectures. Specifically, the additional mixup task is first applied to the same batch of training images, which enriches the high-order feature information transferred by the model. And then, a top-k guided specification filter is added to the teacher model, to reduce the bias induced throughout the knowledge transfer process. Finally, the SHKD utilizes the same and different teacher–student model pairs of the deep neural network, which can be easily adjusted to the computational resources available in the current world. Extensive experiments are conducted to examine the efficiency, and show that the SHKD outperforms state-of-the-art knowledge distillation methods. With the SHKD, the high performance of models will be maintained, and the gap between the teacher model and the student model can be reduced, making the construction of deep neural networks more convenient to the unknown scenarios. The source code is available at https://github.com/lambett/SHKD. [ABSTRACT FROM AUTHOR]

Inference centers need more data to have a more comprehensive and beneficial learning model, and for this purpose, they need to collect data from data providers. On the other hand, data providers are cautious about delivering their datasets to inference centers in terms of privacy considerations. In this paper, by modifying the structure of the autoencoder, we present a method that manages the utility-privacy trade-off well. To be more precise, the data is first compressed using the encoder, then confidential and non-confidential features are separated and uncorrelated using the classifier. The confidential feature is appropriately combined with noise, and the non-confidential feature is enhanced, and at the end, data with the original data format is produced by the decoder. The suggested architecture additionally enables data providers to modify the degree of privacy needed for private features and the level of utility for non-private features. The proposed method has been examined for both image and categorical databases, and the results show a significant performance improvement compared to previous methods. [ABSTRACT FROM AUTHOR]

As the deep neural networks (DNNs) with satisfied expression ability usually require large scale to gain adequate performance, and deploying large-scale DNNs on resource-limited environments is still a challenge, neural network compression becomes a hot topic nowadays. Among the multiple compression methods, tensor decomposition reveals many specific advantages, such as regular data structure comes from linear algebra, convenient approach of training from scratch, ideal compression ratio, etc. Nevertheless, for some compact neural modules such as two-dimensional/three-dimensional (2D/3D) convolutional kernels, traditional tensor decomposition in the way of approximation has encountered intractable obstacles. Fortunately, some works utilize the tensor mapping approach, which just regards the data structure of tensor decomposition as neural layers, to reconstruct the convolutional kernel into a new lightweight module with several thinner convolutional kernels. The only two flies in the ointment are there lack necessary theories of tensor mapping, and there is still no tensor mapping way to compress high-order three-dimensional convolutional neural networks (3DCNNs). In this paper, we first deeply analyze the tensor mapping theory including convergence and precision, which separately establishes the rationality of tensor mapping and its superiority over the traditional tensor approximation, according to the Lottery Ticket Hypothesis. Then we propose an efficient method termed as 3D-KCPNet, to compress 3DCNNs based on the novel Kronecker canonical polyadic (KCP) tensor decomposition. The proposed method can not only compress the 3D convolutional kernels, but also convert a 3D convolution to efficient 1 × 1 × 1 and 2D depthwise convolutions. The experiments on the video recognition datasets VIVA Challenge, UCF11, UCF50, and UCF101 show that the accuracy of 3D-KCPNet can surpass its original baseline model and the corresponding tensor approximation model. [ABSTRACT FROM AUTHOR]