Search

Knowledge distillation is a form of model compression that allows artificial neural networks of different sizes to learn from one another. Its main application is the compactification of large deep neural networks to free up computational resources, in particular on edge devices. In this article, we consider proton-proton collisions at the High-Luminosity Large Hadron Collider (HL-LHC) and demonstrate a successful knowledge transfer from an event-level graph neural network (GNN) to a particle-level small deep neural network (DNN). Our algorithm, DistillNet, is a DNN that is trained to learn about the provenance of particles, as provided by the soft labels that are the GNN outputs, to predict whether or not a particle originates from the primary interaction vertex. The results indicate that for this problem, which is one of the main challenges at the HL-LHC, there is minimal loss during the transfer of knowledge to the small student network, while improving significantly the computational resource needs compared to the teacher. This is demonstrated for the distilled student network on a CPU, as well as for a quantized and pruned student network deployed on an field programmable gate array. Our study proves that knowledge transfer between networks of different complexity can be used for fast artificial intelligence (AI) in high-energy physics that improves the expressiveness of observables over non-AI-based reconstruction algorithms. Such an approach can become essential at the HL-LHC experiments, e.g. to comply with the resource budget of their trigger stages.

Introduction: The accurate and rapid detection of ginseng fruits in natural environments is crucial for the development of intelligent harvesting equipment for ginseng fruits. Due to the complexity and density of the growth environment of ginseng fruits, some newer visual detection methods currently fail to meet the requirements for accurate and rapid detection of ginseng fruits. Therefore, this study proposes the YOLO-Ginseng detection method. Methods: Firstly, this detection method innovatively proposes a plug-and-play deep hierarchical perception feature extraction module called C3f-RN, which incorporates a sliding window mechanism. Its unique structure enables the interactive processing of cross-window feature information, expanding the deep perception field of the network while effectively preserving important weight information. This addresses the detection challenges caused by occlusion or overlapping of ginseng fruits, significantly reducing the overall missed detection rate and improving the long-distance detection performance of ginseng fruits; Secondly, in order to maintain the balance between YOLO-Ginseng detection precision and speed, this study employs a mature channel pruning algorithm to compress the model. Results: The experimental results demonstrate that the compressed YOLO-Ginseng achieves an average precision of 95.6%, which is a 2.4% improvement compared to YOLOv5s and only a 0.2% decrease compared to the uncompressed version. The inference time of the model reaches 7.4ms. The compressed model exhibits reductions of 76.4%, 79.3%, and 74.2% in terms of model weight size, parameter count, and computational load, respectively. Discussion: Compared to other models, YOLO-Ginseng demonstrates superior overall detection performance. During the model deployment experiments, YOLO-Ginseng successfully performs real-time detection of ginseng fruits on the Jetson Orin Nano computing device, exhibiting good detection results. The average detection speed reaches 24.9 fps. The above results verify the effectiveness and practicability of YOLO-Ginseng, which creates primary conditions for the development of intelligent ginseng fruit picking equipment. [ABSTRACT FROM AUTHOR]

Regularization is a common technique to prune neural networks which removes some weights with acceptable accuracy. Moreover, the alternating direction method of multipliers (ADMM) can quickly solve regularization problems. However, for learning sparsity in deep learning, ADMM does not ensure convergence and may stop too early, since the loss is usually not convex and the stopping condition is not suitable for ADMM. In this work, we combine linearized the subproblem and iterations on mini-batches to overcome the shortcomings. To avoid ADMM converging to a local optimum, we linearize the ADMM-based subproblem. After linearizing the ADMM-based sub-problem, the objective function is convex, which can improve the convergence of ADMM theoretically. To alleviate the issue that ADMM stops too early, we increase the number of ADMM iterations by repeating the ADMM framework on mini-batches. After removing redundant parameters, the sparse model is fine-tuned in the standard way on public datasets. The experiment shows no significant accuracy reduction on CIFAR-10 when our method removes 90.55% of parameters from ResNet-20.Besides, with no drop in Top-1 accuracy on CIFAR-10, we prune 89.51% and 98.80% of the parameters for ResNet-56 and VGG-16, respectively. For ResNet-50, 73.52% of parameters are pruned without accuracy loss on ImageNet. Experimental results prove our methods outperform recent state-of-the-art pruning methods. [ABSTRACT FROM AUTHOR]

Knowledge distillation can transfer knowledge from a powerful yet cumbersome teacher model to a less-parameterized student model, thus effectively achieving model compression. Various knowledge distillation methods have mainly focused on the task of knowledge transfer, and distillation location selection, which in turn increases the difficulty of model interpretation on the one hand, and on the other hand, there have been few works on the role of the teacher classifier in distillation. In this study, we propose a novel collaborative multi-knowledge distillation under the influence of softmax regression representation. Firstly, we propose a stage-wise logit knowledge distillation, where the teacher classifier is used as an auxiliary structure to align the features of the student and teacher models. By leveraging the teacher classifier, the student features are aligned with the teacher features in the logits space, eliminating the need for a complex feature projector that requires extensive computation to match the features between the teacher and student models. Secondly, considering the teacher classifier’s adaptability to classification features, we introduce a stage-wise feature knowledge distillation. This mechanism maps the features of the student model to a latent space with the same dimensions as the features of the teacher model, guiding the student’s features to align with the teacher’s final features using a Mean Square Error (MSE) loss. Finally, we propose a pseudo-teacher knowledge distillation loss to optimize the modeling of the deformation relationship between the student and teacher features. This loss provides additional gradient optimization information for the parameters of the feature projector. Extensive experiments on CIFAR-100 and ImageNet datasets demonstrate the superiority of the proposed model compared with the state-of-the-art methods. The code is available at [ABSTRACT FROM AUTHOR]

As globalization deepens and mobile technology rapidly advances, the demand for crosslinguistic communication has been steadily increasing, making real-time speech translation systems a research focus. However, given the limited computational capacity and storage space of mobile devices, optimizing system performance while maintaining translation quality has become a critical challenge. Current optimization approaches for real-time speech translation systems primarily focus on improvements to model architectures and hardware acceleration, often neglecting a systematic study of model compression. This is particularly evident when handling real-time data, where achieving both high efficiency and translation accuracy remains difficult. To address these challenges, a model compression method based on the connectionist temporal classification (CTC) criterion was proposed, along with an in-depth study of parameter compression tailored for mobile applications. The research focuses on two key areas: first, model compression techniques based on the CTC criterion were explored to enhance the efficiency of real-time speech translation; second, parameter compression methods were investigated to significantly reduce resource consumption in mobile applications while preserving translation quality. The aim of this study is to improve the performance and user experience of real-time speech translation systems on mobile devices. [ABSTRACT FROM AUTHOR]

Copyright of Electric Power Automation Equipment / Dianli Zidonghua Shebei is the property of Electric Power Automation Equipment Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

With the rapid growth of big data, extracting meaningful knowledge from data is crucial for machine learning. The existing Swarm Learning data collaboration models face challenges such as data security, model security, high communication overhead, and model performance optimization. To address this, we propose the Swarm Mutual Learning (SML). Firstly, we introduce an Adaptive Mutual Distillation Algorithm that dynamically controls the learning intensity based on distillation weights and strength, enhancing the efficiency of knowledge extraction and transfer during mutual distillation. Secondly, we design a Global Parameter Aggregation Algorithm based on homomorphic encryption, coupled with a Dynamic Gradient Decomposition Algorithm using singular value decomposition. This allows the model to aggregate parameters in ciphertext, significantly reducing communication overhead during uploads and downloads. Finally, we validate the proposed methods on real datasets, demonstrating their effectiveness and efficiency in model updates. On the MNIST dataset and CIFAR-10 dataset, the local model accuracies reached 95.02% and 55.26%, respectively, surpassing those of the comparative models. Furthermore, while ensuring the security of the aggregation process, we significantly reduced the communication overhead for uploading and downloading. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The rapid evolution of deep learning has led to significant achievements in computer vision, primarily driven by complex convolutional neural networks (CNNs). However, the increasing depth and parameter count of these networks often result in overfitting and elevated computational demands. Knowledge distillation (KD) has emerged as a promising technique to address these issues by transferring knowledge from a large, well-trained teacher model to a more compact student model. This paper introduces a novel knowledge distillation method that simplifies the distillation process and narrows the performance gap between teacher and student models without relying on intricate knowledge representations. Our approach leverages a unique teacher network architecture designed to enhance the efficiency and effectiveness of knowledge transfer. Additionally, we introduce a streamlined teacher network architecture that transfers knowledge effectively through a simplified distillation process, enabling the student model to achieve high accuracy with reduced computational demands. Comprehensive experiments conducted on the CIFAR-10 dataset demonstrate that our proposed model achieves superior performance compared to traditional KD methods and established architectures such as ResNet and VGG networks. The proposed method not only maintains high accuracy but also significantly reduces training and validation losses. Key findings highlight the optimal hyperparameter settings (temperature T = 15.0 and smoothing factor α = 0.7), which yield the highest validation accuracy and lowest loss values. This research contributes to the theoretical and practical advancements in knowledge distillation, providing a robust framework for future applications and research in neural network compression and optimization. The simplicity and efficiency of our approach pave the way for more accessible and scalable solutions in deep learning model deployment. [ABSTRACT FROM AUTHOR]

Deep networks play a crucial role in the recognition of agricultural diseases. However, these networks often come with numerous parameters and large sizes, posing a challenge for direct deployment on resource-limited edge computing devices for plant protection robots. To tackle this challenge for recognizing cotton diseases on the edge device, we adopt knowledge distillation to compress the big networks, aiming to reduce the number of parameters and the computational complexity of the networks. In order to get excellent performance, we conduct combined comparison experiments from three aspects: teacher network, student network and distillation algorithm. The teacher networks contain three classical convolutional neural networks, while the student networks include six lightweight networks in two categories of homogeneous and heterogeneous structures. In addition, we investigate nine distillation algorithms using spot-adaptive strategy. The results demonstrate that the combination of DenseNet40 as the teacher and ShuffleNetV2 as the student show best performance when using NST algorithm, yielding a recognition accuracy of 90.59% and reducing FLOPs from 0.29 G to 0.045 G. The proposed method can facilitate the lightweighting of the model for recognizing cotton diseases while maintaining high recognition accuracy and offer a practical solution for deploying deep models on edge computing devices. [ABSTRACT FROM AUTHOR]

Transferring the extensive and varied knowledge contained within multiple complex models into a more compact student model poses significant challenges in multi-teacher knowledge distillation. Traditional distillation approaches often fall short in this context, as they struggle to fully capture and integrate the wide range of valuable information from each teacher. The variation in the knowledge offered by various teacher models complicates the student model's ability to learn effectively and generalize well, ultimately resulting in subpar results. To overcome these constraints, We introduce an innovative method that integrates both localized and globalized frequency attention techniques, aiming to substantially enhance the distillation process. By simultaneously focusing on fine-grained local details and broad global patterns, our approach allows the student model to more effectively grasp the complex and diverse information provided by each teacher, therefore enhancing its learning capability. This dual-attention mechanism allows for a more balanced assimilation of specific details and generalized concepts, resulting in a more robust and accurate student model. Extensive experimental evaluations on standard benchmarks demonstrate that our methodology reliably exceeds the performance of current multi-teacher distillation methods, yielding outstanding outcomes regarding both performance and robustness. Specifically, our approach achieves an average performance improvement of 0.55% over CA-MKD, with a 1.05% gain in optimal conditions. These findings suggest that frequency-based attention mechanisms can unlock new potential in knowledge distillation, model compression, and transfer learning. [ABSTRACT FROM AUTHOR]

Neural network model compression is very important to achieve model deployment based on the memory and storage available in different computing systems. Generally, the continuous drive for higher accuracy in these models increases their size and complexity, making it challenging to deploy them on resource-constrained computing environments. This article proposes various algorithms for model compression by exploiting weight characteristics and conducts an in-depth study of their performance. The algorithms involve manipulating exponents and mantissa in the floating-point representations of weights. In addition, we also present a retraining method that uses the proposed algorithms to further reduce the size of pre-trained models. The results presented in this article are mainly on BFloat16 floating-point format. The proposed weight manipulation algorithms save at least 20% of memory on state-of-the-art image classification models with very minor accuracy loss. This loss is bridged using the retraining method that saves at least 30% of memory, with potential memory savings of up to 43%. We compare the performance of the proposed methods against the state-of-the-art model compression techniques in terms of accuracy, memory savings, inference time, and energy. [ABSTRACT FROM AUTHOR]

Deep learning (DL) models have been deployed in various platforms, including resource-constrained environments such as edge computing, smartphones, and personal devices. Such deployment requires models to have smaller sizes and memory footprints. To this end, many model compression techniques proposed in the literature successfully reduce model sizes and maintain comparable accuracy. However, the robustness of compressed DL models against class imbalance, a natural phenomenon in real-life datasets, is still under-explored. We present a comprehensive experimental study of the performance and robustness of compressed DL models when trained on class-imbalanced datasets. We investigate the robustness of compressed DL models using three popular compression techniques (pruning, quantization, and knowledge distillation) with class-imbalanced variants of the CIFAR-10 dataset and show that compressed DL models are not robust against class imbalance in training datasets. We also show that different compression techniques have varying degrees of impact on the robustness of compressed DL models. [ABSTRACT FROM AUTHOR]

Pruning and quantization are among the most widely used techniques for deep learning model compression. Their combined application holds the potential for even greater performance gains. Most existing works combine pruning and quantization sequentially. However, this separation makes it difficult to fully leverage their complementarity and exploit the potential benefits of joint optimization. To address the limitations of existing methods, we propose A-JOPQ (adaptive joint optimization of pruning and quantization), an adaptive joint optimization framework for pruning and quantization. Starting with a deep neural network, A-JOPQ first constructs a pruning network through adaptive mutual learning with a quantization network. This process compensates for the loss of structural information during pruning. Subsequently, the pruning network is incrementally quantized using adaptive multi-teacher knowledge distillation of itself and the original uncompressed model. This approach effectively mitigates the adverse effects of quantization. Finally, A-JOPQ generates a pruning-quantization network that achieves significant model compression while maintaining high accuracy. Extensive experiments conducted on several public datasets demonstrate the superiority of our proposed method. Compared to existing methods, A-JOPQ achieves higher accuracy with a smaller model size. Additionally, we extend A-JOPQ to federated learning (FL) settings. Simulation experiments show that A-JOPQ can enhance FL by enabling resource-limited clients to participate effectively. [ABSTRACT FROM AUTHOR]

Although deep neural networks have developed at a high level, the large computational requirement limits the deployment in end devices. To this end, a variety of model compression and acceleration techniques have been developed. Among these, knowledge distillation has emerged as a popular approach that involves training a small student model to mimic the performance of a larger teacher model. However, the student architectures used in existing knowledge distillation are not optimal and always have redundancy, which raises questions about the validity of this assumption in practice. This study aims to investigate this assumption and empirically demonstrate that student models could contain redundancy, which can be removed through pruning without significant performance degradation. Therefore, we propose a novel pruning method to eliminate redundancy in student models. Instead of using traditional post-training pruning methods, we perform pruning during knowledge distillation (PDD) to prevent any loss of important information from the teacher models to the student models. This is achieved by designing a differentiable mask for each convolutional layer, which can dynamically adjust the channels to be pruned based on the loss. Experimental results show that with ResNet20 as the student model and ResNet56 as the teacher model, a 39.53%-FLOPs reduction was achieved by removing 32.77% of parameters, while the top-1 accuracy on CIFAR10 increased by 0.17%. With VGG11 as the student model and VGG16 as the teacher model, a 74.96%-FLOPs reduction was achieved by removing 76.43% of parameters, with only a loss of 1.34% in the top-1 accuracy on CIFAR10. Our code is available at https://github.com/YihangZhou0424/PDD-Pruning-during-distillation. [ABSTRACT FROM AUTHOR]

Remote sensing image (RSI) scene classification aims to identify semantic categories in RSI using neural networks. However, high-performance deep neural networks typically demand substantial storage and computational resources, making practical deployment challenging. Knowledge distillation has emerged as an effective technique for developing compact models that maintain high classification accuracy in RSI tasks. Existing knowledge distillation methods often overlook the high inter-class similarity in RSI scenes, leading to low-confidence soft labels from the teacher model, which can mislead the student model. Conversely, overly confident soft labels may discard valuable non-target information. Additionally, the significant intra-class variability in RSI contributes to instability in the model's decision boundaries. To address these challenges, we propose an efficient method called instance-level scaling and dynamic margin-alignment knowledge distillation (ISDM) for RSI scene classification. To balance the target and non-target class influence, we apply an entropy regularization loss to scale the teacher model's target class at the instance level. Moreover, we introduce dynamic margin alignment between the student and teacher models to improve the student's discriminative capability. By optimizing soft labels and enhancing the student's ability to distinguish between classes, our method reduces the effects of inter-class similarity and intra-class variability. Experimental results on three public RSI scene classification datasets (AID, UCMerced, and NWPU-RESISC) demonstrate that our method achieves state-of-the-art performance across all teacher–student pairs with lower computational costs. Additionally, we validate the generalization of our approach on general datasets, including CIFAR-100 and ImageNet-1k. [ABSTRACT FROM AUTHOR]

Pre-trained language models perform well in various natural language processing tasks. However, their large number of parameters poses significant challenges for edge devices with limited resources, greatly limiting their application in practical deployment. This paper introduces a simple and efficient method called Autocorrelation Matrix Knowledge Distillation (AMKD), aimed at improving the performance of smaller BERT models for specific tasks and making them more applicable in practical deployment scenarios. The AMKD method effectively captures the relationships between features using the autocorrelation matrix, enabling the student model to learn not only the performance of individual features from the teacher model but also the correlations among these features. Additionally, it addresses the issue of dimensional mismatch between the hidden states of the student and teacher models. Even in cases where the dimensions are smaller, AMKD retains the essential features from the teacher model, thereby minimizing information loss. Experimental results demonstrate that BERTTINY-AMKD outperforms traditional distillation methods and baseline models, achieving an average score of 83.6% on GLUE tasks. This represents a 4.1% improvement over BERTTINY-KD and exceeds the performance of BERT4-PKD and DistilBERT4 by 2.6% and 3.9%, respectively. Moreover, despite having only 13.3% of the parameters of BERTBASE, the BERTTINY-AMKD model retains over 96.3% of the performance of the teacher model, BERTBASE. [ABSTRACT FROM AUTHOR]

Filter pruning is an essential method for compressing and accelerating deep neural networks on computationally restricted devices. Despite recognizing the high correlation between filter redundancy and samples, existing methods primarily focus on independently searching for optimal subnetworks from individual input while ignoring the relationships among different inputs. In this paper, we propose a novel approach called Gated Filter Pruning based on Sample Manifold Relationships, which exploits and aligns the manifold relationships of all samples during training to obtain an optimal subnetwork. Firstly, we introduce a Gated Filter Normalization Module (GFNM) that excavates the manifold information of each sample, applicable to the operator level without adding many additional parameters. GFNM incorporates explainable control variables jointly optimized with convolutional weights, explicitly determining the competition and cooperation among filters during training. Subsequently, Manifold Regularized Pruning Module (MRPM) measures the manifold relationships between samples and subnetworks, efficiently regularizing the solution space of sample-network pairs. The manifold relationships between samples and subnetworks are aligned in training to derive an effective subnetwork for all input samples. Extensive experimental results validate the effectiveness of our method, demonstrating competitive performance in terms of accuracy and computational cost compared to state-of-the-art (SOTA) methods. [ABSTRACT FROM AUTHOR]

Deep networks play a crucial role in the recognition of agricultural diseases. However, these networks often come with numerous parameters and large sizes, posing a challenge for direct deployment on resource-limited edge computing devices for plant protection robots. To tackle this challenge for recognizing cotton diseases on the edge device, we adopt knowledge distillation to compress the big networks, aiming to reduce the number of parameters and the computational complexity of the networks. In order to get excellent performance, we conduct combined comparison experiments from three aspects: teacher network, student network and distillation algorithm. The teacher networks contain three classical convolutional neural networks, while the student networks include six lightweight networks in two categories of homogeneous and heterogeneous structures. In addition, we investigate nine distillation algorithms using spot-adaptive strategy. The results demonstrate that the combination of DenseNet40 as the teacher and ShuffleNetV2 as the student show best performance when using NST algorithm, yielding a recognition accuracy of 90.59% and reducing FLOPs from 0.29 G to 0.045 G. The proposed method can facilitate the lightweighting of the model for recognizing cotton diseases while maintaining high recognition accuracy and offer a practical solution for deploying deep models on edge computing devices. [ABSTRACT FROM AUTHOR]

Currently, image recognition based on deep neural networks has become the mainstream direction of research; therefore, significant progress has been made in its application in the field of tea detection. Many deep models exhibit high recognition rates in tea leaves detection. However, deploying these models directly on tea-picking equipment in natural environments is impractical; the extremely high parameters and computational complexity of these models make it challenging to perform real-time tea leaves detection. Meanwhile, lightweight models struggle to achieve competitive detection accuracy; therefore, this paper addresses the issue of computational resource constraints in remote mountain areas and proposes Reconstructed Feature and Dual Distillation (RFDD) to enhance the detection capability of lightweight models for tea leaves. In our method, the Reconstructed Feature selectively masks the feature of the student model based on the spatial attention map of the teacher model; it utilizes a generation block to force the student model to generate the teacher's full feature. The Dual Distillation comprises Decoupled Distillation and Global Distillation. Decoupled Distillation divides the reconstructed feature into foreground and background features based on the Ground-Truth. This compels the student model to allocate different attention to foreground and background, focusing on their critical pixels and channels. However, Decoupled Distillation leads to the loss of relation knowledge between foreground and background pixels. Therefore, we further perform Global Distillation to extract this lost knowledge. Since RFDD only requires loss calculation on feature map, it can be easily applied to various detectors. We conducted experiments on detectors with different frameworks, using a tea dataset collected at the Huangshan Houkui Tea Plantation. The experimental results indicate that, under the guidance of RFDD, the student detectors have achieved performance improvements to varying degrees. For instance, a one-stage detector like RetinaNet (ResNet-50) experienced a 3.14% increase in Average Precision (AP) after RFDD guidance. Similarly, a two-stage model like Faster RCNN (ResNet-50) obtained a 3.53% improvement in AP. This offers promising prospects for lightweight models to efficiently perform real-time tea leaves detection tasks. [ABSTRACT FROM AUTHOR]

Knowledge distillation is a commonly used method for model compression that has been widely utilized in various computer vision tasks. Many efforts have utilized attention mechanisms to guide the student networks during training, encouraging them to mimic the important features of the teacher. However, most of these efforts use either the channel attention map or the spatial attention map to guide the student, ignoring the importance of positional features. In this paper, we propose a new distillation framework transferring global and positional features (DFGPD), which consists of three parts: global and positional distillation, a generic teacher framework and a two-stage distillation method. DFGPD takes positional information into consideration for a more effective distillation process. We conduct extensive comparison experiments, ablation studies, and sensitivity studies to demonstrate the effectiveness and stability of DFGPD. Our results show that (1) DFGPD achieves comparable or even better performance compared to state-of-the-art methods; (2) DFGPD can alleviate the bigger-models-not-always-better-teachers issue to a certain extent. [ABSTRACT FROM AUTHOR]

Large-scale deep learning algorithms have emerged as the primary technology for underwater target detection, demonstrating exceptional inference effectiveness and accuracy. However, the real-time capabilities of these high-accuracy algorithms rely heavily on high-performance computing resources like CPUs and GPUs. It presents a challenge for deploying them on underwater embedded devices, where communication is limited and computational and energy resources are constrained. To overcome this, this paper focuses on constructing a lightweight yet highly accurate deep learning model suitable for real-time underwater target detection on edge devices. We develop a new lightweight model, named YOLO-TN, for real-time underwater object recognition on edge devices using a self-constructed image dataset captured by an underwater unmanned vehicle. This model is obtained by compressing the classical YOLO-V5, utilizing a globally differentiable deep neural architecture search method and a network pruning technique. Experimental results show that the YOLO-TN achieves a mean average precision (mAP) of 0.5425 and an inference speed of 28.6 FPS on embedded devices, while its parameter size is between 0.4 M and 0.6 M. This performance is a fifth of the parameter size and twelve times the FPS of the YOLO-V5 model, with almost no loss in inference accuracy. In conclusion, this framework significantly enhances the feasibility of deploying large-scale deep learning models on edge devices with high precision and compactness, ensuring real-time inference and offline deployment capabilities. This research is pivotal in addressing the computational challenges faced in underwater operations. [ABSTRACT FROM AUTHOR]

Recent advancements in natural language processing have given rise to numerous pre-training language models in question-answering systems. However, with the constant evolution of algorithms, data, and computing power, the increasing size and complexity of these models have led to increased training costs and reduced efficiency. This study aims to minimize the inference time of such models while maintaining computational performance. It also proposes a novel Distillation model for PAL-BERT (DPAL-BERT), specifically, employs knowledge distillation, using the PAL-BERT model as the teacher model to train two student models: DPAL-BERT-Bi and DPAL-BERT-C. This research enhances the dataset through techniques such as masking, replacement, and n-gram sampling to optimize knowledge transfer. The experimental results showed that the distilled models greatly outperform models trained from scratch. In addition, although the distilled models exhibit a slight decrease in performance compared to PAL-BERT, they significantly reduce inference time to just 0.25% of the original. This demonstrates the effectiveness of the proposed approach in balancing model performance and efficiency. [ABSTRACT FROM AUTHOR]

Abstract Currently, image recognition based on deep neural networks has become the mainstream direction of research; therefore, significant progress has been made in its application in the field of tea detection. Many deep models exhibit high recognition rates in tea leaves detection. However, deploying these models directly on tea-picking equipment in natural environments is impractical; the extremely high parameters and computational complexity of these models make it challenging to perform real-time tea leaves detection. Meanwhile, lightweight models struggle to achieve competitive detection accuracy; therefore, this paper addresses the issue of computational resource constraints in remote mountain areas and proposes Reconstructed Feature and Dual Distillation (RFDD) to enhance the detection capability of lightweight models for tea leaves. In our method, the Reconstructed Feature selectively masks the feature of the student model based on the spatial attention map of the teacher model; it utilizes a generation block to force the student model to generate the teacher’s full feature. The Dual Distillation comprises Decoupled Distillation and Global Distillation. Decoupled Distillation divides the reconstructed feature into foreground and background features based on the Ground-Truth. This compels the student model to allocate different attention to foreground and background, focusing on their critical pixels and channels. However, Decoupled Distillation leads to the loss of relation knowledge between foreground and background pixels. Therefore, we further perform Global Distillation to extract this lost knowledge. Since RFDD only requires loss calculation on feature map, it can be easily applied to various detectors. We conducted experiments on detectors with different frameworks, using a tea dataset collected at the Huangshan Houkui Tea Plantation. The experimental results indicate that, under the guidance of RFDD, the student detectors have achieved performance improvements to varying degrees. For instance, a one-stage detector like RetinaNet (ResNet-50) experienced a 3.14% increase in Average Precision (AP) after RFDD guidance. Similarly, a two-stage model like Faster RCNN (ResNet-50) obtained a 3.53% improvement in AP. This offers promising prospects for lightweight models to efficiently perform real-time tea leaves detection tasks.

Knowledge distillation (KD) is a technique for transferring knowledge from a teacher model to a student model, commonly used in model compression and transfer learning. While existing KD methods have primarily focused on inter-sample or intra-sample knowledge without fully leveraging supervised labels, we present a novel approach that integrates self-supervised tasks into the KD framework to develop a category-based distillation algorithm. Our approach introduces two distinct memory banks for storing category embeddings predicted by the teacher and student models. By utilizing these memory banks, the student model absorbs knowledge across batches, enhancing the learning process through category embedding contrast rather than relying solely on intra-model knowledge transfer. Additionally, we incorporate self-supervised tasks with diverse data augmentation strategies to guide training in a multi-task framework. Experimental results demonstrate that our method achieves significant improvements, outperforming leading sample-based KD techniques. Specifically, our approach shows superior performance on both CIFAR-100 and ImageNet datasets. [ABSTRACT FROM AUTHOR]

Abstract With the rapid growth of big data, extracting meaningful knowledge from data is crucial for machine learning. The existing Swarm Learning data collaboration models face challenges such as data security, model security, high communication overhead, and model performance optimization. To address this, we propose the Swarm Mutual Learning (SML). Firstly, we introduce an Adaptive Mutual Distillation Algorithm that dynamically controls the learning intensity based on distillation weights and strength, enhancing the efficiency of knowledge extraction and transfer during mutual distillation. Secondly, we design a Global Parameter Aggregation Algorithm based on homomorphic encryption, coupled with a Dynamic Gradient Decomposition Algorithm using singular value decomposition. This allows the model to aggregate parameters in ciphertext, significantly reducing communication overhead during uploads and downloads. Finally, we validate the proposed methods on real datasets, demonstrating their effectiveness and efficiency in model updates. On the MNIST dataset and CIFAR-10 dataset, the local model accuracies reached 95.02% and 55.26%, respectively, surpassing those of the comparative models. Furthermore, while ensuring the security of the aggregation process, we significantly reduced the communication overhead for uploading and downloading.

Machine translation (MT) is the process of using a computer to convert one language into another language with the same semantics. With the introduction of neural network, neural machine translation (NMT), as a powerful machine translation technology, has achieved remarkable success in the field of automatic translation and artificial intelligence. Due to the problem of redundant parameters and structure in traditional neural translation models, knowledge distillation (KD) technology is proposed to compress the model and accelerate the inference of neural machine translation, which has attracted wide attention in the field of machine learning and natural language processing. This paper systematically investigates and compares various translation models with introduction of know-ledge distillation from the perspectives of evaluation indicators and technical innovations. Firstly, this paper briefly reviews the development process, mainstream frameworks and evaluation indicators of machine translation. Secondly, the knowledge distillation technology is introduced in detail. Thirdly, the development direction of neural machine translation based on knowledge distillation is detailed from four perspectives: multi-language model, multi-modal translation, low-resource language, autoregressive and non-autoregressive, and the research status of other fields is briefly introduced. Finally, the problems of existing large language models, zero-resource languages and multi-modal machine translation are analyzed, and the development trend of neural machine translation is prospected.

With the continuous development of mobile communication technology, edge intelligence has received widespread attention from academia. However, when enabling edge intelligence in Unmanned Aerial Vehicle (UAV) networks where drones serve as edge devices, the problem of insufficient computing power often arises due to limited storage and computing resources. In order to solve the problem of insufficient UAV computing power, this paper proposes a distributed cloud-edge collaborative optimization algorithm (DCECOA). The core idea of the DCECOA is to make full use of the local data of edge devices (i.e., UAVs) to optimize the neural network model more efficiently and achieve model volume compression. Compared with the traditional Taylor evaluation criterion, this algorithm consumes less resources on the communication uplink. The neural network model compressed by the proposed optimization algorithm can achieve higher performance under the same compression rate. [ABSTRACT FROM AUTHOR]

Due to the excellent results achieved by transformers in computer vision, more and more scholars have introduced transformers into the field of medical image segmentation. However, the use of transformers will make the model's parameters very large, which occupies a large amount of the computer's resources, making them very time-consuming during training. In order to alleviate this disadvantage, this paper explores a flexible and efficient search strategy that can find the best subnet from a continuous transformer network. The method is based on a learnable and uniform L1 sparsity constraint, which contains factors that reflect the global importance of the continuous search space in different dimensions, while the search process is simple and efficient, containing a single round of training. At the same time, in order to compensate for the loss of accuracy caused by the search, a pixel classification module is introduced into the model to compensate for the loss of accuracy in the model search process. Our experiments show that the model in this paper compresses 30% of the parameters and FLOPs used, while also showing a slight increase in the accuracy of the model on the Automatic Cardiac Diagnosis Challenge (ACDC) dataset. [ABSTRACT FROM AUTHOR]

Human-computer interaction is a discipline that studies the interaction between computer systems and users. Nowadays, it has become an important research topic in the field of robotics, especially in the field of auxiliary robots. As gestures can express rich information, gesture recognition is widely used in machine control, smart home, etc. Vision-based gesture recognition systems can greatly facilitate the lives of the elderly, the sick and the disabled. However, it is also a challenging task to compress the model while maintaining its accuracy in order to achieve efficient recognition on some inexpensive hardware configurations with limited resources. To this end, a lightweight gesture recognition based on an improved gated recurrent unit (GRU) for skeleton dynamic graphs is proposed in this paper. Firstly, the gesture skeleton data is preprocessed, and the coordinates of key joints are selected as input. Then, the mapped gesture feature data is cut to an appropriate size based on an adjustment and clipping module, and then input into an improved GRU for feature learning, which strengthens the ability of the GRU network to learn spatial features, so as to achieve efficient extraction of temporal and spatial features. Finally, the prediction result is output by the classifier. The extensive experiments are conducted on the public datasets DHG14/28 and SHREC'17. In terms of the model size or number of parameters, the proposed model can get the state-of-the-art performance. The average size of the proposed model decreases by more than 30% compared with the general GRU network. Furthermore, the proposed model can get a competitive accuracy performance of 92.13% and 93.98%, on the dataset DHG14/28 and SHREC'17, respectively. It is proved that our model is a lightweight and efficient model for gesture recognition of skeleton dynamic graphs. [ABSTRACT FROM AUTHOR]

Foreign objects in coal flow easily cause damage to conveyor belts, and most foreign objects are often occluded, making them difficult to detect. Aiming at solving the problems of low accuracy and efficiency in the detection of occluded targets in a low-illumination and dust fog environment, an image detection method for foreign objects is proposed. Firstly, YOLOv5s back-end processing is optimized by soft non-maximum suppression to reduce the influence of dense objects. Secondly, SimOTA label allocation is used to reduce the influence of ambiguous samples under dense occlusion. Then, Slide Loss is used to excavate difficult samples, and Inner–SIoU is used to optimize the bounding box regression loss. Finally, Group–Taylor pruning is used to compress the model. The experimental results show that the proposed method has only 4.20 × 105 parameters, a computational amount of 1.00 × 109, a model size of 1.20 MB, and an mAP0.5 of up to 91.30% on the self-built dataset. The detection speed on the different computing devices is as high as 66.31, 41.90, and 33.03 FPS. This proves that the proposed method achieves fast and high-accuracy detection of multi-layer occluded coal flow foreign objects. [ABSTRACT FROM AUTHOR]

Interest in deploying deep reinforcement learning (DRL) models on low-power edge devices, such as Autonomous Mobile Robots (AMRs) and Internet of Things (IoT) devices, has seen a significant rise due to the potential of performing real-time inference by eliminating the latency and reliability issues incurred from wireless communication and the privacy benefits of processing data locally. Deploying such energy-intensive models on power-constrained devices is not always feasible, however, which has led to the development of model compression techniques that can reduce the size and computational complexity of DRL policies. Policy distillation, the most popular of these methods, can be used to first lower the number of network parameters by transferring the behavior of a large teacher network to a smaller student model before deploying these students at the edge. This works well with deterministic policies that operate using discrete actions. However, many real-world tasks that are power constrained, such as in the field of robotics, are formulated using continuous action spaces, which are not supported. In this work, we improve the policy distillation method to support the compression of DRL models designed to solve these continuous control tasks, with an emphasis on maintaining the stochastic nature of continuous DRL algorithms. Experiments show that our methods can be used effectively to compress such policies up to 750% while maintaining or even exceeding their teacher's performance by up to 41% in solving two popular continuous control tasks. [ABSTRACT FROM AUTHOR]

The Transformer network architecture has gained attention due to its ability to learn global relations and its superior performance. To boost performance, it is natural to distill complementary knowledge from a Transformer network to a convolutional neural network (CNN). However, most existing knowledge distillation methods only consider homologous-architecture distillation, which may not be suitable for cross-architecture scenarios, such as from Transformer to CNN. To address this problem, we analyze the globality and transferability of models, which reflect the ability to capture global knowledge and transfer knowledge from teacher to student, respectively. Inspired by our observations, a novel cross-architecture knowledge distillation method is proposed, which supports bi-directional distillation including from Transformer to CNN and from CNN to Transformer. Specifically, rather than directly mimicking the output and intermediate features of the teacher, a partial cross-attention projector (PCA/iPCA) and a group-wise linear projector (GL/iGL) are introduced to align the student features with the teacher's in two projected feature spaces. To better match the teacher's knowledge with the student's knowledge, an adaptive distillation router (ADR) is presented to decide the knowledge from which layer the teacher should be distilled to guide which layer of the student. A multi-view robust training scheme is further presented, to improve the robustness of the framework for distillation. Extensive experiments show that the proposed method outperforms 17 state-of-the-art methods on both small-scale and large-scale datasets. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Railway Science & Engineering is the property of Journal of Railway Science & Engineering Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Massive computational complexity and memory requirement of artificial intelligence models impede their deployability on edge computing devices of the Internet of Things (IoT). While Power-of-Two (PoT) quantization is proposed to improve the efficiency for edge inference of Deep Neural Networks (DNNs), existing PoT schemes require a huge amount of bit-wise manipulation and have large memory overhead, and their efficiency is bounded by the bottleneck of computation latency and memory footprint. To tackle this challenge, we present an efficient inference approach on the basis of PoT quantization and model compression. An integer-only scalar PoT quantization (IOS-PoT) is designed jointly with a distribution loss regularizer, wherein the regularizer minimizes quantization errors and training disturbances. Additionally, two-stage model compression is developed to effectively reduce memory requirement, and alleviate bandwidth usage in communications of networked heterogenous learning systems. The product look-up table (P-LUT) inference scheme is leveraged to replace bit-shifting with only indexing and addition operations for achieving low-latency computation and implementing efficient edge accelerators. Finally, comprehensive experiments on Residual Networks (ResNets) and efficient architectures with Canadian Institute for Advanced Research (CIFAR), ImageNet, and Real-world Affective Faces Database (RAF-DB) datasets, indicate that our approach achieves 2×∼10× improvement in the reduction of both weight size and computation cost in comparison to state-of-the-art methods. A P-LUT accelerator prototype is implemented on the Xilinx KV260 Field Programmable Gate Array (FPGA) platform for accelerating convolution operations, with performance results showing that P-LUT reduces memory footprint by 1.45×, achieves more than 3× power efficiency and 2× resource efficiency, compared to the conventional bit-shifting scheme. [ABSTRACT FROM AUTHOR]

Visual Place Recognition (VPR) aims to determine whether a robot or visual navigation system locates in a previously visited place using visual information. It is an essential technology and challenging problem in computer vision and robotic communities. Recently, numerous works have demonstrated that the performance of Convolutional Neural Network (CNN)-based VPR is superior to that of traditional methods. However, with a huge number of parameters, large memory storage is necessary for these CNN models. It is a great challenge for mobile robot platforms equipped with limited resources. Fortunately, Binary Neural Networks (BNNs) can reduce memory consumption by converting weights and activation values from 32-bit into 1-bit. But current BNNs always suffer from gradients vanishing and a marked drop in accuracy. Therefore, this work proposed a BinVPR model to handle this issue. The solution is twofold. Firstly, a feature restoration strategy was explored to add features into the latter convolutional layers to further solve the gradient-vanishing problem during the training process. Moreover, we identified two principles to address gradient vanishing: restoring basic features and restoring basic features from higher to lower layers. Secondly, considering the marked drop in accuracy results from gradient mismatch during backpropagation, this work optimized the combination of binarized activation and binarized weight functions in the Larq framework, and the best combination was obtained. The performance of BinVPR was validated on public datasets. The experimental results show that it outperforms state-of-the-art BNN-based approaches and full-precision networks of AlexNet and ResNet in terms of both recognition accuracy and model size. It is worth mentioning that BinVPR achieves the same accuracy with only 1% and 4.6% model sizes of AlexNet and ResNet. [ABSTRACT FROM AUTHOR]

In order to improve the efficiency of fine segmentation and obstacle removal in the road of tea plantation in hilly areas, a lightweight and high-precision DR-YOLO instance segmentation algorithm is proposed to realize environment awareness. Firstly, the road data of tea gardens in hilly areas were collected under different road conditions and light conditions, and data sets were generated. YOLOv8n-seg, which has the highest operating efficiency, was selected as the basic model. The MSDA-CBAM and DR-Neck feature fusion network were added to the YOLOv8-seg model to improve the feature extraction capability of the network and the feature fusion capability and efficiency of the model. Experimental results show that, compared with the YOLOv8-seg model, the DR-YOLO model proposed in this study has 2.0% improvement in AP@0.5 and 1.1% improvement in Precision. In this study, the DR-YOLO model is pruned and quantitatively compressed, which greatly improves the model inference speed with little reduction in AP. After deploying on Jetson, compared with the YOLOv8n-seg model, the Precision of DR-YOLO is increased by 0.6%, the AP@0.5 is increased by 1.6%, and the inference time is reduced by 17.1%, which can effectively improve the level of agricultural intelligent automation and realize the efficient operation of the instance segmentation model at the edge. [ABSTRACT FROM AUTHOR]