Search

The article focuses on the Self-Distilled Learning to Rank (SDLR) framework, which is designed to improve the generalizability of ad hoc retrieval models through a self-distillation process. Topics include the methodology of assigning confidence weights to training samples based on feature distributions, the comparison of SDLR's performance against state-of-the-art learning to rank models using evaluation metrics and the effectiveness of the student model trained on a portion of data.

Urolithiasis is a leading urological disorder where accurate preoperative identification of stone types is critical for effective treatment. Deep learning has shown promise in classifying urolithiasis from CT images, yet faces challenges with model size and computational efficiency in real clinical settings. To address these challenges, we developed a non-invasive prediction approach for determining urinary stone types based on CT images. Through the refinement and improvement of the self-distillation architecture, coupled with the incorporation of feature fusion and the Coordinate Attention Module (CAM), we facilitated a more effective and thorough knowledge transfer. This method circumvents the extra computational expenses and performance reduction linked with model compression and removes the reliance on external teacher models, markedly enhancing the efficacy of lightweight models. achieved a classification accuracy of 74.96% on a proprietary dataset, outperforming current techniques. Furthermore, our method demonstrated superior performance and generalizability on two public datasets. This not only validates the effectiveness of our approach in classifying urinary stones but also showcases its potential in other medical image processing tasks. These results further reinforce the feasibility of our model for actual clinical deployment, potentially assisting healthcare professionals in devising more precise treatment plans and reducing patient discomfort. [ABSTRACT FROM AUTHOR]

Self-distillation method guides the model learning via transferring knowledge of the model itself, which has shown the advantages in object segmentation. However, it has been proved that uncertain pixels with predicted probability close to 0.5 will restrict the model performance. The existing self-distillation methods cannot guide the model to enhance its learning ability for uncertain pixels, so the improvement is limited. To boost the student model's learning ability for uncertain pixels, a novel self-distillation method is proposed. Firstly, the predicted probability in the current training sample and the ground truth label are fused to construct the teacher knowledge, as the current predicted information can express the performance of student models and represent the uncertainty of pixels more accurately. Secondly, a quadratic mapping function between the predicted probabilities of the teacher and student model is proposed. Theoretical analysis shows that the proposed method using the mapping function can guide the model to enhance the learning ability for uncertain pixels. Finally, the essential difference of utilizing the predicted probability of the student model in self-distillation is discussed in detail. Extensive experiments were conducted on models with convolutional neural networks and Transformer architectures as the backbone networks. The results on four public datasets demonstrate that the proposed method can effectively improve the student model performance. [ABSTRACT FROM AUTHOR]

Abstract Urolithiasis is a leading urological disorder where accurate preoperative identification of stone types is critical for effective treatment. Deep learning has shown promise in classifying urolithiasis from CT images, yet faces challenges with model size and computational efficiency in real clinical settings. To address these challenges, we developed a non-invasive prediction approach for determining urinary stone types based on CT images. Through the refinement and improvement of the self-distillation architecture, coupled with the incorporation of feature fusion and the Coordinate Attention Module (CAM), we facilitated a more effective and thorough knowledge transfer. This method circumvents the extra computational expenses and performance reduction linked with model compression and removes the reliance on external teacher models, markedly enhancing the efficacy of lightweight models. achieved a classification accuracy of 74.96% on a proprietary dataset, outperforming current techniques. Furthermore, our method demonstrated superior performance and generalizability on two public datasets. This not only validates the effectiveness of our approach in classifying urinary stones but also showcases its potential in other medical image processing tasks. These results further reinforce the feasibility of our model for actual clinical deployment, potentially assisting healthcare professionals in devising more precise treatment plans and reducing patient discomfort.

With the rapid growth of the internet of things (IoT) and smart devices, edge computing has emerged as a critical technology for processing massive amounts of data and protecting user privacy. Split federated learning, an emerging distributed learning framework, enables model training without needing data to leave local devices, effectively preventing data leakage and misuse. However, the disparity in computational capabilities of edge devices necessitates partitioning models according to the least capable client, resulting in a significant portion of the computational load being offloaded to a more capable server-side infrastructure, thereby incurring substantial training overheads. This work proposes a novel method for split federated learning targeting heterogeneous endpoints to address these challenges. The method addresses the problem of heterogeneous training across different clients by adding auxiliary layers, enhances the accuracy of heterogeneous model split training using self-distillation techniques, and leverages the global model from the previous round to mitigate the accuracy degradation during federated aggregation. We conducted validations on the CIFAR-10 dataset and compared it with the existing SL, SFLV1, and SFLV2 methods; our HFSL2 method improved by 3.81%, 13.94%, and 6.19%, respectively. Validations were also carried out on the HAM10000, FashionMNIST, and MNIST datasets, through which we found that our algorithm can effectively enhance the aggregation accuracy of heterogeneous computing capabilities. [ABSTRACT FROM AUTHOR]

The task of visible and infrared person re-identification (VI-ReID) aims to retrieve person images across visible and infrared images. However, the significant modality discrepancy and intra-modality variations render this task extremely challenging. Existing VI-ReID methods ignore the design for lightweight network. To address the above problems, we design a lightweight two-stream network based on omni-scale network (OSNet) for this task, we further explore how many parameters are shared is more efficient for two-stream network. On this basis, we propose a novel self-distillation module (SDM) to improve the feature extraction capability of this two-stream network. The SDM introduces the deepest classifier as a teacher model and constructs three shallow classifiers as student models. Under the guidance of the teacher model, these student models absorb rich deep knowledge from the deepest classifier to achieve optimization of low-level features, thus promoting the improvement of high-level feature representation. Subsequently, in order to extract highly discriminative part-informed features, we introduce a multi-granularity information mining(MGIM) block that not only learns local features but also considers the internal relationships between local features. This helps to fully mine local detail information within the images. The extensive experiments on the SYSU-MM01,RegDB,and LLCM datasets show that our proposed method achieves superior performance. [ABSTRACT FROM AUTHOR]

Traditional knowledge distillation relies on high‐capacity teacher models to supervise the training of compact student networks. To avoid the computational resource costs associated with pretraining high‐capacity teacher models, teacher‐free online knowledge distillation methods have achieved satisfactory performance. Among these methods, feature fusion methods have effectively alleviated the limitations of training without the strong guidance of a powerful teacher model. However, existing feature fusion methods often focus primarily on end‐layer features, overlooking the efficient utilization of holistic knowledge loops and high‐level information within the network. In this article, we propose a new feature fusion‐based mutual learning method called Diversify Feature Enhancement and Fusion for Online Knowledge Distillation (DFEF). First, we enhance advanced semantic information by mapping multiple end‐of‐network features to obtain richer feature representations. Next, we design a self‐distillation module to strengthen knowledge interactions between the deep and shallow network layers. Additionally, we employ attention mechanisms to provide deeper and more diversified enhancements to the input feature maps of the self‐distillation module, allowing the entire network architecture to acquire a broader range of knowledge. Finally, we employ feature fusion to merge the enhanced features and generate a high‐performance virtual teacher to guide the training of the student model. Extensive evaluations on the CIFAR‐10, CIFAR‐100, and CINIC‐10 datasets demonstrate that our proposed method can significantly enhance performance compared to state‐of‐the‐art feature fusion‐based online knowledge distillation methods. Our code can be found at https://github.com/JSJ515-Group/DFEF-Liu. [ABSTRACT FROM AUTHOR]

Few-shot learning (FSL) is a challenging problem. Transfer learning methods offer a straightforward and effective solution to FSL by leveraging pre-trained models and generalizing them to new tasks. However, pre-trained models often lack the ability to highlight and emphasize salient features, a gap that attention mechanisms can fill. Unfortunately, existing attention mechanisms encounter issues such as high complexity and incomplete attention information. To address these issues, we propose a dimensionally enhanced attention (DEA) module for FSL. This DEA module introduces minimal additional computational overhead while fully attending to both channel and spatial information. Specifically, the feature map is first decomposed into 1D tensors of varying dimensions using strip pooling. Next, a multi-dimensional collaborative learning strategy is introduced, enabling cross-dimensional information interactions through 1D convolutions with adaptive kernel sizes. Finally, the feature representation is enhanced by calculating attention weights for each dimension using a sigmoid function and weighting the original input accordingly. This approach ensures comprehensive attention to different dimensions of information, effectively characterizing data in various directions. Additionally, we have found that knowledge distillation significantly improves FSL performance. To this end, we implement a logit standardization self-distillation method tailored for FSL. This method addresses the issue of exact logit matching, which arises from the shared temperature in the self-distillation process, by employing logit standardization. We present experimental results on several benchmark datasets where the proposed method yields significant performance improvements. [ABSTRACT FROM AUTHOR]

Self-supervised monocular depth estimation can exhibit excellent performance in static environments due to the multi-view consistency assumption during the training process. However, it is hard to maintain depth consistency in dynamic scenes when considering the occlusion problem caused by moving objects. For this reason, we propose a method of self-supervised self-distillation for monocular depth estimation (SS-MDE) in dynamic scenes, where a deep network with a multi-scale decoder and a lightweight pose network are designed to predict depth in a self-supervised manner via the disparity, motion information, and the association between two adjacent frames in the image sequence. Meanwhile, in order to improve the depth estimation accuracy of static areas, the pseudo-depth images generated by the LeReS network are used to provide the pseudo-supervision information, enhancing the effect of depth refinement in static areas. Furthermore, a forgetting factor is leveraged to alleviate the dependency on the pseudo-supervision. In addition, a teacher model is introduced to generate depth prior information, and a multi-view mask filter module is designed to implement feature extraction and noise filtering. This can enable the student model to better learn the deep structure of dynamic scenes, enhancing the generalization and robustness of the entire model in a self-distillation manner. Finally, on four public data datasets, the performance of the proposed SS-MDE method outperformed several state-of-the-art monocular depth estimation techniques, achieving an accuracy ( δ 1 ) of 89% while minimizing the error (AbsRel) by 0.102 in NYU-Depth V2 and achieving an accuracy ( δ 1 ) of 87% while minimizing the error (AbsRel) by 0.111 in KITTI. [ABSTRACT FROM AUTHOR]

Emotion recognition in conversations (ERC), which aims to capture the dynamic changes in emotions during conversations, has recently attracted a huge amount of attention due to its importance in providing engaging and empathetic services. Considering that it is difficult for unimodal ERC approaches to capture emotional shifts in conversations, multimodal ERC research is on the rise. However, this still suffers from the following limitations: (1) failing to fully explore richer multimodal interactions and fusion; (2) failing to dynamically model speaker-dependent context in conversations; and (3) failing to employ model-agnostic techniques to eliminate semantic gaps among different modalities. Therefore, we propose a novel hierarchical cross-modal interaction and fusion network enhanced with self-distillation (HCIFN-SD) for ERC. Specifically, HCIFN-SD first proposes three different mask strategies for extracting speaker-dependent cross-modal conversational context based on the enhanced GRU module. Then, the graph-attention-based multimodal fusion (MF-GAT) module constructs three directed graphs for representing different modality spaces, implements in-depth cross-modal interactions for propagating conversational context, and designs a new GNN layer to address over-smoothing. Finally, self-distillation is employed to transfer knowledge from both hard and soft labels to supervise the training process of each student classifier for eliminating semantic gaps between different modalities and improving the representation quality of multimodal fusion. Extensive experimental results on IEMOCAP and MELD demonstrate that HCIFN-SD is superior to the mainstream state-of-the-art baselines by a significant margin. [ABSTRACT FROM AUTHOR]

With the popularity of social media, online opinion analysis is becoming more and more widely and deeply used in management studies. Automatically recognizing the sentiment of user reviews is a crucial tool for opinion analysis research. However, previous studies mainly have focused on specific scenarios or algorithms that cannot be directly applied to real-world opinion analysis. To address this issue, we collect a new dataset of user reviews from multiple real-world scenarios such as e-retail, e-commerce, movie reviews, and social media. Due to the heterogeneity and complexity of this multi-scenario review data, we propose a self-distillation contrastive learning method. Specifically, we utilize two EMA (exponential moving average) models to generate soft labels as additional supervision. Additionally, we introduce the prototypical supervised contrastive learning module to reduce the variability of data in different scenarios by pulling in representations of the same class. Our method has proven to be extremely competitive, outperforming other advanced methods. Specifically, our method achieves an 87.44% F1 score, exceeding the performance of current advanced methods by 1.07%. Experimental results, including examples and visualization analysis, further demonstrate the superiority of our method. [ABSTRACT FROM AUTHOR]

Image sentiment analysis has received a lot of attention due to the fact that users prefer to express their private emotions using images on social platforms. Researchers strive to obtain more discriminative features to characterize these images by designing diverse networks, such as convolutional neural networks (CNNs) and Transformer. This method requires prior knowledge in advance and its feature fusion ignores the correlation between heterogeneous networks. An efficient but effective integration of heterogeneous Transformer and CNN deserves further study. To this end, we propose a novel model called Closed-loop and Knowledge Iterative (ClKI) via self-distillation for image sentiment analysis. First, CNN features at different scales are extracted to capture the complementary multi-scale information, and then input into a Transformer stacked by multiple encoders with the ProbSparse self-attention to acquire more discriminative representations. Second, we adopt the true labels and deepest classification results jointly guide the training procedure of shallow modules via self-distillation. Thus, the ClKI model achieves an efficient interaction between the deep and shallow modules, providing sufficient knowledge for image sentiment analysis. Our model achieves 83.35%, 65.64%, and 91.01% accuracy on the FI, Emotion 6, and Twitter I datasets respectively, which outperforms state-of-the-art methods. Additional experiments further validate that ClKI can resolve the data imbalance problem to some degree and it has good generalization ability. Notably, ClKI is plug-and-play, promoting its practicality. Briefly, this study provides an innovative approach in image sentiment analysis, offering valuable insights for the integration of heterogeneous networks as well as knowledge iterative method. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer-Aided Design & Computer Graphics / Jisuanji Fuzhu Sheji Yu Tuxingxue Xuebao is the property of Gai Kan Bian Wei Hui and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Deep learning has achieved notable success across academia, medicine, and industry. Its ability to identify complex patterns in large-scale data and to manage millions of parameters has made it highly advantageous. However, deploying deep learning models presents a significant challenge due to their high computational demands. Knowledge distillation (KD) has emerged as a key technique for model compression and efficient knowledge transfer, enabling the deployment of deep learning models on resource-limited devices without compromising performance. This survey examines recent advancements in KD, highlighting key innovations in architectures, training paradigms, and application domains. We categorize contemporary KD methods into traditional approaches, such as response-based, feature-based, and relation-based knowledge distillation, and novel advanced paradigms, including self-distillation, cross-modal distillation, and adversarial distillation strategies. Additionally, we discuss emerging challenges, particularly in the context of distillation under limited data scenarios, privacy-preserving KD, and the interplay with other model compression techniques like quantization. Our survey also explores applications across computer vision, natural language processing, and multimodal tasks, where KD has driven performance improvements and enhanced model compression. This review aims to provide researchers and practitioners with a comprehensive understanding of the state-of-the-art in knowledge distillation, bridging foundational concepts with the latest methodologies and practical implications.

Abstract Self-distillation method guides the model learning via transferring knowledge of the model itself, which has shown the advantages in object segmentation. However, it has been proved that uncertain pixels with predicted probability close to 0.5 will restrict the model performance. The existing self-distillation methods cannot guide the model to enhance its learning ability for uncertain pixels, so the improvement is limited. To boost the student model’s learning ability for uncertain pixels, a novel self-distillation method is proposed. Firstly, the predicted probability in the current training sample and the ground truth label are fused to construct the teacher knowledge, as the current predicted information can express the performance of student models and represent the uncertainty of pixels more accurately. Secondly, a quadratic mapping function between the predicted probabilities of the teacher and student model is proposed. Theoretical analysis shows that the proposed method using the mapping function can guide the model to enhance the learning ability for uncertain pixels. Finally, the essential difference of utilizing the predicted probability of the student model in self-distillation is discussed in detail. Extensive experiments were conducted on models with convolutional neural networks and Transformer architectures as the backbone networks. The results on four public datasets demonstrate that the proposed method can effectively improve the student model performance.

Abstract Zero-shot learning represents a formidable paradigm in machine learning, wherein the crux lies in distilling and generalizing knowledge from observed classes to novel ones. The objective is to identify unfamiliar objects that were not included in the model’s training, leveraging learned patterns and knowledge from previously encountered categories. As a crucial subtask of open-world object detection, zero-shot classification can also provide insights and solutions for this field. Despite its potential, current zero-shot classification models often suffer from a performance gap due to limited transfer ability and discriminative capability of learned representations. In pursuit of advancing the subpar state of zero-shot object classification, this paper introduces a novel model for image classification which can be applied to object detection, namely, self-distillation and k-nearest neighbor-based zero-shot classification method. First, we employ a diffusion detector to identify potential objects in images. Then, self-distillation and distance-based classifiers are used for distinguishing unseen objects from seen classes. The k-nearest neighbor-based cluster heads are designed to cluster the unseen objects. Extensive experiments and visualizations were conducted on publicly available datasets on the efficacy of the proposed approach. Precisely, our model demonstrates performance improvement of over 20% compared to contrastive clustering. Moreover, it achieves a precision of 0.910 and a recall of 0.842 on CIFAR-10 datasets, a precision of 0.737, and a recall of 0.688 on CIFAR-100 datasets for the macro average. Compared to a more recent model (SGFR), our model realized improvements of 10.9%, 13.3%, and 7.8% in Sacc, Uacc, and H metrics, respectively. This study aims to introduce fresh ideas into the domain of zero-shot image classification, and it can be applied to open-world object detection tasks. Our code is available at https://www.github.com/CmosWolf1/Code_implementation_for_paper_SKZC .

Automatic classification of stellar spectra contributes to the study of the structure and evolution of the Milky Way and star formation. Currently available methods exhibit unsatisfactory spectral classification accuracy. This study investigates a method called DSRL, which is primarily used for automated and accurate classification of LAMOST stellar spectra based on MK classification criteria. The method utilizes discrete wavelet transform to decompose the spectra into high-frequency and low-frequency information, and combines residual networks and long short-term memory networks to extract both high-frequency and low-frequency features. By introducing self-distillation (DSRL-1, DSRL-2, and DSRL-3), the classification accuracy is improved. DSRL-3 demonstrates superior performance across multiple metrics compared to existing methods. In both three-class(F ,G ,K) and ten-class(A0, A5, F0, F5, G0, G5, K0, K5, M0, M5) experiments, DSRL-3 achieves impressive accuracy, precision, recall, and F1-Score results. Specifically, the accuracy performance reaches 94.50% and 97.25%, precision performance reaches 94.52% and 97.29%, recall performance reaches 94.52% and 97.22%, and F1-Score performance reaches 94.52% and 97.23%. The results indicate the significant practical value of DSRL in the classification of LAMOST stellar spectra. To validate the model, we visualize it using randomly selected stellar spectral data. The results demonstrate its excellent application potential in stellar spectral classification. [ABSTRACT FROM AUTHOR]

Generalized zero-shot learning (GZSL) aims to simultaneously recognize both seen classes and unseen classes by training only on seen class samples and auxiliary semantic descriptions. Recent state-of-the-art methods infer unseen classes based on semantic information or synthesize unseen classes using generative models based on semantic information, all of which rely on the correct alignment of visual–semantic features. However, they often overlook the inconsistency between original visual features and semantic attributes. Additionally, due to the existence of cross-modal dataset biases, the visual features extracted and synthesized by the model may also mismatch with some semantic features, which could hinder the model from properly aligning visual–semantic features. To address this issue, this paper proposes a GZSL framework that enhances the consistency of visual–semantic features using a self-distillation and disentanglement network (SDDN). The aim is to utilize the self-distillation and disentanglement network to obtain semantically consistent refined visual features and non-redundant semantic features to enhance the consistency of visual–semantic features. Firstly, SDDN utilizes self-distillation technology to refine the extracted and synthesized visual features of the model. Subsequently, the visual–semantic features are then disentangled and aligned using a disentanglement network to enhance the consistency of the visual–semantic features. Finally, the consistent visual–semantic features are fused to jointly train a GZSL classifier. Extensive experiments demonstrate that the proposed method achieves more competitive results on four challenging benchmark datasets (AWA2, CUB, FLO, and SUN). [ABSTRACT FROM AUTHOR]

The prediction of typhoon tracks in the Northwest Pacific is key to reducing human casualties and property damage. Traditional numerical forecasting models often require substantial computational resources, are high-cost, and have significant limitations in prediction speed. This research is dedicated to using deep learning methods to address the shortcomings of traditional methods. Our method (AFR-SimVP) is based on a large-kernel convolutional spatio-temporal prediction network combined with multi-feature fusion for forecasting typhoon tracks in the Northwest Pacific. In order to more effectively suppress the effect of noise in the dataset to enhance the generalization ability of the model, we use a multi-branch structure, incorporate an atmospheric reconstruction subtask, and propose a second-order smoothing loss to further improve the prediction ability of the model. More importantly, we innovatively propose a multi-time-step typhoon prediction network (HTAFR-SimVP) that does not use the traditional recurrent neural network family of models at all. Instead, through fine-to-coarse hierarchical temporal feature extraction and dynamic self-distillation, multi-time-step prediction is achieved using only a single regression network. In addition, combined with atmospheric field reconstruction, the network achieves integrated prediction for multiple tasks, which greatly enhances the model's range of applications. Experiments show that our proposed network achieves optimal performance in the 24 h typhoon track prediction task. Our regression network outperforms previous recurrent network-based typhoon prediction models in the multi-time-step prediction task and also performs well in multiple integration tasks. [ABSTRACT FROM AUTHOR]

Zero-shot learning represents a formidable paradigm in machine learning, wherein the crux lies in distilling and generalizing knowledge from observed classes to novel ones. The objective is to identify unfamiliar objects that were not included in the model's training, leveraging learned patterns and knowledge from previously encountered categories. As a crucial subtask of open-world object detection, zero-shot classification can also provide insights and solutions for this field. Despite its potential, current zero-shot classification models often suffer from a performance gap due to limited transfer ability and discriminative capability of learned representations. In pursuit of advancing the subpar state of zero-shot object classification, this paper introduces a novel model for image classification which can be applied to object detection, namely, self-distillation and k-nearest neighbor-based zero-shot classification method. First, we employ a diffusion detector to identify potential objects in images. Then, self-distillation and distance-based classifiers are used for distinguishing unseen objects from seen classes. The k-nearest neighbor-based cluster heads are designed to cluster the unseen objects. Extensive experiments and visualizations were conducted on publicly available datasets on the efficacy of the proposed approach. Precisely, our model demonstrates performance improvement of over 20% compared to contrastive clustering. Moreover, it achieves a precision of 0.910 and a recall of 0.842 on CIFAR-10 datasets, a precision of 0.737, and a recall of 0.688 on CIFAR-100 datasets for the macro average. Compared to a more recent model (SGFR), our model realized improvements of 10.9%, 13.3%, and 7.8% in Sacc, Uacc, and H metrics, respectively. This study aims to introduce fresh ideas into the domain of zero-shot image classification, and it can be applied to open-world object detection tasks. Our code is available at https://www.github.com/CmosWolf1/Code_implementation_for_paper_SKZC. [ABSTRACT FROM AUTHOR]

Scene graph generation (SGG) aims to build a structural representation for the image with the object instance and the relations between object pairs. Due to the long-tail distribution of the dataset labeling, scene graph generation models must adopt the debiasing method during the learning process. In this paper, we propose to integrating a novel self-distillation method into the existing SGG models and the experimental results have shown competitive debiasing performance. Further analysis of its effectiveness with causal inference theory has indicated that our method can be considered as a new intervention method. [ABSTRACT FROM AUTHOR]

Massive tree dieback events triggered by various disturbance agents, such as insect outbreaks, pests, fires, and windstorms, have recently compromised the health of forests in numerous countries with a significant impact on ecosystems. The inventory of forest tree dieback plays a key role in understanding the effects of forest disturbance agents and improving forest management strategies. In this article, we illustrate a deep learning approach that trains a U-Net model for the semantic segmentation of Sentinel-2 images of forest areas. The proposed U-Net architecture integrates an attention mechanism to amplify the crucial information and a self-distillation approach to transfer the knowledge within the U-Net architecture. Experimental results demonstrate the significant contribution of both attention and self-distillation to gaining accuracy in two case studies in which we perform the inventory mapping of forest tree dieback caused by insect outbreaks and wildfires, respectively.

In person re-identification, most prevalent models are predominantly designed for cloud computing environments which introduces complexities that limit their effectiveness in edge computing scenarios. Person re-identification systems optimized for edge computing can achieve real-time or near-real-time responses, providing substantial practical value. Addressing this gap, this paper presents the Attention Knowledge-aided Distillation Lightweight Network (ADLN), a network architecture expressly crafted for edge computing. The ADLN enhances inference speed while maintaining accuracy, which is essential for real-time applications. The core innovation of the ADLN lies in its dimension interaction attention mechanism, strategically integrated into the network to boost recognition performance. This mechanism is complemented by a self-distillation approach, transferring attention knowledge from deeper to shallower layers, thereby streamlining the network and accelerating inference. Moreover, the ADLN employs an optimization strategy combining cross-entropy loss, weighted triplet loss regularization, and center loss, effectively reducing intra-class variances. Tested on Market1501 and DukeMTMC-ReID datasets, experiments indicate that the ADLN significantly reduces the model’s parameter count and identification latency, while largely maintaining accuracy.

Face parsing, the segmentation of facial components at the pixel level, is pivotal for comprehensive facial analysis. However, previous studies encountered challenges, showing reduced performance in areas with small or thin classes like necklaces and earrings, and struggling to adapt to occlusion scenarios such as masks, glasses, caps or hands. To address these issues, this study proposes a robust face parsing technique through the strategic integration of self-attention and self-distillation methods. The self-attention module enhances contextual information, enabling precise feature identification for each facial element. Multi-task learning for edge detection, coupled with a specialized loss function focusing on edge regions, elevates the understanding of fine structures and contours. Additionally, the application of self-distillation for fine-tuning proves highly efficient, producing refined parsing results while maintaining high performance in scenarios with limited labels and ensuring robust generalization. The integration of self-attention and self-distillation techniques addresses challenges of previous studies, particularly in handling small or thin classes. This strategic fusion enhances overall performance, achieving computational efficiency, and aligns with the latest trends in this research area. The proposed approach attains a Mean F1 score of 88.18% on the CelebAMask-HQ dataset, marking a significant advancement in face parsing with state-of-the-art performance. Even in challenging occlusion areas like hands and masks, it demonstrates a remarkable F1 score of over 99%, showcasing robust face parsing capabilities in real-world environments.

Semantic super-resolution (SR) is an approach that improves the SR performance by leveraging semantic information about the scene. This study develops a novel semantic SR method that is based on the generative adversarial network (GAN) framework and self-distillation. A discriminator is adversarially trained along with a generator to extract semantic features from images and distinguish semantic differences between images. To train the generator, an additional adversarial loss is computed from the discriminator’s outputs of SR images belonging to the same category and minimized via self-distillation. This guides the generator to learn implicit category-specific semantic priors. We conducted experiments for SR of text and face images using the Enhanced Deep Super-Resolution (EDSR) generator and the SRGAN discriminator. Experimental results showed that our method can contribute to improving both the quantitative and qualitative quality of SR images. Although the improvement varied depending on image category and dataset, the peak signal-to-noise ratio (PSNR) value increased by up to 0.87 dB and the perceptual index (PI) decreased by up to 0.17 by using our method. Our method outperformed existing semantic SR methods.

Recently, few-shot learning (FSL) has exhibited remarkable performance in computer vision tasks. However, the existing FSL approaches perform poorly when facing data shortages and domain variations between the source and target datasets. This is because the target domain is hidden during training and the strong discriminating ability on the source domain dataset cannot be properly transferred into the good classification precision on the target dataset. To address the optimization problem for cross-domain few-shot image identification, this study proposed the Feature Adaptive Distillation (FAD) method. Specifically, we capture broader variations in feature distributions through a novel Feature Adaptive Distillation method. The two primary components of FAD are the Self-Distillation module (SD) and the Feature Adaptive module (FA). By including additional adaptive parameters for particular tasks in the feature extractor, FA enhances the generalization performance of this method. To enhance it's ability to recognize features and determine the most effective feature extractor, the feature extractor is further self-distilled using SD. The results indicate that this method can greatly enhance the effectiveness of such kind image recognition. [ABSTRACT FROM AUTHOR]

The objective of document-level relation extraction is to retrieve the relations existing between entities within a document. Currently, deep learning methods have demonstrated superior performance in document-level relation extraction tasks. However, to enhance the model's performance, various methods directly introduce additional modules into the backbone model, which often increases the number of parameters in the overall model. Consequently, deploying these deep models in resource-limited environments presents a challenge. In this article, we introduce a self-distillation framework for document-level relational extraction. We partition the document-level relation extraction model into two distinct modules, namely, the entity embedding representation module and the entity pair embedding representation module. Subsequently, we apply separate distillation techniques to each module to reduce the model's size. In order to evaluate the proposed framework's performance, two benchmark datasets for document-level relation extraction, namely GDA and DocRED are used in this study. The results demonstrate that our model effectively enhances performance and significantly reduces the model's size. [ABSTRACT FROM AUTHOR]

Despite the proven effectiveness of data-driven deep learning techniques in urban flood mapping, the availability of annotation data remains a critical factor impeding their timeliness in real applications. Recent progress in weakly supervised semantic segmentation (WSSS) presents promising solutions for addressing this limitation. To accomplish prompt and accurate flood mapping in complex urban areas from high-resolution remote sensing images to support disaster management and response, this study makes contributions in three key aspects: weak training data generation, the improvement of WSSS algorithm, and the construction of benchmark datasets. Firstly, we present a novel yet efficient weak training data generation strategy by decoupling the acquisition of positive and negative samples. This strategy enables the rapid generation of block-level weak annotations assisted by pre-flood river data or the segment anything model (SAM) for zero-shot segmentation, thereby alleviating the burden of weak data labeling. Secondly, to enhance the flood detection results in complex urban landscapes based on low-cost weak labels, we design an end-to-end WSSS framework incorporating tree filtering-based structure constraints and a perturbed dual-branch cross self-distillation mechanism. Lastly, to evaluate the performance of the proposed approach, we constructed two large aerial imagery datasets, namely Calgary-Flood and Huston-Flood. These datasets encompass diverse urban land covers and include challenging scenarios with extensive shadows, providing a robust benchmark for evaluating our method against various urban environments. Experimental results demonstrate that our weak data annotation strategy substantially enhances efficiency. Additionally, the proposed WSSS framework exhibits superior performance in comparison to existing state-of-the-art methods, particularly in terms of edge delineation and algorithmic stability. The advancements in weak data annotation strategy, end-to-end model architecture, and benchmark dataset development in this study collectively exploit the potential of the weakly supervised paradigm for rapid flood mapping in urgent situations. The datasets and code will be publicly available at (https://github.com/YJ-He/Flood_Mapping_WSSS). [ABSTRACT FROM AUTHOR]

With the advent of large-scale datasets, significant advancements have been made in image semantic segmentation. However, the annotation of these datasets necessitates substantial human and financial resources. Therefore, the focus of research has shifted towards few-shot semantic segmentation, which leverages a small number of labeled samples to effectively segment unknown categories. The current mainstream methods are to use the meta-learning framework to achieve model generalization, and the main challenges are as follows. (1) The trained model will be biased towards the seen class, so the model will misactivate the seen class when segmenting the unseen class, which makes it difficult to achieve the idealized class agnostic effect. (2) When the sample size is limited, there exists an intra-class gap between the provided support images and the query images, significantly impacting the model's generalization capability. To solve the above two problems, we propose a network with prototype complementarity characteristics (PCNet). Specifically, we first generate a self-support query prototype based on the query image. Through the self-distillation, the query prototype and the support prototype perform feature complementary learning, which effectively reduces the influence of the intra-class gap on the model generalization. A standard semantic segmentation model is introduced to segment the seen classes during the training process to achieve accurate irrelevant class shielding. After that, we use the rough prediction map to extract its background prototype and shield the background in the query image by the background prototype. In this way, we obtain more accurate fine-grained segmentation results. The proposed method exhibits superiority in extensive experiments conducted on the PASCAL- 5 i and COCO- 20 i datasets. We achieve new state-of-the-art results in the few-shot semantic segmentation task, with an mIoU of 71.27% and 51.71% in the 5-shot setting, respectively. Comprehensive ablation experiments and visualization studies show that the proposed method has a significant effect on small-sample semantic segmentation. [ABSTRACT FROM AUTHOR]

Recognizing human actions can help in numerous ways, such as health monitoring, intelligent surveillance, virtual reality and human–computer interaction. A quick and accurate detection algorithm is required for daily real-time detection. This paper first proposes to generate a lightweight graph neural network by self-distillation for human action recognition tasks. The lightweight graph neural network was evaluated on the NTU-RGB+D dataset. The results demonstrate that, with competitive accuracy, the heavyweight graph neural network can be compressed by up to 80 % . Furthermore, the learned representations have denser clusters, estimated by the Davies–Bouldin index, the Dunn index and silhouette coefficients. The ideal input data and algorithm capacity are also discussed. [ABSTRACT FROM AUTHOR]

Obtaining dense depth ground-truth is not trivial, which leads to the introduction of self-supervised monocular depth estimation. Most self-supervised methods utilize the photometric loss as the primary supervisory signal to optimize a depth network. However, such self-supervised training often falls into an undesirable local minimum due to the ambiguity of the photometric loss. In this paper, we propose a novel self-distillation training scheme that provides a new self-supervision signal, depth consistency among different input resolutions, to the depth network. We further introduce a gradient masking strategy that adjusts the self-supervision signal of the depth consistency during back-propagation to boost the effectiveness of our depth consistency. Experiments demonstrate that our method brings meaningful performance improvements when applied to various depth network architectures. Furthermore, our method outperforms the existing self-supervised methods on KITTI, Cityscapes, and DrivingStereo datasets by a noteworthy margin. • A simple yet effective distillation is proposed for self-supervised monocular depth. • Resolving the resolution bias eventually improves depths in a target resolution. • The proposed depth consistency learning produces reliable self-supervision. • The gradients masking boosts the effectiveness of the depth consistency learning. • The depth consistency improves the generality of self-supervised depth networks. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Guangxi Normal University - Natural Science Edition is the property of Gai Kan Bian Wei Hui and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The self-distillation methods can transfer the knowledge within the network itself to enhance the generalization ability of the network. However, due to the lack of spatially refined knowledge representations, current self-distillation methods can hardly be directly applied to object segmentation tasks. In this paper, we propose a novel self-distillation framework via pyramid knowledge representation and transfer for the object segmentation task. Firstly, a lightweight inference network is built to perform pixel-wise prediction rapidly. Secondly, a novel self-distillation method is proposed. To derive refined pixel-wise knowledge representations, the auxiliary self-distillation network via multi-level pyramid representation branches is built and appended to the inference network. A synergy distillation loss, which utilizes the top-down and consistency knowledge transfer paths, is presented to force more discriminative knowledge to be distilled into the inference network. Consequently, the performance of the inference network is improved. Experimental results on five datasets of object segmentation demonstrate that the proposed self-distillation method helps our inference network perform better segmentation effectiveness and efficiency than nine recent object segmentation network. Furthermore, the proposed self-distillation method outperforms typical self-distillation methods. The source code is publicly available at https://github.com/xfflyer/SKDforSegmentation. [ABSTRACT FROM AUTHOR]

Self-supervised monocular depth estimation can exhibit excellent performance in static environments due to the multi-view consistency assumption during the training process. However, it is hard to maintain depth consistency in dynamic scenes when considering the occlusion problem caused by moving objects. For this reason, we propose a method of self-supervised self-distillation for monocular depth estimation (SS-MDE) in dynamic scenes, where a deep network with a multi-scale decoder and a lightweight pose network are designed to predict depth in a self-supervised manner via the disparity, motion information, and the association between two adjacent frames in the image sequence. Meanwhile, in order to improve the depth estimation accuracy of static areas, the pseudo-depth images generated by the LeReS network are used to provide the pseudo-supervision information, enhancing the effect of depth refinement in static areas. Furthermore, a forgetting factor is leveraged to alleviate the dependency on the pseudo-supervision. In addition, a teacher model is introduced to generate depth prior information, and a multi-view mask filter module is designed to implement feature extraction and noise filtering. This can enable the student model to better learn the deep structure of dynamic scenes, enhancing the generalization and robustness of the entire model in a self-distillation manner. Finally, on four public data datasets, the performance of the proposed SS-MDE method outperformed several state-of-the-art monocular depth estimation techniques, achieving an accuracy (δ1) of 89% while minimizing the error (AbsRel) by 0.102 in NYU-Depth V2 and achieving an accuracy (δ1) of 87% while minimizing the error (AbsRel) by 0.111 in KITTI.