Search

Knowledge Distillation (KD), a learning manner with a larger teacher network guiding a smaller student network, transfers dark knowledge from the teacher to the student via logits or intermediate features, with the aim of producing a well-performed lightweight model. Notably, many subsequent feature-based KD methods outperformed the earliest logit-based KD method and iteratively generated numerous state-of-the-art distillation methods. Nevertheless, recent work has uncovered the potential of the logit-based method, bringing the simple KD form based on logits back into the limelight. Features or logits? They partially implement the KD with entirely distinct perspectives; therefore, choosing between logits and features is not straightforward. This paper provides a unified perspective of feature alignment in order to obtain a better comprehension of their fundamental distinction. Inheriting the design philosophy and insights of feature-based and logit-based methods, we introduce a block-wise logit distillation framework to apply implicit logit-based feature alignment by gradually replacing teacher's blocks as intermediate stepping-stone models to bridge the gap between the student and the teacher. Our method obtains comparable or superior results to state-of-the-art distillation methods. This paper demonstrates the great potential of combining logit and features, and we hope it will inspire future research to revisit KD from a higher vantage point.

Recent insights have revealed that rate-coding is a primary form of information representation captured by surrogate-gradient-based Backpropagation Through Time (BPTT) in training deep Spiking Neural Networks (SNNs). Motivated by these findings, we propose rate-based backpropagation, a training strategy specifically designed to exploit rate-based representations to reduce the complexity of BPTT. Our method minimizes reliance on detailed temporal derivatives by focusing on averaged dynamics, streamlining the computational graph to reduce memory and computational demands of SNNs training. We substantiate the rationality of the gradient approximation between BPTT and the proposed method through both theoretical analysis and empirical observations. Comprehensive experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS validate that our method achieves comparable performance to BPTT counterparts, and surpasses state-of-the-art efficient training techniques. By leveraging the inherent benefits of rate-coding, this work sets the stage for more scalable and efficient SNNs training within resource-constrained environments. Our code is available at https://github.com/Tab-ct/rate-based-backpropagation., Comment: Accepted by NeurIPS 2024

Ultrathin oxide semiconductors with sub-1-nm thickness are promising building blocks for ultrascaled field-effect transistor (FET) applications due to their resilience against short-channel effects, high air stability, and potential for low-energy device operation. However, the n-type dominance of ultrathin oxide FET has hindered their integration into complementary metal-oxide-semiconductor (CMOS) technology, which requires both n-and p-type devices. Here we develop an ab initio device-driven computational screening workflow to identify sub-1-nm thickness oxide semiconductors for sub-5-nm FET applications. We demonstrate that ultrathin CaO2, CaO, and SrO are compatible with p-type device operations under both high-performance (HP) and low-power (LP) requirements specified by the International Technology Roadmap of Semiconductors (ITRS), thereby expanding the limited family of p-type oxide semiconductors. Notably, CaO and SrO emerge as the first-of-kind sub-1-nm thickness oxide semiconductors capable of simultaneously meeting the ITRS HP and LP criteria for both n-and p-type devices. CaO and SrO FETs outperform many existing low-dimensional semiconductors, exhibiting scalability below 5-nm gate length. Our findings offer a pioneering effort in the ab initio, device-driven screening of sub-1-nm thickness oxide semiconductors, significantly broadening the material candidate pool for future CMOS technology nodes., Comment: 23 pages, 5 figures, 1 table

Spiking neural networks (SNNs) have low power consumption and bio-interpretable characteristics, and are considered to have tremendous potential for energy-efficient computing. However, the exploration of SNNs on image generation tasks remains very limited, and a unified and effective structure for SNN-based generative models has yet to be proposed. In this paper, we explore a novel diffusion model architecture within spiking neural networks. We utilize transformer to replace the commonly used U-net structure in mainstream diffusion models. It can generate higher quality images with relatively lower computational cost and shorter sampling time. It aims to provide an empirical baseline for research of generative models based on SNNs. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 datasets demonstrate that our work is highly competitive compared to existing SNN generative models.

Traditional end-to-end (E2E) training of deep networks necessitates storing intermediate activations for back-propagation, resulting in a large memory footprint on GPUs and restricted model parallelization. As an alternative, greedy local learning partitions the network into gradient-isolated modules and trains supervisely based on local preliminary losses, thereby providing asynchronous and parallel training methods that substantially reduce memory cost. However, empirical experiments reveal that as the number of segmentations of the gradient-isolated module increases, the performance of the local learning scheme degrades substantially, severely limiting its expansibility. To avoid this issue, we theoretically analyze the greedy local learning from the standpoint of information theory and propose a ContSup scheme, which incorporates context supply between isolated modules to compensate for information loss. Experiments on benchmark datasets (i.e. CIFAR, SVHN, STL-10) achieve SOTA results and indicate that our proposed method can significantly improve the performance of greedy local learning with minimal memory and computational overhead, allowing for the boost of the number of isolated modules. Our codes are available at https://github.com/Tab-ct/ContSup., Comment: 9 figures, 12 tables

Ultra-thin (UT) oxide semiconductors are promising candidates for back-end-of-line (BEOL) compatible transistors and monolithic three-dimensional integration. Experimentally, UT indium oxide (In$_2$O$_3$) field-effect transistors (FETs) with thicknesses down to 0.4 nm exhibits extremely high drain current (10000 $\mu$A/$\mu$m) and transconductance (4000 $\mu$S/$\mu$m). Here, we employ the ab initio quantum transport simulation to investigate the performance limit of sub-5-nm gate length (Lg) UT In$_2$O$_3$ FET. Based on the International Technology Roadmap for Semiconductors (ITRS) criteria for high-performance (HP) devices, the scaling limit of UT In$_2$O$_3$ FETs can reach 2 nm in terms of on-state current, delay time, and power dissipation. The wide bandgap nature of UT In$_2$O$_3$ (3.15 eV) renders it a suitable candidate for ITRS low-power (LP) electronics with Lg down to 3 nm. Both the HP and LP UT In$_2$O$_3$ FETs exhibit superior energy-delay products as compared to other common 2D semiconductors such as monolayer MoS2 and MoTe2. Our study unveils the immense promise of UT In$_2$O$_3$ for both HP and LP device applications., Comment: 16 pages, 7 figures

Gate-all-around (GAA) nanowire (NW) field-effect transistor (FET) is a promising device architecture due to its superior gate controllability than that of the conventional FinFET architecture. The significantly higher electron mobility of indium phosphide (InP) NW than silicon NW makes it particularly well-suited for high-performance (HP) electronics applications. In this work, we perform an ab initio quantum transport simulation to investigate the performance limit of sub-5-nm gate length (Lg) GAA InP NW FETs. The GAA InP NW FETs with Lg of 4 nm can meet the International Technology Roadmap for Semiconductors (ITRS) requirements for HP devices from the perspective of on-state current, delay time, and power dissipation. We also investigate the impact of strain on 3-nm-Lg GAA InP NW FETs. The application of tensile strain results in a remarkable increase of over 60% in the on-state current. These results highlight the potential of GAA InP NW FETs for HP applications in the sub-5-nm Lg region., Comment: 22 pages, 7 figures

Spiking Neural Networks (SNNs), as one of the algorithmic models in neuromorphic computing, have gained a great deal of research attention owing to temporal information processing capability, low power consumption, and high biological plausibility. The potential to efficiently extract spatio-temporal features makes it suitable for processing the event streams. However, existing synaptic structures in SNNs are almost full-connections or spatial 2D convolution, neither of which can extract temporal dependencies adequately. In this work, we take inspiration from biological synapses and propose a spatio-temporal synaptic connection SNN (STSC-SNN) model, to enhance the spatio-temporal receptive fields of synaptic connections, thereby establishing temporal dependencies across layers. Concretely, we incorporate temporal convolution and attention mechanisms to implement synaptic filtering and gating functions. We show that endowing synaptic models with temporal dependencies can improve the performance of SNNs on classification tasks. In addition, we investigate the impact of performance vias varied spatial-temporal receptive fields and reevaluate the temporal modules in SNNs. Our approach is tested on neuromorphic datasets, including DVS128 Gesture (gesture recognition), N-MNIST, CIFAR10-DVS (image classification), and SHD (speech digit recognition). The results show that the proposed model outperforms the state-of-the-art accuracy on nearly all datasets.

Spiking Neural Network (SNN) is considered more biologically realistic and power-efficient as it imitates the fundamental mechanism of the human brain. Recently, backpropagation (BP) based SNN learning algorithms that utilize deep learning frameworks have achieved good performance. However, bio-interpretability is partially neglected in those BP-based algorithms. Toward bio-plausible BP-based SNNs, we consider three properties in modeling spike activities: Multiplicity, Adaptability, and Plasticity (MAP). In terms of multiplicity, we propose a Multiple-Spike Pattern (MSP) with multiple spike transmission to strengthen model robustness in discrete time-iteration. To realize adaptability, we adopt Spike Frequency Adaption (SFA) under MSP to decrease spike activities for improved efficiency. For plasticity, we propose a trainable convolutional synapse that models spike response current to enhance the diversity of spiking neurons for temporal feature extraction. The proposed SNN model achieves competitive performances on neuromorphic datasets: N-MNIST and SHD. Furthermore, experimental results demonstrate that the proposed three aspects are significant to iterative robustness, spike efficiency, and temporal feature extraction capability of spike activities. In summary, this work proposes a feasible scheme for bio-inspired spike activities with MAP, offering a new neuromorphic perspective to embed biological characteristics into spiking neural networks.