Search

This paper introduces TinySaver, an early-exit-like dynamic model compression approach which employs tiny models to substitute large models adaptively. Distinct from traditional compression techniques, dynamic methods like TinySaver can leverage the difficulty differences to allow certain inputs to complete their inference processes early, thereby conserving computational resources. Most existing early exit designs are implemented by attaching additional network branches to the model's backbone. Our study, however, reveals that completely independent tiny models can replace a substantial portion of the larger models' job with minimal impact on performance. Employing them as the first exit can remarkably enhance computational efficiency. By searching and employing the most appropriate tiny model as the computational saver for a given large model, the proposed approaches work as a novel and generic method to model compression. This finding will help the research community in exploring new compression methods to address the escalating computational demands posed by rapidly evolving AI models. Our evaluation of this approach in ImageNet-1k classification demonstrates its potential to reduce the number of compute operations by up to 90\%, with only negligible losses in performance, across various modern vision models.

Modern deep learning (DL) models necessitate the employment of scaling and compression techniques for effective deployment in resource-constrained environments. Most existing techniques, such as pruning and quantization are generally static. On the other hand, dynamic compression methods, such as early exits, reduce complexity by recognizing the difficulty of input samples and allocating computation as needed. Dynamic methods, despite their superior flexibility and potential for co-existing with static methods, pose significant challenges in terms of implementation due to any changes in dynamic parts will influence subsequent processes. Moreover, most current dynamic compression designs are monolithic and tightly integrated with base models, thereby complicating the adaptation to novel base models. This paper introduces DyCE, an dynamic configurable early-exit framework that decouples design considerations from each other and from the base model. Utilizing this framework, various types and positions of exits can be organized according to predefined configurations, which can be dynamically switched in real-time to accommodate evolving performance-complexity requirements. We also propose techniques for generating optimized configurations based on any desired trade-off between performance and computational complexity. This empowers future researchers to focus on the improvement of individual exits without latent compromise of overall system performance. The efficacy of this approach is demonstrated through image classification tasks with deep CNNs. DyCE significantly reduces the computational complexity by 23.5% of ResNet152 and 25.9% of ConvNextv2-tiny on ImageNet, with accuracy reductions of less than 0.5%. Furthermore, DyCE offers advantages over existing dynamic methods in terms of real-time configuration and fine-grained performance tuning.

We investigate a quantum walk on a ring represented by a directed triangle graph with complex edge weights and monitored at a constant rate until the quantum walker is detected. To this end, the first hitting time statistics is recorded using unitary dynamics interspersed stroboscopically by measurements, which is implemented on IBM quantum computers with a midcircuit readout option. Unlike classical hitting times, the statistical aspect of the problem depends on the way we construct the measured path, an effect that we quantify experimentally. First, we experimentally verify the theoretical prediction that the mean return time to a target state is quantized, with abrupt discontinuities found for specific sampling times and other control parameters, which has a well-known topological interpretation. Second, depending on the initial state, system parameters, and measurement protocol, the detection probability can be less than one or even zero, which is related to dark-state physics. Both, return-time quantization and the appearance of the dark states are related to degeneracies in the eigenvalues of the unitary time evolution operator. We conclude that, for the IBM quantum computer under study, the first hitting times of monitored quantum walks are resilient to noise. Yet, a finite number of measurements leads to broadening effects, which modify the topological quantization and chiral effects of the asymptotic theory with an infinite number of measurements. Our results point the way for the development of novel quantum walk algorithms that exploit measurement-induced effects on quantum computers.

We introduce a novel time-energy uncertainty relationship within the context of restarts in monitored quantum dynamics. Initially, we investigate the concept of ``first hitting time'' in quantum systems using an IBM quantum computer and a three-site ring graph as our starting point. Previous studies have established that the mean recurrence time, which represents the time taken to return to the initial state, is quantized as an integer multiple of the sampling time, displaying pointwise discontinuous transitions at resonances. Our findings demonstrate that, the natural utilization of the restart mechanism in laboratory experiments, driven by finite data collection time spans, leads to a broadening effect on the transitions of the mean recurrence time. Our newly proposed uncertainty relation captures the underlying essence of these phenomena, by connecting the broadening of the mean hitting time near resonances, to the intrinsic energies of the quantum system and to the fluctuations of recurrence time. This work not only contributes to our understanding of fundamental aspects related to quantum measurements and dynamics, but also offers practical insights for the design of efficient quantum algorithms with mid-circuit measurements., Comment: 10 pages, 6 figures

Unsupervised learning methods have become increasingly important in deep learning due to their demonstrated large utilization of datasets and higher accuracy in computer vision and natural language processing tasks. There is a growing trend to extend unsupervised learning methods to other domains, which helps to utilize a large amount of unlabelled data. This paper proposes an unsupervised pre-training technique based on masked autoencoder (MAE) for electrocardiogram (ECG) signals. In addition, we propose a task-specific fine-tuning to form a complete framework for ECG analysis. The framework is high-level, universal, and not individually adapted to specific model architectures or tasks. Experiments are conducted using various model architectures and large-scale datasets, resulting in an accuracy of 94.39% on the MITDB dataset for ECG arrhythmia classification task. The result shows a better performance for the classification of previously unseen data for the proposed approach compared to fully supervised methods., Comment: Accepted by IEEE Biomedical Circuits and Systems (BIOCAS) 2023

In recent years, two competitive time series classification models, namely, ROCKET and MINIROCKET, have garnered considerable attention due to their low training cost and high accuracy. However, they rely on a large number of random 1-D convolutional kernels to comprehensively capture features, which is incompatible with resource-constrained devices. Despite the development of heuristic algorithms designed to recognize and prune redundant kernels, the inherent time-consuming nature of evolutionary algorithms hinders efficient evaluation. To efficiently prune models, this paper eliminates feature groups contributing minimally to the classifier, thereby discarding the associated random kernels without direct evaluation. To this end, we incorporate both group-level ($l_{2,1}$-norm) and element-level ($l_2$-norm) regularizations to the classifier, formulating the pruning challenge as a group elastic net classification problem. An ADMM-based algorithm is initially introduced to solve the problem, but it is computationally intensive. Building on the ADMM-based algorithm, we then propose our core algorithm, POCKET, which significantly speeds up the process by dividing the task into two sequential stages. In Stage 1, POCKET utilizes dynamically varying penalties to efficiently achieve group sparsity within the classifier, removing features associated with zero weights and their corresponding kernels. In Stage 2, the remaining kernels and features are used to refit a $l_2$-regularized classifier for enhanced performance. Experimental results on diverse time series datasets show that POCKET prunes up to 60% of kernels without a significant reduction in accuracy and performs 11$\times$ faster than its counterparts. Our code is publicly available at https://github.com/ShaowuChen/POCKET.

Repeatedly-monitored quantum walks with a rate $1/\tau$ yield discrete-time trajectories which are inherently random. With these paths the first-hitting time with sharp restart is studied. We find an instability in the optimal mean hitting time, which is not found in the corresponding classical random walk process. This instability implies that a small change in parameters can lead to a rather large change of the optimal restart time. We show that the optimal restart time versus $\tau$, as a control parameter, exhibits sets of staircases and plunges. The plunges, are due to the mentioned instability, which in turn is related to the quantum oscillations of the first-hitting time probability, in the absence of restarts. Furthermore, we prove that there are only two patterns of staircase structures, dependent on the parity of the distance between the target and the source in units of lattice constant. The global minimum of the hitting time, is controlled not only by the restart time, as in classical problems, but also by the sampling time $\tau$. We provide numerical evidence that this global minimum occurs for the $\tau$ minimizing the mean hitting time, given restarts taking place after each measurement. Last but not least, we numerically show that the instability found in this work is relatively robust against stochastic perturbations in the sampling time $\tau$., Comment: 17 pages (main text), 15 figures