Search

This paper introduces TinySaver, an early-exit-like dynamic model compression approach which employs tiny models to substitute large models adaptively. Distinct from traditional compression techniques, dynamic methods like TinySaver can leverage the difficulty differences to allow certain inputs to complete their inference processes early, thereby conserving computational resources. Most existing early exit designs are implemented by attaching additional network branches to the model's backbone. Our study, however, reveals that completely independent tiny models can replace a substantial portion of the larger models' job with minimal impact on performance. Employing them as the first exit can remarkably enhance computational efficiency. By searching and employing the most appropriate tiny model as the computational saver for a given large model, the proposed approaches work as a novel and generic method to model compression. This finding will help the research community in exploring new compression methods to address the escalating computational demands posed by rapidly evolving AI models. Our evaluation of this approach in ImageNet-1k classification demonstrates its potential to reduce the number of compute operations by up to 90\%, with only negligible losses in performance, across various modern vision models.

Conventional deep learning (DL) model compression and scaling methods focus on altering the model's components, impacting the results across all samples uniformly. However, since samples vary in difficulty, a dynamic model that adapts computation based on sample complexity offers a novel perspective for compression and scaling. Despite this potential, existing dynamic models are typically monolithic and model-specific, limiting their generalizability as broad compression and scaling methods. Additionally, most deployed DL systems are fixed, unable to adjust their scale once deployed and, therefore, cannot adapt to the varying real-time demands. This paper introduces DyCE, a dynamically configurable system that can adjust the performance-complexity trade-off of a DL model at runtime without requiring re-initialization or redeployment on inference hardware. DyCE achieves this by adding small exit networks to intermediate layers of the original model, allowing computation to terminate early if acceptable results are obtained. DyCE also decouples the design of an efficient dynamic model, facilitating easy adaptation to new base models and potential general use in compression and scaling. We also propose methods for generating optimized configurations and determining the types and positions of exit networks to achieve desired performance and complexity trade-offs. By enabling simple configuration switching, DyCE provides fine-grained performance tuning in real-time. We demonstrate the effectiveness of DyCE through image classification tasks using deep convolutional neural networks (CNNs). DyCE significantly reduces computational complexity by 23.5% for ResNet152 and 25.9% for ConvNextv2-tiny on ImageNet, with accuracy reductions of less than 0.5%.

We investigate a quantum walk on a ring represented by a directed triangle graph with complex edge weights and monitored at a constant rate until the quantum walker is detected. To this end, the first hitting time statistics is recorded using unitary dynamics interspersed stroboscopically by measurements, which is implemented on IBM quantum computers with a midcircuit readout option. Unlike classical hitting times, the statistical aspect of the problem depends on the way we construct the measured path, an effect that we quantify experimentally. First, we experimentally verify the theoretical prediction that the mean return time to a target state is quantized, with abrupt discontinuities found for specific sampling times and other control parameters, which has a well-known topological interpretation. Second, depending on the initial state, system parameters, and measurement protocol, the detection probability can be less than one or even zero, which is related to dark-state physics. Both, return-time quantization and the appearance of the dark states are related to degeneracies in the eigenvalues of the unitary time evolution operator. We conclude that, for the IBM quantum computer under study, the first hitting times of monitored quantum walks are resilient to noise. Yet, a finite number of measurements leads to broadening effects, which modify the topological quantization and chiral effects of the asymptotic theory with an infinite number of measurements. Our results point the way for the development of novel quantum walk algorithms that exploit measurement-induced effects on quantum computers.

We introduce a novel time-energy uncertainty relationship within the context of restarts in monitored quantum dynamics. Initially, we investigate the concept of ``first hitting time'' in quantum systems using an IBM quantum computer and a three-site ring graph as our starting point. Previous studies have established that the mean recurrence time, which represents the time taken to return to the initial state, is quantized as an integer multiple of the sampling time, displaying pointwise discontinuous transitions at resonances. Our findings demonstrate that, the natural utilization of the restart mechanism in laboratory experiments, driven by finite data collection time spans, leads to a broadening effect on the transitions of the mean recurrence time. Our newly proposed uncertainty relation captures the underlying essence of these phenomena, by connecting the broadening of the mean hitting time near resonances, to the intrinsic energies of the quantum system and to the fluctuations of recurrence time. This work not only contributes to our understanding of fundamental aspects related to quantum measurements and dynamics, but also offers practical insights for the design of efficient quantum algorithms with mid-circuit measurements., Comment: 10 pages, 6 figures

Unsupervised learning methods have become increasingly important in deep learning due to their demonstrated large utilization of datasets and higher accuracy in computer vision and natural language processing tasks. There is a growing trend to extend unsupervised learning methods to other domains, which helps to utilize a large amount of unlabelled data. This paper proposes an unsupervised pre-training technique based on masked autoencoder (MAE) for electrocardiogram (ECG) signals. In addition, we propose a task-specific fine-tuning to form a complete framework for ECG analysis. The framework is high-level, universal, and not individually adapted to specific model architectures or tasks. Experiments are conducted using various model architectures and large-scale datasets, resulting in an accuracy of 94.39% on the MITDB dataset for ECG arrhythmia classification task. The result shows a better performance for the classification of previously unseen data for the proposed approach compared to fully supervised methods., Comment: Accepted by IEEE Biomedical Circuits and Systems (BIOCAS) 2023

In recent years, two competitive time series classification models, namely, ROCKET and MINIROCKET, have garnered considerable attention due to their low training cost and high accuracy. However, they rely on a large number of random 1-D convolutional kernels to comprehensively capture features, which is incompatible with resource-constrained devices. Despite the development of heuristic algorithms designed to recognize and prune redundant kernels, the inherent time-consuming nature of evolutionary algorithms hinders efficient evaluation. To efficiently prune models, this paper eliminates feature groups contributing minimally to the classifier, thereby discarding the associated random kernels without direct evaluation. To this end, we incorporate both group-level ($l_{2,1}$-norm) and element-level ($l_2$-norm) regularizations to the classifier, formulating the pruning challenge as a group elastic net classification problem. An ADMM-based algorithm is initially introduced to solve the problem, but it is computationally intensive. Building on the ADMM-based algorithm, we then propose our core algorithm, POCKET, which significantly speeds up the process by dividing the task into two sequential stages. In Stage 1, POCKET utilizes dynamically varying penalties to efficiently achieve group sparsity within the classifier, removing features associated with zero weights and their corresponding kernels. In Stage 2, the remaining kernels and features are used to refit a $l_2$-regularized classifier for enhanced performance. Experimental results on diverse time series datasets show that POCKET prunes up to 60% of kernels without a significant reduction in accuracy and performs 11$\times$ faster than its counterparts. Our code is publicly available at https://github.com/ShaowuChen/POCKET.

Repeatedly-monitored quantum walks with a rate $1/\tau$ yield discrete-time trajectories which are inherently random. With these paths the first-hitting time with sharp restart is studied. We find an instability in the optimal mean hitting time, which is not found in the corresponding classical random walk process. This instability implies that a small change in parameters can lead to a rather large change of the optimal restart time. We show that the optimal restart time versus $\tau$, as a control parameter, exhibits sets of staircases and plunges. The plunges, are due to the mentioned instability, which in turn is related to the quantum oscillations of the first-hitting time probability, in the absence of restarts. Furthermore, we prove that there are only two patterns of staircase structures, dependent on the parity of the distance between the target and the source in units of lattice constant. The global minimum of the hitting time, is controlled not only by the restart time, as in classical problems, but also by the sampling time $\tau$. We provide numerical evidence that this global minimum occurs for the $\tau$ minimizing the mean hitting time, given restarts taking place after each measurement. Last but not least, we numerically show that the instability found in this work is relatively robust against stochastic perturbations in the sampling time $\tau$., Comment: 17 pages (main text), 15 figures