Search

Text-to-image (T2I) generation with Stable Diffusion models (SDMs) involves high computing demands due to billion-scale parameters. To enhance efficiency, recent studies have reduced sampling steps and applied network quantization while retaining the original architectures. The lack of architectural reduction attempts may stem from worries over expensive retraining for such massive models. In this work, we uncover the surprising potential of block pruning and feature distillation for low-cost general-purpose T2I. By removing several residual and attention blocks from the U-Net of SDMs, we achieve 30%~50% reduction in model size, MACs, and latency. We show that distillation retraining is effective even under limited resources: using only 13 A100 days and a tiny dataset, our compact models can imitate the original SDMs (v1.4 and v2.1-base with over 6,000 A100 days). Benefiting from the transferred knowledge, our BK-SDMs deliver competitive results on zero-shot MS-COCO against larger multi-billion parameter models. We further demonstrate the applicability of our lightweight backbones in personalized generation and image-to-image translation. Deployment of our models on edge devices attains 4-second inference. We hope this work can help build small yet powerful diffusion models with feasible training budgets. Code and models can be found at: https://github.com/Nota-NetsPresso/BK-SDM, Comment: Updated results; Preliminary version at ICML Workshop on ES-FoMo (2023): https://openreview.net/forum?id=bOVydU0XKC

We introduce a weighted Sobolev space theory for the non-local elliptic equation $$ \Delta^{\alpha/2}u=f, \quad x\in D\,; \quad u|_{\overline{D}^c}=g $$ as well as for the non-local parabolic equation $$ u_t=\Delta^{\alpha/2}u+f, \quad t>0,\, x\in D \,; \quad u(0,\cdot)|_D=u_0, \,u|_{(0,T)\times \overline{D}^c}=g. $$ Here, $\alpha\in (0,2)$ and $D$ is a $C^{1,1}$ open set. We prove uniqueness and existence results in weighted Sobolev spaces. We measure the Sobolev and H\"older regularities of arbitrary order derivatives of solutions using a system of weights consisting of appropriate powers of the distance to the boundary. One of the most interesting features of our results is that, unlike the classical result in Sobolev spaces without weights, the weighted regularities of solutions in $D$ are barely affected by those of $g$ in $\overline{D}^c$. For instance, even if $g=\delta_{x_0}$, the dirac delta distribution concentrated at $x_0\in \overline{D}^c $, the solution to the elliptic equation given with $f=0$ is infinitely differentiable in $D$, and for any $k=0,1,2, 3,\cdots$, $\varepsilon>0$, and $\delta\in (0,1)$, it holds that $$ |d_x^{-\frac{\alpha}{2}+\varepsilon+k}D^k_xu|_{C(D)} +|d_x^{-\frac{\alpha}{2}+\varepsilon+k+\delta} D^k_xu|_{C^{\delta}(D)}<\infty, $$ where $d_x=dist(x, \partial D)$., Comment: 46 pages

Accurate delineation of key waveforms in an ECG is a critical step in extracting relevant features to support the diagnosis and treatment of heart conditions. Although deep learning based methods using segmentation models to locate P, QRS, and T waves have shown promising results, their ability to handle arrhythmias has not been studied in any detail. In this paper we investigate the effect of arrhythmias on delineation quality and develop strategies to improve performance in such cases. We introduce a U-Net-like segmentation model for ECG delineation with a particular focus on diverse arrhythmias. This is followed by a post-processing algorithm which removes noise and automatically determines the boundaries of P, QRS, and T waves. Our model has been trained on a diverse dataset and evaluated against the LUDB and QTDB datasets to show strong performance, with F1-scores exceeding 99% for QRS and T waves, and over 97% for P waves in the LUDB dataset. Furthermore, we assess various models across a wide array of arrhythmias and observe that models with a strong performance on standard benchmarks may still perform poorly on arrhythmias that are underrepresented in these benchmarks, such as tachycardias. We propose solutions to address this discrepancy.

Very long baseline radio interferometry (VLBI) with ground-based observatories is limited by the size of Earth, the geographic distribution of antennas, and the transparency of the atmosphere. In this whitepaper, we present Capella, a tentative design of a space-only VLBI system. Using four small (<500 kg) satellites in two orthogonal polar low-Earth orbit planes, and single-band heterodyne receivers operating at frequencies around 690 GHz, the interferometer is able to achieve angular resolutions of approximately 7 microarcsec. Within a total observing time of three days, a near-complete uv plane coverage can be reached. The technology for all key components required for Capella - radio telescope, receiver, sampler, recorder, frequency standard, positioning system, data downlink, and pointing control system - is already available, partially off-the-shelf. Making realistic assumptions, the science payload of each satellite has a mass of about 190 kg and consumes about 460 W of power. The data from the telescopes can be correlated on the ground using dedicated versions of existing Fourier transform (FX) software correlators. With the specifications assumed in this whitepaper, Capella will be able to address a range of science cases, including: photon rings around supermassive black holes; the acceleration and collimation zones of plasma jets emitted from the vicinity of supermassive black holes; the chemical composition of accretion flows into active galactic nuclei through observations of molecular absorption lines; mapping supermassive binary black holes; the magnetic activity of stars; and nova eruptions of symbiotic binary stars - and, like any substantially new observing technique, has the potential for unexpected discoveries., Comment: 20 pages, 3 figures, 1 table. Whitepaper version 1.4. Updates: frequency reference, sensitivity, mass and power budgets, list of authors. Abstract abridged. Living document, will be updated when necessary

Selfies represent people's desires, and social media platforms like Instagram have been flooded with them. This study uses selfie data to examine how peoples' desires for ideal facial representations vary by region, particularly in East Asia. Through the analysis, we aim to refute the "all Asians prefer identical visuals," which is a subset of the prevalent Western belief that "all Asians look the same." Our findings, reinforced by postcolonial interpretations, dispute those assumptions. We propose a strategy for resolving the mismatch between real-world desires and the Western beauty market's views. We expect the disparity between hegemonic color schemes and the augmented skin colors shown by our results may facilitate the study of color and Asian identity.

Virtual humans have gained considerable attention in numerous industries, e.g., entertainment and e-commerce. As a core technology, synthesizing photorealistic face frames from target speech and facial identity has been actively studied with generative adversarial networks. Despite remarkable results of modern talking-face generation models, they often entail high computational burdens, which limit their efficient deployment. This study aims to develop a lightweight model for speech-driven talking-face synthesis. We build a compact generator by removing the residual blocks and reducing the channel width from Wav2Lip, a popular talking-face generator. We also present a knowledge distillation scheme to stably yet effectively train the small-capacity generator without adversarial learning. We reduce the number of parameters and MACs by 28$\times$ while retaining the performance of the original model. Moreover, to alleviate a severe performance drop when converting the whole generator to INT8 precision, we adopt a selective quantization method that uses FP16 for the quantization-sensitive layers and INT8 for the other layers. Using this mixed precision, we achieve up to a 19$\times$ speedup on edge GPUs without noticeably compromising the generation quality., Comment: MLSys Workshop on On-Device Intelligence, 2023; Demo: https://huggingface.co/spaces/nota-ai/compressed_wav2lip