Search

On-device inference for Large Language Models (LLMs), driven by increasing privacy concerns and advancements of mobile-sized models, has gained significant interest. However, even mobile-sized LLMs (e.g., Gemma-2B) encounter unacceptably high inference latency, often bottlenecked by the prefill stage in tasks like screen UI understanding. We present llm.npu, the first LLM inference system utilizing on-device Neural Processing Unit (NPU) offloading to reduce prefill latency. llm.npu enhances NPU offloading efficiency by re-constructing the prompt and model in three levels: (1) At prompt level, it divides variable-length prompts into multiple fixed-sized chunks while maintaining data dependencies; (2) At tensor level, it identifies and extracts significant outliers to run on the CPU/GPU in parallel with minimal overhead; (3) At block level, it schedules Transformer blocks in an out-of-order manner to the CPU/GPU and NPU based on their hardware affinity and sensitivity to accuracy. Compared to competitive baselines, llm.npu achieves 22.4x faster prefill speed and 30.7$\times$ energy savings on average, and up to 32.8x speedup in an end-to-end real-world application. For the first time, llm.npu achieves more than 1,000 tokens/sec prefilling for a billion-sized model.

Let $S$ be a subnormal operator on a separable complex Hilbert space $\mathcal H$ and let $\mu$ be the scalar-valued spectral measure for the minimal normal extension $N$ of $S.$ Let $R^\infty (\sigma(S),\mu)$ be the weak-star closure in $L^\infty (\mu)$ of rational functions with poles off $\sigma(S),$ the spectrum of $S.$ The multiplier algebra $M(S)$ consists of functions $f\in L^\infty(\mu)$ such that $f(N)\mathcal H \subset \mathcal H.$ The multiplication operator $M_{S,f}$ of $f\in M(S)$ is defined $M_{S,f} = f(N) |_{\mathcal H}.$ We show that for $f\in R^\infty (\sigma(S),\mu),$ (1) $M_{S,f}$ is invertible iff $f$ is invertible in $M(S)$ and (2) $M_{S,f}$ is Fredholm iff there exists $f_0\in R^\infty (\sigma(S),\mu)$ and a polynomial $p$ such that $f=pf_0,$ $f_0$ is invertible in $M(S),$ and $p$ has only zeros in $\sigma (S) \setminus \sigma_e (S),$ where $\sigma_e (S)$ denotes the essential spectrum of $S.$ Consequently, we characterize $\sigma(M_{S,f})$ and $\sigma_e(M_{S,f})$ in terms of some cluster subsets of $f.$ Moreover, we show that if $S$ is an irreducible subnormal operator and $f \in R^\infty (\sigma(S),\mu),$ then $M_{S,f}$ is invertible iff $f$ is invertible in $R^\infty (\sigma(S),\mu).$ The results answer the second open question raised by J. Dudziak in 1984.

Optical spectroscopy plays an essential role across scientific research and industry for non-contact materials analysis1-3, increasingly through in-situ or portable platforms4-6. However, when considering low-light-level applications, conventional spectrometer designs necessitate a compromise between their resolution and sensitivity7,8, especially as device and detector dimensions are scaled down. Here, we report on a miniaturizable spectrometer platform where light throughput onto the detector is instead enhanced as the resolution is increased. This planar, CMOS-compatible platform is based around metasurface encoders designed to exhibit photonic bound states in the continuum9, where operational range can be altered or extended simply through adjusting geometric parameters. This system can enhance photon collection efficiency by up to two orders of magnitude versus conventional designs; we demonstrate this sensitivity advantage through ultra-low-intensity fluorescent and astrophotonic spectroscopy. This work represents a step forward for the practical utility of spectrometers, affording a route to integrated, chip-based devices that maintain high resolution and SNR without requiring prohibitively long integration times.

The imperative task of revising or updating the knowledge stored within large language models arises from two distinct sources: intrinsic errors inherent in the model which should be corrected and outdated knowledge due to external shifts in the real world which should be updated. Prevailing efforts in model editing conflate these two distinct categories of edits arising from distinct reasons and directly modify the original knowledge in models into new knowledge. However, we argue that preserving the model's original knowledge remains pertinent. Specifically, if a model's knowledge becomes outdated due to evolving worldly dynamics, it should retain recollection of the historical knowledge while integrating the newfound knowledge. In this work, we introduce the task of Temporal Knowledge Editing (TKE) and establish a benchmark AToKe (Assessment of TempOral Knowledge Editing) to evaluate current model editing methods. We find that while existing model editing methods are effective at making models remember new knowledge, the edited model catastrophically forgets historical knowledge. To address this gap, we propose a simple and general framework termed Multi-Editing with Time Objective (METO) for enhancing existing editing models, which edits both historical and new knowledge concurrently and optimizes the model's prediction for the time of each fact. Our assessments demonstrate that while AToKe is still difficult, METO maintains the effectiveness of learning new knowledge and meanwhile substantially improves the performance of edited models on utilizing historical knowledge., Comment: AAAI 2024. 9 pages, 3 figures

Fourier ptychography (FP) is an enabling imaging technique that produces high-resolution complex-valued images with extended field coverages. However, when FP images a phase object with any specific spatial frequency, the captured images contain only constant values, rendering the recovery of the corresponding linear phase ramp impossible. This challenge is not unique to FP but also affects other common microscopy techniques -- a rather counterintuitive outcome given their widespread use in phase imaging. The underlying issue originates from the non-uniform phase transfer characteristic inherent in microscope systems, which impedes the conversion of object wavefields into discernible intensity variations. To address this challenge, we present spatially-coded Fourier ptychography (scFP), a new method that synergizes FP with spatial-domain coded detection for true quantitative phase imaging. In scFP, a flexible and detachable coded thin film is attached atop the image sensor in a regular FP setup. The spatial modulation of this thin film ensures a uniform phase response across the entire synthetic bandwidth. It improves reconstruction quality and corrects refractive index underestimation issues prevalent in conventional FP and related tomographic implementations. The inclusion of the coded thin film further adds a new dimension of measurement diversity in the spatial domain. The development of scFP is expected to catalyse new research directions and applications for phase imaging, emphasizing the need for true quantitative accuracy with uniform frequency response.

For a compact subset $K\subset \mathbb C$ and a positive finite Borel measure $\mu$ supported on $K,$ let $\text{Rat}(K)$ denote the space of rational functions with poles off $K,$ let $R^\infty (K,\mu)$ be the weak-star closure of $\text{Rat}(K)$ in $L^\infty (\mu),$ and let $R^2 (K,\mu)$ be the closure of $\text{Rat}(K)$ in $L^2(\mu).$ We show that there exists a compact subset $K\subset \mathbb C,$ a positive finite Borel measure $\mu$ supported on $K,$ and a function $f\in R^\infty (K,\mu)$ such that $R^\infty (K,\mu)$ has no non-trivial direct $L^\infty$ summands, $f$ is invertible in $R^2 (K,\mu)\cap L^\infty(\mu),$ and $f$ is not invertible in $R^\infty (K,\mu).$ The result answers an open question concerning spectral mapping theorems for subnormal operators raised by J. Dudziak in 1984., Comment: arXiv admin note: text overlap with arXiv:2301.06305

For a compact set $K\subset \mathbb C,$ a finite positive Borel measure $\mu$ on $K,$ and $1 \le t < \i,$ let $\text{Rat}(K)$ be the set of rational functions with poles off $K$ and let $R^t(K, \mu)$ be the closure of $\text{Rat}(K)$ in $L^t(\mu).$ For a bounded Borel subset $\mathcal D\subset \mathbb C,$ let $\area_{\mathcal D}$ denote the area (Lebesgue) measure restricted to $\mathcal D$ and let $H^\i (\mathcal D)$ be the weak-star closed sub-algebra of $L^\i(\area_{\mathcal D})$ spanned by $f,$ bounded and analytic on $\mathbb C\setminus E_f$ for some compact subset $E_f \subset \mathbb C\setminus \mathcal D.$ We show that if $R^t(K, \mu)$ contains no non-trivial direct $L^t$ summands, then there exists a Borel subset $\mathcal R \subset K$ whose closure contains the support of $\mu$ and there exists an isometric isomorphism and a weak-star homeomorphism $\rho$ from $R^t(K, \mu) \cap L^\infty(\mu)$ onto $H^\infty(\mathcal R)$ such that $\rho(r) = r$ for all $r\in\text{Rat}(K).$ Consequently, we obtain some structural decomposition theorems for $\rtkmu$., Comment: arXiv admin note: text overlap with arXiv:2212.10811

Polarimetric imaging provides valuable insights into the polarization state of light interacting with a sample. It can infer crucial birefringence properties of bio-specimens without using any labels, thereby facilitating the diagnosis of diseases such as cancer and osteoarthritis. In this study, we present a novel polarimetric coded ptychography (pol-CP) approach that enables high-resolution, high-throughput gigapixel birefringence imaging on a chip. Our platform deviates from traditional lens-based polarization systems by employing an integrated polarimetric coded sensor for lensless coherent diffraction imaging. Utilizing Jones calculus, we quantitatively determine the birefringence retardance and orientation information of bio-specimens from the recovered images. Our portable pol-CP prototype can resolve the 435-nm linewidth on the resolution target and the imaging field of view for a single acquisition is limited only by the detector size of 41^2. The prototype allows for the acquisition of gigapixel birefringence images with a 180-mm^2 field of view in ~3.5 minutes, a performance that rivals high-end whole slide scanner but a small fraction of the cost. To demonstrate its biomedical applications, we perform high-throughput imaging of malaria-infected blood smears, locating parasites using birefringence contrast. We also generate birefringence maps of label-free thyroid smears to identify thyroid follicles. Notably, the recovered birefringence maps emphasize the same regions as autofluorescence images, underscoring the potential for rapid on-site evaluation of label-free biopsies. Our approach provides a turnkey and portable solution for lensless polarimetric analysis on a chip, with promising applications in disease diagnosis, crystal screening, and label-free chemical imaging, particularly in resource-constrained environments.

For $K\subset \mathbb C$ a compact subset and $\mu$ a positive finite Bore1 measure supported on $K,$ let $R^\infty (K,\mu)$ be the weak-star closure in $L^\infty (\mu)$ of rational functions with poles off $K.$ We show that if $R^\infty (K,\mu)$ has no non-trivial $L^\infty$ summands and $f\in R^\infty (K,\mu),$ then $f$ is invertible in $R^\infty (K,\mu)$ if and only if Chaumat's map for $K$ and $\mu$ applied to $f$ is bounded away from zero on the envelope with respect to $K$ and $\mu.$ The result proves the conjecture $\diamond$ posed by J. Dudziak in 1984., Comment: J. Funct. Anal. (2023). arXiv admin note: text overlap with arXiv:2212.10811

In 1991, J. Thomson obtained a celebrated decomposition theorem for $P^t(\mu),$ the closed subspace of $L^t(\mu)$ spanned by the analytic polynomials, when $1 \le t < \i.$ In 2008, J. Brennan \cite{b08} generalized Thomson's theorem to $R^t(K, \mu),$ the closed subspace of $L^t(\mu)$ spanned by the rational functions with poles off a compact subset $K$ containing the support of $\mu,$ when the diameters of the components of $\mathbb C\setminus K$ are bounded below. We extend the above decomposition theorems for $R^t(K, \mu)$ when the boundary of $K$ is not too wild., Comment: arXiv admin note: text overlap with arXiv:1904.06446, arXiv:2212.05392

In 1991, J. Thomson obtained celebrated structural results for $P^t(\mu).$ Later, J. Brennan (2008) generalized Thomson's theorem to $R^t(K,\mu)$ when the diameters of the components of $\mathbb C\setminus K$ are bounded below. The results indicate that if $R^t(K,\mu)$ is pure, then $R^t(K,\mu) \cap L^\infty (\mu)$ is the "same as" the algebra of bounded analytic functions on $\mbox{abpe}(R^t(K, \mu)),$ the set of analytic bounded point evaluations. We show that if the diameters of the components of $\mathbb C\setminus K$ are allowed to tend to zero, then even though $\text{int}(K) = \mbox{abpe}(R^t(K, \mu))$ and $K =\overline {\text{int}(K)},$ the algebra $R^t(K,\mu) \cap L^\infty (\mu)$ may "be equal to" a proper sub-algebra of bounded analytic functions on $\text{int}(K),$ where functions in the sub-algebra are "continuous" on certain portions of the inner boundary of $K.$, Comment: arXiv admin note: text overlap with arXiv:1904.06446