Search

Hyperspectral imaging provides high-dimensional spatial–temporal–spectral information showing intrinsic matter characteristics1–5. Here we report an on-chip computational hyperspectral imaging framework with high spatial and temporal resolution. By integrating different broadband modulation materials on the image sensor chip, the target spectral information is non-uniformly and intrinsically coupled to each pixel with high light throughput. Using intelligent reconstruction algorithms, multi-channel images can be recovered from each frame, realizing real-time hyperspectral imaging. Following this framework, we fabricated a broadband visible–near-infrared (400–1,700 nm) hyperspectral image sensor using photolithography, with an average light throughput of 74.8% and 96 wavelength channels. The demonstrated resolution is 1,024 × 1,024 pixels at 124 fps. We demonstrated its wide applications, including chlorophyll and sugar quantification for intelligent agriculture, blood oxygen and water quality monitoring for human health, textile classification and apple bruise detection for industrial automation, and remote lunar detection for astronomy. The integrated hyperspectral image sensor weighs only tens of grams and can be assembled on various resource-limited platforms or equipped with off-the-shelf optical systems. The technique transforms the challenge of high-dimensional imaging from a high-cost manufacturing and cumbersome system to one that is solvable through on-chip compression and agile computation.

The recently developed image-free sensing technique decouples semantic information directly from compressed measurements without image reconstruction, which maintains the advantages of both the light hardware and software. However, the existing attempts have failed to classify multi-semantic information with multiple targets in the practical fieldof-view. In this Letter, we report a novel image-free sensing technique to tackle the multi-target recognition challenge for the first time, to the best of our knowledge. Different from the convolutional layer stack of image-free single-pixel networks, the reported convolutional recurrent neural network (CRNN) uses the bidirectional LSTM architecture to predict the distribution of multiple characters simultaneously. The framework enables capture of the long-range dependencies, providing a high recognition accuracy of multiple characters. We demonstrate the technique’s effectiveness in license plate detection, which achieves a recognition accuracy of 87.60% at a sampling rate of 5% with a refresh rate higher than 100 FPS.

Single-molecule localization microscopy (SMLM) can bypass the diffraction limit of optical microscopes and greatly improve the resolution in fluorescence microscopy. By introducing the point spread function (PSF) engineering technique, we can customize depth varying PSF to achieve higher axial resolution. However, most existing 3D single-molecule localization algorithms require excited fluorescent molecules to be sparse and captured at high signal-to-noise ratios, which results in a long acquisition time and precludes SMLM’s further applications in many potential fields. To address this problem, we propose a novel 3D single-molecular localization method based on a multi-channel neural network based on U-Net. By leveraging the deep network’s great advantages in feature extraction, the proposed network can reliably discriminate dense fluorescent molecules with overlapped PSFs and corrupted by sensor noise. Both simulated and real experiments demonstrate its superior performance in PSF engineered microscopes with short exposure and dense excitations, which holds great potential in fast 3D super-resolution microscopy.

Ptychography is a predominant non-interferometric technique to image large complex fields but with quite a narrow working spectrum, because diffraction measurements require dense array detection with an ultra-high dynamic range. Here we report a single-pixel ptychography technique that realizes non-interferometric and non-scanning complex-field imaging in a wide waveband, where 2D dense detector arrays are not available. A single-pixel detector is placed in the far field to record the DC-only component of the diffracted wavefront scattered from the target field, which is illuminated by a sequence of binary modulation patterns. This decreases the measurements’ dynamic range by several orders of magnitude. We employ an efficient single-pixel phase-retrieval algorithm to jointly recover the field’s 2D amplitude and phase maps from the 1D intensity-only measurement sequence. No a priori object information is needed in the recovery process. We validate the technique’s quantitative phase imaging nature using both calibrated phase objects and biological samples and demonstrate its wide working spectrum with both 488-nm visible light and 980-nm near-infrared light.

Conventional coherent diffraction imaging (CDI) suffers from inherent phase retrieval ambiguity due to limited intensity-only measurements. Coded illumination with multiple modulations has been introduced to tackle the underdetermination challenge, which however slows down imaging speed. In addition, the required high-dynamic-range acquisition at the Fourier plane is also time consuming. To increase imaging speed, we report an accelerated coded CDI method in this Letter. It requires only three binarized intensity patterns to illuminate the full field, which can be implemented at ∼22kHz using a digital micromirror device. Each diffraction pattern at the Fourier plane is acquired in a single shot without high-dynamic-range synthesis, resulting in three intensity-only images corrupted with underexposed pixels. We develop an adaptive phase retrieval algorithm to adaptively remove the negative influence of underexposure and recover both the object’s amplitude and phase. Both simulations and experiments validate that the method enables fast and high-fidelity complex-field imaging.

In this Letter, we present, to the best of our knowledge, a novel illumination modulation method for reflective and fluorescent separation by using only one spectral image. Specifically, we present an optical system using off-the-shelf devices to generate high frequency illumination, which is desirable in reflective-fluorescent separation tasks. In addition, we employ the total variation regularization scheme to account for spectral–spatial correlation, which makes our method robust to noise. Experiments on both simulated and real data verify the effectiveness and practicality of our method.

Supplemental Digital Content is available in the text.

Single-pixel imaging (SPI) is a novel technique that captures 2D images using a photodiode, instead of conventional 2D array sensors. SPI has high signal-to-noise ratio, wide spectral range, low cost, and robustness to light scattering. Various algorithms have been proposed for SPI reconstruction, including linear correlation methods, the alternating projection (AP) method, and compressive sensing (CS) based methods. However, there has been no comprehensive review discussing respective advantages, which is important for SPI’s further applications and development. In this paper, we review and compare these algorithms in a unified reconstruction framework. We also propose two other SPI algorithms, including a conjugate gradient descent (CGD) based method and a Poisson maximum-likelihood-based method. Both simulations and experiments validate the following conclusions: to obtain comparable reconstruction accuracy, the CS-based total variation (TV) regularization method requires the fewest measurements and consumes the least running time for small-scale reconstruction, the CGD and AP methods run fastest in large-scale cases, and the TV and AP methods are the most robust to measurement noise. In a word, there are trade-offs in capture efficiency, computational complexity, and robustness to noise among different SPI algorithms. We have released our source code for non-commercial use.

Fourier ptychography is a novel imaging technique with high space-bandwidth product (SBP) on the scale of gigapixel, and has been successfully applied for high-resolution and large field-of-view (FOV) microscopy, termed Fourier ptychographic microscopy (FPM). FPM utilizes a low-numerical-aperture objective lens to capture multiple large FOV but low-resolution images under angularly varying illumination, and uses phase-retrieval algorithms to reconstruct the sample’s high-resolution amplitude and phase information. FPM is advantageous over conventional high SBP microscopy techniques in many aspects, including no mechanical scanning, computational acquisition of both amplitude and phase, extended depth of focus, long working distance, and high compatibility with current microscopes. In this paper, we review FPM with its principles, multiple techniques to improve its performance, and its various applications and extensions.

Fourier ptychography (FP) is a recently proposed computational imaging technique for high space-bandwidth product imaging. In real setups such as endoscope and transmission electron microscope, the common sample motion largely degrades the FP reconstruction and limits its practicability. In this paper, we propose a novel FP reconstruction method to efficiently correct for unknown sample motion. Specifically, we adaptively update the sample’s Fourier spectrum from low spatial-frequency regions towards high spatial-frequency ones, with an additional motion recovery and phase-offset compensation procedure for each sub-spectrum. Benefiting from the phase retrieval redundancy theory, the required large overlap between adjacent sub-spectra offers an accurate guide for successful motion recovery. Experimental results on both simulated data and real captured data show that the proposed method can correct for unknown sample motion with its standard deviation being up to 10% of the field-of-view scale. We have released our source code for non-commercial use, and it may find wide applications in related FP platforms such as endoscopy and transmission electron microscopy.

Structured illumination technique enhances the lateral resolution by projecting non-uniform intensity patterns on a sample. In a typical implementation, three lateral phase shifts (0, 2π/3, 4π/3) are needed for each orientation of the sinusoidal pattern, and 3 different orientations are needed to double the bandwidth isotopically in the Fourier domain. To this end, 9 incoherent images are needed in the acquisition process. In this paper, we discuss an imaging strategy for the structured illumination technique and demonstrate the use of a modified incoherent Fourier ptychographic procedure for reducing the number of acquisitions. In the first implementation, we used complementary sinusoidal patterns for sample illumination. We show that, the number of lateral phase shifts can be reduced from 3 to 2 for each orientation of the sinusoidal pattern and the total number of image acquisitions can be reduced to 6 with 3 orientations. In the second implementation, we further reduce the number of image acquisitions to 4. We also show that, the resolution-doubled image can be recovered even with unknown phases of the sinusoidal patterns. We validate the proposed imaging procedure with non-fluorescence samples. The reported approach may shorten the acquisition time of super-resolution imaging and reduce phototoxicity of biological samples.

Fourier ptychography (FP) is a recently reported technique, for large field-of-view and high-resolution imaging. Specifically, FP captures a set of low-resolution images, under angularly varying illuminations, and stitches them together in the Fourier domain. One of FP’s main disadvantages is its long capturing process, due to the requisite large number of incident illumination angles. In this Letter, utilizing the sparsity of natural images in the Fourier domain, we propose a highly efficient method, termed adaptive Fourier ptychography (AFP), which applies content adaptive illumination for FP, to capture the most informative parts of the scene’s spatial spectrum. We validate the effectiveness and efficiency of the reported framework, with both simulated and real experiments. Results show that the proposed AFP could shorten the acquisition time of conventional FP, by around 30%–60%.

Previous studies showed an inverse association between ideal cardiovascular health (CVH) metrics and the total risk of cardiovascular diseases and stroke. This study aimed to investigate the relationship between ideal CVH metrics and the risks of ischemic and hemorrhagic stroke, respectively.

High-throughput computational imaging requires efficient processing algorithms to retrieve multi-dimensional and multi-scale information. In computational phase imaging, phase retrieval (PR) is required to reconstruct both amplitude and phase in complex space from intensity-only measurements. The existing PR algorithms suffer from the tradeoff among low computational complexity, robustness to measurement noise and strong generalization on different modalities. In this work, we report an efficient large-scale phase retrieval technique termed as LPR. It extends the plug-and-play generalized-alternating-projection framework from real space to nonlinear complex space. The alternating projection solver and enhancing neural network are respectively derived to tackle the measurement formation and statistical prior regularization. This framework compensates the shortcomings of each operator, so as to realize high-fidelity phase retrieval with low computational complexity and strong generalization. We applied the technique for a series of computational phase imaging modalities including coherent diffraction imaging, coded diffraction pattern imaging, and Fourier ptychographic microscopy. Extensive simulations and experiments validate that the technique outperforms the existing PR algorithms with as much as 17dB enhancement on signal-to-noise ratio, and more than one order-of-magnitude increased running efficiency. Besides, we for the first time demonstrate ultra-large-scale phase retrieval at the 8K level (7680×4320pixels) in minute-level time.