Search

To study the behavior of freely moving model organisms such as zebrafish (Danio rerio) and fruit flies (Drosophila) across multiple spatial scales, it would be ideal to use a light microscope that can resolve 3D information over a wide field of view (FOV) at high speed and high spatial resolution. However, it is challenging to design an optical instrument to achieve all of these properties simultaneously. Existing techniques for large-FOV microscopic imaging and for 3D image measurement typically require many sequential image snapshots, thus compromising speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over a 135-cm^2 area, achieving up to 230 frames per second at throughputs exceeding 5 gigapixels (GPs) per second. 3D-RAPID features a 3D reconstruction algorithm that, for each synchronized temporal snapshot, simultaneously fuses all 54 images seamlessly into a globally-consistent composite that includes a coregistered 3D height map. The self-supervised 3D reconstruction algorithm itself trains a spatiotemporally-compressed convolutional neural network (CNN) that maps raw photometric images to 3D topography, using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. As a result, our end-to-end 3D reconstruction algorithm is robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. The scalable hardware and software design of 3D-RAPID addresses a longstanding problem in the field of behavioral imaging, enabling parallelized 3D observation of large collections of freely moving organisms at high spatiotemporal throughputs, which we demonstrate in ants (Pogonomyrmex barbatus), fruit flies, and zebrafish larvae.

This paper experimentally examines different configurations of a multi-camera array microscope (MCAM) imaging technology. The MCAM is based upon a densely packed array of “micro-cameras” to jointly image across a large field-of-view (FOV) at high resolution. Each micro-camera within the array images a unique area of a sample of interest, and then all acquired data with 54 micro-cameras are digitally combined into composite frames, whose total pixel counts significantly exceed the pixel counts of standard microscope systems. We present results from three unique MCAM configurations for different use cases. First, we demonstrate a configuration that simultaneously images and estimates the 3D object depth across a 100×135mm2 FOV at approximately 20 µm resolution, which results in 0.15 gigapixels (GP) per snapshot. Second, we demonstrate an MCAM configuration that records video across a continuous 83×123mm2 FOV with twofold increased resolution (0.48 GP per frame). Finally, we report a third high-resolution configuration (2 µm resolution) that can rapidly produce 9.8 GP composites of large histopathology specimens.

We present a multi-modal fiber array snapshot technique (M-FAST) based on an array of 96 compact cameras placed behind a primary objective lens and a fiber bundle array. Our technique is capable of large-area, high-resolution, multi-channel video acquisition. The proposed design provides two key improvements to prior cascaded imaging system approaches: a novel optical arrangement that accommodates the use of planar camera arrays, and a new ability to acquire multi-modal image data acquisition. M-FAST is a multi-modal, scalable imaging system that can acquire snapshot dual-channel fluorescence images as well as differential phase contrast measurements over a large 6.59 mm × 9.74 mm field-of-view at 2.2-μm center full-pitch resolution.

Optical coherence tomography (OCT) is a cross-sectional, micrometre-scale imaging modality with widespread clinical application. Typical OCT systems sacrifice lateral resolution to achieve long depths of focus for bulk tissue imaging, and therefore tend to have better axial than lateral resolution. Such anisotropic resolution can obscure fine ultrastructural features. Furthermore, conventional OCT suffers from refraction-induced image distortions. Here, we introduce optical coherence refraction tomography (OCRT), which extends the superior axial resolution to the lateral dimension, synthesizing undistorted cross-sectional image reconstructions from multiple conventional images acquired with angular diversity. In correcting refraction-induced distortions to register the OCT images, OCRT also achieves spatially resolved refractive index imaging. We demonstrate greater than threefold improvement in lateral resolution as well as speckle reduction in imaging the tissue ultrastructure, consistent with histology. With further optimization in optical designs to incorporate angular diversity into clinical instruments, OCRT could be widely applied as an enhancement over conventional OCT. By synthesizing undistorted cross-sectional image reconstructions from multiple conventional images acquired with angular diversity, optical coherence refraction tomography offers greater than threefold improvement in lateral resolution and speckle reduction in imaging tissue ultrastructure, and reconstructs the tissue’s internal refractive index distribution.

Optical coherence tomography (OCT) has seen widespread success as an in vivo clinical diagnostic 3D imaging modality, impacting areas including ophthalmology, cardiology, and gastroenterology. Despite its many advantages, such as high sensitivity, speed, and depth penetration, OCT suffers from several shortcomings that ultimately limit its utility as a 3D microscopy tool, such as its pervasive coherent speckle noise and poor lateral resolution required to maintain millimeter-scale imaging depths. Here, we present 3D optical coherence refraction tomography (OCRT), a computational extension of OCT that synthesizes an incoherent contrast mechanism by combining multiple OCT volumes, acquired across two rotation axes, to form a resolution-enhanced, speckle-reduced, refraction-corrected 3D reconstruction. Our label-free computational 3D microscope features a novel optical design incorporating a parabolic mirror to enable the capture of 5D plenoptic datasets, consisting of millimetric 3D fields of view over up to ± 75 ∘ without moving the sample. We demonstrate that 3D OCRT reveals 3D features unobserved by conventional OCT in fruit fly, zebrafish, and mouse samples.

This paper presents a microscopic imaging technique that uses variable-angle illumination to recover the complex polarimetric properties of a specimen at high resolution and over a large field-of-view. The approach extends Fourier ptychography, which is a synthetic aperture-based imaging approach to improve resolution with phaseless measurements, to additionally account for the vectorial nature of light. After images are acquired using a standard microscope outfitted with an LED illumination array and two polarizers, our vectorial Fourier ptychography (vFP) algorithm solves for the complex 2x2 Jones matrix of the anisotropic specimen of interest at each resolved spatial location. We introduce a new sequential Gauss-Newton-based solver that additionally jointly estimates and removes polarization-dependent imaging system aberrations. We demonstrate effective vFP performance by generating large-area (29 mm2), high-resolution (1.24 μm full-pitch) reconstructions of sample absorption, phase, orientation, diattenuation, and retardance for a variety of calibration samples and biological specimens.

This work demonstrates a multi-lens microscopic imaging system that overlaps multiple independent fields of view on a single sensor for high-efficiency automated specimen analysis. Automatic detection, classification and counting of various morphological features of interest is now a crucial component of both biomedical research and disease diagnosis. While convolutional neural networks (CNNs) have dramatically improved the accuracy of counting cells and sub-cellular features from acquired digital image data, the overall throughput is still typically hindered by the limited space-bandwidth product (SBP) of conventional microscopes. Here, we show both in simulation and experiment that overlapped imaging and co-designed analysis software can achieve accurate detection of diagnostically-relevant features for several applications, including counting of white blood cells and the malaria parasite, leading to multi-fold increase in detection and processing throughput with minimal reduction in accuracy.

We present a real-time spectral-scanning frequency-modulated continuous wave (FMCW) 3D imaging and velocimetry system that can produce 3D depth maps at 33 Hz, with 48° × 68° field of view (FOV) and 32.8-cm depth range. Each depth map consists of 507 × 500 pixels, with 0.095° × 0.14° angular resolution and 2.82-mm depth resolution. The system employs a grating for beam steering and a telescope for angular FOV magnification. Quantitative depth, reflectivity, and axial velocity measurements of a static 3D printed depth variation target and a moving robotic arm are demonstrated.

Anterior uveitis is the most common form of intraocular inflammation, and one of its main signs is the presence of white blood cells (WBCs) in the anterior chamber (AC). Clinically, the true composition of cells can currently only be obtained using AC paracentesis, an invasive procedure to obtain AC fluid requiring needle insertion into the AC. We previously developed a spectroscopic optical coherence tomography (SOCT) analysis method to differentiate between populations of RBCs and subtypes of WBCs, including granulocytes, lymphocytes and monocytes, both in vitro and in ACs of excised porcine eyes. We have shown that different types of WBCs have distinct characteristic size distributions, extracted from the backscattered reflectance spectrum of individual cells using Mie theory. Here, we further develop our method to estimate the composition of blood cell mixtures, both in vitro and in vivo. To do so, we estimate the size distribution of unknown cell mixtures by fitting the distribution observed using SOCT with a weighted combination of reference size distributions of each WBC type calculated using kernel density estimation. We validate the accuracy of our estimation in an in vitro study, by comparing our results for a given WBC sample mixture with the cellular concentrations measured by a hemocytometer and SOCT images before mixing. We also conducted a small in vivo quantitative cell mixture validation pilot study which demonstrates congruence between our method and AC paracentesis in two patients with uveitis. The SOCT based method appears promising to provide quantitative diagnostic information of cellular responses in the ACs of patients with uveitis.

We present a general theory of optical coherence tomography (OCT), which synthesizes the fundamental concepts and implementations of OCT under a common 3D k -space framework. At the heart of this analysis is the Fourier diffraction theorem, which relates the coherent interaction between a sample and plane wave to the Ewald sphere in the 3D k -space representation of the sample. While only the axial dimension of OCT is typically analyzed in k -space, we show that embracing a fully 3D k -space formalism allows explanation of nearly every fundamental physical phenomenon or property of OCT, including contrast mechanism, resolution, dispersion, aberration, limited depth of focus, and speckle. The theory also unifies diffraction tomography, confocal microscopy, point-scanning OCT, line-field OCT, full-field OCT, Bessel beam OCT, transillumination OCT, interferometric synthetic aperture microscopy (ISAM), and optical coherence refraction tomography (OCRT), among others. Our unified theory not only enables clear understanding of existing techniques but also suggests new research directions to continue advancing the field of OCT.

We present a tomographic imaging technique, termed Deep Prior Diffraction Tomography (DP-DT), to reconstruct the 3D refractive index (RI) of thick biological samples at high resolution from a sequence of low-resolution images collected under angularly varying illumination. DP-DT processes the multi-angle data using a phase retrieval algorithm that is extended by a deep image prior (DIP), which reparameterizes the 3D sample reconstruction with an untrained, deep generative 3D convolutional neural network (CNN). We show that DP-DT effectively addresses the missing cone problem, which otherwise degrades the resolution and quality of standard 3D reconstruction algorithms. As DP-DT does not require pre-captured data or pre-training, it is not biased towards any particular dataset. Hence, it is a general technique that can be applied to a wide variety of 3D samples, including scenarios in which large datasets for supervised training would be infeasible or expensive. We applied DP-DT to obtain 3D RI maps of bead phantoms and complex biological specimens, both in simulation and experiment, and show that DP-DT produces higher-quality results than standard regularization techniques. We further demonstrate the generality of DP-DT, using two different scattering models, the first Born and multi-slice models. Our results point to the potential benefits of DP-DT for other 3D imaging modalities, including X-ray computed tomography, magnetic resonance imaging, and electron microscopy.

Traditional imaging systems exhibit a well-known trade-off between the resolution and the field of view of their captured images. Typical cameras and microscopes can either “zoom in” and image at high-resolution, or they can “zoom out” to see a larger area at lower resolution, but can rarely achieve both effects simultaneously. In this review, we present details about a relatively new procedure termed Fourier ptychography (FP), which addresses the above trade-off to produce gigapixel-scale images without requiring any moving parts. To accomplish this, FP captures multiple low-resolution, large field-of-view images and computationally combines them in the Fourier domain into a high-resolution, large field-of-view result. Here, we present details about the various implementations of FP and highlight its demonstrated advantages to date, such as aberration recovery, phase imaging, and 3D tomographic reconstruction, to name a few. After providing some basics about FP, we list important details for successful experimental implementation, discuss its relationship with other computational imaging techniques, and point to the latest advances in the field while highlighting persisting challenges.

Illuminating or imaging samples from a broad angular range is essential in a wide variety of computational 3D imaging and resolution-enhancement techniques, such as optical projection tomography, optical diffraction tomography, synthetic aperture microscopy, Fourier ptychographic microscopy, structured illumination microscopy, photogrammetry, and optical coherence refraction tomography. The wider the angular coverage, the better the resolution enhancement or 3D-resolving capabilities. However, achieving such angular ranges is a practical challenge, especially when approaching ± 90 ∘ or beyond. Often, researchers resort to expensive, proprietary high numerical aperture (NA) objectives or to rotating the sample or source-detector pair, which sacrifices temporal resolution or perturbs the sample. Here, we propose several new strategies for multiangle imaging approaching 4pi steradians using concave parabolic or ellipsoidal mirrors and fast, low rotational inertia scanners, such as galvanometers. We derive theoretically and empirically relations between a variety of system parameters (e.g., NA, wavelength, focal length, telecentricity) and achievable fields of view (FOVs) and importantly show that intrinsic tilt aberrations do not restrict FOV for many multiview imaging applications, contrary to conventional wisdom. Finally, we present strategies for avoiding spherical aberrations at obliquely illuminated flat boundaries. Our simple designs allow for high-speed multiangle imaging for microscopic, mesoscopic, and macroscopic applications.

Combining principles of computed tomography with modern machine-learning tools significantly improves OCT’s resolution while extending imaging depth, reducing noise and reconstructing refractive-index maps of biological samples.

There is potential clinical significance in identifying cellular responses in the anterior chamber (AC) of the eye, which can indicate hyphema (an accumulation of red blood cells [RBCs]) or aberrant intraocular inflammation (an accumulation of white blood cells [WBCs]). In this work, we developed a spectroscopic OCT analysis method to differentiate between populations of RBCs and subtypes of WBCs, including granulocytes, lymphocytes and monocytes, both in vitro and in ACs of porcine eyes. We developed an algorithm to track single cells within OCT data sets, and extracted the backscatter reflectance spectrum of each single cell from the detected interferograms using the short-time Fourier transform (STFT). A look-up table of Mie back-scattering spectra was generated and used to correlate the backscatter spectral features of single cells to their characteristic sizes. The extracted size distributions based on the best Mie spectra fit were significantly different between each cell type. We also studied theoretical backscattering models of single RBCs to further validate our experimental results. The described work is a promising step towards clinically differentiating and quantifying AC blood cell types.

Since its invention, the microscope has been optimized for interpretation by a human observer. With the recent development of deep learning algorithms for automated image analysis, there is now a clear need to re-design the microscope's hardware for specific interpretation tasks. To increase the speed and accuracy of automated image classification, this work presents a method to co-optimize how a sample is illuminated in a microscope, along with a pipeline to automatically classify the resulting image, using a deep neural network. By adding a ""physical layer"" to a deep classification network, we are able to jointly optimize for specific illumination patterns that highlight the most important sample features for the particular learning task at hand, which may not be obvious under standard illumination. We demonstrate how our learned sensing approach for illumination design can automatically identify malaria-infected cells with up to 5-10% greater accuracy than standard and alternative microscope lighting designs. We show that this joint hardware-software design procedure generalizes to offer accurate diagnoses for two different blood smear types, and experimentally show how our new procedure can translate across different experimental setups while maintaining high accuracy.

In optical coherence tomography (OCT), the axial resolution is often superior to the lateral resolution, which is sacrificed for long imaging depths. To address this anisotropy, we previously developed optical coherence refraction tomography (OCRT), which uses images from multiple angles to computationally reconstruct an image with isotropic resolution, given by the OCT axial resolution. On the other hand, spectroscopic OCT (SOCT), an extension of OCT, trades axial resolution for spectral resolution and hence often has superior lateral resolution. Here, we present spectroscopic OCRT (SOCRT), which uses SOCT images from multiple angles to reconstruct a spectroscopic image with isotropic spatial resolution limited by the OCT lateral resolution. We experimentally show that SOCRT can estimate bead size based on Mie theory at simultaneously high spectral and isotropic spatial resolution. We also applied SOCRT to a biological sample, achieving axial resolution enhancement limited by the lateral resolution.

More people die from melanoma after a stage I diagnosis than after a stage IV diagnosis, because the tools available to clinicians do not readily identify which early-stage cancers will be aggressive. Near-infrared pump-probe microscopy detects fundamental differences in melanin structure between benign human moles and melanoma and also correlates with metastatic potential. However, the biological mechanisms of these changes have been difficult to quantify, as many different mechanisms can contribute to the pump-probe signal. We use model systems (sepia, squid, and synthetic eumelanin), cellular uptake studies, and a range of pump and probe wavelengths to demonstrate that the clinically observed effects come from alterations of the aggregated mode from “thick oligomer stacks” to “thin oligomer stacks” (due to changes in monomer composition) and (predominantly) deaggregation of the assembled melanin structure. This provides the opportunity to use pump-probe microscopy for the detection and study of melanin-associated diseases.

We integrate spectroscopic optical coherence tomography (SOCT) with stimulated Raman scattering (SRS) to enable simultaneously multiplexed spatial and spectral imaging with sensitivity to many endogenous biochemical species that play an important role in biology and medicine. The combined approach, termed SRS-SOCT, overcomes the limitations of each individual method. Ultimately, SRS-SOCT has the potential to achieve fast, volumetric, and highly sensitive label-free molecular imaging. We demonstrate the approach by imaging excised human adipose tissue and detecting the lipids' Raman signatures in the high-wavenumber region.

We present a new OCT method for flow speed quantification and directional velocimetry: particle streak velocimetry-OCT (PSV-OCT). PSV-OCT generates two-dimensional, 2.5-vector component (vx ,|vy |,vz ) maps of microscale flow velocity fields. Knowledge of 2.5-vector components also enables the estimation of total flow speed. The enabling insight behind PSV-OCT is that tracer particles in sparsely-seeded fluid flow trace out streaks in (x,z,t)-space. The streak orientations in x-t and z-t yield vx and vz , respectively. The in-plane (x-z plane) residence time yields the out-of-plane speed |vy |. Vector component values are generated by fitting streaks to a model of image formation that incorporates equations of motion in 3D space. We demonstrate cross-sectional estimation of (vx ,|vy |,vz ) in two important animal models in ciliary biology: Xenopus embryos (tadpoles) and mouse trachea.

OCT is a popular cross-sectional microscale imaging modality in medicine and biology. While structural imaging using OCT is a mature technology in many respects, flow and motion estimation using OCT remains an intense area of research. In particular, there is keen interest in maximizing information extraction from the complex-valued OCT signal. Here, we introduce a Bayesian framework into the data workflow in OCT-based velocimetry. We demonstrate that using prior information in this Bayesian framework can significantly improve velocity estimate precision in a correlation-based, model-based framework for Doppler and transverse velocimetry. We show results in calibrated flow phantoms as well as in vivo in a Drosophila melanogaster (fruit fly) heart. Thus, our work improves upon the current approaches in terms of improved information extraction from the complex-valued OCT signal.