Search

This article is the second within a three-part series on Fourier ptychography, which is a computational microscopy technique for high-resolution, large field-of-view imaging. While the first article laid out the basics of Fourier ptychography, this second part sheds light on its algorithmic ingredients. We present a non-technical discussion of phase retrieval, which allows for the synthesis of high-resolution images from a sequence of low-resolution raw data. Fourier ptychographic phase retrieval can be carried out on standard, widefield microscopy platforms with the simple addition of a low-cost LED array, thus offering a convenient alternative to other phase-sensitive techniques that require more elaborate hardware such as differential interference contrast and digital holography.

Fourier ptychography is an emerging computational microscopy technique that can generate gigapixel-scale images of biological samples. With only the addition of a low-cost LED array to a standard digital microscope and a reconstruction algorithm, Fourier ptychography overcomes the fundamental trade-off between a microscope's resolution and field-of-view without any moving parts. This article is the first in a three-part series that aims to introduce the fundamentals of the technology to the broader microscopy community and beyond, using intuitive explanations.

SUMMARYDevelopmentally programmed polyploidy (whole-genome-duplication) of cardiomyocytes is common across evolution. Functions of such polyploidy are essentially unknown. Here, we reveal roles for precise polyploidy levels in cardiac tissue. We highlight a conserved asymmetry in polyploidy level between cardiac chambers inDrosophilalarvae and humans. InDrosophila, differential Insulin Receptor (InR) sensitivity leads the heart chamber to reach a higher ploidy/cell size relative to the aorta chamber. Cardiac ploidy-reduced animals exhibit reduced heart chamber size, stroke volume, cardiac output, and acceleration of circulating hemocytes. TheseDrosophilaphenotypes mimic systemic human heart failure. Using human donor hearts, we reveal asymmetry in nuclear volume (ploidy) and insulin signaling between the left ventricle and atrium. Our results identify productive and likely conserved roles for polyploidy in cardiac chambers and suggest precise ploidy levels sculpt many developing tissues. These findings of productive cardiomyocyte polyploidy impact efforts to block developmental polyploidy to improve heart injury recovery.

To study the behavior of freely moving model organisms such as zebrafish (Danio rerio) and fruit flies (Drosophila) across multiple spatial scales, it would be ideal to use a light microscope that can resolve 3D information over a wide field of view (FOV) at high speed and high spatial resolution. However, it is challenging to design an optical instrument to achieve all of these properties simultaneously. Existing techniques for large-FOV microscopic imaging and for 3D image measurement typically require many sequential image snapshots, thus compromising speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over a 135-cm^2 area, achieving up to 230 frames per second at throughputs exceeding 5 gigapixels (GPs) per second. 3D-RAPID features a 3D reconstruction algorithm that, for each synchronized temporal snapshot, simultaneously fuses all 54 images seamlessly into a globally-consistent composite that includes a coregistered 3D height map. The self-supervised 3D reconstruction algorithm itself trains a spatiotemporally-compressed convolutional neural network (CNN) that maps raw photometric images to 3D topography, using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. As a result, our end-to-end 3D reconstruction algorithm is robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. The scalable hardware and software design of 3D-RAPID addresses a longstanding problem in the field of behavioral imaging, enabling parallelized 3D observation of large collections of freely moving organisms at high spatiotemporal throughputs, which we demonstrate in ants (Pogonomyrmex barbatus), fruit flies, and zebrafish larvae.

This paper experimentally examines different configurations of a multi-camera array microscope (MCAM) imaging technology. The MCAM is based upon a densely packed array of “micro-cameras” to jointly image across a large field-of-view (FOV) at high resolution. Each micro-camera within the array images a unique area of a sample of interest, and then all acquired data with 54 micro-cameras are digitally combined into composite frames, whose total pixel counts significantly exceed the pixel counts of standard microscope systems. We present results from three unique MCAM configurations for different use cases. First, we demonstrate a configuration that simultaneously images and estimates the 3D object depth across a 100×135mm2 FOV at approximately 20 µm resolution, which results in 0.15 gigapixels (GP) per snapshot. Second, we demonstrate an MCAM configuration that records video across a continuous 83×123mm2 FOV with twofold increased resolution (0.48 GP per frame). Finally, we report a third high-resolution configuration (2 µm resolution) that can rapidly produce 9.8 GP composites of large histopathology specimens.

Ophthalmic microsurgery is technically difficult because the scale of required surgical tool manipulations challenge the limits of the surgeon's visual acuity, sensory perception, and physical dexterity. Intraoperative optical coherence tomography (OCT) imaging with micrometer-scale resolution is increasingly being used to monitor and provide enhanced real-time visualization of ophthalmic surgical maneuvers, but surgeons still face physical limitations when manipulating instruments inside the eye. Autonomously controlled robots are one avenue for overcoming these physical limitations. In this article, we demonstrate the feasibility of using learning from demonstration and reinforcement learning with an industrial robot to perform OCT-guided corneal needle insertions in an ex vivo model of deep anterior lamellar keratoplasty (DALK) surgery. Our reinforcement learning agent trained on ex vivo human corneas, then outperformed surgical fellows in reaching a target needle insertion depth in mock corneal surgery trials. This article shows the combination of learning from demonstration and reinforcement learning is a viable option for performing OCT-guided robotic ophthalmic surgery.

We present a multi-modal fiber array snapshot technique (M-FAST) based on an array of 96 compact cameras placed behind a primary objective lens and a fiber bundle array. Our technique is capable of large-area, high-resolution, multi-channel video acquisition. The proposed design provides two key improvements to prior cascaded imaging system approaches: a novel optical arrangement that accommodates the use of planar camera arrays, and a new ability to acquire multi-modal image data acquisition. M-FAST is a multi-modal, scalable imaging system that can acquire snapshot dual-channel fluorescence images as well as differential phase contrast measurements over a large 6.59 mm × 9.74 mm field-of-view at 2.2-μm center full-pitch resolution.

Optical coherence tomography (OCT) is a cross-sectional, micrometre-scale imaging modality with widespread clinical application. Typical OCT systems sacrifice lateral resolution to achieve long depths of focus for bulk tissue imaging, and therefore tend to have better axial than lateral resolution. Such anisotropic resolution can obscure fine ultrastructural features. Furthermore, conventional OCT suffers from refraction-induced image distortions. Here, we introduce optical coherence refraction tomography (OCRT), which extends the superior axial resolution to the lateral dimension, synthesizing undistorted cross-sectional image reconstructions from multiple conventional images acquired with angular diversity. In correcting refraction-induced distortions to register the OCT images, OCRT also achieves spatially resolved refractive index imaging. We demonstrate greater than threefold improvement in lateral resolution as well as speckle reduction in imaging the tissue ultrastructure, consistent with histology. With further optimization in optical designs to incorporate angular diversity into clinical instruments, OCRT could be widely applied as an enhancement over conventional OCT. By synthesizing undistorted cross-sectional image reconstructions from multiple conventional images acquired with angular diversity, optical coherence refraction tomography offers greater than threefold improvement in lateral resolution and speckle reduction in imaging tissue ultrastructure, and reconstructs the tissue’s internal refractive index distribution.

We present a new multi-camera array microscope (MCAM) that can capture images at single micrometer resolution over 54 cm2 fields of view, which contain up to 16 gigapixels, and associated software for seamless viewing and analysis.

We present a novel spectral-scanning FMCW 3D imaging and velocimetry system that can produce 3D depth maps of 507 X 500 pixels at 33 Hz, with 48° X 68° FOV and 32.8 cm depth range.

Optical coherence tomography (OCT) has seen widespread success as an in vivo clinical diagnostic 3D imaging modality, impacting areas including ophthalmology, cardiology, and gastroenterology. Despite its many advantages, such as high sensitivity, speed, and depth penetration, OCT suffers from several shortcomings that ultimately limit its utility as a 3D microscopy tool, such as its pervasive coherent speckle noise and poor lateral resolution required to maintain millimeter-scale imaging depths. Here, we present 3D optical coherence refraction tomography (OCRT), a computational extension of OCT that synthesizes an incoherent contrast mechanism by combining multiple OCT volumes, acquired across two rotation axes, to form a resolution-enhanced, speckle-reduced, refraction-corrected 3D reconstruction. Our label-free computational 3D microscope features a novel optical design incorporating a parabolic mirror to enable the capture of 5D plenoptic datasets, consisting of millimetric 3D fields of view over up to ± 75 ∘ without moving the sample. We demonstrate that 3D OCRT reveals 3D features unobserved by conventional OCT in fruit fly, zebrafish, and mouse samples.

This paper presents a microscopic imaging technique that uses variable-angle illumination to recover the complex polarimetric properties of a specimen at high resolution and over a large field-of-view. The approach extends Fourier ptychography, which is a synthetic aperture-based imaging approach to improve resolution with phaseless measurements, to additionally account for the vectorial nature of light. After images are acquired using a standard microscope outfitted with an LED illumination array and two polarizers, our vectorial Fourier ptychography (vFP) algorithm solves for the complex 2x2 Jones matrix of the anisotropic specimen of interest at each resolved spatial location. We introduce a new sequential Gauss-Newton-based solver that additionally jointly estimates and removes polarization-dependent imaging system aberrations. We demonstrate effective vFP performance by generating large-area (29 mm2), high-resolution (1.24 μm full-pitch) reconstructions of sample absorption, phase, orientation, diattenuation, and retardance for a variety of calibration samples and biological specimens.

We present a real-time spectral-scanning frequency-modulated continuous wave (FMCW) 3D imaging and velocimetry system that can produce 3D depth maps at 33 Hz, with 48° × 68° field of view (FOV) and 32.8-cm depth range. Each depth map consists of 507 × 500 pixels, with 0.095° × 0.14° angular resolution and 2.82-mm depth resolution. The system employs a grating for beam steering and a telescope for angular FOV magnification. Quantitative depth, reflectivity, and axial velocity measurements of a static 3D printed depth variation target and a moving robotic arm are demonstrated.

Illuminating or imaging samples from a broad angular range is essential in a wide variety of computational 3D imaging and resolution-enhancement techniques, such as optical projection tomography, optical diffraction tomography, synthetic aperture microscopy, Fourier ptychographic microscopy, structured illumination microscopy, photogrammetry, and optical coherence refraction tomography. The wider the angular coverage, the better the resolution enhancement or 3D-resolving capabilities. However, achieving such angular ranges is a practical challenge, especially when approaching ± 90 ∘ or beyond. Often, researchers resort to expensive, proprietary high numerical aperture (NA) objectives or to rotating the sample or source-detector pair, which sacrifices temporal resolution or perturbs the sample. Here, we propose several new strategies for multiangle imaging approaching 4pi steradians using concave parabolic or ellipsoidal mirrors and fast, low rotational inertia scanners, such as galvanometers. We derive theoretically and empirically relations between a variety of system parameters (e.g., NA, wavelength, focal length, telecentricity) and achievable fields of view (FOVs) and importantly show that intrinsic tilt aberrations do not restrict FOV for many multiview imaging applications, contrary to conventional wisdom. Finally, we present strategies for avoiding spherical aberrations at obliquely illuminated flat boundaries. Our simple designs allow for high-speed multiangle imaging for microscopic, mesoscopic, and macroscopic applications.

Combining principles of computed tomography with modern machine-learning tools significantly improves OCT’s resolution while extending imaging depth, reducing noise and reconstructing refractive-index maps of biological samples.