Search

X-ray ptychographic tomography is a nondestructive method for three dimensional (3D) imaging with nanometer-sized resolvable features. The size of the volume that can be imaged is almost arbitrary, limited only by the penetration depth and the available scanning time. Here we present a method that rapidly accelerates the imaging operation over a given volume through acquiring a limited set of data via large angular reduction and compensating for the resulting ill-posedness through deeply learned priors. The proposed 3D reconstruction method “RAPID” relies initially on a subset of the object measured with the nominal number of required illumination angles and treats the reconstructions from the conventional two-step approach as ground truth. It is then trained to reproduce equal fidelity from much fewer angles. After training, it performs with similar fidelity on the hitherto unexamined portions of the object, previously not shown during training, with a limited set of acquisitions. In our experimental demonstration, the nominal number of angles was 349 and the reduced number of angles was 21, resulting in a $$\times 140$$ × 140 aggregate speedup over a volume of $$4.48\times 93.18\times 3.92\, \upmu \text {m}^3$$ 4.48 × 93.18 × 3.92 μ m 3 and with $$(14\,\text {nm})^3$$ ( 14 nm ) 3 feature size, i.e. $$\sim 10^8$$ ∼ 10 8 voxels. RAPID’s key distinguishing feature over earlier attempts is the incorporation of atrous spatial pyramid pooling modules into the deep neural network framework in an anisotropic way. We found that adjusting the atrous rate improves reconstruction fidelity because it expands the convolutional kernels’ range to match the physics of multi-slice ptychography without significantly increasing the number of parameters.

We realize low-dose X-ray ptychography via Spatial Pyramid Atrous Convolution End-to-end Reconstruction (PtychoSPACER), which offers nanometer-scale resolution on the complex index of refraction for spatially extended samples non-destructively and reduces the risk of radiation damage.

Limited-angle tomography of an interior volume is a challenging, highly ill-posed problem with practical implications in medical and biological imaging, manufacturing, automation, and environmental and food security. Regularizing priors are necessary to reduce artifacts by improving the condition of such problems. Recently, it was shown that one effective way to learn the priors for strongly scattering yet highly structured 3D objects, e.g. layered and Manhattan, is by a static neural network [Goy et al. Proc. Natl. Acad. Sci. 116, 19848–19856 (2019)]. Here, we present a radically different approach where the collection of raw images from multiple angles is viewed analogously to a dynamical system driven by the object-dependent forward scattering operator. The sequence index in the angle of illumination plays the role of discrete time in the dynamical system analogy. Thus, the imaging problem turns into a problem of nonlinear system identification, which also suggests dynamical learning as a better fit to regularize the reconstructions. We devised a Recurrent Neural Network (RNN) architecture with a novel Separable-Convolution Gated Recurrent Unit (SC-GRU) as the fundamental building block. Through a comprehensive comparison of several quantitative metrics, we show that the dynamic method is suitable for a generic interior-volumetric reconstruction under a limited-angle scheme. We show that this approach accurately reconstructs volume interiors under two conditions: weak scattering, when the Radon transform approximation is applicable and the forward operator well defined; and strong scattering, which is nonlinear with respect to the 3D refractive index distribution and includes uncertainty in the forward operator.

Each image metric represents different characteristics of images. For instance, similarity metrics, e.g. Structural Similarity Index Metric (SSIM) or Pearson Correlation Coefficient (PCC), utilize correlation between two images to calculate similarity between them; error metrics, e.g. Mean Absolute Error (MAE) or Root-Mean Squared Error (RMSE), compute the pixel-wise error between them according to different norms. As each of them highlights different aspects, a choice of the metric for an application may depend upon characteristics of the images. In this paper, we will show some tomographic reconstructions of dense-layered binary-phase objects, and as the objects are binary, we propose Probability of Error (PE) as an image metric for the assessment of the reconstructions in contrast to other metrics that are not constrained to the range of values. PE is equivalent to Bit-Error Rate (BER) in digital communications as both of the signals of interest are binary, and we are interested to a bit-wise deviation of the reconstructions of their corresponding ground truth images.

In high-contrast imaging applications, such as the direct imaging of exoplanets, a coronagraph is used to suppress the light from an on-axis star so that a dimmer, off-axis object can be imaged. To maintain a high-contrast dark region in the image, optical aberrations in the instrument must be minimized. The use of phase-contrast-based Zernike Wavefront Sensors (ZWFS) to measure and correct for aberrations has been studied for large segmented aperture telescopes and ZWFS are planned for the coronagraph instrument on the Roman Space Telescope (RST). ZWFS enable subnanometer wavefront sensing precision, but their response is nonlinear. Lyot-based Low-OrderWavefront Sensors (LLOWFS) are an alternative technique, where light rejected from a coronagraph's Lyot stop is used for linear measurement of small wavefront displacements. Recently, the use of Deep Neural Networks (DNNs) to enable phase retrieval from intensity measurements has been demonstrated in several optical configurations. In a LLOWFS system, the use of DNNs rather than linear regression has been shown to greatly extend the sensor's usable dynamic range. In this work, we investigate the use of two different types of machine learning algorithms to extend the dynamic range of the ZWFS. We present static and dynamic deep learning architectures for single- and multi-wavelength measurements, respectively. Using simulated ZWFS intensity measurements, we validate the network training technique and present phase reconstruction results. We show an increase in the capture range of the ZWFS sensor by a factor of 3.4 with a single wavelength and 4.5 with four wavelengths.

Sensing and correction of low-order wavefront aberrations is critical for high-contrast astronomical imaging. State of the art coronagraph systems typically use image-based sensing methods that exploit the rejected on-axis light, such as Lyot-based low order wavefront sensors (LLOWFS); these methods rely on linear least-squares fitting to recover Zernike basis coefficients from intensity data. However, the dynamic range of linear recovery is limited. We propose the use of deep neural networks with residual learning techniques for non-linear wavefront sensing. The deep residual learning approach extends the usable range of the LLOWFS sensor by more than an order of magnitude compared to the conventional methods, and can improve closed-loop control of systems with large initial wavefront error. We demonstrate that the deep learning approach performs well even in low-photon regimes common to coronagraphic imaging of exoplanets.

Deep learning (DL) has been applied extensively in many computational imaging problems, often leading to superior performance over traditional iterative approaches. However, two important questions remain largely unanswered: first, how well can the trained neural network generalize to objects very different from the ones in training? This is particularly important in practice, since large-scale annotated examples similar to those of interest are often not available during training. Second, has the trained neural network learnt the underlying (inverse) physics model, or has it merely done something trivial, such as memorizing the examples or point-wise pattern matching? This pertains to the interpretability of machine-learning based algorithms. In this work, we use the Phase Extraction Neural Network (PhENN) [Optica 4, 1117-1125 (2017)], a deep neural network (DNN) for quantitative phase retrieval in a lensless phase imaging system as the standard platform and show that the two questions are related and share a common crux: the choice of the training examples. Moreover, we connect the strength of the regularization effect imposed by a training set to the training process with the Shannon entropy of images in the dataset. That is, the higher the entropy of the training images, the weaker the regularization effect can be imposed. We also discover that weaker regularization effect leads to better learning of the underlying propagation model, i.e. the weak object transfer function, applicable for weakly scattering objects under the weak object approximation. Finally, simulation and experimental results show that better cross-domain generalization performance can be achieved if DNN is trained on a higher-entropy database, e.g. the ImageNet, than if the same DNN is trained on a lower-entropy database, e.g. MNIST, as the former allows the underlying physics model be learned better than the latter.

The quality of inverse problem solutions obtained through deep learning is limited by the nature of the priors learned from examples presented during the training phase. Particularly in the case of quantitative phase retrieval, spatial frequencies that are underrepresented in the training database, most often at the high band, tend to be suppressed in the reconstruction. Ad hoc solutions have been proposed, such as pre-amplifying the high spatial frequencies in the examples; however, while that strategy improves the resolution, it also leads to high-frequency artefacts, as well as low-frequency distortions in the reconstructions. Here, we present a new approach that learns separately how to handle the two frequency bands, low and high, and learns how to synthesize these two bands into full-band reconstructions. We show that this “learning to synthesize” (LS) method yields phase reconstructions of high spatial resolution and without artefacts and that it is resilient to high-noise conditions, e.g., in the case of very low photon flux. In addition to the problem of quantitative phase retrieval, the LS method is applicable, in principle, to any inverse problem where the forward operator treats different frequency bands unevenly, i.e., is ill-posed., Image reconstruction: teaching algorithms to look high and low A computational approach that avoids artifacts when extracting data about the phase of light from noisy intensity signals can improve imaging of transparent objects including cells. Mo Deng from the Massachusetts Institute of Technology, United States, and colleagues have developed a procedure that separates raw intensity signals into high-frequency and low-frequency spectral channels. Then deep neural network algorithms, trained to operate in these two frequency bands, retrieve the phase information. A final algorithm recombines the two channels into a new phase image. The team’s experiments revealed that this method avoids the tendency of recent automatic phase extraction programs to over-represent low frequency signals, a quirk that reduces image resolution. The new reconstruction scheme is especially adept at handling noisy signal inputs, such as low light conditions.

Scattering generally worsens the condition of inverse problems, with the severity depending on the statistics of the refractive index gradient and contrast. Removing scattering artifacts from images has attracted much work in the literature, including recently the use of static neural networks. S. Li et al. [Optica 5(7), 803 (2018)10.1364/OPTICA.5.000803] trained a convolutional neural network to reveal amplitude objects hidden by a specific diffuser; whereas Y. Li et al. [Optica 5(10), 1181 (2018)10.1364/OPTICA.5.001181] were able to deal with arbitrary diffusers, as long as certain statistical criteria were met. Here, we propose a novel dynamical machine learning approach for the case of imaging phase objects through arbitrary diffusers. The motivation is to strengthen the correlation among the patterns during the training and to reveal phase objects through scattering media. We utilize the on-axis rotation of a diffuser to impart dynamics and utilize multiple speckle measurements from different angles to form a sequence of images for training. Recurrent neural networks (RNN) embedded with the dynamics filter out useful information and discard the redundancies, thus quantitative phase information in presence of strong scattering. In other words, the RNN effectively averages out the effect of the dynamic random scattering media and learns more about the static pattern. The dynamical approach reveals transparent images behind the scattering media out of speckle correlation among adjacent measurements in a sequence. This method is also applicable to other imaging applications that involve any other spatiotemporal dynamics.

The interiors of integrated circuits (ICs) are imaged by X-ray translational scanning ptychography as the sole acquisition method of raw data. This is unlike ptycho- tomographic and ptycho-laminographic schemes, which also require the angle of illumi- nation to be scanned. The computational reconstruction is reconstructed by a Transformer, a form of machine learning algorithm also used in dynamical image processing. During training, the Transformer learns the rich priors that define the IC interior. Even though 3D reconstructions obtained from experimental raw data are not error-free, the technique shows promise toward drastically reducing the overall scanning time.