Search

Super cells or multi-layer metasurfaces are used to realize various multi-functional and exotic functional devices. In such methods, the design space expands exponentially as more variable parameters are introduced; however, this will necessitate huge computational effort without special treatment. The function of a metasurface can be described mathematically by using a Jones matrix. When the gap between adjacent atoms is sufficiently large, the overall Jones matrix of a 3D lattice which is composed of multiple meta-atoms can be obtained by adding or multiplying each meta-atom's Jones matrix for a parallel or cascaded arrangement, respectively. Reversely, an arbitrary Jones matrix can be decomposed to achieve a combination of diagonal and rotation matrices. This means that the devices with various functions can be constructed by combining, cascading, and rotating a kind of atom, and thus the computation requirements will be reduced significantly. In this work, the feasibility of this approach is demonstrated with two cases, circular polarization selective transmission and resemble optical activity. Both the simulation and experiment are consistent with the hypothesis. This method can manipulate all degrees of freedom in a Jones matrix and reduce design complexity and may find applications to extend the scope of meta-optics.

In this Letter, we present a learning-based method for efficient Fourier single-pixel imaging (FSI). Based on the auto-encoder, the proposed adaptive under-sampling technique (AuSamNet) manages to optimize a sampling mask and a deep neural network at the same time to achieve both under-sampling of the object image’s Fourier spectrum and high-quality reconstruction from the under-sampled measurements. It is thus helpful in determining the best encoding and decoding scheme for FSI. Simulation and experiments demonstrate that AuSamNet can reconstruct high-quality natural color images even when the sampling ratio is as low as 7.5%. The proposed adaptive under-sampling strategy can be used for other computational imaging modalities, such as tomography and ptychography. We have released our source code.

Matrix multiplication (MM) is a fundamental operation in various scientific and engineering computations, as well as in artificial intelligence algorithms. Efficient implementation of MM is crucial for speeding up numerous applications. Photonics presents an opportunity for efficient acceleration of dense matrix computation, owing to its intrinsic advantages, such as huge parallelism, low latency, and low power consumption. However, most optical matrix computing architectures have been limited to realizing single-channel vector–matrix multiplication or using complex configurations to expand the number of channels, which does not fully exploit the parallelism of optics. In this study, we propose a novel, to the best of our knowledge, scheme for the implementation of large-scale two-dimensional optical MM with truly massive parallelism based on a specially designed Dammann grating. We demonstrate a sequence of MMs of 50 pairs of randomly generated 4 × 8 and 8 × 4 matrices in our proof-of-principle experiment. The results indicate that the mean relative error is approximately 0.048, thereby demonstrating optical robustness and high accuracy.

In this Letter we present a physics-enhanced deep learning approach for speckle correlation imaging (SCI), i.e., DeepSCI. DeepSCI incorporates the theoretical model of SCI into both the training and test stages of a neural network to achieve interpretable data preprocessing and model-driven fine-tuning, allowing the full use of data and physics priors. It can accurately reconstruct the image from the speckle pattern and is highly scalable to both medium perturbations and domain shifts. Our experimental results demonstrate the suitability and effectiveness of DeepSCI for solving the problem of limited generalization generally encountered in data-driven approaches.

Combustion diagnostics play an essential role in energy engineering, transportation, and aerospace industries, which has great potential in combustion efficiency improvement and polluting emission control. The three-dimensional (3D) visualization of the combustion field and the measurement of key physical parameters such as temperature, species concentration, and velocity during the combustion process are important topics in the field of combustion diagnostics. Benefiting from the non-contact and non-intrusive advantages of the optical detection method as well as the advantages of the 3D full-field measurement of the measured field by computational tomography, flame chemiluminescence tomography (FCT) has the ability to realize non-intrusive and instantaneous 3D quantitative measurement and 3D full-field visualization of key physical parameters in the combustion process, which has crucial research significance in combustion diagnostics. In this study, we review the progress of FCT technique. First, we provide an extensive review of practical applications of FCT in state-of-the-art combustion diagnostics and research. Then, the basic concepts and mathematical theory of FCT are elaborated. Finally, we introduce the conventional reconstruction algorithm and proceed to more popular artificial intelligence-based algorithms.

In addition to enhancing confinement, restricting optical systems to two dimensions gives rise to new photonic states, modified transport and distinct nonlinear effects. Here we explore these properties in combination and experimentally demonstrate a Berezinskii–Kosterlitz–Thouless phase transition in a nonlinear photonic lattice. In this topological transition, vortices are created in pairs and then unbind, changing the dynamics from that of a photonic fluid to that of a plasma-like gas of free (topological) charges. We explicitly measure the number and correlation properties of free vortices, for both repulsive and attractive interactions (the photonic equivalent of ferromagnetic and antiferromagnetic conditions), and confirm the traditional thermodynamics of the Berezinskii–Kosterlitz–Thouless transition. We also suggest a purely fluid interpretation, in which vortices are nucleated by inhomogeneous flow and driven by seeded instability. The results are fundamental to optical hydrodynamics and can impact two-dimensional photonic devices if temperature and interactions are not controlled properly. A topological transition in a nonlinear photonic lattice results in new vortex dynamics and a change from photonic fluid behaviour to that of a plasma-like gas.

Optical technologies have been widely used in information security owing to its parallel and high-speed processing capability. However, the most critical problem with current optical encryption techniques is that the cyphertext is linearly related with the plaintext, leading to the possibility that one can crack the system by solving a set of linear equations with only two cyphertext from the same encryption machine. Many efforts have been taken in the last decade to resolve the linearity issue, but none of these offers a true nonlinear solution. Inspired by the recent advance in spatial nonlinear optics, here we demonstrate a true nonlinear optical encryption technique. We show that, owing to the self-phase modulation effect of the photorefractive crystal, the proposed nonlinear optical image encryption technique is robust against the known plaintext attack based on phase retrieval. This opens up a new avenue for optical encryption in the spatial nonlinear domain.

We implement image prior and lightweight convolution in our PhysenNet for phase retrieval. Our results show that better image can be reconstructed while significantly reducing the computation expenses and training time.

Light sheet fluorescence microscopy (LSFM) utilizes a light sheet to optically section samples, giving it the advantages of high signal-to-noise ratio, low phototoxicity and rapid three-dimensional imaging, which is an ideal tool for long-time microscopic observation of living samples. However, due to the inherent characteristics of Gaussian light sheet, there is a contradiction between the field of view and depth resolution of LSFM. In life science research, high depth resolution is necessary to resolve fine structures of biological systems such as neurons. However, in order to maintain a large field of view, the thickness of Gaussian light sheet is limited. Here, we propose a depth resolution enhancement method for LSFM. The incorporation of light field imaging into LSFM allows the acquirement of light field information within the Gaussian light sheet. By using the light field back projection (LFBP) reconstruction algorithm and depth estimation algorithm, it is possible to obtain the depth information in a light sheet, so as to improve the depth resolution. We built a light field light sheet fluorescence microscope (LFLSFM), and demonstrated it by fluorescent microsphere experiments. The proposed method can achieve a depth resolution of 0.5um when the thickness of the light sheet is kept at 6um to ensure a large field of view. Therefore, the method has potentialities in visualizing neural network of tissues and organs such as mouse brain.

Deep neural networks have been successfully applied to constrained object priors and parameter optimization. Here, we propose a novel learning-free deep neural network architecture to tackle uncertain system optimization. This blind system constraint deep neural network(BlindNet) need not to know all the parameters of the system and can simultaneously acquire the desired image and system parameters. In order to do so, we showed that the BlindNet can perform phase retrieval on the diffraction pattern with unknown diffraction distance.

Metasurface provides subwavelength structures for manipulating wavefronts of light. The benefits of subwavelength components offer a continuous modulation of amplitude, phase, and polarization, thus eliminating the production of higher-order images and improving the utilization of light intensity. Despite the rapid progress in this field, multiparameter control of light using single layer metasurface is rarely reported. In fact, multiparameter control of light helps to improve information storage capacity and image fidelity. With simultaneous manipulation of polarization and amplitude at each pixel, it is possible to encode two separate images into one metasurface and reconstruct them under proper conditions. In a proof of concept experiment, we demonstrate an independent display of two binary images at the same position with polarization de-multiplexing from a single metasurface. This unique technology of encoding two images through amplitude and polarization manipulation provides a new opportunity for various applications in, such as encryption, information storage, polarization holograms, optical communications and fundamental physics.

Deep learning (DL) is a powerful tool in computational imaging for many applications. A common strategy is to use a preprocessor to reconstruct a preliminary image as the input to a neural network to achieve an optimized image. Usually, the preprocessor incorporates knowledge of the physics priors in the imaging model. One outstanding challenge, however, is errors that arise from imperfections in the assumed model. Model mismatches degrade the quality of the preliminary image and therefore affect the DL predictions. Another main challenge is that many imaging inverse problems are ill-posed and the networks are over-parameterized; DL networks have flexibility to extract features from the data that are not directly related to the imaging model. This can lead to suboptimal training and poorer image reconstruction results. To solve these challenges, a two-step training DL (TST-DL) framework is proposed for computational imaging without physics priors. First, a single fully-connected layer (FCL) is trained to directly learn the inverse model with the raw measurement data as the inputs and the images as the outputs. Then, this pre-trained FCL is fixed and concatenated with an un-trained deep convolutional network with a U-Net architecture for a second-step training to optimize the output image. This approach has the advantage that does not rely on an accurate representation of the imaging physics since the first-step training directly learns the inverse model. Furthermore, the TST-DL approach mitigates network over-parameterization by separately training the FCL and U-Net. We demonstrate this framework using a linear single-pixel camera imaging model. The results are quantitatively compared with those from other frameworks. The TST-DL approach is shown to perform comparable to approaches which incorporate perfect knowledge of the imaging model, to be robust to noise and model ill-posedness, and to be more robust to model mismatch than approaches which incorporate imperfect knowledge of the imaging model. Furthermore, TST-DL yields better results than end-to-end training while suffering from less overfitting. Overall, this TST-DL framework is a flexible approach for image reconstruction without physics priors, applicable to diverse computational imaging systems.

In this paper, we propose a single-shot three-dimensional imaging technique. This is achieved by simply placing a normal thin scattering layer in front of a two-dimensional image sensor, making it a light-field-like camera. The working principle of the proposed technique is based on the statistical independence and spatial ergodicity of the speckle produced by the scattering layer. Thus, the local point responses of the scattering layer should be measured in advance and are used for image reconstruction. We demonstrate the proposed method with proof-of-concept experiments and analyze the factors that affect its performance.

When an optically rough surface is illuminated by the coherent light, we can only capture speckle pattern due to random constructive and destructive interference between light waves scattered from elementary areas of the rough surface. Here we compare two different correlography-based speckle imaging models, the results demonstrate that there is no obvious difference in imaging capabilities between two models. The spatial light propagation processes are both assumed to be far-filed Fraunhofer diffraction in two models. In order to verify the far-field distance, we capture speckle patterns in different imaging distance (range from 5cm to 100cm). We found that the far-field diffraction is always applicable in our experiments if the speckle size satisfies the Nyquist criterion, and we analyzed the relationship between speckle contrast and imaging distance, the curve of the relationship between reconstructed image size and imaging distance is fitted, the physical interpretation of the function parameters is given.

the supplementary material for the revised manuscript

A two-step deep learning approach is proposed for computational imaging. This approach is robust to image model mismatches since physics priors are not needed and mitigates over-parameterization by training the network in 2 steps.

Mathematical Derivation

The formulation and training of unitary neural networks is the basis of an active modulation diffractive deep neural network. In this Letter, an optical random phase DropConnect is implemented on an optical weight to manipulate a jillion of optical connections in the form of massively parallel sub-networks, in which a micro-phase assumed as an essential ingredient is drilled into Bernoulli holes to enable training convergence, and malposed deflections of the geometrical phase ray are reformulated constantly in epochs, allowing for enhancement of statistical inference. Optically, the random micro-phase-shift acts like a random phase sparse griddle with respect to values and positions, and is operated in the optical path of a projective imaging system. We investigate the performance of the full-drilling and part-drilling phenomena. In general, random micro-phase-shift part-drilling outperforms its full-drilling counterpart both in the training and inference since there are more possible recombinations of geometrical ray deflections induced by random phase DropConnect.

We experimentally investigate image reconstruction through a scattering medium under white-light illumination. To solve the inverse problem of noninvasive scattering imaging, a modified iterative algorithm is employed with an interpretable constraint on the optical transfer function (OTF). As a result, a sparse and real object can be retrieved whether it is illuminated with a narrowband or broadband light. Compared with the well-known speckle correlation technique (SCT), the proposed method requires no restrictions on the speckle autocorrelation and shows a potential advantage in scattering imaging.

It is well known that neural networks including deep learning have been widely employed to solve the problems in recognition and classification. It was not until recently that people started to use them to solve imaging problems. In this talk, we focus on how to use deep learning to solve phase retrieval problems.

Many methods have been demonstrated that it is possible to reconstruct an object hidden scattering layers. However, it is still a big challenge when suffer from dynamic and/or time-variant scattering media. Speckle correlation is a breakthrough technique which can noninvasively retrieve the image of object from a single-shot captured pattern but it does not allow for imaging in real time as the complicated iteration process. Recently, deep learning has attracted great attention in scattering imaging but they usually employ end-to-end mode so that the scattering medium must be fixed during the training and testing process. Here, we develop a two-step deep learning strategy for imaging through moving scattering layers. In our proposed scheme, speckle autocorrelation de-noising and object image reconstruction from autocorrelation are trained respectively by using two convolution neural network. Optical experiments show that our proposed scheme has outstanding performance for real-time imaging through moving scattering layers.

Recently artificial intelligent techniques such as deep learning (DL) have shown great potential in solving various inverse problems in computational imaging. In this presentation we will focus on the use of DL for computational imaging under certain extreme conditions. Three use cases will be discussed. Two out of the three are about the environment conditions such as imaging with extremely low light, and through very thick scattering media. The third one is about the neural network itself. We demonstrate that a neural network does not need to train at all before it can be used for some specific computational imaging tasks.

Single-pixel imaging (SPI) is a typical computational imaging modality that allows two- and three-dimensional image reconstruction from a one-dimensional bucket signal acquired under structured illumination. It is in particular of interest for imaging under low light conditions and in spectral regions where good cameras are unavailable. However, the resolution of the reconstructed image in SPI is strongly dependent on the number of measurements in the temporal domain. Data-driven deep learning has been proposed for high-quality image reconstruction from a undersampled bucket signal. But the generalization issue prohibits its practical application. Here we propose a physics-enhanced deep learning approach for SPI. By blending a physics-informed layer and a model-driven fine-tuning process, we show that the proposed approach is generalizable for image reconstruction. We implement the proposed method in an in-house SPI system and an outdoor single-pixel LiDAR system, and demonstrate that it outperforms some other widespread SPI algorithms in terms of both robustness and fidelity. The proposed method establishes a bridge between data-driven and model-driven algorithms, allowing one to impose both data and physics priors for inverse problem solvers in computational imaging, ranging from remote sensing to microscopy.

Non-line-of-sight (NLOS) imaging has received considerable attentions for its ability to recover occluded objects from an indirect view. Various NLOS imaging techniques have been demonstrated recently. Here, we propose a white-light NLOS imaging method that is equipped only with an ordinary camera, and not necessary to operate under active coherent illumination as in other existing NLOS systems. The central idea is to incorporate speckle correlation-based model into a deep neural network (DNN), and form a two-step DNN strategy that endeavors to learn the optimization of the scattered pattern autocorrelation and object image reconstruction, respectively. Optical experiments are carried out to demonstrate the proposed method.

Image fusion and reconstruction from muldti-images taken by distributed or mobile cameras need accurate calibration to avoid image mismatching. This calibration process becomes difficult in fog when no clear nearby reference is available. In this work, the fusion of multi-view images taken in fog by two cameras fixed on a moving platform is realized. The positions and aiming directions of the cameras are determined by taking a close visible object as a reference. One camera with a large field of view (FOV) is applied to acquire images of a short-distance object which is still visible in fog. This reference is then adopted to the calibration of the camera system to determine the positions and pointing directions at each viewpoint. The extrinsic parameter matrices are obtained with these data, which are applied for the image fusion of distant images captured by another camera beyond visibility. The experimental verification was carried out in a fog chamber and the technique is shown to be valid for imaging reconstruction in fog without a prior in-plane. The synthetic image, accumulated and averaged by ten-view images, is shown to perform potential applicability for fog removal. The enhanced structure similarity is discussed and compared in detail with conventional single-view defogging techniques.

Unitary learning is a backpropagation (BP) method that serves to update unitary weights in fully connected deep complex-valued neural networks, meeting a prior unitary in an active modulation diffractive deep neural network. However, the square matrix characteristic of unitary weights in each layer results in its learning belonging to a small-sample training, which produces an almost useless network that has a fairly poor generalization capability. To alleviate such a serious over-fitting problem, in this Letter, optical random phase dropout is formulated and designed. The equivalence between unitary forward and diffractive networks deduces a synthetic mask that is seamlessly compounded with a computational modulation and a random sampling comb called dropout. The dropout is filled with random phases in its zero positions that satisfy the Bernoulli distribution, which could slightly deflect parts of transmitted optical rays in each output end to generate statistical inference networks. The enhancement of generalization benefits from the fact that massively parallel full connection with different optical links is involved in the training. The random phase comb introduced into unitary BP is in the form of conjugation, which indicates the significance of optical BP.