Search

Permutation matrices form an important computational building block frequently used in various fields including e.g., communications, information security and data processing. Optical implementation of permutation operators with relatively large number of input-output interconnections based on power-efficient, fast, and compact platforms is highly desirable. Here, we present diffractive optical networks engineered through deep learning to all-optically perform permutation operations that can scale to hundreds of thousands of interconnections between an input and an output field-of-view using passive transmissive layers that are individually structured at the wavelength scale. Our findings indicate that the capacity of the diffractive optical network in approximating a given permutation operation increases proportional to the number of diffractive layers and trainable transmission elements in the system. Such deeper diffractive network designs can pose practical challenges in terms of physical alignment and output diffraction efficiency of the system. We addressed these challenges by designing misalignment tolerant diffractive designs that can all-optically perform arbitrarily-selected permutation operations, and experimentally demonstrated, for the first time, a diffractive permutation network that operates at THz part of the spectrum. Diffractive permutation networks might find various applications in e.g., security, image encryption and data processing, along with telecommunications; especially with the carrier frequencies in wireless communications approaching THz-bands, the presented diffractive permutation networks can potentially serve as channel routing and interconnection panels in wireless networks., Comment: 22 Pages, 6 Figures

Pathological diagnosis relies on the visual inspection of histologically stained thin tissue specimens, where different types of stains are applied to bring contrast to and highlight various desired histological features. However, the destructive histochemical staining procedures are usually irreversible, making it very difficult to obtain multiple stains on the same tissue section. Here, we demonstrate a virtual stain transfer framework via a cascaded deep neural network (C-DNN) to digitally transform hematoxylin and eosin (H&E) stained tissue images into other types of histological stains. Unlike a single neural network structure which only takes one stain type as input to digitally output images of another stain type, C-DNN first uses virtual staining to transform autofluorescence microscopy images into H&E and then performs stain transfer from H&E to the domain of the other stain in a cascaded manner. This cascaded structure in the training phase allows the model to directly exploit histochemically stained image data on both H&E and the target special stain of interest. This advantage alleviates the challenge of paired data acquisition and improves the image quality and color accuracy of the virtual stain transfer from H&E to another stain. We validated the superior performance of this C-DNN approach using kidney needle core biopsy tissue sections and successfully transferred the H&E-stained tissue images into virtual PAS (periodic acid-Schiff) stain. This method provides high-quality virtual images of special stains using existing, histochemically stained slides and creates new opportunities in digital pathology by performing highly accurate stain-to-stain transformations., 14 Pages, 4 Figures, 1 Table

Early detection and identification of pathogenic bacteria such as Escherichia coli (E. coli) is an essential task for public health. The conventional culture-based methods for bacterial colony detection usually take >24 hours to get the final read-out. Here, we demonstrate a bacterial colony-forming-unit (CFU) detection system exploiting a thin-film-transistor (TFT)-based image sensor array that saves ~12 hours compared to the Environmental Protection Agency (EPA)-approved methods. To demonstrate the efficacy of this CFU detection system, a lensfree imaging modality was built using the TFT image sensor with a sample field-of-view of ~10 cm^2. Time-lapse images of bacterial colonies cultured on chromogenic agar plates were automatically collected at 5-minute intervals. Two deep neural networks were used to detect and count the growing colonies and identify their species. When blindly tested with 265 colonies of E. coli and other coliform bacteria (i.e., Citrobacter and Klebsiella pneumoniae), our system reached an average CFU detection rate of 97.3% at 9 hours of incubation and an average recovery rate of 91.6% at ~12 hours. This TFT-based sensor can be applied to various microbiological detection methods. Due to the large scalability, ultra-large field-of-view, and low cost of the TFT-based image sensors, this platform can be integrated with each agar plate to be tested and disposed of after the automated CFU count. The imaging field-of-view of this platform can be cost-effectively increased to >100 cm^2 to provide a massive throughput for CFU detection using, e.g., roll-to-roll manufacturing of TFTs as used in the flexible display industry., 18 Pages, 6 Figures

Fluorescence lifetime imaging microscopy (FLIM) is a powerful imaging technique that enables the visualization of biological samples at the molecular level by measuring the fluorescence decay rate of fluorescent probes. This provides critical information about molecular interactions, environmental changes, and localization within biological systems. However, creating high-resolution lifetime maps using conventional FLIM systems can be challenging, as it often requires extensive scanning that can significantly lengthen acquisition times. This issue is further compounded in three-dimensional (3D) imaging because it demands additional scanning along the depth axis. To tackle this challenge, we developed a novel computational imaging technique called light field tomographic FLIM (LIFT-FLIM). Our approach allows for the acquisition of volumetric fluorescence lifetime images in a highly data-efficient manner, significantly reducing the number of scanning steps required compared to conventional point-scanning or line-scanning FLIM imagers. Moreover, LIFT-FLIM enables the measurement of high-dimensional data using low-dimensional detectors, which are typically low-cost and feature a higher temporal bandwidth. We demonstrated LIFT-FLIM using a linear single-photon avalanche diode array on various biological systems, showcasing unparalleled single-photon detection sensitivity. Additionally, we expanded the functionality of our method to spectral FLIM and demonstrated its application in high-content multiplexed imaging of lung organoids. LIFT-FLIM has the potential to open up new avenues in both basic and translational biomedical research.

A unidirectional imager would only permit image formation along one direction, from an input field-of-view (FOV) A to an output FOV B, and in the reverse path, the image formation would be blocked. Here, we report the first demonstration of unidirectional imagers, presenting polarization-insensitive and broadband unidirectional imaging based on successive diffractive layers that are linear and isotropic. These diffractive layers are optimized using deep learning and consist of hundreds of thousands of diffractive phase features, which collectively modulate the incoming fields and project an intensity image of the input onto an output FOV, while blocking the image formation in the reverse direction. After their deep learning-based training, the resulting diffractive layers are fabricated to form a unidirectional imager. As a reciprocal device, the diffractive unidirectional imager has asymmetric mode processing capabilities in the forward and backward directions, where the optical modes from B to A are selectively guided/scattered to miss the output FOV, whereas for the forward direction such modal losses are minimized, yielding an ideal imaging system between the input and output FOVs. Although trained using monochromatic illumination, the diffractive unidirectional imager maintains its functionality over a large spectral band and works under broadband illumination. We experimentally validated this unidirectional imager using terahertz radiation, very well matching our numerical results. Using the same deep learning-based design strategy, we also created a wavelength-selective unidirectional imager, where two unidirectional imaging operations, in reverse directions, are multiplexed through different illumination wavelengths. Diffractive unidirectional imaging using structured materials will have numerous applications in e.g., security, defense, telecommunications and privacy protection., 27 Pages, 10 Figures

Compartmentalization, leveraging microfluidics, enables highly sensitive assays; but the requirement for significant infrastructure for their design, build, and operation limits access. Newer multi-material particle-based technologies thermodynamically stabilize monodisperse droplets as individual reaction compartments with simple liquid handling steps, precluding the need for expensive microfluidic equipment. Here, we further improve the accessibility of this lab on a particle technology to resource-limited settings by combining this assay system with a portable multi-modal reader, thus enabling nanoliter droplet assays in an accessible platform. We show the utility of this platform in measuring N-terminal propeptide B-type natriuretic peptide (NT-proBNP), a heart failure biomarker, in complex medium and patient samples. We report a limit of detection of ∼0.05 ng/ml and a linear response between 0.2 – 2 ng/ml in spiked plasma samples. We also show that, owing to the plurality of measurements per sample, “swarm” sensing acquires better statistical quantitation with a portable reader. Monte Carlo simulations show the increasing capability of this platform to differentiate between negative and positive samples, i.e. below or above the clinical cut-off for acute heart failure (∼0.1ng/ml), as a function of the number of particles measured. Our platform measurements correlate with gold standard ELISA measurement in cardiac patient samples, and achieve lower variation in measurement across samples compared to the standard well plate-based ELISA. Thus, we show the capabilities of a cost-effective droplet-reader system in accurately measuring biomarkers in nanoliter droplets for diseases that disproportionally affect underserved communities in resource-limited settings.

Classification of an object behind a random and unknown scattering medium sets a challenging task for computational imaging and machine vision fields. Recent deep learning-based approaches demonstrated the classification of objects using diffuser-distorted patterns collected by an image sensor. These methods demand relatively large-scale computing using deep neural networks running on digital computers. Here, we present an all-optical processor to directly classify unknown objects through unknown, random phase diffusers using broadband illumination detected with a single pixel. A set of transmissive diffractive layers, optimized using deep learning, forms a physical network that all-optically maps the spatial information of an input object behind a random diffuser into the power spectrum of the output light detected through a single pixel at the output plane of the diffractive network. We numerically demonstrated the accuracy of this framework using broadband radiation to classify unknown handwritten digits through random new diffusers, never used during the training phase, and achieved a blind testing accuracy of 88.53%. This single-pixel all-optical object classification system through random diffusers is based on passive diffractive layers that process broadband input light and can operate at any part of the electromagnetic spectrum by simply scaling the diffractive features proportional to the wavelength range of interest. These results have various potential applications in, e.g., biomedical imaging, security, robotics, and autonomous driving., Comment: 28 Pages, 6 Figures

A dual assay cartridge was developed and used in conjunction with a handheld reader for sensing % glycated albumin to monitor gestational diabetes at home.

We demonstrate multiplexed computational sensing with a point-of-care serodiagnosis assay to simultaneously quantify three biomarkers of acute cardiac injury. This point-of-care sensor includes a paper-based fluorescence vertical flow assay (fxVFA) processed by a low-cost mobile reader, which quantifies the target biomarkers through trained neural networks, all within 0.9 linearity and < 15 % coefficient of variation. The competitive performance of this multiplexed computational fxVFA along with its inexpensive paper-based design and handheld footprint make it a promising point-of-care sensor platform that could expand access to diagnostics in resource-limited settings., 17 Pages, 6 Figures

We report deep learning-based design of a massively parallel broadband diffractive neural network for all-optically performing a large group of arbitrarily-selected, complex-valued linear transformations between an input and output field-of-view, each with N_i and N_o pixels, respectively. This broadband diffractive processor is composed of N_w wavelength channels, each of which is uniquely assigned to a distinct target transformation. A large set of arbitrarily-selected linear transformations can be individually performed through the same diffractive network at different illumination wavelengths, either simultaneously or sequentially (wavelength scanning). We demonstrate that such a broadband diffractive network, regardless of its material dispersion, can successfully approximate N_w unique complex-valued linear transforms with a negligible error when the number of diffractive neurons (N) in its design matches or exceeds 2 x N_w x N_i x N_o. We further report that the spectral multiplexing capability (N_w) can be increased by increasing N; our numerical analyses confirm these conclusions for N_w > 180, which can be further increased to e.g., ~2000 depending on the upper bound of the approximation error. Massively parallel, wavelength-multiplexed diffractive networks will be useful for designing high-throughput intelligent machine vision systems and hyperspectral processors that can perform statistical inference and analyze objects/scenes with unique spectral properties., 30 Pages, 9 Figures

The application of deep learning techniques has greatly enhanced holographic imaging capabilities, leading to improved phase recovery and image reconstruction. Here, we introduce a deep neural network termed enhanced Fourier Imager Network (eFIN) as a highly generalizable framework for hologram reconstruction with pixel super-resolution and image autofocusing. Through holographic microscopy experiments involving lung, prostate and salivary gland tissue sections and Papanicolau (Pap) smears, we demonstrate that eFIN has a superior image reconstruction quality and exhibits external generalization to new types of samples never seen during the training phase. This network achieves a wide autofocusing axial range of 0.35 mm, with the capability to accurately predict the hologram axial distances by physics-informed learning. eFIN enables 3x pixel super-resolution imaging and increases the space-bandwidth product of the reconstructed images by 9-fold with almost no performance loss, which allows for significant time savings in holographic imaging and data processing steps. Our results showcase the advancements of eFIN in pushing the boundaries of holographic imaging for various applications in e.g., quantitative phase imaging and label-free microscopy., 10 Pages, 4 Figures

Diffractive optical networks provide rich opportunities for visual computing tasks since the spatial information of a scene can be directly accessed by a diffractive processor without requiring any digital pre-processing steps. Here we present data class-specific transformations all-optically performed between the input and output fields-of-view (FOVs) of a diffractive network. The visual information of the objects is encoded into the amplitude (A), phase (P), or intensity (I) of the optical field at the input, which is all-optically processed by a data class-specific diffractive network. At the output, an image sensor-array directly measures the transformed patterns, all-optically encrypted using the transformation matrices pre-assigned to different data classes, i.e., a separate matrix for each data class. The original input images can be recovered by applying the correct decryption key (the inverse transformation) corresponding to the matching data class, while applying any other key will lead to loss of information. The class-specificity of these all-optical diffractive transformations creates opportunities where different keys can be distributed to different users; each user can only decode the acquired images of only one data class, serving multiple users in an all-optically encrypted manner. We numerically demonstrated all-optical class-specific transformations covering A-->A, I-->I, and P-->I transformations using various image datasets. We also experimentally validated the feasibility of this framework by fabricating a class-specific I-->I transformation diffractive network using two-photon polymerization and successfully tested it at 1550 nm wavelength. Data class-specific all-optical transformations provide a fast and energy-efficient method for image and data encryption, enhancing data security and privacy., 27 Pages, 9 Figures, 1 Table

Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72 times the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially-repeating virtual spectral filter array with 2x2=4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available., 24 Pages, 9 Figures

Reconstruction of in-line holograms of unknown objects in general suffers from twin-image artifacts due to the appearance of an out-of-focus image overlapping with the desired image to be reconstructed. Computer-based iterative phase retrieval algorithms and learning-based methods have been used for the suppression of such image artifacts in digital holography. Here we report an all-optical hologram reconstruction method that can instantly retrieve the image of an unknown object from its in-line hologram and eliminate twin-image artifacts without using a digital processor or a computer. Multiple transmissive diffractive layers are trained using deep learning so that the diffracted light from an arbitrary input hologram is processed all-optically, through light-matter interaction, to reconstruct the image of an unknown object at the speed of light propagation and without the need for any external power. This passive all-optical processor composed of spatially-engineered transmissive layers forms a diffractive network, which successfully generalizes to reconstruct in-line holograms of unknown, new objects and exhibits improved diffraction efficiency as well as extended depth-of-field at the hologram recording distance. This all-optical hologram processor and the underlying design framework can find numerous applications in coherent imaging and holographic display-related applications owing to its major advantages in terms of image reconstruction speed and computer-free operation., Comment: 19 Pages, 5 Figures

We report the design of diffractive surfaces to all-optically perform arbitrary complex-valued linear transformations between an input (N_i) and output (N_o), where N_i and N_o represent the number of pixels at the input and output fields-of-view (FOVs), respectively. First, we consider a single diffractive surface and use a matrix pseudoinverse-based method to determine the complex-valued transmission coefficients of the diffractive features/neurons to all-optically perform a desired/target linear transformation. In addition to this data-free design approach, we also consider a deep learning-based design method to optimize the transmission coefficients of diffractive surfaces by using examples of input/output fields corresponding to the target transformation. We compared the all-optical transformation errors and diffraction efficiencies achieved using data-free designs as well as data-driven (deep learning-based) diffractive designs to all-optically perform (i) arbitrarily-chosen complex-valued transformations including unitary, nonunitary and noninvertible transforms, (ii) 2D discrete Fourier transformation, (iii) arbitrary 2D permutation operations, and (iv) high-pass filtered coherent imaging. Our analyses reveal that if the total number (N) of spatially-engineered diffractive features/neurons is N_i x N_o or larger, both design methods succeed in all-optical implementation of the target transformation, achieving negligible error. However, compared to data-free designs, deep learning-based diffractive designs are found to achieve significantly larger diffraction efficiencies for a given N and their all-optical transformations are more accurate for N < N_i x N_o. These conclusions are generally applicable to various optical processors that employ spatially-engineered diffractive surfaces., Comment: 46 Pages, 12 Figures

The emergence of epigenetic gene regulation and its role in disease have motivated a growing field of epigenetic diagnostics for risk assessment and screening. In particular, irregular cytosine DNA base methylation has been implicated in several diseases, yet the methods for detecting these epigenetic marks are limited to lengthy protocols requiring bulky and costly equipment. We demonstrate a simple workflow for detecting methylated CpG dinucleotides in synthetic and genomic DNA samples using methylation-sensitive restriction enzyme digestion followed by loop-mediated isothermal amplification. We additionally demonstrate a cost-effective mobile fluorescence reader comprising a light-emitting diode bundle, a mirror, and optical fibers to transduce fluorescence signals associated with DNA amplification. The workflow can be performed in approximately 1 h, requiring only a simple heat source, and can therefore provide a foundation for distributable point-of-care testing of DNA methylation levels.

Chronic conditions like diabetes require monitoring of vital biomarkers over extended periods of time. Monitoring gestational diabetes mellitus (GDM) is crucial to avoid short- and long-term adverse effects on both mother and infant. Providing monitoring systems to patients at the point-of-care (POC) has the potential to help mitigate these effects. In this manuscript, we propose the use of a sensing system combining lateral flow assays (LFAs) with a handheld colorimetric reader for use in tracking the glycemic status of a GDM patient at the POC. Current strategies of GDM monitoring include glucose and HbA1c measurements. These are often too frequent or not frequent enough for effective monitoring. Hence, we have developed a sensor for an intermediate interval biomarker - glycated albumin (GA). Based on the half-life of the protein, GA is measured once every 2-3 weeks. Here we first present two lateral flow assays, one for GA and another for total serum albumin used in conjunction with a handheld reader to read the colorimetric signals. Both assays have a sandwich aptamer format and measure the target proteins in their physiologically relevant ranges. The GA assay has a dynamic range of 3-20 mg ml

Deep learning demonstrates the ability to significantly increase the scanning speed of whole slide imaging in histology. This transformative solution can be used to further accelerate the adoption of digital pathology.

Over the past several decades the dramatic increase in the availability of computational resources, coupled with the maturation of machine learning, has profoundly impacted sensor technology. In this Perspective, we discuss computational sensing with a focus on intelligent sensor system design. By leveraging inverse design and machine learning techniques, data acquisition hardware can be fundamentally redesigned to ‘lock-in’ to the optimal sensing data with respect to a user-defined cost function or design constraint. We envision a new generation of computational sensing systems that reduce the data burden while also improving sensing capabilities, enabling low-cost and compact sensor implementations engineered through iterative analysis of data-driven sensing outcomes. We believe that the methodologies discussed in this Perspective will permeate the design phase of sensing hardware, and thereby will fundamentally change and challenge traditional, intuition-driven sensor and readout designs in favour of application-targeted and perhaps highly non-intuitive implementations. Such computational sensors enabled by machine learning can therefore foster new and widely distributed applications that will benefit from ‘big data’ analytics and the internet of things to create powerful sensing networks, impacting various fields, including for example, biomedical diagnostics, environmental sensing and global health, among others. Traditional sensing techniques apply computational analysis at the output of the sensor hardware to separate signal from noise. A new, more holistic and potentially more powerful approach proposed in this Perspective is designing intelligent sensor systems that ‘lock-in’ to optimal sensing of data, making use of machine leaning strategies.

We present a virtual image refocusing method over an extended depth of field (DOF) enabled by cascaded neural networks and a double-helix point-spread function (DH-PSF). This network model, referred to as W-Net, is composed of two cascaded generator and discriminator network pairs. The first generator network learns to virtually refocus an input image onto a user-defined plane, while the second generator learns to perform a cross-modality image transformation, improving the lateral resolution of the output image. Using this W-Net model with DH-PSF engineering, we extend the DOF of a fluorescence microscope by ~20-fold. This approach can be applied to develop deep learning-enabled image reconstruction methods for localization microscopy techniques that utilize engineered PSFs to improve their imaging performance, including spatial resolution and volumetric imaging throughput., 7 Pages, 3 Figures, 1 Table

Exposure to bio-aerosols such as mold spores and pollen can lead to adverse health effects. There is a need for a portable and cost-effective device for long-term monitoring and quantification of various bio-aerosols. To address this need, we present a mobile and cost-effective label-free bio-aerosol sensor that takes holographic images of flowing particulate matter concentrated by a virtual impactor, which selectively slows down and guides particles larger than ~6 microns to fly through an imaging window. The flowing particles are illuminated by a pulsed laser diode, casting their inline holograms on a CMOS image sensor in a lens-free mobile imaging device. The illumination contains three short pulses with a negligible shift of the flowing particle within one pulse, and triplicate holograms of the same particle are recorded at a single frame before it exits the imaging field-of-view, revealing different perspectives of each particle. The particles within the virtual impactor are localized through a differential detection scheme, and a deep neural network classifies the aerosol type in a label-free manner, based on the acquired holographic images. We demonstrated the success of this mobile bio-aerosol detector with a virtual impactor using different types of pollen (i.e., bermuda, elm, oak, pine, sycamore, and wheat) and achieved a blind classification accuracy of 92.91%. This mobile and cost-effective device weighs ~700 g and can be used for label-free sensing and quantification of various bio-aerosols over extended periods since it is based on a cartridge-free virtual impactor that does not capture or immobilize particulate matter., 23 Pages, 5 Figures, 1 Table