Search

Inverse problems in scientific imaging often seek physical characterization of heterogeneous scene materials. The scene is thus represented by physical quantities, such as the density and sizes of particles (microphysics) across a domain. Moreover, the forward image formation model is physical. An important case is that of clouds, where microphysics in three dimensions (3D) dictate the cloud dynamics, lifetime and albedo, with implications to Earth's energy balance, sustainable energy and rainfall. Current methods, however, recover very degenerate representations of microphysics. To enable 3D volumetric recovery of all the required microphysical parameters, we introduce the neural microphysics field (NeMF). It is based on a deep neural network, whose input is multi-view polarization images. NeMF is pre-trained through supervised learning. Training relies on polarized radiative transfer, and noise modeling in polarization-sensitive sensors. The results offer unprecedented recovery, including droplet effective variance. We test NeMF in rigorous simulations and demonstrate it using real-world polarization-image data.

We propose that spaceborne polarimetric imagers can be calibrated, or self-calibrated using zodiacal light (ZL). ZL is created by a cloud of interplanetary dust particles. It has a significant degree of polarization in a wide field of view. From space, ZL is unaffected by terrestrial disturbances. ZL is insensitive to the camera location, so it is suited for simultaneous cross-calibration of satellite constellations. ZL changes on a scale of months, thus being a quasi-constant target in realistic calibration sessions. We derive a forward model for polarimetric image formation. Based on it, we formulate an inverse problem for polarimetric calibration and self-calibration, as well as an algorithm for the solution. The methods here are demonstrated in simulations. Towards these simulations, we render polarized images of the sky, including ZL from space, polarimetric disturbances, and imaging noise.

Night beats with alternating current (AC) illumination. By passively sensing this beat, we reveal new scene information which includes: the type of bulbs in the scene, the phases of the electric grid up to city scale, and the light transport matrix. This information yields unmixing of reflections and semi-reflections, nocturnal high dynamic range, and scene rendering with bulbs not observed during acquisition. The latter is facilitated by a dataset of bulb response functions for a range of sources, which we collected and provide. To do all this, we built a novel coded-exposure high-dynamic-range imaging technique, specifically designed to operate on the grid's AC lighting.

Scattering-based computed tomography (CT) recovers a heterogeneous volumetric scattering medium using images taken from multiple directions. It is a nonlinear problem. Prior art mainly approached it by explicit physics-based optimization of image-fitting, being slow and difficult to scale. Scale is particularly important when the objects constitute large cloud fields, where volumetric recovery is important for climate studies. Besides speed, imaging and recovery need to be flexible, to efficiently handle variable viewing geometries and resolutions. These can be caused by perturbation in camera poses or fusion of data from different types of observational sensors. There is a need for fast variable imaging projection scattering tomography of clouds (VIP-CT). We develop a learning-based solution, using a deep-neural network (DNN) which trains on a large physics-based labeled volumetric dataset. The DNN parameters are oblivious to the domain scale, hence the DNN can work with arbitrarily large domains. VIP-CT offers much better quality than the state of the art. The inference speed and flexibility of VIP-CT make it effectively real-time in the context of spaceborne observations. The paper is the first to demonstrate CT of a real cloud using empirical data directly in a DNN. VIP-CT may offer a model for a learning-based solution to nonlinear CT problems in other scientific domains. Our code is available online.

Plankton interact with the environment according to their size and three-dimensional (3D) structure. To study them outdoors, these translucent specimens are imaged in situ. Light projects through a specimen in each image. The specimen has a random scale, drawn from the population's size distribution and random unknown pose. The specimen appears only once before drifting away. We achieve 3D tomography using such a random ensemble to statistically estimate an average volumetric distribution of the plankton type and specimen size. To counter errors due to non-rigid deformations, we weight the data, drawing from advanced models developed for cryo-electron microscopy. The weights convey the confidence in the quality of each datum. This confidence relies on a statistical error model. We demonstrate the approach on live plankton using an underwater field microscope.

We present a wide-field imaging approach to optically sense underwater sediment resuspension events. It uses wide-field multi-directional views and diffuse backlight. Our approach algorithmically quantifies the amount of material resuspended and its spatiotemporal distribution. The suspended particles affect the radiation that reaches the cameras, hence the captured images. By measuring the radiance during and prior to resuspension, we extract the optical depth on the line of sight per pixel. Using computed tomography (CT) principles, the optical depths yield estimation of the extinction coefficient of the suspension, per voxel. The suspended density is then derived from the reconstructed extinction coefficient.

Random refraction occurs in turbulence and through a wavy water-air interface. It creates distortion that changes in space, time and with viewpoint. Localizing objects in three dimensions (3D) despite this random distortion is important to some predators and also to submariners avoiding the salient use of periscopes. We take a multiview approach to this task. Refracted distortion statistics induce a probabilistic relation between any pixel location and a line of sight in space. Measurements of an object's random projection from multiple views and times lead to a likelihood function of the object's 3D location. The likelihood leads to estimates of the 3D location and its uncertainty. Furthermore, multiview images acquired simultaneously in a wide stereo baseline have uncorrelated distortions. This helps reduce the acquisition time needed for localization. The method is demonstrated in stereoscopic video sequences, both in a lab and a swimming pool.

Many studies analyze resolution limits in single-channel, pan-chromatic systems. However, color imaging is popular. Thus, there is a need for its modeling in terms of resolving capacity under noise. This work analyzes the probability of resolving details as a function of spatial frequency in color imaging. The analysis introduces theoretical bounds for performance, using optimal linear filtering and fusion operations. The work focuses on resolution loss caused strictly by noise, without the presence of imaging blur. It applies to full-field color systems, which do not compromise resolution by spatial multiplexing. The framework allows us to assess and optimize the ability of an imaging system to distinguish an object of given size and color under image noise.

Aerosols affect climate, health and aviation. Currently, their retrieval assumes a plane-parallel atmosphere and solely vertical radiative transfer. We propose a principle to estimate the aerosol distribution as it really is: a three dimensional (3D) volume. The principle is a type of tomography. The process involves wide angle integral imaging of the sky on a very large scale. The imaging can use an array of cameras in visible light. We formulate an image formation model based on 3D radiative transfer. Model inversion is done using optimization methods, exploiting a closed-form gradient which we derive for the model-fit cost function. The tomography model is distinct, as the radiation source is unidirectional and uncontrolled, while off-axis scattering dominates the images.

Refraction causes random dynamic distortions in atmospheric turbulence and in views across a water interface. The latter scenario is experienced by submerged animals seeking to detect prey or avoid predators, which may be airborne or on land. Man encounters this when surveying a scene by a submarine or divers while wishing to avoid the use of an attention-drawing periscope. The problem of inverting random refracted dynamic distortions is difficult, particularly when some of the objects in the field of view (FOV) are moving. On the other hand, in many cases, just those moving objects are of interest, as they reveal animal, human, or machine activity. Furthermore, detecting and tracking these objects does not necessitate handling the difficult task of complete recovery of the scene. We show that moving objects can be detected very simply, with low false-positive rates, even when the distortions are very strong and dominate the object motion. Moreover, the moving object can be detected even if it has zero mean motion. While the object and distortion motions are random and unknown, they are mutually independent. This is expressed by a simple motion feature which enables discrimination of moving object points versus the background.

Systems in which the point spread function (PSF) is a rotating beam have increasing use in three-dimensional (3D) microscopy and depth estimation. We analyze in several ways the 3D optical transfer function (OTF) of Gauss Laguerre modes and rotating beams. This is based on analysis of 3D OTFs of general aperture functions. Consequently, we suggest a criterion for depth resolution based on an effective cutoff of the axial frequency response. This criterion can be used to optimize PSFs explicitly and directly, to maximize axial resolution.

Image recovery under noise is widely studied. However, there is little emphasis on performance as a function of object size. In this work we analyze the probability of recovery as a function of object spatial frequency. The analysis uses a physical model for the acquired signal and noise, and also accounts for potential postacquisition noise filtering. Linear-systems analysis yields an effective cutoff frequency, which is induced by noise, despite having no optical blur in the imaging model. This means that a low signal-to-noise ratio (SNR) in images causes resolution loss, similar to image blur. We further consider the effect on SNR of pointwise image formation models, such as added specular or indirect reflections, additive scattering, radiance attenuation in haze, and flash photography. The result is a tool that assesses the ability to recover (within a desirable success rate) an object or feature having a certain size, distance from the camera, and radiance difference from its nearby background, per attenuation coefficient of the medium. The bounds rely on the camera specifications.

Underwater, natural illumination typically varies strongly temporally and spatially. The reason is that waves on the water surface refract light into the water in a spatiotemporally varying manner. The resulting underwater illumination field forms a caustic network and is known as flicker. This work shows that caustics can be useful for stereoscopic vision, naturally leading to range mapping of the scene. Range triangulation by stereoscopic vision requires the determination of correspondence between image points in different viewpoints, which is often a difficult problem. We show that the spatiotemporal caustic pattern very effectively establishes stereo correspondences. Thus, we term the use of this effect as CauStereo. The temporal radiance variations due to flicker are unique to each object point, thus disambiguating the correspondence, with very simple calculations. Theoretical limitations of the method are analyzed using ray-tracing simulations. The method is demonstrated by underwater in situ experiments.

Polarization may be sensed by imaging modules. This is done in various engineering systems as well as in biological systems, specifically by insects and some marine species. However, polarization per pixel is usually not the direct variable of interest. Rather, polarization-related data serve as a cue for recovering task-specific scene information. How should polarization-picture post-processing (P(4)) be done for the best scene understanding? Answering this question is not only helpful for advanced engineering (computer vision), but also to prompt hypotheses as to the processing occurring within biological systems. In various important cases, the answer is found by a principled expression of scene recovery as an inverse problem. Such an expression relies directly on a physics-based model of effects in the scene. The model includes analysis that depends on the different polarization components, thus facilitating the use of these components during the inversion, in a proper, even if non-trivial, manner. We describe several examples for this approach. These include automatic removal of path radiance in haze or underwater, overcoming partial semireflections and visual reverberations; three-dimensional recovery and distance-adaptive denoising. The resulting inversion algorithms rely on signal-processing methods, such as independent component analysis, deconvolution and optimization.

Vision in scattering media is important but challenging. Images suffer from poor visibility due to backscattering and attenuation. Most prior methods for scene recovery use active illumination scanners (structured and gated), which can be slow and cumbersome, while natural illumination is inapplicable to dark environments. The current paper addresses the need for a non-scanning recovery method, that uses active scene irradiance. We study the formation of images under widefield artificial illumination. Based on the formation model, the paper presents an approach for recovering the object signal. It also yields rough information about the 3D scene structure. The approach can work with compact, simple hardware, having active widefield, polychromatic polarized illumination. The camera is fitted with a polarization analyzer. Two frames of the scene are taken, with different states of the analyzer or polarizer. A recovery algorithm follows the acquisition. It allows both the backscatter and the object reflection to be partially polarized. It thus unifies and generalizes prior polarization-based methods, which had assumed exclusive polarization of either of these components. The approach is limited to an effective range, due to image noise and illumination falloff. Thus, the limits and noise sensitivity are analyzed. We demonstrate the approach in underwater field experiments.

Outdoor imaging in haze is plagued by poor visibility. A major problem is spatially-varying reduction of contrast by airlight, which is scattered by the haze particles towards the camera. However, images can be compensated for haze, and even yield a depth map of the scene. A key step in such scene recovery is subtraction of the airlight. In particular, this can be achieved by analyzing polarization-filtered images. This analysis requires parameters of the airlight, particularly its degree of polarization (DOP). These parameters were estimated in past studies by measuring pixels in sky areas. However, the sky is often unseen in the field of view. This paper derives several methods for estimating these parameters, when the sky is not in view. The methods are based on minor prior knowledge about a couple of scene points. Moreover, we propose blind estimation of the DOP, based on the image data. This estimation is based on independent component analysis (ICA). The methods were demonstrated in field experiments.

Ultrasound images are very noisy. Along with system noise, a significant noise source is the speckle phenomenon caused by interference in the viewed object. Most of the past approaches for denoising ultrasound images essentially blur the image and they do not handle attenuation. We discuss an approach that does not blur the image and handles attenuation. It is based on frequency compounding, in which images of the same object are acquired in different acoustic frequencies and, then, compounded. Existing frequency compounding methods have been based on simple averaging, and have achieved only limited enhancement. The reason is that the statistical and physical characteristics of the signal and noise vary with depth, and the noise is correlated between acoustic frequencies. Hence, we suggest two spatially varying frequency compounding methods, based on the understanding of these characteristics. As demonstrated in experiments, the proposed approaches suppress various noise sources and also recover attenuated objects while maintaining a high resolution.

Measuring an array of variables is central to many systems, including imagers (array of pixels), spectrometers (array of spectral bands) and lighting systems. Each of the measurements, however, is prone to noise and potential sensor saturation. It is recognized by a growing number of methods that such problems can be reduced by multiplexing the measured variables. In each measurement, multiple variables (radiation channels) are mixed (multiplexed) by a code. Then, after data acquisition, the variables are decoupled computationally in post processing. Potential benefits of the use of multiplexing include increased signal-to-noise ratio and accommodation of scene dynamic range. However, existing multiplexing schemes, including Hadamard-based codes, are inhibited by fundamental limits set by sensor saturation and Poisson distributed photon noise, which is scene dependent. There is thus a need to find optimal codes that best increase the signal to noise ratio, while accounting for these effects. Hence, this paper deals with the pursuit of such optimal measurements that avoid saturation and account for the signal dependency of noise. The paper derives lower bounds on the mean square error of demultiplexed variables. This is useful for assessing the optimality of numerically-searched multiplexing codes, thus expediting the numerical search. Furthermore, the paper states the necessary conditions for attaining the lower bounds by a general code.We show that graph theory can be harnessed for finding such ideal codes, by the use of strongly regular graphs.

When imaging in scattering media, visibility degrades as objects become more distant. Visibility can be significantly restored by computer vision methods that account for physical processes occurring during image formation. Nevertheless, such recovery is prone to noise amplification in pixels corresponding to distant objects, where the medium transmittance is low. We present an adaptive filtering approach that counters the above problems: while significantly improving visibility relative to raw images, it inhibits noise amplification. Essentially, the recovery formulation is regularized, where the regularization adapts to the spatially varying medium transmittance. Thus, this regularization does not blur close objects. We demonstrate the approach in atmospheric and underwater experiments, based on an automatic method for determining the medium transmittance.

Imaging of objects under variable lighting directions is an important and frequent practice in computer vision, machine vision, and image-based rendering. Methods for such imaging have traditionally used only a single light source per acquired image. They may result in images that are too dark and noisy, e.g., due to the need to avoid saturation of highlights. We introduce an approach that can significantly improve the quality of such images, in which multiple light sources illuminate the object simultaneously from different directions. These illumination-multiplexed frames are then computationally demultiplexed. The approach is useful for imaging dim objects, as well as objects having a specular reflection component. We give the optimal scheme by which lighting should be multiplexed to obtain the highest quality output, for signal-independent noise. The scheme is based on Hadamard codes. The consequences of imperfections such as stray light, saturation, and noisy illumination sources are then studied. In addition, the paper analyzes the implications of shot noise, which is signal-dependent, to Hadamard multiplexing. The approach facilitates practical lighting setups having high directional resolution. This is shown by a setup we devise, which is flexible, scalable, and programmable. We used it to demonstrate the benefit of multiplexing in experiments.

Some scenarios require performance estimation of an imaging or a computer vision system prior to its actual operation such as in system design, as well as in tasks of high risk or cost. To predict the performance, we propose an image-based approach that accounts for underlying image-formation processes while using real image data. We give a detailed description of image formation from scene photons to image gray levels. This analysis includes all the optical, electrical, and digital sources of signal distortion and noise. On the basis of this analysis and our access to the camera parameters, we devise a simple image-based algorithm. It transforms a baseline high-quality image to render an estimated outcome of the system we wish to operate or design. We demonstrate our approach on thermal imaging systems (infrared spectrum, 3-5 microm).

The accuracy of depth estimation based on defocus effects has been essentially limited by the depth of field of the imaging system. We show that depth estimation can be improved significantly relative to classical methods by exploiting three-dimensional diffraction effects. We formulate the problem by using information theory analysis and present, to the best of our knowledge, a new paradigm for depth estimation based on spatially rotating point-spread functions (PSFs). Such PSFs are fundamentally more sensitive to defocus thanks to their first-order axial variation. Our system acquires a frame by using a rotating PSF and jointly processes it with an image acquired by using a standard PSF to recover depth information. Analytical, numerical, and experimental evidence suggest that the approach is suitable for applications such as microscopy and machine vision.

Nonuniform exposures often affect imaging systems, e.g., owing to vignetting. Moreover, the sensor's radiometric response may be nonlinear. These characteristics hinder photometric measurements. They are particularly annoying in image mosaicking, in which images are stitched to enhance the field of view. Mosaics suffer from seams stemming from radiometric inconsistencies between raw images. Prior methods feathered the seams but did not address their root cause. We handle these problems in a unified framework. We suggest a method for simultaneously estimating the radiometric response and the camera nonuniformity, based on a frame sequence acquired during camera motion. The estimated functions are then compensated for. This permits image mosaicking, in which no seams are apparent. There is no need to resort to dedicated seam-feathering methods. Fundamental ambiguities associated with this estimation problem are stated.

We present an approach for easily removing the effects of haze from passively acquired images. Our approach is based on the fact that usually the natural illuminating light scattered by atmospheric particles (airlight) is partially polarized. Optical filtering alone cannot remove the haze effects, except in restricted situations. Our method, however, stems from physics-based analysis that works under a wide range of atmospheric and viewing conditions, even if the polarization is low. The approach does not rely on specific scattering models such as Rayleigh scattering and does not rely on the knowledge of illumination directions. It can be used with as few as two images taken through a polarizer at different orientations. As a byproduct, the method yields a range map of the scene, which enables scene rendering as if imaged from different viewpoints. It also yields information about the atmospheric particles. We present experimental results of complete dehazing of outdoor scenes, in far-from-ideal conditions for polarization filtering. We obtain a great improvement of scene contrast and correction of color.