Search

Event cameras or dynamic vision sensors (DVS) record asynchronous response to brightness changes instead of conventional intensity frames, and feature ultra-high sensitivity at low bandwidth. The new mechanism demonstrates great advantages in challenging scenarios with fast motion and large dynamic range. However, the recorded events might be highly sparse due to either limited hardware bandwidth or extreme photon starvation in harsh environments. To unlock the full potential of event cameras, we propose an inventive event sequence completion approach conforming to the unique characteristics of event data in both the processing stage and the output form. Specifically, we treat event streams as 3D event clouds in the spatiotemporal domain, develop a diffusion-based generative model to generate dense clouds in a coarse-to-fine manner, and recover exact timestamps to maintain the temporal resolution of raw data successfully. To validate the effectiveness of our method comprehensively, we perform extensive experiments on three widely used public datasets with different spatial resolutions, and additionally collect a novel event dataset covering diverse scenarios with highly dynamic motions and under harsh illumination. Besides generating high-quality dense events, our method can benefit downstream applications such as object classification and intensity frame reconstruction. [ABSTRACT FROM AUTHOR]

Multi-spectral imaging is a fundamental tool characterizing the constituent energy of scene radiation. However, current multi-spectral video cameras cannot scale up beyond megapixel resolution due to optical constraints and the complexity of the reconstruction algorithms. To circumvent the above issues, we propose a tens-of-megapixel handheld multi-spectral videography approach (THETA), with a proof-of-concept camera achieving 65-megapixel videography of 12 wavebands within visible light range. The high performance is brought by multiple designs: We propose an imaging scheme to fabricate a thin mask for encoding spatio-spectral data using a conventional film camera. Afterwards, a fiber optic plate is introduced for building a compact prototype supporting pixel-wise encoding with a large space-bandwidth product. Finally, a deep-network-based algorithm is adopted for large-scale multi-spectral data decoding, with the coding pattern specially designed to facilitate efficient coarse-to-fine model training. Experimentally, we demonstrate THETA's advantageous and wide applications in outdoor imaging of large macroscopic scenes. Current multispectral video cameras can hardly scale up beyond megapixel resolution and do not apply to large-scale scenes. The authors design a tens-ofmegapixel handheld multi-spectral imaging scheme, build a setup achieving 65-megapixel videography of 12 wavebands, and show its wide applications. [ABSTRACT FROM AUTHOR]

High-throughput imaging is highly desirable in intelligent analysis of computer vision tasks. In conventional design, throughput is limited by the separation between physical image capture and digital post processing. Computational imaging increases throughput by mixing analog and digital processing through the image capture pipeline. Yet, recent advances of computational imaging focus on the "compressive sampling", this precludes the wide applications in practical tasks. This paper presents a systematic analysis of the next step for computational imaging built on snapshot compressive imaging (SCI) and semantic computer vision (SCV) tasks, which have independently emerged over the past decade as basic computational imaging platforms. SCI is a physical layer process that maximizes information capacity per sample while minimizing system size, power and cost. SCV is an abstraction layer process that analyzes image data as objects and features, rather than simple pixel maps. In current practice, SCI and SCV are independent and sequential. This concatenated pipeline results in the following problems: i) a large amount of resources are spent on task-irrelevant computation and transmission, ii) the sampling and design efficiency of SCI is attenuated, and iii) the final performance of SCV is limited by the reconstruction errors of SCI. Bearing these concerns in mind, this paper takes one step further aiming to bridge the gap between SCI and SCV to take full advantage of both approaches. After reviewing the current status of SCI, we propose a novel joint framework by conducting SCV on raw measurements captured by SCI to select the region of interest, and then perform reconstruction on these regions to speed up processing time. We use our recently built SCI prototype to verify the framework. Preliminary results are presented and the prospects for a joint SCI and SCV regime are discussed. By conducting computer vision tasks in the compressed domain, we envision that a new era of snapshot compressive imaging with limited end-to-end bandwidth is coming. [ABSTRACT FROM AUTHOR]

Large-scale imaging of biological dynamics with high spatiotemporal resolution is indispensable to system biology studies. However, conventional microscopes have an inherent compromise between the achievable field of view and spatial resolution due to the space–bandwidth product theorem. In addition, a further challenge is the ability to handle the enormous amount of data generated by a large-scale imaging platform. Here, we break these bottlenecks by proposing the use of a flat–curved–flat imaging strategy, in which the sample plane is magnified onto a large spherical image surface and then seamlessly conjugated to multiple planar sensors. Our real-time, ultra-large-scale, high-resolution (RUSH) imaging platform operates with a 10 × 12 mm2 field of view, a uniform resolution of ~1.20 μm after deconvolution and a data throughput of 5.1 gigapixels per second. We use the RUSH platform to perform video-rate, gigapixel imaging of biological dynamics at centimetre scale and micrometre resolution, including brain-wide structural imaging and functional imaging in awake, behaving mice. Video-rate imaging of the brains of awake mice with a field of view of 10 × 12 mm2 and a spatial resolution of 1.2 µm is accomplished. [ABSTRACT FROM AUTHOR]

Markov random field (MRF), as one of special undirected graphs, is widely used in modeling priors of natural images. Targeting to learn better prior models from a given database, we explore the natural image statistics at different scales and build normalized filter pool, a kind of high-order MRF, for prior learning of nature images. The main contribution of the proposed model is that we construct a multi-scale MRF model through constraining the norms of filters in kernel space and integrate all the filtering responses in a unified framework. We formulate both learning and inference as constrained optimization problems and solve them using augmented Lagrange method. The experiment results demonstrate that the normalization of filters at different scales helps to achieve fast convergence in learning stage and obtain superior performance in image restoration, e.g., image denoising and image inpainting. [ABSTRACT FROM AUTHOR]

Computational photography is an emerging multidisciplinary field. Over the last two decades, it has integrated studies across computer vision, computer graphics, signal processing, applied optics and related disciplines. Researchers are exploring new ways to break through the limitations of traditional digital imaging for the benefit of photographers, vision and graphics researchers, and image processing programmers. Thanks to much effort in various associated fields, the large variety of issues related to these new methods of photography are described and discussed extensively in this paper. To give the reader the full picture of the voluminous literature related to computational photography, this paper briefly reviews the wide range of topics in this new field, covering a number of different aspects, including: (i) the various elements of computational imaging systems and new sampling and reconstruction mechanisms; (ii) the different image properties which benefit from computational photography, e.g. depth of field, dynamic range; and (iii) the sampling subspaces of visual scenes in the real world. Based on this systematic review of the previous and ongoing work in this field, we also discuss some open issues and potential new directions in computational photography. This paper aims to help the reader get to know this new field, including its history, ultimate goals, hot topics, research methodologies, and future directions, and thus build a foundation for further research and related developments. [ABSTRACT FROM AUTHOR]

The demand for compact cameras capable of recording high-speed scenes with high resolution is steadily increasing. However, achieving such capabilities often entails high bandwidth requirements, resulting in bulky, heavy systems unsuitable for low-capacity platforms. To address this challenge, leveraging a coded exposure setup to encode a frame sequence into a blurry snapshot and subsequently retrieve the latent sharp video presents a lightweight solution. Nevertheless, restoring motion from blur remains a formidable challenge due to the inherent ill-posedness of motion blur decomposition, the intrinsic ambiguity in motion direction, and the diverse motions present in natural videos. In this study, we propose a novel approach to address these challenges by combining the classical coded exposure imaging technique with the emerging implicit neural representation for videos. We strategically embed motion direction cues into the blurry image during the imaging process. Additionally, we develop a novel implicit neural representation based blur decomposition network to sequentially extract the latent video frames from the blurry image, leveraging the embedded motion direction cues. To validate the effectiveness and efficiency of our proposed framework, we conduct extensive experiments using benchmark datasets and real-captured blurry images. The results demonstrate that our approach significantly outperforms existing methods in terms of both quality and flexibility. The code for our work is available at https://github.com/zhihongz/BDINR. [ABSTRACT FROM AUTHOR]

Computational ghost imaging (CGI) achieves single-pixel imaging by using a Spatial Light Modulator (SLM) to generate structured illuminations for spatially resolved information encoding. The imaging speed of CGI is limited by the modulation frequency of available SLMs, and sets back its practical applications. This paper proposes to bypass this limitation by trading off SLM's redundant spatial resolution for multiplication of the modulation frequency. Specifically, a pair of galvanic mirrors sweeping across the high resolution SLM multiply the modulation frequency within the spatial resolution gap between SLM and the final reconstruction. A proof-of-principle setup with two middle end galvanic mirrors achieves ghost imaging as fast as 42 Hz at 80 × 80-pixel resolution, 5 times faster than state-of-the-arts, and holds potential for one magnitude further multiplication by hardware upgrading. Our approach brings a significant improvement in the imaging speed of ghost imaging and pushes ghost imaging towards practical applications. [ABSTRACT FROM AUTHOR]

Combining spectral imaging with compressive sensing (CS) enables efficient data acquisition by fully utilizing the intrinsic redundancies in natural images. Current compressive multispectral imagers, which are mostly based on array sensors (e.g, CCD or CMOS), suffer from limited spectral range and relatively low photon efficiency. To address these issues, this paper reports a multispectral imaging scheme with a single-pixel detector. Inspired by the spatial resolution redundancy of current spatial light modulators (SLMs) relative to the target reconstruction, we design an all-optical spectral splitting device to spatially split the light emitted from the object into several counterparts with different spectrums. Separated spectral channels are spatially modulated simultaneously with individual codes by an SLM. This no-moving-part modulation ensures a stable and fast system, and the spatial multiplexing ensures an efficient acquisition. A proof-of-concept setup is built and validated for 8-channel multispectral imaging within 420~720 nm wavelength range on both macro and micro objects, showing a potential for efficient multispectral imager in macroscopic and biomedical applications. [ABSTRACT FROM AUTHOR]