Dual-Tap Computational Photography Image Sensor With Per-Pixel Pipelined Digital Memory for Intra-Frame Coded Multi-Exposure

A coded-exposure-pixel image sensor for computational imaging applications is presented. Each frame exposure time is divided into ${N}$ subframes. Within each subframe, each pixel sorts photo-generated charge into two charge taps depending on that pixel’s 1-bit binary code. ${N}$ global updates of arbitrary pixel-wise codes are implemented in each frame to enable ${N}$ short global pixel-specific subexposures within one frame. To make these subexposures global, two latches per pixel are utilized in a pipelined fashion. The code for the next subframe is loaded into latch 1 in a row parallel fashion, while the code for the current subframe is being applied by latch 2 globally for photo-generated charge sorting during the current subexposure. A $280^{H}\times 176^{V}$ image sensor prototype with $11.2{\text{-}}\mu \text{m}$ pixel pitch has been fabricated in a $0.11{\text{-}}\mu \text{m}$ CMOS image sensor (CIS) technology. The image sensor has been demonstrated in two computational photography applications, each using only a single frame of a video: 1) computing both albedo (a measure of reflectivity) and 3-D depth maps by means of structured-light imaging and 2) computing surface normals (3-D orientations) map by means of photometric stereo imaging. These demonstrations experimentally validate some of the unique capabilities of this computational image sensor, such as accurate 3-D visual scene reconstruction using only one camera, while maintaining its native specifications: the full spatial resolution and the maximum frame rate.