Search

Self-driving vehicles demand efficient and reliable depth-sensing technologies. Lidar, with its capability for long-distance, high-precision measurement, is a crucial component in this pursuit. However, conventional mechanical scanning implementations suffer from reliability, cost, and frame rate limitations. Solid-state lidar solutions have emerged as a promising alternative, but the vast amount of photon data processed and stored using conventional direct time-of-flight (dToF) prevents long-distance sensing unless power-intensive partial histogram approaches are used. In this paper, we introduce a groundbreaking 'guided' dToF approach, harnessing external guidance from other onboard sensors to narrow down the depth search space for a power and data-efficient solution. This approach centers around a dToF sensor in which the exposed time window of independent pixels can be dynamically adjusted. We utilize a 64-by-32 macropixel dToF sensor and a pair of vision cameras to provide the guiding depth estimates. Our demonstrator captures a dynamic outdoor scene at 3 fps with distances up to 75 m. Compared to a conventional full histogram approach, on-chip data is reduced by over twenty times, while the total laser cycles in each frame are reduced by at least six times compared to any partial histogram approach. The capability of guided dToF to mitigate multipath reflections is also demonstrated. For self-driving vehicles where a wealth of sensor data is already available, guided dToF opens new possibilities for efficient solid-state lidar. [ABSTRACT FROM AUTHOR]

Establishing the biological basis of cognition and its disorders will require high precision spatiotemporal measurements of neural activity. Recently developed genetically encoded voltage indicators (GEVIs) report both spiking and subthreshold activity of identified neurons. However, maximally capitalizing on the potential of GEVIs will require imaging at millisecond time scales, which remains challenging with standard camera systems. Here, application of single photon avalanche diode (SPAD) sensors is reported to image neural activity at kilohertz frame rates. SPADs are electronic devices that when activated by a single photon cause an avalanche of electrons and a large electric current. An array of SPAD sensors is used to image individual neurons expressing the GEVI Voltron‐JF525‐HTL. It is shown that subthreshold and spiking activity can be resolved with shot noise limited signals at frame rates of up to 10 kHz. SPAD imaging is able to reveal millisecond scale synchronization of neural activity in an ex vivo seizure model. SPAD sensors may have widespread applications for investigation of millisecond timescale neural dynamics. [ABSTRACT FROM AUTHOR]

This article provides a tutorial introduction to the direct Time-of-Flight (dToF) signal chain and typical artifacts introduced due to detector and processing electronic limitations. We outline the memory requirements of embedded histograms related to desired precision and detectability, which are often the limiting factor in the array resolution. A survey of integrated CMOS dToF arrays is provided highlighting future prospects to further scaling through process optimization or smart embedded processing. [ABSTRACT FROM AUTHOR]

A miniaturized 1.4 mm $\times \,\, 1.4$ mm, $128\times120$ single-photon avalanche diode (SPAD) image sensor with a five-wire interface is designed for time-resolved fluorescence microendoscopy. This is the first endoscopic chip-on-tip sensor capable of fluorescence lifetime imaging microscopy (FLIM). The sensor provides a novel, compact means to extend the photon counting dynamic range (DR) by partitioning the required bit depth between in-pixel counters and off-pixel noiseless frame summation. The sensor is implemented in STMicroelectronics 40-/90-nm 3-D-stacked backside-illuminated (BSI) CMOS process with 8- $\mu \text{m}$ pixels and 45% fill factor. The sensor capabilities are demonstrated through FLIM examples, including ex vivo human lung tissue, obtained at video rate. [ABSTRACT FROM AUTHOR]

A 128 $\times $ 128 single-photon avalanche diode (SPAD) motion detection-triggered time-of-flight (ToF) sensor is implemented in STMicroelectronics 40 nm CMOS SPAD foundry process. The sensor combines vision and ToF ranging functions to acquire depth frames only when inter-frame intensity changes are detected. The 40 $\mu \text{m}\,\,\times $ 20 $\mu \text{m}$ pixels integrate two 16-bit time-gated counters to acquire ToF histograms and repurpose them to compare two vision frames without the requirement for additional out-of-pixel frame memory resources. An embedded column-parallel ToF and vision processor performs on-chip vision frame comparison and binary frame output compression as well as controlling the time-resolved histogram sampling. The sensor achieves a maximum 32.5 kframes/s in vision modality and 500 frames/s in motion detection-triggered ToF over a measured 3.5 m distance with 1.5 cm accuracy. The vision function reduces the sensor power consumption by 70% over continuous ToF operation and allows the sensor to gate the ToF laser emitter to reduce the system power when no motion activity is observed. [ABSTRACT FROM AUTHOR]

A 256 $\times $ 256 single-photon avalanche diode (SPAD) sensor integrated into a 3-D-stacked 90-nm 1P4M/40-nm 1P8M process is reported for flash light detection and ranging (LIDAR) or high-speed direct time-of-flight (ToF) 3-D imaging. The sensor bottom tier is composed of a 64 $\times $ 64 matrix of 36.72- $\mu \text{m}$ pitch modular photon processing units which operate from shared $4\,\,\times $ 4 SPADs at 9.18- $\mu \text{m}$ pitch and 51% fill-factor. A 16 $\times $ 14 bit counter array integrates photon counts or events to compress data to 31.4 Mb/s at 30-frame/s readout over 8 I/O operating at 100 MHz. The pixel-parallel multi-event time-to-digital converter (TDC) approach employs a programmable internal or external clock for 0.56–560-ns time bin resolution. In conjunction with a per-pixel correlator, the power is reduced to less than 100 mW in practical daylight ranging scenarios. Examples of ranging and high-speed 3-D ToF applications are given. [ABSTRACT FROM AUTHOR]

Advanced CMOS Single Photon Avalanche Diode Arrays have the potential to reveal characteristics of electronic display panels that have, until now, been extremely challenging or impossible to measure routinely. We demonstrate the use of a CMOS SPAD array to make optical measurements of pixels of an OLED microdisplay at very high sampling rates, very low light levels and over a very wide dynamic range. [ABSTRACT FROM AUTHOR]

Abstract: Advanced complementary metal‐oxide semiconductor‐compatible single‐photon avalanche diode array technology is progressing rapidly and is being deployed in a wide range of applications. We report for the first time the use of a complementary metal‐oxide semiconductor‐compatible single‐photon avalanche diode array to perform detailed optical measurements on pixels of an organic light‐emitting diode microdisplay at very high sampling rate, very low light level, and over a very wide dynamic range of luminance. This offers a clear demonstration of the huge potential of this single‐photon avalanche diode technology to reveal hitherto obscure details of the optical characteristics of individual and groups of organic light‐emitting diode pixels. [ABSTRACT FROM AUTHOR]

The technology of CMOS‐compatible Single Photon Avalanche Diodes is evolving rapidly and has matured to the point at which it can address the requirements of a range of imaging applications. In this report we consider the current suitability and future potential of CMOS‐compatible Single Photon Avalanche Diodes to address the particular application of display metrology. [ABSTRACT FROM AUTHOR]

A 256 x 256 single-photon avalanche diode image sensor operating at 100 kfpswith fill factor of 61% and pixel pitch of 16 µmis reported. An all-nMOS 7T pixel allows gated operation down to 4 ns and ~600-ps fall time with on-chip delay generation. The sensor operates with 0.996 temporal aperture ratio in rolling shutter. Gating and cooling allow the suppression of dark noise, which, in combination with the high fill factor, enables competitive low-light performance with electronmultiplying charge-coupled devices while offering time-resolved imaging modes. [ABSTRACT FROM AUTHOR]

Quanta Imager Sensors provide photon detections at high frame rates, with negligible read-out noise, making them ideal for high-speed optical tracking. At the basic level of bit-planes or binary maps of photon detections, objects may present limited detail. However, through motion estimation and spatial reassignment of photon detections, the objects can be reconstructed with minimal motion artefacts. We here present the first demonstration of high-speed two-dimensional (2D) tracking and reconstruction of rigid, planar objects with a Quanta Image Sensor, including a demonstration of depth-resolved tracking. [ABSTRACT FROM AUTHOR]

SPAD-based solid state CMOS image sensors utilising analogue integrators have attained deep sub-electron read noise (DSERN) permitting single photon counting (SPC) imaging. A new method is proposed to determine the read noise in DSERN image sensors by evaluating the peak separation and width (PSW) of single photon peaks in a photon counting histogram (PCH). The technique is used to identify and analyse cumulative noise in analogue integrating SPC SPAD-based pixels. The DSERN of our SPAD image sensor is exploited to confirm recent multi-photon threshold quanta image sensor (QIS) theory. Finally, various single and multiple photon spatio-temporal oversampling techniques are reviewed. [ABSTRACT FROM AUTHOR]

A CMOS single-photon avalanche diode (SPAD)-based quarter video graphics array image sensor with 8- \mu \textm pixel pitch and 26.8% fill factor (FF) is presented. The combination of analog pixel electronics and scalable shared-well SPAD devices facilitates high-resolution, high-FF SPAD imaging arrays exhibiting photon shot-noise-limited statistics. The SPAD has 47 counts/s dark count rate at 1.5 V excess bias (EB), 39.5% photon detection probability (PDP) at 480 nm, and a minimum of 1.1 ns dead time at 1 V EB. Analog single-photon counting imaging is demonstrated with maximum 14.2-mV/SPAD event sensitivity and 0.06e− minimum equivalent read noise. Binary quanta image sensor (QIS) 16-kframes/s real-time oversampling is shown, verifying single-photon QIS theory with $4.6\times $ overexposure latitude and 0.168e− read noise. [ABSTRACT FROM AUTHOR]

The advent of circular wave tanks, with wave-making segments all around the perimeter, brings potential advantages over standard, rectangular wave tanks where the wave-maker is confined to one or two adjacent sides of the tank. It is now possible to reproduce seas with full 360° directionality, enhancing the range of possible test scenarios. However, this additional capability also presents technical challenges: waves generated on "one side" of the tank must be absorbed on the opposite side, together with any waves reflected or radiated by the model under test, to prevent contamination of the wave field. This paper reviews the theory of wave generation and absorption in a circular tank, before proceeding to identify an appropriate control scheme for the University of Edinburgh's "FloWave" combined wave/current basin. Numerical simulations, based on linear multi-chromatic waves, are carried out using WAMIT to assess the suitability of wave-maker control schemes suggested in literature. For the first time a round tank's ability to reproduce sea spectra is assessed numerically. The simulations suggest that the generation of "peaked" spectra is possible to an accurate degree, with an overall standard deviation error of less than 2% over a designated "test zone". However, there are difficulties in producing "wide" spectra, as effective dynamic wave absorption cannot be ensured over the whole frequency range. This may have important repercussions, not just for the usage of FloWave, but also in terms of the design of future round basins. [ABSTRACT FROM AUTHOR]

Abstract: Obtaining a hydrodynamic model for a wave tank has many benefits, from allowing the useable test zone to be identified, to helping with the tuning of the wavemaker controllers. This paper explores a first-order, boundary element method (BEM) that has been previously proposed for modelling wave tanks, applying the method to a tank with a unique, curved geometry. In a series of experiments, the model is shown to provide a good representation of the wave profile across the tank. Inherent limitations in the method are also identified: in the case when only a single paddle is moved, significant, un-modelled second-order spurious waves are found to emerge. Moreover, the representation of the wave absorbers by a simple, partially reflecting surface does not adequately reproduce the measured spatial variation in the reflection coefficient. [Copyright &y& Elsevier]

Single molecule localisation microscopy (SMLM) has become an essential part of the super-resolution toolbox for probing cellular structure and function. The rapid evolution of these techniques has outstripped detector development and faster, more sensitive cameras are required to further improve localisation certainty. Single-photon avalanche photodiode (SPAD) array cameras offer single-photon sensitivity, very high frame rates and zero readout noise, making them a potentially ideal detector for ultra-fast imaging and SMLM experiments. However, performance traditionally falls behind that of emCCD and sCMOS devices due to lower photon detection efficiency. Here we demonstrate, both experimentally and through simulations, that the sensitivity of a binary SPAD camera in SMLM experiments can be improved significantly by aggregating only frames containing signal, and that this leads to smaller datasets and competitive performance with that of existing detectors. The simulations also indicate that with predicted future advances in SPAD camera technology, SPAD devices will outperform existing scientific cameras when capturing fast temporal dynamics. [ABSTRACT FROM AUTHOR]

We investigate the rate of convergence of the Wong-Zakai approximations for second-order stochastic PDEs of parabolic type driven by a multi-dimensional Wiener process W. [ABSTRACT FROM AUTHOR]

The article focuses on single-photon–sensitive depth sensors that are being increasingly used in next-generation electronics for human pose and gesture recognition. Topics include examines cost-effective sensors typically have a low spatial resolution, restricting their use to basic motion identification and simple object detection.

Single-Photon Avalanche Diode (SPAD) direct Time-of-Flight (dToF) sensors provide depth imaging over long distances, enabling the detection of objects even in the absence of contrast in colour or texture. However, distant objects are represented by just a few pixels and are subject to noise from solar interference, limiting the applicability of existing computer vision techniques for high-level scene interpretation. We present a new SPAD-based vision system for human activity recognition, based on convolutional and recurrent neural networks, which is trained entirely on synthetic data. In tests using real data from a 64×32 pixel SPAD, captured over a distance of 40 m, the scheme successfully overcomes the limited transverse resolution (in which human limbs are approximately one pixel across), achieving an average accuracy of 89% in distinguishing between seven different activities. The approach analyses continuous streams of video-rate depth data at a maximal rate of 66 FPS when executed on a GPU, making it well-suited for real-time applications such as surveillance or situational awareness in autonomous systems.

3D time-of-flight (ToF) image sensors are used widely in applications such as self-driving cars, augmented reality (AR), and robotics. When implemented with single-photon avalanche diodes (SPADs), compact, array format sensors can be made that offer accurate depth maps over long distances, without the need for mechanical scanning. However, array sizes tend to be small, leading to low lateral resolution, which combined with low signal-to-background ratio (SBR) levels under high ambient illumination, may lead to difficulties in scene interpretation. In this paper, we use synthetic depth sequences to train a 3D convolutional neural network (CNN) for denoising and upscaling (×4) depth data. Experimental results, based on synthetic as well as real ToF data, are used to demonstrate the effectiveness of the scheme. With GPU acceleration, frames are processed at >30 frames per second, making the approach suitable for low-latency imaging, as required for obstacle avoidance.

3D time-of-flight (ToF) imaging is used in a variety of applications such as augmented reality (AR), computer interfaces, robotics and autonomous systems. Single-photon avalanche diodes (SPADs) are one of the enabling technologies providing accurate depth data even over long ranges. By developing SPADs in array format with integrated processing combined with pulsed, flood-type illumination, high-speed 3D capture is possible. However, array sizes tend to be relatively small, limiting the lateral resolution of the resulting depth maps and, consequently, the information that can be extracted from the image for applications such as object detection. In this paper, we demonstrate that these limitations can be overcome through the use of convolutional neural networks (CNNs) for high-performance object detection. We present outdoor results from a portable SPAD camera system that outputs 16-bin photon timing histograms with 64×32 spatial resolution, with each histogram containing thousands of photons. The results, obtained with exposure times down to 2 ms (equivalent to 500 FPS) and in signal-to-background (SBR) ratios as low as 0.05, point to the advantages of providing the CNN with full histogram data rather than point clouds alone. Alternatively, a combination of point cloud and active intensity data may be used as input, for a similar level of performance. In either case, the GPU-accelerated processing time is less than 1 ms per frame, leading to an overall latency (image acquisition plus processing) in the millisecond range, making the results relevant for safety-critical computer vision applications which would benefit from faster than human reaction times.

Light-in-flight (LIF) imaging is the measurement and reconstruction of light's path as it moves and interacts with objects. It is well known that relativistic effects can result in apparent velocities that differ significantly from the speed of light. However, less well known is that Rayleigh scattering and the effects of imaging optics can lead to observed intensities changing by several orders of magnitude along light's path. We develop a model that enables us to correct for all of these effects, thus we can accurately invert the observed data and reconstruct the true intensity-corrected optical path of a laser pulse as it travels in air. We demonstrate the validity of our model by observing the photon arrival time and intensity distribution obtained from single-photon avalanche detector (SPAD) array data for a laser pulse propagating towards and away from the camera. We can then reconstruct the true intensity-corrected path of the light in four dimensions (three spatial dimensions and time).

The number of applications that use depth imaging is increasing rapidly, e.g. self-driving autonomous vehicles and auto-focus assist on smartphone cameras. Light detection and ranging (LIDAR) via single-photon sensitive detector (SPAD) arrays is an emerging technology that enables the acquisition of depth images at high frame rates. However, the spatial resolution of this technology is typically low in comparison to the intensity images recorded by conventional cameras. To increase the native resolution of depth images from a SPAD camera, we develop a deep network built to take advantage of the multiple features that can be extracted from a camera's histogram data. The network is designed for a SPAD camera operating in a dual-mode such that it captures alternate low resolution depth and high resolution intensity images at high frame rates, thus the system does not require any additional sensor to provide intensity images. The network then uses the intensity images and multiple features extracted from down-sampled histograms to guide the up-sampling of the depth. Our network provides significant image resolution enhancement and image denoising across a wide range of signal-to-noise ratios and photon levels. Additionally, we show that the network can be applied to other data types of SPAD data, demonstrating the generality of the algorithm.

This paper examines methods to best exploit the High Dynamic Range (HDR) of the single photon avalanche diode (SPAD) in a high fill-factor HDR photon counting pixel that is scalable to megapixel arrays. The proposed method combines multi-exposure HDR with temporal oversampling in-pixel. We present a silicon demonstration IC with 96 × 40 array of 8.25 µm pitch 66% fill-factor SPAD-based pixels achieving >100 dB dynamic range with 3 back-to-back exposures (short, mid, long). Each pixel sums 15 bit-planes or binary field images internally to constitute one frame providing 3.75× data compression, hence the 1k frames per second (FPS) output off-chip represents 45,000 individual field images per second on chip. Two future projections of this work are described: scaling SPAD-based image sensors to HDR 1 MPixel formats and shrinking the pixel pitch to 1-3 µm., Competing Interests: The authors declare no conflict of interest.

A CMOS single-photon avalanche diode (SPAD) quanta image sensor is used to reconstruct depth and intensity profiles when operating in a range-gated mode used in conjunction with pulsed laser illumination. By designing the CMOS SPAD array to acquire photons within a pre-determined temporal gate, the need for timing circuitry was avoided and it was therefore possible to have an enhanced fill factor (61% in this case) and a frame rate (100,000 frames per second) that is more difficult to achieve in a SPAD array which uses time-correlated single-photon counting. When coupled with appropriate image reconstruction algorithms, millimeter resolution depth profiles were achieved by iterating through a sequence of temporal delay steps in synchronization with laser illumination pulses. For photon data with high signal-to-noise ratios, depth images with millimeter scale depth uncertainty can be estimated using a standard cross-correlation approach. To enhance the estimation of depth and intensity images in the sparse photon regime, we used a bespoke clustering-based image restoration strategy, taking into account the binomial statistics of the photon data and non-local spatial correlations within the scene. For sparse photon data with total exposure times of 75 ms or less, the bespoke algorithm can reconstruct depth images with millimeter scale depth uncertainty at a stand-off distance of approximately 2 meters. We demonstrate a new approach to single-photon depth and intensity profiling using different target scenes, taking full advantage of the high fill-factor, high frame rate and large array format of this range-gated CMOS SPAD array.

Single-photon avalanche photodiode (SPAD) image sensors offer time-gated photon counting, at high binary frame rates of >100 kFPS and with no readout noise. This makes them well-suited to a range of scientific applications, including microscopy, sensing and quantum optics. However, due to the complex electronics required, the fill factor tends to be significantly lower (< 10%) than that of EMCCD and sCMOS cameras (>90%), whilst the pixel size is typically larger, impacting the sensitivity and practicalities of the SPAD devices. This paper presents the first characterisation of a cylindrical-shaped microlens array applied to a small, 8 micron, pixel SPAD imager. The enhanced fill factor, ≈50% for collimated light, is the highest reported value amongst SPAD sensors with comparable resolution and pixel pitch. We demonstrate the impact of the increased sensitivity in single-molecule localisation microscopy, obtaining a resolution of below 40nm, the best reported figure for a SPAD sensor.