Search

Abstract Optical spectroscopy is an indispensable technique in almost all areas of scientific research and industrial applications. After its acquisition, an optical spectrum is usually further processed using a mathematical algorithm to classify or quantify the measurement results. Here we present the design and realization of a smart photodetector that provides such information directly without the need to explicitly record a spectrum. This is achieved by tailoring the spectral responsivity of the device to a specific purpose. In-sensor computation is performed at the lowest possible level of the sensor system hierarchy – the physical level of photon detection – and does not require any external processing of the measurement data. The device can be programmed to cover different types of spectral regression or classification tasks. We present the analysis of spectral mixtures as an example, but the scheme can also be applied to any other algorithm that can be represented by a linear operator. Our prototype physical implementation utilizes an ensemble of optical cavity-enhanced MoS2 photodetectors with different center wavelengths and individually adjustable peak responsivities. This spectroscopy method represents a significant advance in miniaturized and energy-efficient optical sensing.

Abstract Pixel binning is a technique, widely used in optical image acquisition and spectroscopy, in which adjacent detector elements of an image sensor are combined into larger pixels. This reduces the amount of data to be processed as well as the impact of noise, but comes at the cost of a loss of information. Here, we push the concept of binning to its limit by combining a large fraction of the sensor elements into a single “superpixel” that extends over the whole face of the chip. For a given pattern recognition task, its optimal shape is determined from training data using a machine learning algorithm. We demonstrate the classification of optically projected images from the MNIST dataset on a nanosecond timescale, with enhanced dynamic range and without loss of classification accuracy. Our concept is not limited to imaging alone but can also be applied in optical spectroscopy or other sensing applications.

Abstract As conventional frame-based cameras suffer from high energy consumption and latency, several new types of image sensors have been devised, with some of them exploiting the sparsity of natural images in some transform domain. Instead of sampling the full image, those devices capture only the coefficients of the most relevant spatial frequencies. The number of samples can be even sparser if a signal only needs to be classified rather than being fully reconstructed. Based on the corresponding mathematical framework, we developed an image sensor that can be trained to classify optically projected images by reading out the few most relevant pixels. The device is based on a two-dimensional array of metal–semiconductor–metal photodetectors with individually tunable photoresponsivity values. We demonstrate its use for the classification of handwritten digits with an accuracy comparable to that achieved by readout of the full image, but with lower delay and energy consumption.