Search

Modern image sensor devices regularly aim for low-noise and high frame-rate performances, among other key characteristics. However, not all sensor applications require both low-noise and fast output image acquisition as part of the same sensor. In slow object motion, at low-light illumination levels (such as for night vision), low-noise image performance is necessary but only at a reasonable sensor frame rate. This paper presents a theoretical analysis of the most common pixel readout methods in modern imagers, at two distinct operation modes. One mode is the classical active pixel sensor circuit and the other is a more recent operation mode based on a pulsed biasing scheme introduced for specific application cases, in which the sensor speed is not of much concern. From this perspective, this paper aims to compare the theoretical output thermal noise powers of both kinds of readouts in order to identify which readout method is the read option most suitable for low-light vision without the need to employ advanced and complex techniques to obtain a low-noise imager. In addition to the theoretical results, simulation results are presented as well, as a means to validate the theoretical work.

This study focuses on a CMOS sensing system for Radon and alpha radiation, which is based on a semiconductor device that is integrated monolithically on a single chip with the Readout Circuitry, thus allowing fabrication of a low-power and low-cost sensing system. The new sensor is based on a new mosaic design of an array of Floating Gate non-volatile memory-like transistors, which are implemented in a standard CMOS technology, with a single polysilicon layer. The transistors are electrically combined in parallel and are operated at subthreshold, thus achieving very high sensitivity and reduced noise. The sensing system’s architecture and design is presented, along with key operation concepts, characterization, and analysis results. Alpha and radon exposure results are compared to commercial radon detectors. The new sensor, dubbed TODOS-Radon sensor, measures continuously, is battery operated and insensitive to humidity.

This article presents a fully integrated multi-modal CMOS cellular sensor/stimulator array chip with 21 952 pixels and 1568-pixel concurrent readouts, while each array pixel can be independently reconfigured to support three sensing and one stimulation modalities: cellular potential recording, optical-point impedance sensing, and bi-phasic current stimulation. The CMOS sensor/stimulator array chip provides 3.6 mm $\times $ 1.6 mm active field-of-view (FoV). Each pixel contains one 8 $\mu \text{m}\,\,\times $ 8 $\mu \text{m}$ gold deposited electrode, one 6 $\mu \text{m}\,\,\times $ 6 $\mu \text{m}$ photodiode, and in-pixel circuits within 8 $\mu \text{m}\,\,\times $ 11 $\mu \text{m}$ pixel area, while the pixel-to-pixel pitch is 16 $\mu \text{m}\,\,\times $ 16 $\mu \text{m}$ . As a proof of concept, the CMOS array is implemented in a standard 130-nm BiCMOS process. Comprehensive electrical testing (potential/optical/four-point impedance/stimulation) and biological measurements (potential/optical/four-point impedance) with on-chip rat cardiomyocytes demonstrate the functionalities and unique advantages of this multi-modality cellular array. With high throughput multi-modal cellular sensing supported at the pixel level, this array chip enables holistic characterization of on-chip cellular samples with single-cell resolution.

Comparing with the application of charge coupled device (CCD) sensor, a kind of interference stripe noise (ISN) can be observed easily from the complementary metal-oxide-semiconductor (CMOS) sensor-based digital holographic measurement (DHM) system because a series of optic coatings and filters are used in front of that sensor in order to improve its imaging effect. The improper assembly or the micro manufacturing error of these optic components may create the unexpected interference phenomena in the imaging sensor. To reduce that noise, an effective image processing method is developed. On one hand the Laplacian pyramid method is used to highlight the ISN and a cartoon-texture decomposition technique is carried out to remove the additive ISN. On the other hand, an improved Lee filtering is performed to restrain the multiplicative ISN. Extensive experiment results have shown that the image quality can be improved after the ISN reduction and this method is fit for solving a deionized water temperature estimation application where the texture complexity of target image is relatively low.

Due to the delay in the row-wise exposure and the lack of stable support when a photographer holds a CMOS camera, video jitter and rolling shutter distortion are closely coupled degradations in the captured videos. However, previous methods have rarely considered both phenomena and usually treat them separately, with stabilization approaches that are unable to handle the rolling shutter effect and rolling shutter removal algorithms that are incapable of addressing motion shake. To tackle this problem, we propose a novel method that simultaneously stabilizes and rectifies a rolling shutter shaky video. The key issue is to estimate both inter-frame motion and intra-frame motion. Specifically, for each pair of adjacent frames, we first estimate a set of spatially variant inter-frame motions using a neighbor-motion-aware local motion model, where the classical mesh-based model is improved by introducing a new constraint to enhance the neighbor motion consistency. Then, different from other 2D rolling shutter removal methods that assume the pixels in the same row have a single intra-frame motion, we build a novel mesh-based intra-frame motion calculation model to cope with the depth variation in a mesh row and obtain more faithful estimation results. Finally, temporal and spatial motion constraints and an adaptive weight assignment strategy are considered together to generate the optimal warping transformations for different motion situations. Experimental results demonstrate the effectiveness and superiority of the proposed method when compared with other state-of-the-art methods.

The adaptive driving beam headlamp system is helpful to solve the traffic safety problems caused by the abuse of high beams at night. The fundamental problem for the adaptive driving beam system is to detect and track the vehicles at specific night environment for light shape control. This paper proposes a new and efficient algorithm for detecting and tracking based on video collected by a low-cost camera. The related automobile regulations are analyzed for classifying the application scenarios of the adaptive driving beam system and summarizing the performance requirements of the algorithm. A standard CMOS camera mounted behind windshield captures the test video from different scenarios. Some preprocessing approaches are designed to optimize the captured video so that the algorithm can work independently on specific camera. The color and morphological characteristics of the rear lights are utilized to extract the rear lamps of the vehicle. The symmetry of rear lights is checked by a correlation coefficient method to pair the rear lamps and determine the vehicle ahead preliminary. Then, the Hungarian algorithm and Kalman filter are performed to track the multiple occurrences in two consecutive frames and correct the detection results. Finally, an estimation method is given for calculating the vehicle position in real world. The experiments are designed according to referred regulations and the test video is obtained from a low-cost camera mounted on test vehicles driving in specific scenarios. The experimental results demonstrate that the algorithm can get high detection rates and adaptability to the working condition of adaptive driving beam system. Moreover, the proposed algorithm has low time cost and can be applied in embedded devices of the vehicle.

In this paper, a low-power readout using reconfigurable cyclic ADC for CMOS image sensors is proposed. It reduces the total number of pixels to be read by taking advantage of pixel correlation. The required number of ADC operations is reduced, resulting in power saving. In contrast to the existing pixel correlation-based approaches, which focus only on the intensity differences, in the proposed method, the polarity of the differences is also taken into account. It helps in preserving fine edges representing features such as texture. The discarded or unread pixels are interpolated on-chip while reconfiguring the ADC input range according to the interpolation step size. Furthermore, this reduces the number of ADC conversion cycles by 25% to 50% for interpolation steps of 16 LSB and 64 LSBs, respectively. The ADC is designed and fabricated in UMC 180-nm CMOS technology, and the proposed method is verified for standard test images. The reconstructed images incorporating ADC non-linearities result in average Pratt’s FoM values of 0.88, 0.86, and 0.81 for 60%, 70%, and 74% compression, respectively. The corresponding best values achieved by the existing approaches are 0.86, 0.80, and 0.77, respectively. The improvement in FoM is observed due to the consideration of polarity information. The proposed technique results from 33% to 50% power saving for 80% compression in $512\times512$ image, using reconfigurable ADC. Therefore, it is suitable for a power efficient CMOS sensor design.

We demonstrate a versatile thin lensless camera with a designed phase-mask placed at sub-2 mm from an imaging CMOS sensor. Using wave optics and phase retrieval methods, we present a general-purpose framework to create phase-masks that achieve desired sharp point-spread-functions (PSFs) for desired camera thicknesses. From a single 2D encoded measurement, we show the reconstruction of high-resolution 2D images, computational refocusing, and 3D imaging. This ability is made possible by our proposed high-performance contour-based PSF. The heuristic contour-based PSF is designed using concepts in signal processing to achieve maximal information transfer to a bit-depth limited sensor. Due to the efficient coding, we can use fast linear methods for high-quality image reconstructions and switch to iterative nonlinear methods for higher fidelity reconstructions and 3D imaging.

In order to better solve the problem of automatic identification of quality control line and detection line in the detection of gold standard test strip, this paper proposes to collect the image information of gold standard test strip after color rendering through CMOS sensor, preprocess the obtained information, transform RGB image into gray image, build cloud model in the CIELAB/HSV/HSL space, and apply the improved AdaBoost algorithm to determine the position of detection line and quality control line Place. Compared with the traditional template matching method, it improves the accuracy and accuracy of recognition.

Over the last half century, photography has evolved from photochemical reactions induced on a photosensitive film to optoelectronic devices employed in digital image. Video has done the same migration from classical film movies to optoelectronic acquisition and storage. Digital and video cameras share optics with traditional film cameras, but the image recording is realized by an optoelectronic device –a CCD or CMOS sensor– rather than a film. The quantity of digital photo and video images has dramatically increased in the last decade, arising from the ubiquitous cameras of mobile phones. Despite their popularity, condensed tutorial articles on the subject are still scarce. This work intends to contribute to the knowledge of this field with a medium-level short course that is currently being taughtin a broader subject (Consumer Photonics) to undergraduate students of Telecommunication Engineers at the Universidad Politecnica de Madrid. The course can be followed by students having a basic background on Physics and specifically on Optics and Semiconductors. The course is divided into two chapters. An introduction to optics of cameras and basic concepts of photography has been considered necessary to follow the second part, which is entirely devoted to optoelectronic sensors

Frame-based optical flow (OF) methods struggle in the presence of large motion and occlusion due to slow frame rates. Optical flow of dynamic vision sensor (DVS) has gained attention recently as a way to overcome these shortcomings. DVS reports “events” corresponding log intensity changes exceeding a specific threshold, with an accuracy of microsecond order. The increased temporal sampling rate are indeed helpful, but the poor spatial fidelity of DVS outputs make events-based OF less reliable overall. In this work, we consider a new sensor called DAViS that combines the conventional active pixel sensor (APS) and DVS circuitries, yielding a conventional intensity image frames as well as the events. We propose a novel optical flow method designed specifically for a DAViS camera that leverages the high spatial fidelity of intensity image frames and the high temporal resolution of events generated by DVS. Hence, the proposed DAViS-OF method yields reliable motion vector estimates while overcoming the fast motion and occlusion problems. The proposed DAViS-OF method is computationally efficient and is suitable for real-time implementation.

Several recent studies on compressive video sensing realized scene capture beyond the fundamental trade-off limit between spatial resolution and temporal resolution using random space-time sampling. However, most of these studies obtained results for higher-frame-rate video that was produced in simulation experiments or using an optically simulated random sampling camera, because there are currently no commercially available image sensors with random exposure or sampling capabilities. We fabricated a prototype complementary metal-oxide-semiconductor (CMOS) image sensor with quasi pixel-wise exposure timing that can realize nonuniform space-time sampling. The prototype sensor resets exposures independently by columns and fixes these exposures by rows for each 8 × 8-pixel block. This CMOS sensor is not fully controllable the pixels and has line-dependent control, but offers greater flexibility when compared with regular CMOS or charge-coupled device sensors with global or rolling shutters. We propose a method of realizing pseudo-random sampling for high-speed video acquisition that uses the flexibility of the CMOS sensor. We reconstruct the high-speed video sequence from images produced in pseudo-random sampling using a pre-learned decoder.

Despite CNN-based deblur models have shown their superiority when solving motion blurs, restoring a photorealistic image from severe motion blur remains an ill-posed problem due to the loss of temporal information and textures. Event cameras such as Dynamic and Active-pixel Vision Sensor (DAVIS) can simultaneously produce gray-scale Active Pixel Sensor (APS) frames and events, which can capture fast motions as events of very high temporal resolution, i. e., $1~\mu s$ , can provide extra information for blurry APS frames. Due to the natural noise and sparsity of events, we employ a recurrent encoder-decoder architecture to generate dense recurrent event representations, which encode the overall historical information. We concatenate the original blurry image with the event representation as our hybrid input, from which the network learns to restore the sharp output. We conduct extensive experiments on GoPro dataset and a real event blurry dataset captured by DAVIS240C. Our experimental results on both synthetic and real images demonstrate state-of-the-art performance for $1280\times 720 $ images at 30 fps.

A novel image motion self-registration algorithm was proposed and realized to improve the effectiveness of Time-Delay-Integration image processing. First, to overcome the defects of the on-chip accumulator, a CMOS camera with off-chip digital accumulator was designed. Compared to the traditional on-chip Time-Delay-Integration, the new off-chip digital Time-Delay-Integration cost none silicon area and can integrate signal processing easily. The novel algorithm was developed to work with this new off-chip digital Time-Delay-Integration. To assess the proposed algorithm, simulations and experiments were carried out. With the new algorithm, the cross-correlation similarity of the Time-Delay-Integration images improved from 0.9257 to 0.9587, even when the image motion mismatch rate is 0.1. Further experiments proved the effectiveness of the proposed algorithm when the image motion mismatch is existing. This algorithm is suitable for applications in remote sensing systems with characters of low-illumination and high scanning speed.

A method is presented for scene detection and estimation using high-resolution imagery acquired through autonomous drone navigation aided with landmark detection and recognition. The proposed system comprises a drone platform that facilitates efficient autonomous flight; it can capture images and provide real-time video streaming of the ground cover using a camera equipped with a 14-megapixel CMOS sensor and a fish-eye lens. In addition, landmark detection and recognition was performed by applying the histogram of oriented gradients and linear support vector machine methods on each frame of the video stream. The high spatial resolution of the acquired drone images makes the detection and interpretation of environments less complicated. First, through image processing, orthomosaic images and 3-D environment reconstruction (point clouds) of the scene are generated from a set of drone images by using an automatic photogrammetric technique called “structure from motion.” Subsequently, an unsupervised classification method is used to detect and differentiate environmental classes (scene interpretation) in the target or investigated area by using the high-resolution images. Finally, the results of the proposed method are evaluated by comparing them against ground-truth points.

The Coded Access Optical Sensor (CAOS) camera is a novel, single unit, full spectrum (UV to short-wave IR bands), linear, high dynamic range (HDR) camera. In this paper, calibrated color target imaging using the CAOS camera and a comparison to a commercial HDR CMOS camera is demonstrated for the first time. The first experiment using a calibrated color check chart indicates that although the CMOS sensor-based camera has an 87 dB manufacturer-specified HDR range, unrestricted usage of this CMOS camera’s output range greatly fails quality color recovery. On the other hand, the intrinsically linear full dynamic range operation CAOS camera color image recovery generally matches the restricted linear-mode commercial CMOS sensor-based camera recovery for the presented 39.5 dB non-HDR target that also matches the near 40 dB linear camera response function (CRF) range of the CMOS camera. Specifically, compared to the color checker chart manufacturer provided XYZ values for the calibrated target, percentage XYZ mean errors of 8.3% and 10.9% are achieved for the restricted linear range CMOS camera and CAOS camera, respectively. An alternate color camera assessment gives CIE ΔE00 mean values of 4.59 and 5.7 for the restricted linear range CMOS camera and CAOS camera, respectively. Unlike the CMOS camera lens optics and its photo-detection electronics, no special linear response optics and photo-detector designs were used for the experimental CAOS camera, nevertheless, a good and equivalent color recovery was achieved. Given the limited HDR linear range capabilities of a CMOS camera and the intrinsically wide linear HDR capability of a CAOS camera, a combined CAOS-CMOS mode of the CAOS smart camera is prudent and can empower HDR color imaging. Applications for such a hybrid camera includes still photography imaging, especially for quantitative imaging of biological samples, valuable artworks and archaeological artefacts that require authentic color data generation for reliable medical decisions as well as forgery preventing verifications.

The measurement of a wide temperature range in a scene requires hardware capable of high dynamic range imaging. We describe a novel near-infrared thermal imaging system operating at a wavelength of 940 nm based on a commercial photovoltaic mode high dynamic range camera and analyse its measurement uncertainty. The system is capable of measuring over an unprecedently wide temperature range, however, this comes at the cost of a reduced temperature resolution and increased uncertainty compared to a conventional CMOS camera operating in photodetective mode. Despite this, the photovoltaic mode thermal camera has an acceptable level of uncertainty for most thermal imaging applications with an NETD of 4–12 °C and a combined measurement uncertainty of approximately 1% K if a low pixel clock is used. We discuss the various sources of uncertainty and how they might be minimised to further improve the performance of the thermal camera. The thermal camera is a good choice for imaging low frame rate applications that have a wide inter-scene temperature range.

The focal-plane camera on the Rockets for Extended-source X-ray Spectroscopy (tREXS) is a large-area detector array that takes advantage of the large-format, 3-side-buttable design of the Teledyne e2v Vega-CIS113 CMOS sensor. This paper discusses the initial design of the focal plane camera, results from testing that identified read noise performance issues, mechanical and electrical challenges of this initial design, and supply chain problems. The changes to the focal plane camera that were made due to these challenges are then presented, along with the final flight camera that has been designed to optimize noise performance and be able to be built within the schedule constraints of the tREXS mission.

This paper introduces an ambient light rejection (ALR) circuit for the autonomous adaptation of a subretinal implant system. The sub-retinal implants, located beneath a bipolar cell layer, are known to have a significant advantage in spatial resolution by integrating more than a thousand pixels, compared to epi-retinal implants. However, challenges remain regarding current dispersion in high-density retinal implants, and ambient light induces pixel saturation. Thus, the technical issues of ambient light associated with a conventional image processing technique, which lead to high power consumption and area occupation, are still unresolved. Thus, it is necessary to develop a novel image-processing unit to handle ambient light, considering constraints related to power and area. In this paper, we present an ALR circuit as an image-processing unit for sub-retinal implants. We first introduced an ALR algorithm to reduce the ambient light in conventional retinal implants, next, we implemented the ALR algorithm as an application-specific integrated chip (ASIC). The ALR circuit was fabricated using a standard 0.35-μm CMOS process along with an image-sensor-based stimulator, a sensor pixel, and digital blocks. As experimental results, the ALR circuit occupies an area of 190 µm2, consumes a power of 3.2 mW and shows a maximum response time of 1.6 s at a light intensity of 20,000 lux. The proposed ALR circuit also has a pixel loss rate of 0.3%. The experimental results show that the ALR circuit leads to a sensor pixel (SP) being autonomously adjusted, depending on the light intensity.

An event camera reports per-pixel intensity differences as an asynchronous stream of events with low latency, high dynamic range (HDR), and low power consumption. This stream of sparse/dense events limits the direct use of well-known computer vision applications for event cameras. Further applications of event streams to vision tasks that are sensitive to image quality issues, such as spatial resolution and blur, e.g., object detection, would benefit from a higher resolution of image reconstruction. Moreover, despite the recent advances in spatial resolution in event camera hardware, the majority of commercially available event cameras still have relatively low spatial resolutions when compared to conventional cameras. We propose an end-to-end recurrent network to reconstruct high-resolution, HDR, and temporally consistent grayscale or color frames directly from the event stream, and extend it to generate temporally consistent videos. We evaluate our algorithm on real-world and simulated sequences and verify that it reconstructs fine details of the scene, outperforming previous methods in quantitative quality measures. We further investigate how to (1) incorporate active pixel sensor frames (produced by an event camera) and events together in a complementary setting and (2) reconstruct images iteratively to create an even higher quality and resolution in the images.

An exceptionally simple and versatile advance in super-resolution microscopy has been created by adding a new birefringent FINCH holographic lens system including an inexpensive uncooled CMOS camera to a standard microscope. Resolution, after only a single image capture, is equivalent to or better than other more complex popular methods such as SIM, Airyscan and a number of image scanning microscopy methods that boost resolution about two-fold. This new FINCH implementation uniquely works for any objective power and NA and is solid state, fast, and calibration-free. In addition to being as easy to operate and maintain as a standard fluorescence microscope, it can uniquely create super-resolved images with any type or wavelength of light including fluorescence, bioluminescence or reflected light because its principle depends only on emitted light from objects and requires no prior training or knowledge about the sample being imaged. This microscope technique increases the utility and availability of super-resolution microscopy for any user in any research lab.

Electroluminescence images of modules obtained with InGaAs, scientific Si CMOS, cooled Si CCD, and DSLR CMOS camera technologies have been compared and analyzed. The quality of the images have been examined in terms of feature visibility. Calibrated electroluminescence image analysis was performed on many of these images to extract quantitative cell-level performance characteristics and the results from each of the cameras were compared. Image processing techniques were explored to examine their effect on image quality and quantitative analysis.

An image pattern recognition system, consisting of a CMOS camera, a set of illumination devices, and a lab-developed machine vision image analysis software, has been developed for evaluation of tool scraping wear conditions. The images of the scraped surface texture on the workpiece were generated by lab-developed automatic scraping machine under controlled conditions; the image was first transformed by Haar wavelet transform with Otsu’s algorithm to discriminate the vertical details of surface patterns, and they were then emphasized by weighted calculations according to tool scraping wear conditions. The experimental results show that the average difference between the data obtained from the surface image analysis and the roughness measurement is about ± 2%, proving the feasibility of the proposed system. Furthermore, a statistic analysis reveals that the standard deviation of the non-zero proportion of the medium frequency domain obtained by image processing might be considered a significant reference for tool replacement. For the field applications, the proposed system provides the capability of online scraping tool wear evaluation for scraping process in manufacturing site, compared with the traditional scraping tool wear evaluation method, which was time consuming; user needs to remove the workpiece to measure instrument.

In this article, we proposed a low-temperature poly-silicon (LTPS) active pixel sensor (APS) circuit with threshold voltage ( ${V}_{\text {th}}$ ) compensation. LTPS thin-film transistors (TFTs) are known to degrade under long-term irradiation and this is quantified. Simulation results and experimental data confirm that the proposed APS circuit compensates the instability of LTPS TFTs and is a promising, high-performance alternative to current amorphous silicon (a-Si) pixel-based X-ray detectors.

Portable, low-cost smartphone platform microscopic systems have emerged as a potential tool for imaging of various micron and submicron scale particles in recent years (Ozcan; Pirnstill and Cote; Breslauer et al.; Zhu et al.). In most of the reported works, it involves either the use of sophisticated optical set-ups along with a high-end computational tool for postprocessing of the captured images, or it requires a high-end configured smartphone to obtain enhanced imaging of the sample. Present work reports the working of a low-cost, field-portable 520× optical microscope using a smartphone. The proposed smartphone microscopic system has been designed by attaching a 3D printed compact optical set-up to the rear camera of a regular smartphone. By using cloud-based services, an image processing algorithm has been developed which can be accessed anytime through a mobile broadband network. Using this facility, the quality of the captured images can be further enhanced, thus obviating the need for dedicated computational tools for postprocessing of the images. With the designed microscopic system, an optical resolution ∼2 µm has been obtained. Upon postprocessing, the resolution of the captured images can be improved further. It is envisioned that with properly designed optical set-up in 3D printer and by developing an image processing application in the cloud, it is possible to obtain a low-cost, user-friendly, field-portable optical microscope on a regular smartphone that performs at par with that of a laboratory-grade microscope. LAY DESCRIPTION: With the ever-improving features both in hardware and software part, smartphone becomes ubiquitous in the modern civilised society with approximately 8.1 billion cell phone users across the world, and ∼40% of them can be considered as smartphones. This technology is undoubtedly the leading technology of the 21st century. Very recently, various researchers across the globe have utilised different sensing components embedded in the smartphone to convert it into a field-portable low-cost and user-friendly tool which can be used for different sensing and imaging purposes. By using simple optical components such as lens, pinhole, diffuser etc. and the camera of the smartphone, various groups have converted the phone into a microscopic imaging system. Again, by removing the camera lenses of the phone, holography images of microscopic particles by directly casting its shadows on the CMOS sensor on the phone has been demonstrated. The holographic images have subsequently been processed using the dedicated computational tool, and the original photos of the samples can be obtained. All the reported smartphone-based microscopic systems either suffer from relatively low field-of-view (FOV), resolution or it needs a high computational platform. Present work, demonstrate an alternative approach by which a reasonably good resolution (

In this paper, dynamic experiments of various amplitudes with a broad band of frequencies were carried out using a shaking table in order to validate the wide-range reliability of a vision-based displacement measurement system. In addition, the dynamic characteristics of a multiple-degree-of-freedom model structures were estimated using the vision-based measurement method. As a result of error analysis through comparison with a laser displacement sensor (LDS), it was determined that the accuracy of dynamic displacement measurement by the camcorder in the vision-based system is affected by the frequency of the measurement object. From these results, it is judged that the CMOS sensor type has the rolling shutter effect when the velocity between the model structure and the camcorder is high. The accuracy of dynamic characteristics estimation was determined by comparing the dynamic characteristics estimated using the LDSs and the accelerometers. The vision-based measurement system using the camcorder in both time domain and frequency domains showed very similar results to the existing LDS and accelerometer measurements.

Recently, visible light communication (VLC) based on complementary metal–oxide–semiconductor (CMOS) sensor has been widely studied, and most of the research uses the modulation and demodulation method, which modulates the light emitting diode (LED) light to transmit data and demodulates bright and dark stripes on the image captured by the CMOS sensor to get data. However, the method have some defects. Firstly, as the distance increases to a certain extent, the data frame structure will be partially lost. Secondly, the image captured by the CMOS sensor must be strictly synchronized, which is hard to guarantee. Thirdly, the focus of recent related research is mainly on the real-time nature of communication, which is difficult to achieve due to the complex image processing methods at this stage. What is more, for many application scenarios recognizing the LED information in the image captured by the CMOS sensor is enough. So, in this paper, we introduce the RGB-LED and propose an optical fringe code (LED-OFC) modulation and recognition algorithm based on VLC using convolutional neural network (CNN). The RGB-LED is modulated to assign different features to the LED-OFC instead of transmitting data. Then the CNN is employed to recognize LED-OFC with different features. The experiment results show that both the recognition accuracy, the recognition amount, the maximum recognition distance and the robustness are greatly improved by the proposed method compared with the traditional modulation and demodulation method, which has broad application prospects.

In this paper, a positioning system framework based on a single light emitting diode (LED) lamp is introduced. The LED lamp is used to transmit the positioning information, and the commercial mobile phone is used as the receiver. A fringe image carrying information is acquired by the rolling shutter effect of the complementary metal oxide semiconductor (CMOS) image sensor. Based on this, a piecewise fitting method is proposed to instead of the overall fitting method, which improves the decoding success rate. We also propose a method of LED light source recognition, which detects the edge of the light source based on brightness weakening, the Sobel operator and the least-squares fitting. In addition, the imaging principle of the camera in horizontal and tilted scenarios is analyzed. Accurate indoor positioning is achieved by the imaging principle of the camera lens and the attitude information of the mobile phone. The experimental results show that the system established in this paper can achieve centimeter-level positioning accuracy whether the mobile phone is placed horizontally or obliquely.

An integrated calibration method of the imaging sensor and the fiber optic taper-caused distortion in an intensified Complementary Metal-Oxide-Semiconductor (CMOS) camera for the X-ray capture application is proposed. Generally, four kinds of distortions need to be corrected in this system: the column noise, the inconsistent pixel intensity response of whole frame, the nonlinear intensity response of each pixel, and the geometry distortion of whole frame. To conquer these problems, first, the column noise is estimated and subtracted from the captured image. The multiple measurement data processing method and the strip noise estimation technique are used. Second, a kind of two-step correction technique of the pixel intensity response is proposed to solve middle two distortion problems. The first step builds the ideal intensity response line for all pixels in image; and the second step estimates and compensates the correction error for each pixel by a polynomial function. Third a polynomial fitting method is also utilized to correct the geometry distortion. The RANdom SAmple Consensus (RANSC) computation mechanism is used to improve the calibration precision further. The optic indices of fiber optic taper and camera are considered to assist the selection of key parameters of these calibration method above. Many experiment results have proved the correctness and effectiveness of proposed method.

This paper presents a visible light-communication-based vehicle-to-vehicle tracking system using a new positioning algorithm and modified version of the Kalman filter. In this system, LED head and tail lamps on the vehicles are used to transmit the positioning signals to other vehicles. Two CMOS dashboard cameras on each vehicle are used to receive these signals. From the geometric relationship between two cameras and the images of LEDs captured by these two cameras, the instantaneous position of the target vehicle can be determined, given that at least one LED of the target vehicle is in the view frame of the two cameras. The discrete positioning result always contains unavoidable errors, which consist of systematic errors caused by the CMOS rolling shutter artifact and the weak spatial separability of the sensor, and other random errors. The contribution of this paper is twofold. First, a new positioning algorithm with two compensation mechanisms is proposed to eliminate systematic errors. Second, a modified Kalman filter is proposed to filter out random errors to achieve a smooth and accurate tracking result for the vehicle position. The performance of the system is verified through simulations.

The development of a bioinspired image sensor, which can match the functionality of the vertebrate retina, has provided new opportunities for vision systems and processing through the realization of new architectures. Research in both retinal cellular systems and nanodriven memristive technology has made a challenging arena more accessible to emulate features of the retina that are closer to biological systems. This paper synthesizes the signal flow path of photocurrent throughout a retina in a scalable 180-nm CMOS technology, which initiates at a $128\times 128$ active pixel image sensor, and converges to a $16\times 16$ array, where each node emits a spike train synonymous to the function of the retinal ganglionic output cell. This signal can be sent to the visual cortex for image interpretation as part of an artificial vision system. Layers of memristive networks are used to emulate the functions of horizontal and amacrine cells in the retina, which average and converge signals. The resulting image matches biologically verified results within an error margin of 6% and exhibits the following features of the retina: lateral inhibition, asynchronous adaptation, and a low-dynamic-range integration active pixel sensor to perceive a high-dynamic-range scene.

This paper introduces a machine vision-based system design that is a promising solution for such a problem like a technology detecting the fatigue crack propagation. Fatigue cracks propagate because of alternating mechanical stresses. In technical components they usually start on surfaces at points of stress concentration. The system uses USB 3.0 camera with the Sony IMX264 CMOS sensor delivers 35 frames per second at 5.0 MP resolution in cooperation of PC based NI VisionBuilder software. Various segmentation methods have been explored to extract region of interest in acquired images and compared based on the capability of segmentation in real-time. The paper presents results of algorithm outputs taken on fatigue crack propagation samples and implementation of a machine vision system.

Modern implementations of widefield fluorescence microscopy often rely on sCMOS cameras, but this camera architecture inherently features pixel-to-pixel variations. Such variations lead to image artifacts and render quantitative image interpretation difficult. Although a variety of algorithmic corrections exists, they require a thorough characterization of the camera, which typically is not easy to access or perform. Here, we developed a fully automated pipeline for camera characterization based solely on thermally generated signal, and implemented it in the popular open-source software Micro-Manager and ImageJ/Fiji. Besides supplying the conventional camera maps of noise, offset and gain, our pipeline also gives access to dark current and thermal noise as functions of the exposure time. This allowed us to avoid structural bias in single-molecule localization microscopy (SMLM), which without correction is substantial even for scientific-grade, cooled cameras. In addition, our approach enables high-quality 3D super-resolution as well as live-cell time-lapse microscopy with cheap, industry-grade cameras. As our approach for camera characterization does not require any user interventions or additional hardware implementations, numerous correction algorithms demanding camera characterization become easily applicable.

This paper focuses on space cooperative target identification and ranging using monocular CMV4000 CMOS sensor and Xilinx Zynq-7000 Soc based platform without the operating system that adopts Histograms of Oriented Gradients (HOG) descriptor and Support Vector Machine (SVM) for classification. With Vivado High-Level Synthesis to accelerate IP creation by enabling C specifications to be directly targeted into devices, this implementation allows for two high-resolution images ($2048 \times2048$ pixels) processing per second. The hardware and software co-design of Xilinx Zynq7000 promises the accuracy of space target classification and distance measurement and thus meets the requirement of real-time space application.

The low-light-level sensing technology has been subject to the development of image sensors. The later, developed in the recent decades, has been improved from the low light level image intensifier, EBCCD, to recently, the CMOS technology. New technologies, for instance, APS(Active Pixel Sensor), back illuminated, etc. have been supplied following the CMOS process technology, resulting in a series of CMOS detectors with high sensitivity, lower noise and fewer operation restrictions. However, as a weak signal detection system, the operation condition, for example, the temperature characteristics related with the exposure time, ambient brightness, and target brightness, also cannot be ignored. In this paper, we presented a detailed temperature characteristic analysis based on a low-light-level CMOS system. The dark current was verified based on all black tests. The calculation as followed was drawn based on the varying ambient conditions and settings of the system:1, A typical dark current curve was obtained from the experiment, double every 9℃ with environment temperature increased.2, the information acquisition ability is affected by the exposure time, ambient brightness, and target brightness, reflected by the dark current and photon noise. Meanwhile, it was proofed in this paper, in a low light level system, the traditional signal to noise ratio calculation could not delaminate the brightness caused by the dominate photo noise, resulting an error strong enough to affect signal. Therefore, in this paper, a SNR evaluate method which could improve the objectivity of the evaluation of the low light level image was provided. It was believed to be helpful to the future research.

In this paper we present a sun position sensor platform with a scalable approach for the 3D integration of the sensor optics. This would facilitate the sun position sensor miniaturization, reduces fabrication cost and mitigates the need for sensor calibration. The sun position sensor platform is implemented in a seven mask BICMOS technology with optical windows between the light masking layer and CMOS image sensor implemented by adhesively bonded glass windows. The CMOS sensor functionality is experimentally verified by modulation of a light spot using a laser. The proposed approach enables wafer-scale fabrication of the 3D optics that includes the wafer stepper accurate overlay alignment of the apertures. This mitigates the need for cumbersome alignment of the apertures at die or package level and facilitates further miniaturization, accuracy and sensor cost.

Current methods of human body movement recognition neglect the depth denoising and edge restoration of movement image, which leads to great error in athletes’ wrong movement recognition and poor application intelligence. Therefore, an intelligent recognition method based on image vision for sports athletes’ wrong actions is proposed. The basic principle, structure, and 3D application of computer image vision technology are defined. Capturing the human body image and point cloud data, the three-dimensional dynamic model of sports athletes action is constructed. The color camera including CCD sensor and CMOS sensor is selected to collect the wrong movement image of athlete and provide image data for the recognition of wrong movement. Wavelet transform coefficient and quantization matrix threshold are introduced to denoise the wrong motion images of athletes. Based on this, the feature of sports athlete’s motion contour image is extracted in spatial frequency domain, and the edge of the image is further recovered by Canny operator. Experimental results show that the proposed method can accurately identify the wrong movements of athletes, and there is no redundancy in the recognition results. Image denoising effect is good and less time-consuming and can provide a reliable basis for related fields.

CLICTD is a monolithic silicon pixel sensor fabricated in a modified 180 nm CMOS imaging process with a small collection electrode design and a high-resistivity epitaxial layer. It features an innovative sub-pixel segmentation scheme and is optimised for fast charge collection and high spatial resolution. The sensor was developed to target the requirements for the tracking detector of the proposed future Compact Linear Collider (CLIC). Most notably, a temporal resolution of a few nanoseconds and a spatial resolution below 7 μm are demanded. In this contribution, the sensor performance measured in beam tests is presented with emphasis on recent studies using assemblies with different thicknesses (down to 50 μm to minimize the material budget) and inclined particle tracks. CLICTD is a monolithic silicon pixel sensor fabricated in a modified 180 nm CMOS imaging process with a small collection electrode design and a high-resistivity epitaxial layer. It features an innovative sub-pixel segmentation scheme and is optimised for fast charge collection and high spatial resolution. The sensor was developed to target the requirements for the tracking detector of the proposed future Compact Linear Collider (CLIC). Most notably, a temporal resolution of a few nanoseconds and a spatial resolution below 7 µm are demanded. In this contribution, the sensor performance measured in beam tests is presented with emphasis on recent studies using assemblies with different thicknesses (down to 50 µm to minimize the material budget) and inclined particle tracks. CLICTD is a monolithic silicon pixel sensor fabricated in a modified 180 nm CMOS imaging process with a small collection electrode design and a high-resistivity epitaxial layer. It features an innovative sub-pixel segmentation scheme and is optimised for fast charge collection and high spatial resolution. The sensor was developed to target the requirements for the tracking detector of the proposed future Compact Linear Collider (CLIC). Most notably, a temporal resolution of a few nanoseconds and a spatial resolution below 7 microns are demanded. In this contribution, the sensor performance measured in beam tests is presented with emphasis on recent studies using assemblies with different thicknesses (down to 50 microns to minimize the material budget) and inclined particle tracks.

Despite CNN-based deblurring models have shown their superiority on solving motion blurs, how to restore photorealistic images from severe motion blurs remains an ill-posed problem due to the loss of temporal information and textures. Video deblurring methods try to extract meaningful temporal information in multiple consecutive blurry frames during a long period of time. However, the information obtained in such a coarse period is overcomplicated, and all frames may suffer from severe motion blurs. Event cameras such as Dynamic and Active Pixel Vision Sensor (DAVIS) can simultaneously produce gray-scale Active Pixel Sensor (APS) frames and events, which capture motions as the events at very high temporal resolution, i.e., \(1\, \upmu \mathrm{s} \), and provide useful information for blurry APS frames. In this paper, we propose a deep fine-grained video deblurring pipeline consisting of a deblurring module and a recurrent module to address severe motion blurs. Concatenating the blurry image with event representations at a fine-grained temporal period, our proposed model achieves state-of-the-art performance on both popular GoPro and real blurry datasets captured by DAVIS, and is capable of generating high frame-rate video by applying a tiny shift to event representations in the recurrent module.

This work explore approach for image denoising of received by CMOS sensor. Proposed pipeline solves the problem of unsupervised training neural network architectures for image denoising which uses datasets without clean data. This approach bases on theoretical background about image restoration proposed by Nvidia researchers. We implemented custom denoising neural network architectures using specifics of noise distribution. Networks are trained on custom images collection.

We propose a calibration methodology for a novel type of inward-looking spherical light field acquisition device comprised of a moving CMOS camera with two angular degrees of freedom around an object. We designed a calibration cube covered with ChArUco markers to resolve viewpoint ambiguity. Our calibration model includes 20 unknowns describing the spherical parameters and the camera's geometrical and internal properties. These parameters are jointly optimized to reduce the reprojection error of the calibration cube's markers from multiple viewpoints, resulting in a model that can predict the pose of the camera from any other viewpoint. We successfully tested this calibrated model with a photogrammetry experiment followed by view synthesis of novel views using the resulting depth maps. Results show that the reconstructed image is highly accurate when compared to a real-life capturing of the same view.

Structural color filters (i.e. plasmonics and nano-cavities) provide vivid and robust color filtering in applications such as CMOS image sensors but lack simplicity in fabrication and dynamic tuning. Here we report a dynamically tunable, transmissive color filter by incorporating an ultra-thin phase change layer inside a thin-film optical resonator. The transmitted color spectrum can be designed over the entire visible range and shifted by around 50 nm after phase transition. Angle dependence shows little color variation within a ±30° viewing angle. Crucially, only film deposition is required to fabricate our phase change color filter, showing great potential for large-scale and inexpensive production. The dynamically tunable color filter, described in this paper, could be a promising component in display, CMOS sensor, and solar cell technology.

A method for extracting lettuce phenotypic features using an ARTY A7-35T FPGA platform is proposed. Image acquisition is done by interfacing an OV7670 CMOS camera with FPGA and saving image data on DDR3 memory. The image processing techniques firstly involve color model conversion for saturation enhancement. Then, binarization is done as an initial step in background discrimination, using a threshold value on the green channel of image. To construct a solid figure for foreground, morphological transformations are implemented. Then, pixel count of foreground white pixels are used as an argument in the computation of lettuce canopy area. The FPGA implementation consumes 85.89% of total LUTs resources of the chosen FPGA board with errors as low as 1.28% on the computed canopy area value compared with a MATLAB benchmark. Power consumption reached up to 1.73W, with a total calculated latency of 596.51 ms from image acquisition to canopy area value.

This paper proposes a improved method for measuring the focal length of a telephoto lens. In this paper, a photoelectric theodolite with high accuracy of angle measurement is used. Besides, a CMOS camera is used to take a position image of the center point of the theodolite, and the corresponding azimuth displayed by the theodolite is recorded. Leveling the optoelectronic theodolite on the open field, Polaris is used as a point light source outdoors. Adjusting the point target on the CMOS image, by which the target is placed in the view center of the field (crosshair center). And then, the pitch and azimuth data of the device are recorded. Adding disturbance to the photoelectric theodolite, the CMOS images is recorded. Adapting methods of target feature segmentation, sub-pixel positioning is performed on the point target image which captured by the CMOS camera. And the point target coordinates is obtained. Finally, the result based on the theodolite angle offset, sub-pixel offset and elevation angle data is calculated. The perturbation data is averaged to calculate the focal length of the telephoto lens. Compared with equipment testing result of focal length, experiments have achieved high measurement accuracy. And this method is suitable for detecting the focal length of an optoelectronic device in an outdoor environment.

International audience; In this paper, we introduce a novel color pixel reconstruction algorithm for a RGBZ CMOS imager. A RGBZ imager is a Vision System on Chip (VSoC) sensor that captures simultaneously color and depth information with a hybrid pixel matrix. The imager pattern is based on a Bayer filter array where the Z pixel covers an equivalent area of a 2x2-color-pixel size, and one quarter of the total color information is missing. We propose an original adaptive algorithm to reconstruct the color information using a new operator called semi-gradient. The results show improvements on edges, corners, and narrow lines reconstruction, and a reduction of color and structural artefacts compared to classical reconstruction algorithms.

Cubert GmbH (Germany) has developed a novel snapshot hyperspectral imaging approach. Our camera is based on light field technology, where both the intensity and direction of incident light rays are used to produce spectral images. The camera combines a continuously variable bandpass filter, a multi-lens array and a 20-MP CMOS sensor. This spectral camera comes with a native resolution of 410x410 pixel, resulting in 168,100 pixels, each with 125 spectral bands covering 450-950nm, taken in a single snapshot. Our snapshot imaging spectrometer captures the entire scene in a few milliseconds or less, which makes it a game-changer in real-time spectral imaging and video spectroscopy.

Multispectral imaging (MI) techniques are being used very often to identify different properties of nature in several domains, going from precision agriculture to environmental studies, not to mention quality inspection of pharmaceutical production, art restoration, biochemistry, forensic sciences or geology, just to name some. Different implementations are commercially available from the industry and yet there is quite an interest from the scientific community to spread its use to the majority of society by means of cost effectiveness and ease of use for solutions. These devices make the most sense when combined with unmanned aerial vehicles (UAVs), going a step further and alleviating repetitive routines which could be strenuous if traditional methods were adopted. In this work, a low cost and modular solution for a multispectral camera is presented, based on the use of a single panchromatic complementary metal oxide semiconductor (CMOS) sensor combined with a rotating wheel of interchangeable band pass optic filters. The system is compatible with open source hardware permitting one to capture, process, store and/or transmit data if needed. In addition, a calibration and characterization methodology has been developed for the camera, allowing not only for quantifying its performance, but also able to characterize other CMOS sensors in the market in order to select the one that best suits the budget and application. The process was experimentally validated by mounting the camera in a Dji Matrice 600 UAV to uncover vegetation indices in a reduced area of palm trees plantation. Results are presented for the normalized difference vegetation index (NDVI) showing a generated colored map with the captured information.

Neuromorphic event cameras are useful for dynamic vision problems under difficult lighting conditions. To enable studies of using event cameras in automobile driving applications, this paper reports a new end-to-end driving dataset called DDD20. The dataset was captured with a DAVIS camera that concurrently streams both dynamic vision sensor (DVS) brightness change events and active pixel sensor (APS) intensity frames. DDD20 is the longest event camera end-to-end driving dataset to date with 51h of DAVIS event+frame camera and vehicle human control data collected from 4000km of highway and urban driving under a variety of lighting conditions. Using DDD20, we report the first study of fusing brightness change events and intensity frame data using a deep learning approach to predict the instantaneous human steering wheel angle. Over all day and night conditions, the explained variance for human steering prediction from a Resnet-32 is significantly better from the fused DVS+APS frames (0.88) than using either DVS (0.67) or APS (0.77) data alone., Accepted in The 23rd IEEE International Conference on Intelligent Transportation Systems (Special Session: Beyond Traditional Sensing for Intelligent Transportation)

This paper proposes a new frame recovery method for screen to camera communication (SCC). SCC is a signal transmission system using a display as transmitter and a camera as receiver. In case of using a CMOS sensor camera, a captured video is distorted by rolling shutter effect which causes bit errors. Against the rolling shutter distortion, we propose a model of distorted video frame and a frame recovery method. In order to recover the frame, we insert known symbols to a transmitted frame. The frame is recovered by solving an optimization problem using the inserted known symbols. To evaluate the proposed method performance, we conduct an experiment using commercial devices. The result shows that the proposed method can accurately recover the transmitted frame and improve bit error rate compared to a simply binarization method and an adaptive binarization method.

To realize the application of super-resolution technology from theory to practice, and to improve microsatellite spatial resolution, we propose a special super-resolution algorithm based on the multi-modality super-CMOS sensor which can adapt to the limited operation capacity of microsatellite computers. First, we designed an oblique sampling mode with the sensor rotated at an angle of 26.56 ∘ ( arctan 1 2 ) to obtain high overlap ratio images with sub-pixel displacement. Secondly, the proposed super-resolution algorithm was applied to reconstruct the final high-resolution image. Because the satellite equipped with this sensor is scheduled to be launched this year, we also designed the simulation mode of conventional sampling and the oblique sampling of the sensor to obtain the comparison and experimental data. Lastly, we evaluated the super-resolution quality of images, the effectiveness, the practicality, and the efficiency of the algorithm. The results of the experiments showed that the satellite-using super-resolution algorithm combined with multi-modality super-CMOS sensor oblique-mode sampling can increase the spatial resolution of an image by about 2 times. The algorithm is simple and highly efficient, and can realize the super-resolution reconstruction of two remote-sensing images within 0.713 s, which has good performance on the microsatellite.