Search

In this paper, we propose an effective object detection framework based on proposal fusion of multiple sensors such as infrared camera, RGB cameras, radar and LiDAR. Our framework first applies the Selective Search (SS) method on RGB image data to extract possible candidate proposals which likely contain the objects of interest. Then it uses the information from other sensors in order to reduce the number of generated proposals by SS and find more dense proposals. Finally, the class of objects within the final proposals are identified by Convolutional Neural Network (CNN). Experimental results on real dataset demonstrate that our framework can precisely detect meaningful object regions using a smaller number of proposals than other object proposals methods. Further, our framework can achieve reliable object detection and classification results in maritime environments.

Robust real-time object detection and tracking are challenging problems in autonomous transportation systems due to operation of algorithms in inherently uncertain and dynamic environments and rapid movement of objects. Therefore, tracking and detection algorithms must cooperate with each other to achieve smooth tracking of detected objects that later can be used by the navigation system. In this paper, we first present an efficient multi-sensor fusion approach based on the probabilistic data association method in order to achieve accurate object detection and tracking results. The proposed approach fuses the detection results obtained independently from four main sensors: radar, LiDAR, RGB camera and infrared camera. It generates object region proposals based on the fused detection result. Then, a Convolutional Neural Network (CNN) approach is used to identify the object categories within these regions. The CNN is trained on a real dataset from different ferry driving scenarios. The experimental results of tracking and classification on real datasets show that the proposed approach provides reliable object detection and classification results in maritime environments.

Online monitoring and closed-loop control of laser welding offer great possibilities for achieving better weld quality. Earlier work on visual laser welding monitoring has mainly focused on aluminum and fairly thin steel used, for example, in car production. We extend this work by focusing on the automated analysis of the phenomena present in the laser welding of thick steel, where all of the phenomena related to the weld quality are still not well understood or controlled. This paper presents the implementation, test results and analysis for weld monitoring methods implemented on a compact smart camera system. The applied embedded sensor–processor platform allows for high-speed implementation of image capture and dynamic range compression, real-time seam tracking and spatter feature extraction. The paper describes experimental results from implemented real-time algorithms for seam tracking and spatter extraction and additional off-line analysis of methods for spatter tracking and seam widening detection, which are also feasible for future online hardware implementation. The results suggest that it is possible to integrate a compact laser welding analysis system, which achieves analysis rates that are sufficient for real-time process control.

We describe a novel data-processing and analysis pipeline for optical observations of moving objects, either of natural (asteroids, meteors) or artificial origin (satellites, space debris). The monitoring of the space object populations requires reliable acquisition of observational data, to support the development and validation of population models and to build and maintain catalogues of orbital elements. The orbital catalogues are, in turn, needed for the assessment of close approaches (for asteroids, with the Earth; for satellites, with each other) and for the support of contingency situations or launches. For both types of populations, there is also increasing interest to detect fainter objects corresponding to the small end of the size distribution. The ESA-funded StreakDet (streak detection and astrometric reduction) activity has aimed at formulating and discussing suitable approaches for the detection and astrometric reduction of object trails, or streaks, in optical observations. Our two main focuses are objects in lower altitudes and space-based observations (i.e., high angular velocities), resulting in long (potentially curved) and faint streaks in the optical images. In particular, we concentrate on single-image (as compared to consecutive frames of the same field) and low-SNR detection of objects. Particular attention has been paid to the process of extraction of all necessary information from one image (segmentation), and subsequently, to efficient reduction of the extracted data (classification). We have developed an automated streak detection and processing pipeline and demonstrated its performance with an extensive database of semisynthetic images simulating streak observations both from ground-based and space-based observing platforms. The average processing time per image is about 13 s for a typical 2k-by-2k image. For long streaks (length >100 pixels), primary targets of the pipeline, the detection sensitivity (true positives) is about 90% for both scenarios for the bright streaks ( SNR > 1 ), while in the low-SNR regime, the sensitivity is still 50% at SNR = 0.5 .

We present design principles for implementing decoders for low-density parity check codes in CMOL-type memristive circuits. The programmable nonvolatile connectivity enabled by the nanowire arrays in such circuits is used to map the parity check matrix of an LDPC code in the decoder, while decoding operations are realized by a cellular CMOS circuit structure. We perform detailed performance analysis and circuit simulations of example decoders, and estimate how CMOL and memristor characteristics such as the memristor OFF/ON resistance ratio, nanowire resistance, and the total capacitance of the nanowire array affect decoder specification and performance. We also analyze how variation in circuit characteristics and persistent device defects affect the decoders.

The high speed thermal imaging is necessary in many applications. However, the traditional column-wise readout implementations reduce the achievable frame-rate. Also, the analog integration of each individual pixel is not possible without sacrificing the pixel area. In this paper we present an algorithm that can be used in parallel focal-plane processing for fast imaging. First we prove that our method is valid in the ideal case. Then we add non-idealities such as noise and show means to cope with them.

In this paper we present a CMOS implementation of a 512×512-cell Associative Content Addressable Memory (ACAM) in 180 nm CMOS. The memory can be operated either as an associative CAM or it can be configured into a Willshaw memory for operating with sparse data. The vector matching operation can use a tunable hit threshold or the strongest hit can be selected with a winner-take-all (WTA) network. Built-in row and column circuitry can perform logic operations on the contents of the row and column memories. The operation of the circuit is verified experimentally with an example on computing with random vectors.

This paper presents the implementation of optical seam tracking for a laser welding process with parallel image processing on a smart camera. Real-time extraction of the seam location and direction is enabled by a combination of massively parallel focal plane processing on a Vision Chip and Hough-transform based analysis on an FPGA. The implementation of all analysis within a single embedded camera system offers low-lag control for correcting the path of a welding robot. The vision chip provides illumination adaptive image sensing and neighborhood based feature segmentation. To extract the dominant line from the segmented data, an auxiliary FPGA processor performs Hough analysis on the reduced binary data from the vision chip. The system can achieve a line extraction speed of more than 1000 fps, which enables real-time visual seam tracking and robot control. Evaluation results from on-line laser welding tests with processing performed on the camera system are reported in the article.

This paper presents the development of methods for real-time fine-tuning of a high power laser welding process of thick steel by using a compact smart camera system. When performing welding in butt-joint configuration, the laser beam’s location needs to be adjusted exactly according to the seam line in order to allow the injected energy to be absorbed uniformly into both steel sheets. In this paper, on-line extraction of seam parameters is targeted by taking advantage of a combination of dynamic image intensity compression, image segmentation with a focal-plane processor ASIC, and Hough transform on an associated FPGA. Additional filtering of Hough line candidates based on temporal windowing is further applied to reduce unrealistic frame-to-frame tracking variations. The proposed methods are implemented in Matlab by using image data captured with adaptive integration time. The simulations are performed in a hardware oriented way to allow real-time implementation of the algorithms on the smart camera system.

In this paper we describe the application of an embedded camera system consisting of a Vision Chip and an FPGA for fast line detection in industrial monitoring. The vision chip is used to perform low level image content reduction tasks where the result is a binary image consisting of e.g. edges in the image. In order to extract the equation of the dominant line from the Vision Chip output results, an FPGA auxiliary processor has been designed to perform the Hough transform for the binary result from the Vision Chip. The system is currently capable of achieving a 200-670fps line recognition speed using realistic input images. The achieved level of performance is sufficient for many industrial applications, e.g. in laser welding process for optical seam tracking. Because the Vision Chip operation is not affected by the Hough estimation, which is performed independently by the FPGA, other welding process analysis can be performed simultaneously at much higher frame rates (upto 10 000 fps demonstrated). The internal statistics of the Hough unit can also be used as feedback to enable parameter adjustment in the Vision Chip, yielding better end results.

This chapter introduces the concept of a memcapacitor, and reviews different approaches to its physical realization. Also, practical constraints for their usage are assessed. Because of their compatibility with traditional circuit integration technologies, two approaches are particularly interesting: the ferroelectric capacitor and the memcapacitor constructed by appending metal-insulator-metal (MIM) capacitor with a memristive switching layer. Ferroelectric capacitors have already been in use for many years so the properties of this technology are relatively well researched. The MIM-memristor hybrid structure can take advantage of the vital research on memristive memories. With sufficiently large ratio of the OFF and ON resistances of a memristive material, the compound structure behaves as a memcapacitive system. Finally, the potential of memcapacitors for memory and logic applications as well as for artificial neural networks are discussed.

Laser welding has become a very important method for industrial manufacturing. Despite of the inherent accuracy of laser welding, the resulting weld quality may still be affected by many dynamic conditions related to the operating parameters and to the properties of the welded material. Methods for monitoring the laser welding process are therefore needed to guarantee consistent manufacturing quality. In this paper, we present a method for characterizing spatters in laser welding of thick steel. Pre-processing and edge detection steps of the proposed algorithm are performed on-line with a very high speed by using a dedicated KOVA1 massively parallel image processing chip, and the actual characterization of the spatters is carried out off-line in Matlab. The methods proposed are simple and efficient, thus also facilitating possible integration of the whole algorithm for on-line processing.

The research reported in this paper focuses on the modeling of Local Binary Patterns (LBPs) and presents an a priori model where LBPs are considered as combinations of permutations. The aim is to increase the understanding of the mechanisms related to the formation of uniform LBPs. Uniform patterns are known to exhibit high discriminative capability; however, so far the reasons for this have not been fully explored. We report an observation that although the overall a priori probability of uniform LBPs is high, it is mostly due to the high probability of only certain classes of patterns, while the a priori probability of other patterns is very low. In order to examine this behavior, the relationship between the runs up and down test for randomness of permutations and the uniform LBPs was studied. Quantitative experiments were then carried out to show that the relative effect of uniform patterns to the LBP histogram is strengthened with deterministic data, in comparison with the i.i.d. model. This was verified by using an a priori model as well as through experiments with natural image data. It was further illustrated that specific uniform LBP codes can also provide responses to salient shapes, that is, to monotonically changing intensity functions and edges within the image microstructure.

This chapter describes MIPA4k, a 64 ×64 cell mixed-mode image processor array chip. Each cell includes an image sensor, A/D/A conversion, embedded digital and analog memories, and hardware-optimized grey-scale and binary processing cores. We describe the architecture of the processor cell, go through the different functional blocks and explore its processing capabilities. The processing capabilities of the cells include programmable space-dependent neighbourhood connections, ranked-order filtering, rank identification and anisotropic resistive filtering. For example, asynchronous analog morphological reconstruction operation can be performed with MIPA4k. The image sensor has an option for locally adaptive exposure time. Also, the peripheral circuitry can highlight windows of activation, and pattern matching can be performed on these regions of interest (ROI) with the aid of parallel write operation to the active window. As the processing capabilities are complemented with global OR and global sum operations, MIPA4k is an effective tool for high-speed image analysis.

In this paper, an approach to computing arbitrary Boolean functions using a continuous-state cellular neural/nonlinear/nanoscale network (CNN) architecture with local static memory is presented. We explain how any given Boolean function can be mapped to a CNN array and how the function is executed using a sequence of wave operations. Furthermore, we explain how gray-scale waves could reduce the number of CNN cells required to perform a certain logic operation. The main benefits of our approach are simple implementation of arbitrary Boolean functions, asynchronous operation, and applicability in multi-state computation.

Emerging 3D-integration enables integrating high quality image sensors with various massively parallel processing elements. Analog motion estimation is one potential application, which is likely to result in significant benefits in the form of low power or high frame-rate 3D-integrated image sensor-processors. The system-level operation of a proposed analog motion estimation array, enabling all various block sizes from 4×4 to 16×16 is examined. The analog motion estimation circuitry has been designed as a 32×32 test array in 0.13 µm CMOS technology. The transistor-level simulation results combined with H.264/AVC JM 14.2 show equivalent rate-distortion results with SAD as the error measure and an approximately 7% increase in bitrate with a slight increase in image quality for SSE.

This paper presents an implementation of locally adaptive image sensing with the MIPA4k image processor array. The implemented adaptive sensor circuitry allows the extraction of image detail in the presence of large dynamic intra-scene lighting variations. The imple­mented circuit method is based on the use of nonlinear resistive network filtering during image integration for controlling the pixel integration times locally, through the extraction and examination of average local intensity. Measured examples of the adaptive sensing operation performed with the MIPA4k processor array are presented.

This paper proposes a new active quenching and ac­tive recharge circuit for single photon avalanche diodes (SPADs). Thanks to the simple ring oscillator architecture, the circuit is one of the most compact active quenching circuits published in the literature. Additionally, a simple modification to the basic version of the circuit allows to measure the hold-off delay and the recharge pulse width without any high-speed time interval measurement electronics by configuring the circuit as a free-running ring oscillator which can be used for testing purposes and for implementation of a hold-off delay control. A test chip has been fabricated in a standard 0.13 μm CMOS technology and simulations and experimental results are reported.

This live demonstration and the accompanying full-length paper are related to the ISCAS track “Cellular Neural Networks and Array Computing/Implementations of Topographical Microprocessors”.

A cellular analog processor array for use in variable block-size motion estimation with a new simple method for shifting reference image data is presented. The new shift method leads to a greatly reduced number of neighborhood connections for each cell of the array, and allows for all shifts within the [8,8] search area to be performed in a single step, with simple digital controls. The new shift circuitry, together with some other cell and system level optimizations, reduces silicon area and array layout complexity, enabling faster and more efficient parallel full search motion estimation hardware. A Open image in new window cell parallel analog test array for reference-shift with a maximum block-size of Open image in new window, as well as absolute value/quadratic processing for variable block-size analog motion estimation (AME) has been designed in a 0.13 Open image in new windowm CMOS technology.

This paper examines the effects of device mismatch in analog gray-scale morphological processing. A hardware oriented simulation environment is used. This makes it possible to simulate the contributions of mismatch on the level of a large array, which would not be possible in practice with circuit level Monte Carlo simulations because of the excessive computational requirements. In particular, this paper analyses the implementation of morphological gray-scale reconstruction, in the presence of process variations.