Real-time image signal processor for SOC/FPGA with direct GPU communication

Many real-time computer image processing applications, e.g., autonomous driving or machine control, need large processing power but also require little space and power consumption. The preprocessing of video data (demosaic, convolutional filtering, statistics, noise shaping etc.), which is normally done in a GPU/CPU, is a major part and consumes lot of resources. The Institute of Embedded Systems of ZHAW developed an Image Signal Processor (ISP), which takes care of demosaicing the raw camera data and has automatic white balancing, automatic exposure time, configurable convolution filtering, image sharpening, color space conversion, gamma correction, collecting statistic data, etc. The output of the video processing is fed directly into the GPU via a newly developed frame-based DMA (FDMA), avoiding the delay that would otherwise be introduced by the processor's video pipeline. The ISP consists of a pipeline of multiple video processing functions that process the raw camera data without CPU involvement. The ISP operates independently. Camera data can be fed into the ISP via MIPI-CSI. In addition to a detailed discussion of the modular ISP, we also discuss the frame base DMA (FDMA) method, which allows data to be fed directly into the GPU via PCI-e without CPU involvement. The implementation can be demonstrated on a specially designed video capture card with six camera interfaces.