Search

This paper proposes an edge-preserving regularization technique to solve the color image demosaicing problem in the realistic case of noisy data. We enforce intra-channel local smoothness of the intensity (low-frequency components) and inter-channel local similarity of the depth of object borders and textures (high-frequency components). Discontinuities of both the low-frequency and high-frequency components are accounted for implicitly, i.e., through suitable functions of the proper derivatives. For the treatment of even the finest image details, derivatives of first, second, and third orders are considered. The solution to the demosaicing problem is defined as the minimizer of an energy function, accounting for all these constraints plus a data fidelity term. This non-convex energy is minimized via an iterative deterministic algorithm, applied to a family of approximating functions, each implicitly referring to geometrically consistent image edges. Our method is general because it does not refer to any specific color filter array. However, to allow quantitative comparisons with other published results, we tested it in the case of the Bayer CFA and on the Kodak 24-image dataset, the McMaster (IMAX) 18-image dataset, the Microsoft Demosaicing Canon 57-image dataset, and the Microsoft Demosaicing Panasonic 500-image dataset. The comparisons with some of the most recent demosaicing algorithms show the good performance of our method in both the noiseless and noisy cases. [ABSTRACT FROM AUTHOR]

In modern digital photogrammetry, an image is usually registered via a digital matrix with an array of colour filters. From the registration of the image until feature points are detected on the image, the image is subjected to a series of calculations, i.e., demosaicing and conversion to greyscale, among others. These algorithms respond differently to the varying light spectrum of the scene, which consequently results in the feature location changing. In this study, the effect of scene illumination on the localisation of a feature in an image is presented. The demosaicing and greyscale conversion algorithms that produce the largest and smallest deviation of the feature from the reference point were assessed. Twelve different illumination settings from polychromatic light to monochromatic light were developed and performed, and five different demosaicing algorithms and five different methods of converting a colour image to greyscale were analysed. A total of 300 different cases were examined. As the study shows, the lowest deviation in the polychromatic light domain was achieved for light with a colour temperature of 5600 K and 5000 K, while in the monochromatic light domain, it was achieved for light with a green colour. Demosaicing methods have a significant effect on the localisation of a feature, and so the smallest feature deviation was achieved for smooth hue-type demosaicing, while for greyscale conversion, it was achieved for the mean type. Demosaicing and greyscale conversion methods for monochrome light had no effect. The article discusses the problem and concludes with recommendations and suggestions in the area of illuminating the scene with artificial light and the application of the algorithms, in order to achieve the highest accuracy using photogrammetric methods. [ABSTRACT FROM AUTHOR]

Image demosaicing and denoising play a critical role in the raw imaging pipeline. These processes have often been treated as independent, without considering their interactions. Indeed, most classic denoising methods handle noisy RGB images, not raw images. Conversely, most demosaicing methods address the demosaicing of noise free images. The real problem is to jointly denoise and demosaic noisy raw images. But the question of how to proceed is still not clarified. In this paper, we carry out extensive experiments and a mathematical analysis to tackle this problem by low complexity algorithms. Indeed, both problems have only been addressed jointly by end-to-end heavy-weight convolutional neural networks (CNNs), which are currently incompatible with low-power portable imaging devices and remain by nature domain (or device) dependent. Our study leads us to conclude that, with moderate noise, demosaicing should be applied first, followed by denoising. This requires a simple adaptation of classic denoising algorithms to demosaiced noise, which we justify and specify. Although our main conclusion is 'demosaic first, then denoise, ' we also discover that for high noise, there is a moderate PSNR gain by a more complex strategy: partial CFA denoising followed by demosaicing and by a second denoising on the RGB image. These surprising results are obtained by a black-box optimization of the pipeline, which could be applied to any other pipeline. We validate our results on simulated and real noisy CFA images obtained from several benchmarks. [ABSTRACT FROM AUTHOR]

Image demosaicing is an important step in the image processing pipeline for digital cameras. In data centric approaches, such as deep learning, the distribution of the dataset used for training can impose a bias on the networks' outcome. For example, in natural images most patches are smooth, and high-content patches are much rarer. This can lead to a bias in the performance of demosaicing algorithms. Most deep learning approaches address this challenge by utilizing specific losses or designing special network architectures. We propose a novel approach SDAT, Sub-Dataset Alternation Training, that tackles the problem from a training protocol perspective. SDAT is comprised of two essential phases. In the initial phase, we employ a method to create sub-datasets from the entire dataset, each inducing a distinct bias. The subsequent phase involves an alternating training process, which uses the derived sub-datasets in addition to training also on the entire dataset. SDAT can be applied regardless of the chosen architecture as demonstrated by various experiments we conducted for the demosaicing task. The experiments are performed across a range of architecture sizes and types, namely CNNs and transformers. We show improved performance in all cases. We are also able to achieve state-of-the-art results on three highly popular image demosaicing benchmarks.

Multi-spectral imaging technologies have made great progress in the past few decades. The development of snapshot cameras equipped with a specific multi-spectral filter array (MSFA) allow dynamic scenes to be captured on a miniaturized platform across multiple spectral bands, opening up extensive applications in quantitative and visualized analysis. However, a snapshot camera based on MSFA captures a single band per pixel; thus, the other spectral band components of pixels are all missed. The raw images, which are captured by snapshot multi-spectral imaging systems, require a reconstruction procedure called demosaicing to estimate a fully defined multi-spectral image (MSI). With increasing spectral bands, the challenge of demosaicing becomes more difficult. Furthermore, the existing demosaicing methods will produce adverse artifacts and aliasing because of the adverse effects of spatial interpolation and the inadequacy of the number of layers in the network structure. In this paper, a novel multi-spectral demosaicing method based on a deep convolution neural network (CNN) is proposed for the reconstruction of full-resolution multi-spectral images from raw MSFA-based spectral mosaic images. The CNN is integrated with the channel attention mechanism to protect important channel features. We verify the merits of the proposed method using 5 × 5 raw mosaic images on synthetic as well as real-world data. The experimental results show that the proposed method outperforms the existing demosaicing methods in terms of spatial details and spectral fidelity. [ABSTRACT FROM AUTHOR]

Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales. However, recent advanced deep convolutional neural network-based methods for image restoration do not take full advantage of self-similarities by relying on self-attention neural modules that only process information at the same scale. To solve this problem, we present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid. Inspired by the fact that corruptions, such as noise or compression artifacts, drop drastically at coarser image scales, our attention module is designed to be able to borrow clean signals from their "clean" correspondences at the coarser levels. The proposed pyramid attention module is a generic building block that can be flexibly integrated into various neural architectures. Its effectiveness is validated through extensive experiments on multiple image restoration tasks: image denoising, demosaicing, compression artifact reduction, and super resolution. Without any bells and whistles, our PANet (pyramid attention module with simple network backbones) can produce state-of-the-art results with superior accuracy and visual quality. Our code is available at https://github.com/SHI-Labs/Pyramid-Attention-Networks [ABSTRACT FROM AUTHOR]

Deep learning-based image signal processor (ISP) models for mobile cameras can generate high-quality images that rival those of professional DSLR cameras. However, their computational demands often make them unsuitable for mobile settings. Additionally, modern mobile cameras employ non-Bayer color filter arrays (CFA) such as Quad Bayer, Nona Bayer, and $\text{Q}\times \text{Q}$ Bayer to enhance image quality, yet most existing deep learning-based ISP (or demosaicing) models focus primarily on standard Bayer CFAs. In this study, we present PyNET- $\text{Q}\times \text{Q}$ , a lightweight demosaicing model specifically designed for $\text{Q}\times \text{Q}$ Bayer CFA patterns, which is derived from the original PyNET. We also propose a knowledge distillation method called progressive distillation to train the reduced network more effectively. Consequently, PyNET- $\text{Q}\times \text{Q}$ contains less than 2.5% of the parameters of the original PyNET while preserving its performance. Experiments using $\text{Q}\times \text{Q}$ images captured by a prototype $\text{Q}\times \text{Q}$ camera sensor show that PyNET- $\text{Q}\times \text{Q}$ outperforms existing conventional algorithms in terms of texture and edge reconstruction, despite its significantly reduced parameter count. Code and partial datasets can be found at https://github.com/Minhyeok01/PyNET-QxQ.

Multi-spectral imaging using a camera with more than three channels is an efficient method to acquire and reconstruct spectral data and is used extensively in tasks like object recognition, relighted rendering, and color constancy. Recently developed methods are used to only guide content-dependent filter selection where the set of spectral reflectances to be recovered are known a priori. We present the first content-independent spectral imaging pipeline that allows optimal selection of multiple channels. We also present algorithms for optimal placement of the channels in the color filter array yielding an efficient demosaicing order resulting in accurate spectral recovery of natural reflectance functions. These reflectance functions have the property that their power spectrum statistically exhibits a power-law behavior. Using this property, we propose power-law based error descriptors that are minimized to optimize the imaging pipeline. We extensively verify our models and optimizations using large sets of commercially available wide-band filters to demonstrate the greater accuracy and efficiency of our multi-spectral imaging pipeline over existing methods.

Recent studies in the light field imaging have shown the potential and advantages of different light field information processes. In most of the existing techniques, the processing pipeline of light field has been treated in a step-by-step manner, and each step is considered to be independent from the others. For example, in light field color demosaicing, inferring the scene geometry is treated as an irrelevant and negligible task, and vice versa. Such processing techniques may fail due to the inherent connection among different steps, and result in both corrupted post-processing and defective pre-processing results. In this paper, we address the interaction between color interpolation and depth estimation in light field, and propose a probabilistic approach to handle these two processing steps jointly. This probabilistic framework is based on a Markov Random Fields —Collaborative Graph Model for simultaneous Demosaicing and Depth Estimation (CGMDD)—to explore the color-depth interdependence from general light field sampling. Experimental results show that both image interpolation quality and depth estimation can benefit from their interaction, mainly for processes such as image demosaicing which are shown to be sensitive to depth information, especially for light field sampling with large baselines.

Multispectral imaging devices incorporating up to 256 different spectral channels have recently become available for various healthcare applications, as e.g. laparoscopy, gastroscopy, dermatology or perfusion imaging for wound analysis. Currently, the use of such devices is limited due to very high investment costs and slow capture times. To compensate these shortcomings, single sensors with spectral masking on the pixel level have been proposed. Hence, adequate spectral reconstruction methods are needed. Within this work, two deep convolutional neural networks (DCNN) architectures for spectral image reconstruction from single sensors are compared with each other. Training of the networks is based on a huge collection of different MSI imagestacks, which have been subsampled, simulating 16-channel single sensors with spectral masking. We define a training, validation and test set (‘HITgoC’) resulting in 351 training (631.128 sub-images), 99 validation (163.272 sub-images) and 51 test images. For the application in the field of neurosurgery an additional testing set of 36 image stacks from the Nimbus data collection is used, depicting MSI brain data during open surgery. Two DCNN architectures were compared to bilinear interpolation (BI) and an intensity difference (ID) algorithm. The DCNNs (ResNet-Shinoda) were trained on HITgoC and consist of a preprocessing step using BI or ID and a refinement part using a ResNet structure. Similarity measures used were PSNR, SSIM and MSE between predicted and reference images. We calculated the similarity measures for HitgoC and Nimbus data and determined differences of the mean similarity measure values achieved with the ResNet-ID and baseline algorithms such as BI algorithm and ResNet-Shinoda. The proposed method achieved better results against BI in SSIM (.0644 vs. .0252), PSNR (15.3 dB vs. 9.1 dB) and 1-MSE*100 (.0855 vs. .0273) and compared to ResNet-Shinoda in SSIM (.0103 vs. .0074), PSNR (3.8 dB vs. 3.6 dB) and 1-MSE*100 (.0075 vs. .0047) for HITgoC/Nimbus. In this study, significantly better results for spectral reconstruction in MSI images of open neurosurgery was achieved using a combination of ID-interpolation and ResNet structure compared to standard methods.

Polarisation Filter Array (PFA) cameras allow the analysis of light polarisation state in a simple and cost-effective manner. Such filter arrays work as the Bayer pattern for colour cameras, sharing similar advantages and drawbacks. Among the others, the raw image must be demosaiced considering the local variations of the PFA and the characteristics of the imaged scene. Non-linear effects, like the cross-talk among neighbouring pixels, are difficult to explicitly model and suggest the potential advantage of a data-driven learning approach. However, the PFA cannot be removed from the sensor, making it difficult to acquire the ground-truth polarization state for training. In this work we propose a novel CNN-based model which directly demosaics the raw camera image to a per-pixel Stokes vector. Our contribution is twofold. First, we propose a network architecture composed by a sequence of Mosaiced Convolutions operating coherently with the local arrangement of the different filters. Second, we introduce a new method, employing a consumer LCD screen, to effectively acquire real-world data for training. The process is designed to be invariant by monitor gamma and external lighting conditions. We extensively compared our method against algorithmic and learning-based demosaicing techniques, obtaining a consistently lower error especially in terms of polarisation angle. [ABSTRACT FROM AUTHOR]

Spatially varying exposure (SVE) is a promising choice for high-dynamic-range (HDR) imaging (HDRI). The SVE-based HDRI, which is called single-shot HDRI, is an efficient solution to avoid ghosting artifacts. However, it is very challenging to restore a full-resolution HDR image from a real-world image with SVE because: a) only one-third of pixels with varying exposures are captured by camera in a Bayer pattern, b) some of the captured pixels are over- and under-exposed. For the former challenge, a spatially varying convolution (SVC) is designed to process the Bayer images carried with varying exposures. For the latter one, an exposure-guidance method is proposed against the interference from over- and under-exposed pixels. Finally, a joint demosaicing and HDRI deep learning framework is formalized to include the two novel components and to realize an end-to-end single-shot HDRI. Experiments indicate that the proposed end-to-end framework avoids the problem of cumulative errors and surpasses the related state-of-the-art methods. Related codes and datasets will be provided at https://github.com/yilun-xu/SVEHDRI/. [ABSTRACT FROM AUTHOR]

Objective. Despite the demonstrated benefits of hyperspectral formalism for stem effect corrections in the context of fiber dose measurements, this approach has not been yet translated into volumetric measurements where cameras are typically used for their distinguishing spatial resolution. This work investigates demosaicing algorithms for polychromatic cameras based spectral imaging. Approach. The scintillation and Cherenkov signals produced in a radioluminescent phantom are imaged by a polychromatic camera and isolated using the spectral formalism. To do so, five demosaicing algorithms are investigated from calibration to measurements: a clustering method and four interpolation algorithms. The resulting accuracy of scintillation and Cherenkov images is evaluated with measurements of the differences (mean ± standard deviation) between the obtained and expected signals from profiles drawn across a scintillation spot. Signal-to-noise ratio and signal-to-background ratio are further measured and compared in the resulting scintillation images. Finally, the resulting differences on the scintillation signal from a 0.2 × 0.2 cm2 region-of-interest (ROI) were reported. Main results. Clustering, OpenCV, bilinear, Malvar and Menon demosaicing algorithms respectively yielded differences of 3 ± 5%, 1 ± 3%, 1 ± 3%, 1 ± 2% and 2 ± 4% in the resulting scintillation images. For the Cherenkov images, all algorithms provided differences below 1%. All methods enabled measurements over the detectability (SBR > 2) and sensitivity (SNR > 5) thresholds with the bilinear algorithm providing the best SNR value. Clustering, OpenCV, bilinear, Malvar and Menon demosaicing algorithms respectively provided differences on the ROI analysis of 7 ± 5%, 3 ± 2%, 3 ± 2%, 4 ± 2%, 7 ± 3%. Significance. Radioluminescent signals can accurately be isolated using a single polychromatic camera. Moreover, demosaicing using a bilinear kernel provided the best results and enabled Cherenkov signal subtraction while preserving the full spatial resolution of the camera. [ABSTRACT FROM AUTHOR]

Modern digital cameras use specific arrangement of Color Filter Array to sample light wavelength corresponding to visible colors. The most common Color Filter Array is the Bayer filter that samples only one color per pixel. To recover the full resolution image, an interpolation algorithm can be used. This process is called demosaicing and it is one of the first processing stages of a digital imaging pipeline. We introduce a novel data-driven model for demosaicing that takes into account the different requirements for reconstruction of the image Luma and Chrominance channels. The final model is a parallel composition of two reconstruction networks with individual architecture and trained with distinct loss functions. In order to solve the overfitting problem, we prepared a dataset that contains groups of patches that share common chromatic and spectral characteristics. We reported the reconstruction error on noise-free images and measured the effect of random noise and quantization noise in the demosaicing reconstruction. To test our model performance, we implemented the network on NVIDIA Jetson Nano, obtaining an end-to-end running time of less than one second for a full frame 12 MPixel image. [ABSTRACT FROM AUTHOR]

Color demosaicing is an essential step in the camera image processing pipeline, especially on hardware platforms for real-time video applications. Though many demosaicing algorithms have been proposed in the past two decades, there remains a substantial gap between industrial needs and academic research. On one hand, industry requires a high perceptual quality, artifact-free, and low-cost demosaicing algorithm that is noniterative and small window based ready to be implemented on a hardware platform with a limited line buffer. On the other hand, academia is targeting high PSNR/SSIM, with the computation cost and line buffer receiving second priority, and often a frame buffer and iterative operation are used. The cost, large line buffer and iteration requirement make most existing demosaicing algorithms inapplicable on hardware platforms. In this paper, we introduce a novel low-cost demosaicing algorithm to narrow the gap. We keep the operation window size and computation cost as the first priority and achieve both high PSNR/SSIM and visual perceptual quality compared to previous state-of-the-art methods on standard test datasets. We fully investigate the a priori knowledge of natural scene raw images and find several key cues that are beneficial to demosaicing. Our demosaicing algorithm is a smart fusion of these useful cues. Furthermore, we solve typical demosaicing issues that occur in many traditional methods, including false color artifacts, T-section closing, and zippering artifacts. The proposed method has no learning stage, no iteration operation, a small line buffer and a limited number of parameters, so it can easily be applied to hardware platforms. [ABSTRACT FROM AUTHOR]

Single-chip imaging devices featuring vertically stacked photodiodes and pixelated spectral filters are advancing multi-dye imaging methods for cancer surgeries, though this innovation comes with a compromise in spatial resolution. To mitigate this drawback, we developed a deep convolutional neural network (CNN) aimed at demosaicing the color and near-infrared (NIR) channels, with its performance validated on both pre-clinical and clinical datasets. We introduce an optimized deep CNN designed for demosaicing both color and NIR images obtained using a hexachromatic imaging sensor. A residual CNN was fine-tuned and trained on a dataset of color images and subsequently assessed on a series of dual-channel, color, and NIR images to demonstrate its enhanced performance compared with traditional bilinear interpolation. Our optimized CNN for demosaicing color and NIR images achieves a reduction in the mean square error by 37% for color and 40% for NIR, respectively, and enhances the structural dissimilarity index by 37% across both imaging modalities in pre-clinical data. In clinical datasets, the network improves the mean square error by 35% in color images and 42% in NIR images while enhancing the structural dissimilarity index by 39% in both imaging modalities. We showcase enhancements in image resolution for both color and NIR modalities through the use of an optimized CNN tailored for a hexachromatic image sensor. With the ongoing advancements in graphics card computational power, our approach delivers significant improvements in resolution that are feasible for real-time execution in surgical environments. [ABSTRACT FROM AUTHOR]

Despite the fact real-world video deinterlacing and demosaicing are well-suited to supervised learning from synthetically degraded data because the degradation models are known and fixed, learned video deinterlacing and demosaicing have received much less attention compared to denoising and super-resolution tasks. We propose a new multi-picture architecture for video deinterlacing or demosaicing by aligning multiple supporting pictures with missing data to a reference picture to be reconstructed, benefiting from both local and global spatio-temporal correlations in the feature space using modified deformable convolution blocks and a novel residual efficient top- k self-attention (kSA) block, respectively. Separate reconstruction blocks are used to estimate different types of missing data. Our extensive experimental results, on synthetic or real-world datasets, demonstrate that the proposed novel architecture provides superior results that significantly exceed the state-of-the-art for both tasks in terms of PSNR, SSIM, and perceptual quality. Ablation studies are provided to justify and show the benefit of each novel modification made to the deformable convolution and residual efficient kSA blocks. Code is available: https://github.com/KUIS-AI-Tekalp-Research-Group/Video-Deinterlacing. • Our model achieves state-of-the-art results for deinterlacing and demosaicing tasks. • We propose modified deformable convolution (DfConv) block to extract local features. • Our efficient residual top-k self-attention (EkSA) block extracts global features. • We additively combine local and global spatio-temporal features computed in parallel. • Separate reconstruction blocks are used for each field/channel to be reconstructed. [ABSTRACT FROM AUTHOR]

In this paper, the impact of demosaicing on gradient extraction is studied and a gradient-based feature extraction pipeline based on raw Bayer pattern images is proposed. It is shown both theoretically and experimentally that the Bayer pattern images are applicable to the central difference gradient-based feature extraction algorithms with negligible performance degradation, as long as the arrangement of color filter array (CFA) patterns matches the gradient operators. The color difference constancy assumption, which is widely used in various demosaicing algorithms, is applied in the proposed Bayer pattern image-based gradient extraction pipeline. Experimental results show that the gradients extracted from Bayer pattern images are robust enough to be used in histogram of oriented gradients (HOG)-based pedestrian detection algorithms and shift-invariant feature transform (SIFT)-based matching algorithms. By skipping most of the steps in the image signal processing (ISP) pipeline, the computational complexity and power consumption of a computer vision system can be reduced significantly. [ABSTRACT FROM AUTHOR]

Abstract Color modification is one of the popular image forgery techniques. It can be used to eliminate criminal evidence in various ways, such as modifying the color of a car used in a crime. If the color of a digital image is modified, the locations of the interpolated and original samples may be changed. Because the original and interpolated pixels have different statistical characteristics, these differences can serve as a basic clue for estimating the degree of color modification. It is assumed that the variance of original samples is greater than that of the interpolated samples. Therefore, we present a novel algorithm for color modification estimation using the variance ratio of color difference images in the wavelet domain. The color difference model is used to emphasize the differences between the original and interpolated samples. For color difference images, we execute a wavelet transform and use the highest frequency subband to calculate variances. We define a variance ratio measurement to quantify the level of color modification. Additionally, changed color local regions can be efficiently detected using the proposed algorithm. Experimental results demonstrate that the proposed method generates accurate estimation results for detecting color modification. Compared to the conventional method, our method provides superior color modification detection performance.

A mosaic of color filter arrays (CFAs) is commonly used in digital cameras as a spectrally selective filter to capture color images. The captured raw image is then processed by a demosaicing algorithm to recover the full-color image. In this paper, we formulate demosaicing as a restoration problem and solve it by minimizing the difference between the input raw image and the sampled full-color result. This under-constrained minimization is then solved with a novel convolutional neural network that estimates a linear subspace for the result at local image patches. In this way, the result in an image patch is determined by a few combination coefficients of the subspace bases, which makes the minimization problem tractable. This approach further allows joint learning of the CFA and demosaicing network. We demonstrate the superior performance of the proposed method by comparing it with state-of-the-art methods in both settings of noise-free and noisy data. [ABSTRACT FROM AUTHOR]

In this paper we present a dataset of images to test the performance of image processing algorithms, in particular demosaicing and denoising methods. Despite the plethora of demosaicing and denoising algorithms present in the literature, only few benchmarks are available to test their performance, and most of them are quite old, thus inadequate to represent the images captured by modern devices. The proposed dataset is composed by twenty 16 bit-depth images that can be used to test full-reference image quality metrics. More specifically, twelve pictures have been synthetically created by means of 2D or 3D softwares, while eight images have been captured by a high-end digital camera. [ABSTRACT FROM AUTHOR]

The truthfulness of digital images can be evaluated by investigating the CFA artifacts introduced due to the interpolation process of the image acquisition phase. In this paper, an image tampering detection technique is proposed by exposing the CFA artifacts in difference domain through the higher-order statistical analysis based on the Markov transition probability matrix (MTPM). Firstly, the given image is re-interpolated with most commonly used four Bayer CFA patterns. The re-interpolation process is performed by using bilinear interpolation scheme for simplicity purpose. Then, the difference between the given image and its re-interpolated versions is evaluated to analyze the CFA inconsistencies. The target difference image is selected corresponding to the maximum sum which is further processed to evaluate the MTPM based second-order statistical feature. The recommended approach is assessed on different images from UCID dataset and various social networking websites based on scalar-based and SVM/machine learning based forensic detectors. The experiment results confirm that the projected method offers improved efficiency in comparison to the existing techniques based on different forgery scenarios. [ABSTRACT FROM AUTHOR]

• Image reconstruction techniques in the multispectral domain were investigated. • Design of an eight-band multispectral filter array imaging system. • A deep guidance network modeling algorithm is proposed. • Outperforms other existing methods in both quantitative and qualitative results. Imaging techniques using multispectral filter arrays (MSFA)have become a research hotspot with the rapid development of spectroscopic techniques. Among them, exploiting the correlation of color channels in the raw data and reconstructing raw images with high sparsity is a bottleneck and constraint in multi-band MSFA imaging systems. Therefore, this paper proposes a 4 × 4 eight-band MSFA imaging system containing a high sampling rate all-pass band. The all-pass band with a 1/2 high sampling rate contains rich color texture information to provide more features. A depth-guided reconstruction network (DGRN), including a depth-guided model (DGM) and a channel adaptive convolution model (CACM), is established to reconstruct the original spectral images. DGM extracts the color texture information of all-pass band images as the guide feature, which is combined with the initially processed eight-band shallow features to be the input of CACM to assign different guide features to different bands adaptively for learning and aggregation. The spatial correlation and spectral correlation of multiple bands are jointly learned using spectral and spatial properties to make the network flexible for MSFA imaging systems. The experimental results show that the method can effectively remove the artifacts of reconstructed images and improve the edge texture clarity. [ABSTRACT FROM AUTHOR]

Residual interpolations are effective methods to reduce the instantaneous field-of-view error of division of focal plane (DoFP) polarimeters. However, their guide-image selection strategies are improper, and do not consider the DoFP polarimeters’ spatial sampling modes. Thus, we propose a residual interpolation method with a new guide-image selection strategy based on the spatial layout of the pixeled polarizer array to improve the sampling rate of the guide image. The interpolation performance is also improved by the proposed pixel-by-pixel, adaptive iterative process and the weighted average fusion of the results of the minimized residual and minimized Laplacian energy guide filters. Visual and objective evaluations demonstrate the proposed method’s superiority to the existing state-of-the-art methods. The proposed method proves that considering the spatial layout of the pixeled polarizer array on the physical level is vital to improving the performance of interpolation methods for DoFP polarimeters.

In this paper, we propose a deep-image-prior-based demosaicing method for a random RGBW color filter array (CFA). The color reconstruction from the random RGBW CFA is performed by the deep image prior network, which uses only the RGBW CFA image as the training data. To our knowledge, this work is a first attempt to reconstruct the color image with a neural network using only a single RGBW CFA in the training. Due to the White pixels in the RGBW CFA, more light is transmitted through the CFA than in the case with the conventional RGB CFA. As the image sensor can detect more light, the signal-to-noise-ratio (SNR) increases and the proposed demosaicing method can reconstruct the color image with a higher visual quality than other existing demosaicking methods, especially in the presence of noise. We propose a loss function that can train the deep image prior (DIP) network to reconstruct the colors from the White pixels as well as from the red, green, and blue pixels in the RGBW CFA. Apart from using the DIP network, no additional complex reconstruction algorithms are required for the demosaicing. The proposed demosaicing method becomes useful in situations when the noise becomes a major problem, for example, in low light conditions. Experimental results show the validity of the proposed method for joint demosaicing and denoising.

Modern digital cameras use specific arrangement of Color Filter Array to sample light wavelength corresponding to visible colors. The most common Color Filter Array is the Bayer filter that samples only one color per pixel. To recover the full resolution image, an interpolation algorithm can be used. This process is called demosaicing and it is one of the first processing stages of a digital imaging pipeline. We introduce a novel data-driven model for demosaicing that takes into account the different requirements for reconstruction of the image Luma and Chrominance channels. The final model is a parallel composition of two reconstruction networks with individual architecture and trained with distinct loss functions. In order to solve the overfitting problem, we prepared a dataset that contains groups of patches that share common chromatic and spectral characteristics. We reported the reconstruction error on noise-free images and measured the effect of random noise and quantization noise in the demosaicing reconstruction. To test our model performance, we implemented the network on NVIDIA Jetson Nano, obtaining an end-to-end running time of less than one second for a full frame 12 MPixel image.

The Curiosity rover has landed on Mars since 2012. One of the instruments onboard the rover is a pair of multispectral cameras known as Mastcams, which act as eyes of the rover. In this paper, we summarize our recent studies on some interesting image processing projects for Mastcams. In particular, we will address perceptually lossless compression of Mastcam images, debayering and resolution enhancement of Mastcam images, high resolution stereo and disparity map generation using fused Mastcam images, and improved performance of anomaly detection and pixel clustering using combined left and right Mastcam images. The main goal of this review paper is to raise public awareness about these interesting Mastcam projects and also stimulate interests in the research community to further develop new algorithms for those applications.

Abstract An image sensor can measure only one color per pixel through the color filter array. Missing pixels are estimated using an interpolation process. For this reason, a captured pixel and interpolated pixel have different statistical characteristics. Because the pattern of a color filter array is changed when the image is manipulated or forged, this pattern change can be a clue to detect image forgery. However, the majority of forgery detection algorithms assume that they know the color filter array pattern. Therefore, estimating the configuration of the color filter array can have an important role as a precondition for image forgery detection. In this paper, we propose an efficient algorithm for estimating the Bayer color filter array configuration. We first construct a color difference image to reflect the characteristics of different demosaicing methods. To identify the color filter array pattern, we employ singular value decomposition. The truncated sum of the singular values is used to distinguish the color filter array pattern. Experimental results confirm that the proposed method generates acceptable estimation results in identifying color filter array patterns. Compared with conventional methods, the proposed method provides superior performance.

The RGBW color filter arrays (CFA), also known as CFA2.0, contains R, G, B, and white (W) pixels. It is a 4 × 4 pattern that has 8 white pixels, 4 green pixels, 2 red pixels, and 2 blue pixels. The pattern repeats itself over the whole image. In an earlier conference paper, we cast the demosaicing process for CFA2.0 as a pansharpening problem. That formulation is modular and allows us to insert different pansharpening algorithms for demosaicing. New algorithms in interpolation and demosaicing can also be used. In this paper, we propose a new enhancement of our earlier approach by integrating a deep learning-based algorithm into the framework. Extensive experiments using IMAX and Kodak images clearly demonstrated that the new approach improved the demosaicing performance even further. [ABSTRACT FROM AUTHOR]

Autofluorescence imaging (AFI) systems are widely used in the detection of precancerous lesions. Fluorescence images of precancerous tissue are usually red (R) or blue (B), so this kind of system has high requirement for colour recovery, especially in R and B channels. Besides, AFI system requires bulk data transmission with no time delay. Existing colour recovery algorithms focus more on green (G) channel, overlooking R and B channels. Although the state-of-art demosaicing algorithms can perform well in colour recovery, they often have high computational cost and high hardware requirements. We propose an efficient interpolation algorithm with low complexity to solve the problem. When calculating R and B channel values, we innovatively propose the diagonal direction to select the interpolation direction, and apply colour difference law to make full use of the correlation between colour channels. The experimental results show that the peak signal-to-noise ratios (PSNRs) of G, R and B channels reach 37.54, 37.40 and 38.22 dB, respectively, which shows good performance in recovery of R and B channels. In conclusion, the algorithm proposed in this paper can be used as an alternative to the existing demosaicing algorithms for AFI system. [ABSTRACT FROM AUTHOR]

A mosaic of color filter arrays (CFAs) is commonly used in digital cameras as a spectrally selective filter to capture color images. The captured raw image is then processed by a demosaicing algorithm to recover the full-color image. In this paper, we formulate demosaicing as a restoration problem and solve it by minimizing the difference between the input raw image and the sampled full-color result. This under-constrained minimization is then solved with a novel convolutional neural network that estimates a linear subspace for the result at local image patches. In this way, the result in an image patch is determined by a few combination coefficients of the subspace bases, which makes the minimization problem tractable. This approach further allows joint learning of the CFA and demosaicing network. We demonstrate the superior performance of the proposed method by comparing it with state-of-the-art methods in both settings of noise-free and noisy data.