Search

This paper introduces a new approach to single image blind deblurring via Bayesian estimation using expected loss, diverging from traditional maximum a posteriori (MAP) estimation methods that are limited by the delta kernel problem-a phenomenon where blur kernels are inaccurately estimated as delta functions due to the disparity of the number of pixels between the latent image and the kernel, leading to blurry latent images. We introduce the concept of robust intensity patch (RIP) values, which are median pixel values within local image patches, demonstrating remarkable stability through the blurring process. These RIP values are proposed as feasible substitutes for ground truth images, which are often unavailable in real-world scenarios due to the inherent difficulty of capturing both blurred and perfectly sharp versions of the same scene under identical conditions. We have developed a new loss function named robust intensity loss (RIL), designed to selectively penalize the latent image. This function aims to equalize the imbalance in the number of pixels between the latent image and the kernel. Through this approach, we have successfully redefined Bayesian estimation using expected loss as an optimization problem. Our contributions include the first application of Bayesian estimation using expected loss for single image deblurring, the introduction of RIP values, the development of the RIL function, and the integration with an advanced optimization scheme to significantly enhance deblurring accuracy. Our empirical results demonstrate the effectiveness of our method across various benchmark datasets, representing the way for future advancements in image deblurring.

This paper proposes a method for demosaicing raw images captured by multispectral cameras. The proposed method estimates a pseudo-panchromatic image (PPI) via an iterative-linear-regression model and utilizes the estimated PPI for multispectral demosaicing. The PPI is estimated through horizontal and vertical guided filtering, with the subsampled multispectral-filter-array-(MSFA) image and low-pass-filtered MSFA as the guide image and filtering input, respectively. The number of iterations is automatically determined according to a predetermined criterion. Spectral differences between the estimated PPI and MSFA are calculated for each channel, and each spectral difference is interpolated using directional interpolation. The weights are calculated from the estimated PPI, and each interpolated spectral difference is combined using the weighted sum. The experimental results indicate that the proposed method outperforms the State-of-the-Art methods with regard to spatial and spectral fidelity for both synthetic and real-world images.

Various super-resolution (SR) kernels in the degradation model deteriorate the performance of the SR algorithms, showing unpleasant artifacts in the output images. Hence, SR kernel estimation has been studied to improve the SR performance in several ways for more than a decade. In particular, a conventional research named KernelGAN has recently been proposed. To estimate the SR kernel from a single image, KernelGAN introduces generative adversarial networks(GANs) that utilize the recurrence of similar structures across scales. Subsequently, an enhanced version of KernelGAN, named E-KernelGAN, was proposed to consider image sharpness and edge thickness. Although it is stable compared to the earlier method, it still encounters challenges in estimating sizable and anisotropic kernels because the structural information of an input image is not sufficiently considered. In this paper, we propose a kernel estimation algorithm called Total Variation Guided KernelGAN (TVG-KernelGAN), which efficiently enables networks to focus on the structural information of an input image. The experimental results show that the proposed algorithm accurately and stably estimates kernels, particularly sizable and anisotropic kernels, both qualitatively and quantitatively. In addition, we compared the results of the non-blind SR methods, using SR kernel estimation techniques. The results indicate that the performance of the SR algorithms was improved using our proposed method.

As the demand for thermal information increases in industrial fields, numerous studies have focused on enhancing the quality of infrared images. Previous studies have attempted to independently overcome one of the two main degradations of infrared images, fixed pattern noise (FPN) and blurring artifacts, neglecting the other problems, to reduce the complexity of the problems. However, this is infeasible for real-world infrared images, where two degradations coexist and influence each other. Herein, we propose an infrared image deconvolution algorithm that jointly considers FPN and blurring artifacts in a single framework. First, an infrared linear degradation model that incorporates a series of degradations of the thermal information acquisition system is derived. Subsequently, based on the investigation of the visual characteristics of the column FPN, a strategy to precisely estimate FPN components is developed, even in the presence of random noise. Finally, a non-blind image deconvolution scheme is proposed by analyzing the distinctive gradient statistics of infrared images compared with those of visible-band images. The superiority of the proposed algorithm is experimentally verified by removing both artifacts. Based on the results, the derived infrared image deconvolution framework successfully reflects a real infrared imaging system.

Image haze removal is essential in autonomous driving as the outdoor images captured during unfavorable weather conditions, such as haze or snow, are affected by poor visibility. Much research has been done to overcome image degradation such as low contrast and faded color due to haze. However, in the traditional model, a phenomenon is neglected that several particles simultaneously involved in light acquisition. To address this problem, we propose a novel single image dehazing method based on the spatially adaptive atmospheric point spread function (APSF). We developed a module that estimates the APSF to overcome the limitations of the spatially invariant APSF which used in existing dehazing algorithms. The key factor in the estimation is that road scenes with haze have different statistical characteristic from common hazy images in color and resolution. Furthermore, the APSF on the traffic signs or lights is estimated by generating superpixels to prevent halo artifacts around the sharp edges of the images. We adopted the total variation model as a regularization functional to reduce halo and unnatural artifacts that may occur during deconvolution. The haze-free images from the proposed method tested whether the proposed method can enhance the performance of vision algorithms for autonomous driving. The experimental results demonstrate that the proposed method outperforms state-of-the-art image dehazing methods enhancing the performance of the vision algorithms. Moreover, additional experiments demonstrated the effectiveness of the proposed method for quantitative and qualitative comparison with the state-of-the-art algorithms.

Images acquired from the outdoors are often degraded owing to bad weather conditions, such as haze, fog, and rain. Various algorithms for dehazing daytime images have been introduced to eliminate the effects of haze. However, unlike daytime conditions, where sunlight is the main source of light, there are multiple regional light sources in nighttime conditions. Thus, it is difficult to effectively eliminate the effects of haze at night using dehazing algorithms that primarily target daytime hazy images. In addition, most nighttime dehazing algorithms adopt the dark channel prior (DCP) to estimate the transmission, but this approach has problems due to the spatially variant illuminations. In this paper, we propose a nighttime single image dehazing algorithm based on the structural patch decomposition. First, atmospheric light is estimated by separating the intensity and color of light. To estimate the transmission, the candidates for the transmission values are set, and the final transmission value is obtained based on a weighted summation of these candidates for each image pixel. The weights of the candidates are determined according to the structural patch information of the scene radiances estimated with these candidates. Using this approach, an appropriate transmission map can be obtained, and the dehazing procedure is shown to be robust to the mixed illumination conditions observed in the nighttime conditions. The experimental results show that the proposed algorithm effectively eliminate the effect of haze in images based on quantitative and qualitative evaluations.

Noise reduction is important for X-ray images because it can reduce radiation exposure to patients. X-ray image noise has a Poisson-Gaussian distribution, and recently, noise analysis and removal in multiscale transformations have been widely implemented. The nonsubsampled contourlet transform (NSCT) is a multiscale transformation suitable for medical images that separates the scale and direction. This study proposes a Poisson-Gaussian noise-removal method using NSCT shrinkage that is based on the characteristics of Poisson-Gaussian noise in NSCT domain. It has the structure of a block-matching 3D filtering algorithm in the form of basic estimation and noise removal process; however, the main processes are modified to consider Poisson-Gaussian noise characteristics. In the basic estimation process, an NSCT shrinkage method that is suitable for Poisson-Gaussian noise characteristics is developed by optimizing the local linear minimum mean square error estimator in the NSCT domain. In the denoising step, the noise term of the Wiener filter is determined using the result of the NSCT shrinkage, and finally, the denoised image is obtained. The proposed method is applied to simulated and real X-ray images and is compared with other state-of-the-art Poisson-Gaussian noise removal methods; it exhibits excellent results in both quantitative and qualitative aspects.

In this paper, we propose a multi-frame depth super-resolution (SR) method based on L1 data fidelity with the total variation regularization (TV-L1) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB sensors and the improvement of the depth image resolution is an inherently ill-posed problem. To overcome this under-determined problem, the solution space is limited by the regularization term through prior knowledge and the data fidelity term using statistical information of the noise. Firstly, the statistical characteristics of ToF depth images are analyzed to specify the appropriate observation model. Thereafter, the objective function for multi-frame depth SR based on the TV-L1 model is designed by considering the prior knowledge of the depth images. This approach enables the sharpness of the edges to be preserved and the noise amplification to be suppressed simultaneously. Furthermore, an efficient solver based on half-quadratic splitting is proposed. The algorithm minimizes the objective function for the multi-frame SR problem consisting of the TV regularization term and L1 data fidelity term. The proposed method is verified on a synthetic dataset and real-world data acquired from a ToF sensor. The experimental results demonstrate that the proposed method can substantially reconstruct high-resolution depth images in terms of preserving sharp depth discontinuities, without any obvious artifacts, and can increase robustness to noise.

In this paper, we propose a crosstalk correction method for color filter array (CFA) image sensors based on Lp-regularized multi-channel deconvolution. Most imaging systems with CFA exhibit a crosstalk phenomenon caused by the physical limitations of the image sensor. In general, this phenomenon produces both color degradation and spatial degradation, which are respectively called desaturation and blurring. To improve the color fidelity and the spatial resolution in crosstalk correction, the feasible solution of the ill-posed problem is regularized by image priors. First, the crosstalk problem with complex spatial and spectral degradation is formulated as a multi-channel degradation model. An objective function with a hyper-Laplacian prior is then designed for crosstalk correction. This approach enables the simultaneous improvement of the color fidelity and the sharpness restoration of the details without noise amplification. Furthermore, an efficient solver minimizes the objective function for crosstalk correction consisting of Lp regularization terms. The proposed method was verified on synthetic datasets according to various crosstalk and noise levels. Experimental results demonstrated that the proposed method outperforms the conventional methods in terms of the color peak signal-to-noise ratio and structural similarity index measure.

In recent years, red, green, blue, and white (RGBW) color filter arrays (CFAs) have been developed to solve the problem of low-light conditions. In this paper, we propose a new color demosaicing algorithm for RGBW CFAs using a Laplacian pyramid. Because the white channel has a high correlation to the red, green, and blue channels, the white channel is interpolated first using each color difference channel. After we estimate the white channel, the red, green, and blue channels are interpolated using the Laplacian pyramid decomposition of the estimated white channel. Our proposed method using Laplacian pyramid restoration works with Canon-RGBW CFAs and any other periodic CFAs. The experimental results demonstrated that the proposed method shows superior performance compared with other conventional methods in terms of the color peak signal-to-noise ratio, structural similarity index measure, and average execution time.

Image quality assessment without a reference image is essential for evaluating the performance of image enhancements. Much research has been done to develop an objective image quality measurement that is relevant to perceived quality evaluations. Because the contrast of an image is one of the important factors for a user to evaluate the image quality, methods for improving the contrast of images are also being extensively studied, but the assessment algorithms for evaluating them are limited. In this paper, we propose a contrast measurement of a color image based on a stimulus in the human visual system (HVS). The proposed method evaluates the luminance component of the image based on just-noticeable-difference to reflect the local contrast perceived in the HVS. The contrast in the color component is evaluated based on a model of the stimulus of the color component in the primary visual cortex (V1). The region response factor, which reflects the relative change of the color saturation in the different luminance values in V1, is used to image contrast. We tested the validity of the proposed method using various image databases and subjective tests. The experimental results showed that the proposed method had higher correlation with the evaluation of people than conventional methods of measuring image contrast without the original image.

An imaging system has natural statistics that reflect its intrinsic characteristics. For example, the gradient histogram of a visible light image generally obeys a heavy-tailed distribution, and its restoration considers natural statistics. Thermal imaging cameras detect infrared radiation, and their signal processors are specialized according to the optical and sensor systems. Thermal images, also known as long wavelength infrared (LWIR) images, suffer from distinct degradations of LWIR sensors and residual nonuniformity (RNU). However, despite the existence of various studies on the statistics of thermal images, thermal image processing has seldom attempted to incorporate natural statistics. In this study, natural statistics of thermal imaging sensors are derived, and an optimization method for restoring thermal images is proposed. To verify our hypothesis about the thermal images, high-frequency components of thermal images from various datasets are analyzed with various measures (correlation coefficient, histogram intersection, chi-squared test, Bhattacharyya distance, and Kullback–Leibler divergence), and generalized properties are derived. Furthermore, cost functions accommodating the validated natural statistics are designed and minimized by a pixel-wise optimization method. The proposed algorithm has a specialized structure for thermal images and outperforms the conventional methods. Several image quality assessments are employed for quantitatively demonstrating the performance of the proposed method. Experiments with synthesized images and real-world images are conducted, and the results are quantified by reference image assessments (peak signal-to-noise ratio and structural similarity index measure) and no-reference image assessments (Roughness (Ro) and Effective Roughness (ERo) indices).

Recently, the white (w) channel has been incorporated in various forms into color filter arrays (CFAs). The advantage of using theWchannel is thatWpixels have less noise than red (R), green (G), or blue (B) (RGB) pixels; therefore, under low-light conditions, pixels with high fidelity can be obtained. However, RGBW CFAs normally suffer from spatial resolution degradation due to a smaller number of color pixels than in RGB CFAs. Therefore, even though the reconstructed colors have higher sensitivity, which results in larger Color Peak Signal-to-Noise Ratio (CPSNR) values, there are some color aliasing artifacts due to a low resolution. In this paper, we propose a rank minimization-based color interpolation method with a colorization constraint for the RGBW format with a large number ofWpixels. The rank minimization can achieve a broad interpolation and preserve the structure in the image, and it thereby eliminates the color artifacts. However, the colors fade from this global process. Therefore, we further incorporate a colorization constraint into the rank minimization process for the better reproduction of the colors. The experimental results show that the images can be reconstructed well, even from noisy pattern images obtained under low-light conditions.

Recently, several red-green-blue near-infrared (RGB-NIR) multispectral filter arrays (MFAs), which include near infrared (NIR) pixels, have been proposed. For extremely low light scenes, the RGB-NIR MFA sensor has been extended to receive NIR light, by adding NIR pixels to supplement for the insufficient visible band light energy. However, the resolution reconstruction of the RGB-NIR MFA, using demosaicing and color restoration methods, is based on the correlation between the NIR pixels and the pixels of other colors; this does not improve the RGB channel sensitivity with respect to the NIR channel sensitivity. In this paper, we propose a color restored image post-processing method to improve the sensitivity and resolution of an RGB-NIR MFA. Although several linear regression based color channel reconstruction methods have taken advantage of the high sensitivity NIR channel, it is difficult to accurately estimate the linear coefficients because of the high level of noise in the color channels under extremely low light conditions. The proposed method solves this problem in three steps: guided filtering, based on the linear similarity between the NIR and color channels, edge preserving smoothing to improve the accuracy of linear coefficient estimation, and residual compensation for lost spatial resolution information. The results show that the proposed method is effective, while maintaining the NIR pixel resolution characteristics, and improving the sensitivity in terms of the signal-to-noise ratio by approximately 13 dB.

Recently, several red-green-blue-white (RGBW) color filter arrays (CFAs), which include highly sensitive W pixels, have been proposed. However, RGBW CFA patterns suffer from spatial resolution degradation owing to the sensor composition having more color components than the Bayer CFA pattern. RGBW CFA demosaicing methods reconstruct resolution using the correlation between white (W) pixels and pixels of other colors, which does not improve the red-green-blue (RGB) channel sensitivity to the W channel level. In this paper, we thus propose a demosaiced image post-processing method to improve the RGBW CFA sensitivity and resolution. The proposed method decomposes texture components containing image noise and resolution information. The RGB channel sensitivity and resolution are improved through updating the W channel texture component with those of RGB channels. For this process, a cross multilateral filter (CMF) is proposed. It decomposes the smoothness component from the texture component using color difference information and distinguishes color components through that information. Moreover, it decomposes texture components, luminance noise, color noise, and color aliasing artifacts from the demosaiced images. Finally, by updating the texture of the RGB channels with the W channel texture components, the proposed algorithm improves the sensitivity and resolution. Results show that the proposed method is effective, while maintaining W pixel resolution characteristics and improving sensitivity from the signal-to-noise ratio value by approximately 4.5 dB.

The noise distribution of images obtained by X-ray sensors in low-dosage situations can be analyzed using the Poisson and Gaussian mixture model. Multiscale conversion is one of the most popular noise reduction methods used in recent years. Estimation of the noise distribution of each subband in the multiscale domain is the most important factor in performing noise reduction, with non-subsampled contourlet transform (NSCT) representing an effective method for scale and direction decomposition. In this study, we use artificially generated noise to analyze and estimate the Poisson–Gaussian noise of low-dose X-ray images in the NSCT domain. The noise distribution of the subband coefficients is analyzed using the noiseless low-band coefficients and the variance of the noisy subband coefficients. The noise-after-transform also follows a Poisson–Gaussian distribution, and the relationship between the noise parameters of the subband and the full-band image is identified. We then analyze noise of actual images to validate the theoretical analysis. Comparison of the proposed noise estimation method with an existing noise reduction method confirms that the proposed method outperforms traditional methods.

In this paper, a spatio-spectral-temporal filter considering an inter-channel correlation is proposed for the denoising of a color filter array (CFA) sequence acquired by CCD/CMOS image sensors. Owing to the alternating under-sampled grid of the CFA pattern, the inter-channel correlation must be considered in the direct denoising process. The proposed filter is applied in the spatial, spectral, and temporal domain, considering the spatio-tempo-spectral correlation. First, nonlocal means (NLM) spatial filtering with patch-based difference (PBD) refinement is performed by considering both the intra-channel correlation and inter-channel correlation to overcome the spatial resolution degradation occurring with the alternating under-sampled pattern. Second, a motion-compensated temporal filter that employs inter-channel correlated motion estimation and compensation is proposed to remove the noise in the temporal domain. Then, a motion adaptive detection value controls the ratio of the spatial filter and the temporal filter. The denoised CFA sequence can thus be obtained without motion artifacts. Experimental results for both simulated and real CFA sequences are presented with visual and numerical comparisons to several state-of-the-art denoising methods combined with a demosaicing method. Experimental results confirmed that the proposed frameworks outperformed the other techniques in terms of the objective criteria and subjective visual perception in CFA sequences.

In this paper, we propose a green (G)-channel restoration for a red–white–blue (RWB) color filter array (CFA) image sensor using the dual sampling technique. By using white (W) pixels instead of G pixels, the RWB CFA provides high-sensitivity imaging and an improved signal-to-noise ratio compared to the Bayer CFA. However, owing to this high sensitivity, the W pixel values become rapidly over-saturated before the red–blue (RB) pixel values reach the appropriate levels. Because the missing G color information included in the W channel cannot be restored with a saturated W, multiple captures with dual sampling are necessary to solve this early W-pixel saturation problem. Each W pixel has a different exposure time when compared to those of the R and B pixels, because the W pixels are double-exposed. Therefore, a RWB-to-RGB color conversion method is required in order to restore the G color information, using a double-exposed W channel. The proposed G-channel restoration algorithm restores G color information from the W channel by considering the energy difference caused by the different exposure times. Using the proposed method, the RGB full-color image can be obtained while maintaining the high-sensitivity characteristic of the W pixels.

A multispectral filter array (MSFA) image sensor with red, green, blue and near-infrared (NIR) filters is useful for various imaging applications with the advantages that it obtains color information and NIR information simultaneously. Because the MSFA image sensor needs to acquire invisible band information, it is necessary to remove the IR cut-offfilter (IRCF). However, without the IRCF, the color of the image is desaturated by the interference of the additional NIR component of each RGB color channel. To overcome color degradation, a signal processing approach is required to restore natural color by removing the unwanted NIR contribution to the RGB color channels while the additional NIR information remains in the N channel. Thus, in this paper, we propose a color restoration method for an imaging system based on the MSFA image sensor with RGBN filters. To remove the unnecessary NIR component in each RGB color channel, spectral estimation and spectral decomposition are performed based on the spectral characteristics of the MSFA sensor. The proposed color restoration method estimates the spectral intensity in NIR band and recovers hue and color saturation by decomposing the visible band component and the NIR band component in each RGB color channel. The experimental results show that the proposed method effectively restores natural color and minimizes angular errors.

This paper proposes a novel way of combining color demosaicing and the auto white balance (AWB) method, which are important parts of image processing. Performance of the AWB is generally affected by demosaicing results because most AWB algorithms are performed posterior to color demosaicing. In this paper, in order to increase the performance and efficiency of the AWB algorithm, the color constancy problem is examined during the color demosaicing step. Initial estimates of the directional luminance and chrominance values are defined for estimating edge direction and calculating the AWB gain. In order to prevent color failure in conventional edge-based AWB methods, we propose a modified edge-based AWB method that used a predefined achromatic region. The estimation of edge direction is performed region adaptively by using the local statistics of the initial estimates of the luminance and chrominance information. Simulated and real Bayer color filter array (CFA) data are used to evaluate the performance of the proposed method. When compared to conventional methods, the proposed method shows significant improvements in terms of visual and numerical criteria.